* [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 16:09 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 16:09 UTC (permalink / raw) To: linux-fsdevel, linux-mm; +Cc: linux-kernel The bug manifests as a soft lockup in kswapd: [ 155.759084] netconsole: network logging started [ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46] [ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] [ 598.920834] CPU 1 [ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] [ 598.926818] [ 598.928043] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 598.929299] RIP: 0010:[<ffffffffa007ff9b>] [<ffffffffa007ff9b>] i915_gem_inactive_shrink+0x78/0x194 [i915] [ 598.930603] RSP: 0018:ffff8800709ebd50 EFLAGS: 00000216 [ 598.931867] RAX: ffff88006ec5c6b0 RBX: 00000000000000c0 RCX: 0000000000000000 [ 598.933135] RDX: ffff880037e59638 RSI: 0000000000000000 RDI: ffff880037876020 [ 598.934408] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 000000000001bd90 [ 598.935680] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e [ 598.936944] R13: ffff8800709ebcf0 R14: ffff8800709ebcf8 R15: ffffffff810dd64d [ 598.938221] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 [ 598.939525] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 598.940813] CR2: 00007fe16a34d380 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 598.942114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 598.943441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 598.944744] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006df8ae00) [ 598.946044] Stack: [ 598.947317] ffff8800709ebd90 ffff880037e59638 ffff8800709ebd60 ffff880037e595f0 [ 598.948654] 0000000000000000 0000000000000000 00000000000000d0 000000000004c24e [ 598.949986] ffff8800709ebde0 ffffffff810e1f89 000000000000003d 0000000000000080 [ 598.951329] Call Trace: [ 598.952673] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 598.954003] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 598.955331] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 598.956683] [<ffffffff8106e157>] kthread+0x84/0x8c [ 598.958022] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 598.959379] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 598.960709] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 598.962032] Code: 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00 48 05 b0 00 00 00 48 39 d8 0f 18 0a [ 598.962192] e1 e9 da 00 00 00 4c 89 f7 e8 c6 fd ff ff 48 8b 43 48 4c 8b [ 598.965009] Call Trace: [ 598.966421] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 598.966423] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 598.966426] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 598.966429] [<ffffffff8106e157>] kthread+0x84/0x8c [ 598.966432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 598.966435] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 598.966438] [<ffffffff81483760>] ? gs_change+0x13/0x13 The traces are slightly different each reboot cycle, but it's always in kswapd and usually in shrink_slab. Once it happens, anything that touches the filesystem hangs in D wait, so the machine is basically toast. The box is a Lenovo T420s sandybridge core i5 based laptop with 2GB of memory. There is a corresponding Red Hat bugzilla report here: https://bugzilla.redhat.com/show_bug.cgi?id=694818 And I've verified that the bug also shows in the 2.6.38.3 stable kernel. If anyone wants me to try anything, please let me know. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 16:09 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 16:09 UTC (permalink / raw) To: linux-fsdevel, linux-mm; +Cc: linux-kernel The bug manifests as a soft lockup in kswapd: [ 155.759084] netconsole: network logging started [ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46] [ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] [ 598.920834] CPU 1 [ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] [ 598.926818] [ 598.928043] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 598.929299] RIP: 0010:[<ffffffffa007ff9b>] [<ffffffffa007ff9b>] i915_gem_inactive_shrink+0x78/0x194 [i915] [ 598.930603] RSP: 0018:ffff8800709ebd50 EFLAGS: 00000216 [ 598.931867] RAX: ffff88006ec5c6b0 RBX: 00000000000000c0 RCX: 0000000000000000 [ 598.933135] RDX: ffff880037e59638 RSI: 0000000000000000 RDI: ffff880037876020 [ 598.934408] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 000000000001bd90 [ 598.935680] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e [ 598.936944] R13: ffff8800709ebcf0 R14: ffff8800709ebcf8 R15: ffffffff810dd64d [ 598.938221] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 [ 598.939525] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 598.940813] CR2: 00007fe16a34d380 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 598.942114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 598.943441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 598.944744] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006df8ae00) [ 598.946044] Stack: [ 598.947317] ffff8800709ebd90 ffff880037e59638 ffff8800709ebd60 ffff880037e595f0 [ 598.948654] 0000000000000000 0000000000000000 00000000000000d0 000000000004c24e [ 598.949986] ffff8800709ebde0 ffffffff810e1f89 000000000000003d 0000000000000080 [ 598.951329] Call Trace: [ 598.952673] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 598.954003] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 598.955331] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 598.956683] [<ffffffff8106e157>] kthread+0x84/0x8c [ 598.958022] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 598.959379] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 598.960709] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 598.962032] Code: 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00 48 05 b0 00 00 00 48 39 d8 0f 18 0a [ 598.962192] e1 e9 da 00 00 00 4c 89 f7 e8 c6 fd ff ff 48 8b 43 48 4c 8b [ 598.965009] Call Trace: [ 598.966421] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 598.966423] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 598.966426] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 598.966429] [<ffffffff8106e157>] kthread+0x84/0x8c [ 598.966432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 598.966435] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 598.966438] [<ffffffff81483760>] ? gs_change+0x13/0x13 The traces are slightly different each reboot cycle, but it's always in kswapd and usually in shrink_slab. Once it happens, anything that touches the filesystem hangs in D wait, so the machine is basically toast. The box is a Lenovo T420s sandybridge core i5 based laptop with 2GB of memory. There is a corresponding Red Hat bugzilla report here: https://bugzilla.redhat.com/show_bug.cgi?id=694818 And I've verified that the bug also shows in the 2.6.38.3 stable kernel. If anyone wants me to try anything, please let me know. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 16:09 ` James Bottomley @ 2011-04-27 16:33 ` Chris Mason -1 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-27 16:33 UTC (permalink / raw) To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel Excerpts from James Bottomley's message of 2011-04-27 12:09:13 -0400: > The bug manifests as a soft lockup in kswapd: > > [ 155.759084] netconsole: network logging started > [ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46] > [ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] > [ 598.920834] CPU 1 > [ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] > [ 598.926818] Probably easier to debug with a sysrq-l and sysrq-w. If you get stuck on the filesystem, it is probably waiting on ram, which it probably can't get because kswapd is spinning. Eventually everyone backs up waiting for the transaction that never ends. If we're really lucky it is just GFP_KERNEL where it should NOFS. Since you're often stuck in different spots inside shrink_slab, we're probably not stuck on a lock. But, trying with lock debugging, lockdep enabled and preempt on is a good idea to rule out locking mistakes. Does the fedora debug kernel enable preempt? -chris ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 16:33 ` Chris Mason 0 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-27 16:33 UTC (permalink / raw) To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel Excerpts from James Bottomley's message of 2011-04-27 12:09:13 -0400: > The bug manifests as a soft lockup in kswapd: > > [ 155.759084] netconsole: network logging started > [ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46] > [ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] > [ 598.920834] CPU 1 > [ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole] > [ 598.926818] Probably easier to debug with a sysrq-l and sysrq-w. If you get stuck on the filesystem, it is probably waiting on ram, which it probably can't get because kswapd is spinning. Eventually everyone backs up waiting for the transaction that never ends. If we're really lucky it is just GFP_KERNEL where it should NOFS. Since you're often stuck in different spots inside shrink_slab, we're probably not stuck on a lock. But, trying with lock debugging, lockdep enabled and preempt on is a good idea to rule out locking mistakes. Does the fedora debug kernel enable preempt? -chris -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 16:33 ` Chris Mason (?) @ 2011-04-27 16:50 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 16:50 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote: > Probably easier to debug with a sysrq-l and sysrq-w. Yes, just enabling them for the repro ... I didn't echo 1 > /proc/sys/kernel/sysrq first time around > If you get stuck > on the filesystem, it is probably waiting on ram, which it probably > can't get because kswapd is spinning. Eventually everyone backs up > waiting for the transaction that never ends. If we're really lucky it > is just GFP_KERNEL where it should NOFS. > > Since you're often stuck in different spots inside shrink_slab, we're > probably not stuck on a lock. But, trying with lock debugging, lockdep > enabled and preempt on is a good idea to rule out locking mistakes. > > Does the fedora debug kernel enable preempt? No ... I've taken the Fedora 15 kernel config directly from their install CD for this (with a bit of munging to get it to work on -rc4) James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 16:50 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 16:50 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote: > Probably easier to debug with a sysrq-l and sysrq-w. Yes, just enabling them for the repro ... I didn't echo 1 > /proc/sys/kernel/sysrq first time around > If you get stuck > on the filesystem, it is probably waiting on ram, which it probably > can't get because kswapd is spinning. Eventually everyone backs up > waiting for the transaction that never ends. If we're really lucky it > is just GFP_KERNEL where it should NOFS. > > Since you're often stuck in different spots inside shrink_slab, we're > probably not stuck on a lock. But, trying with lock debugging, lockdep > enabled and preempt on is a good idea to rule out locking mistakes. > > Does the fedora debug kernel enable preempt? No ... I've taken the Fedora 15 kernel config directly from their install CD for this (with a bit of munging to get it to work on -rc4) James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 16:50 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 16:50 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote: > Probably easier to debug with a sysrq-l and sysrq-w. Yes, just enabling them for the repro ... I didn't echo 1 > /proc/sys/kernel/sysrq first time around > If you get stuck > on the filesystem, it is probably waiting on ram, which it probably > can't get because kswapd is spinning. Eventually everyone backs up > waiting for the transaction that never ends. If we're really lucky it > is just GFP_KERNEL where it should NOFS. > > Since you're often stuck in different spots inside shrink_slab, we're > probably not stuck on a lock. But, trying with lock debugging, lockdep > enabled and preempt on is a good idea to rule out locking mistakes. > > Does the fedora debug kernel enable preempt? No ... I've taken the Fedora 15 kernel config directly from their install CD for this (with a bit of munging to get it to work on -rc4) James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 16:50 ` James Bottomley @ 2011-04-27 16:54 ` Chris Mason -1 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-27 16:54 UTC (permalink / raw) To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel Excerpts from James Bottomley's message of 2011-04-27 12:50:00 -0400: > On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote: > > Probably easier to debug with a sysrq-l and sysrq-w. > > Yes, just enabling them for the repro ... I didn't echo 1 > > /proc/sys/kernel/sysrq first time around > > > If you get stuck > > on the filesystem, it is probably waiting on ram, which it probably > > can't get because kswapd is spinning. Eventually everyone backs up > > waiting for the transaction that never ends. If we're really lucky it > > is just GFP_KERNEL where it should NOFS. > > > > Since you're often stuck in different spots inside shrink_slab, we're > > probably not stuck on a lock. But, trying with lock debugging, lockdep > > enabled and preempt on is a good idea to rule out locking mistakes. > > > > Does the fedora debug kernel enable preempt? > > No ... I've taken the Fedora 15 kernel config directly from their > install CD for this (with a bit of munging to get it to work on -rc4) Ok, I'd try turning it on so we catch the sleeping with a spinlock held case better. -chris ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 16:54 ` Chris Mason 0 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-27 16:54 UTC (permalink / raw) To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel Excerpts from James Bottomley's message of 2011-04-27 12:50:00 -0400: > On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote: > > Probably easier to debug with a sysrq-l and sysrq-w. > > Yes, just enabling them for the repro ... I didn't echo 1 > > /proc/sys/kernel/sysrq first time around > > > If you get stuck > > on the filesystem, it is probably waiting on ram, which it probably > > can't get because kswapd is spinning. Eventually everyone backs up > > waiting for the transaction that never ends. If we're really lucky it > > is just GFP_KERNEL where it should NOFS. > > > > Since you're often stuck in different spots inside shrink_slab, we're > > probably not stuck on a lock. But, trying with lock debugging, lockdep > > enabled and preempt on is a good idea to rule out locking mistakes. > > > > Does the fedora debug kernel enable preempt? > > No ... I've taken the Fedora 15 kernel config directly from their > install CD for this (with a bit of munging to get it to work on -rc4) Ok, I'd try turning it on so we catch the sleeping with a spinlock held case better. -chris -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 16:54 ` Chris Mason (?) @ 2011-04-27 17:21 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 17:21 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > case better. Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? This is the trace with sysrq-l and sysrq-w The repro this time doesn't have a soft lockup, just the tar is hung and one of my CPUs is in 99% system. James --- [ 453.351255] SysRq : Show backtrace of all active CPUs [ 453.352601] sending NMI to all CPUs: [ 453.353849] NMI backtrace for cpu 3 [ 453.355545] CPU 3 [ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.363188] [ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38 [ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246 [ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001 [ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820 [ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad [ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000 [ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820 [ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 [ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000) [ 453.392649] Stack: [ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5 [ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000 [ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89 [ 453.400015] Call Trace: [ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] [ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 [ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c [ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17 [ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f [ 453.421534] Call Trace: [ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] [ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 [ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c [ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 453.439654] NMI backtrace for cpu 2 [ 453.441508] CPU 2 [ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.449517] [ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046 [ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 [ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640 [ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5 [ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030 [ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002 [ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700) [ 453.479024] Stack: [ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290 [ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170 [ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868 [ 453.486518] Call Trace: [ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 453.499958] Call Trace: [ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.509598] NMI backtrace for cpu 1 [ 453.511390] CPU 1 [ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.519288] [ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 [ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 [ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 [ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 [ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 [ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001 [ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 [ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) [ 453.549274] Stack: [ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 [ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 [ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 [ 453.557156] Call Trace: [ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 453.570292] Call Trace: [ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.579811] NMI backtrace for cpu 0 [ 453.581279] CPU 0 [ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.587576] [ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14 [ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883 [ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040 [ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d [ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000 [ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d [ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001 [ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 [ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0 [ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) [ 453.611134] Stack: [ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb [ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002 [ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77 [ 453.617101] Call Trace: [ 453.618492] <IRQ> [ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 [ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 [ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11 [ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e [ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 453.649695] <EOI> [ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55 [ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2 [ 453.663728] Call Trace: [ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 [ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 [ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11 [ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e [ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 454.680802] SysRq : Show Blocked State [ 454.683427] task PC stack pid father [ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000 [ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000 [ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40 [ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db [ 454.696969] Call Trace: [ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde [ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c [ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17 [ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101 [ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 [ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f [ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c [ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 [ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3 [ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353 [ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121 [ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea [ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe [ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a [ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18 [ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 [ 454.753048] Call Trace: [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 [ 454.839453] Call Trace: [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 [ 454.899530] Call Trace: [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000 [ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000 [ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40 [ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8 [ 454.975323] Call Trace: [ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12 [ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b [ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68 [ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43 [ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640 [ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff [ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63 [ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 [ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f [ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8 [ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277 [ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a [ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0 [ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 [ 455.027487] ktime : 455879.226285 [ 455.030142] sched_clk : 455024.886257 [ 455.032786] cpu_clk : 455024.886397 [ 455.035352] jiffies : 4295123185 [ 455.037904] sched_clock_stable : 1 [ 455.040413] [ 455.042892] sysctl_sched [ 455.045306] .sysctl_sched_latency : 18.000000 [ 455.047775] .sysctl_sched_min_granularity : 2.250000 [ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000 [ 455.052643] .sysctl_sched_child_runs_first : 0 [ 455.055034] .sysctl_sched_features : 7279 [ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic) [ 455.059829] [ 455.059830] cpu#0, 2491.994 MHz [ 455.064443] .nr_running : 0 [ 455.066757] .load : 0 [ 455.069054] .nr_switches : 146510 [ 455.071353] .nr_load_updates : 233084 [ 455.073642] .nr_uninterruptible : 2 [ 455.075894] .next_balance : 4295.122831 [ 455.078152] .curr->pid : 0 [ 455.080396] .clock : 454680.481348 [ 455.082634] .cpu_load[0] : 0 [ 455.084867] .cpu_load[1] : 0 [ 455.087065] .cpu_load[2] : 0 [ 455.089233] .cpu_load[3] : 0 [ 455.091390] .cpu_load[4] : 0 [ 455.093499] .yld_count : 0 [ 455.095605] .sched_switch : 0 [ 455.097667] .sched_count : 149062 [ 455.099765] .sched_goidle : 62756 [ 455.101781] .avg_idle : 1000000 [ 455.103807] .ttwu_count : 77219 [ 455.105958] .ttwu_local : 74144 [ 455.107957] .bkl_count : 0 [ 455.109914] [ 455.109915] cfs_rq[0]:/ [ 455.113642] .exec_clock : 20017.048374 [ 455.115515] .MIN_vruntime : 0.000001 [ 455.117353] .min_vruntime : 28900.800090 [ 455.119185] .max_vruntime : 0.000001 [ 455.121028] .spread : 0.000000 [ 455.122820] .spread0 : 0.000000 [ 455.124581] .nr_spread_over : 54 [ 455.126318] .nr_running : 0 [ 455.128045] .load : 0 [ 455.129743] .load_avg : 0.000000 [ 455.131414] .load_period : 0.000000 [ 455.133052] .load_contrib : 0 [ 455.134692] .load_tg : 0 [ 455.136305] [ 455.136306] runnable tasks: [ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.136310] ---------------------------------------------------------------------------------------------------------- [ 455.142856] [ 455.142857] cpu#1, 2491.994 MHz [ 455.146215] .nr_running : 0 [ 455.147931] .load : 0 [ 455.149648] .nr_switches : 62175 [ 455.151348] .nr_load_updates : 217324 [ 455.153045] .nr_uninterruptible : 2 [ 455.154747] .next_balance : 4295.123306 [ 455.156457] .curr->pid : 0 [ 455.158149] .clock : 455157.364642 [ 455.159867] .cpu_load[0] : 0 [ 455.161594] .cpu_load[1] : 0 [ 455.163283] .cpu_load[2] : 0 [ 455.164954] .cpu_load[3] : 0 [ 455.166575] .cpu_load[4] : 0 [ 455.168185] .yld_count : 60 [ 455.169791] .sched_switch : 0 [ 455.171401] .sched_count : 62899 [ 455.172984] .sched_goidle : 27394 [ 455.174580] .avg_idle : 1000000 [ 455.176171] .ttwu_count : 30510 [ 455.177739] .ttwu_local : 25277 [ 455.179292] .bkl_count : 0 [ 455.180882] [ 455.180883] cfs_rq[1]:/ [ 455.183954] .exec_clock : 10655.021809 [ 455.185550] .MIN_vruntime : 0.000001 [ 455.187141] .min_vruntime : 19718.135550 [ 455.188771] .max_vruntime : 0.000001 [ 455.190407] .spread : 0.000000 [ 455.192016] .spread0 : -9182.664540 [ 455.193634] .nr_spread_over : 80 [ 455.195242] .nr_running : 0 [ 455.196848] .load : 0 [ 455.198441] .load_avg : 0.000000 [ 455.200059] .load_period : 0.000000 [ 455.201654] .load_contrib : 0 [ 455.203240] .load_tg : 0 [ 455.204836] [ 455.204837] runnable tasks: [ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.204841] ---------------------------------------------------------------------------------------------------------- [ 455.211403] [ 455.211404] cpu#2, 2491.994 MHz [ 455.214958] .nr_running : 0 [ 455.216771] .load : 0 [ 455.218564] .nr_switches : 37444 [ 455.220394] .nr_load_updates : 219042 [ 455.222202] .nr_uninterruptible : 0 [ 455.224039] .next_balance : 4295.123375 [ 455.225897] .curr->pid : 0 [ 455.227769] .clock : 455227.204963 [ 455.229681] .cpu_load[0] : 0 [ 455.231548] .cpu_load[1] : 0 [ 455.233353] .cpu_load[2] : 0 [ 455.235059] .cpu_load[3] : 0 [ 455.236676] .cpu_load[4] : 0 [ 455.238280] .yld_count : 0 [ 455.239879] .sched_switch : 0 [ 455.241471] .sched_count : 37815 [ 455.243075] .sched_goidle : 16831 [ 455.244674] .avg_idle : 1000000 [ 455.246270] .ttwu_count : 18348 [ 455.247849] .ttwu_local : 16899 [ 455.249430] .bkl_count : 0 [ 455.250992] [ 455.250993] cfs_rq[2]:/ [ 455.254040] .exec_clock : 6758.351942 [ 455.255630] .MIN_vruntime : 0.000001 [ 455.257236] .min_vruntime : 13719.942866 [ 455.258861] .max_vruntime : 0.000001 [ 455.260497] .spread : 0.000000 [ 455.262122] .spread0 : -15180.857224 [ 455.263753] .nr_spread_over : 21 [ 455.265389] .nr_running : 0 [ 455.267018] .load : 0 [ 455.268637] .load_avg : 0.000000 [ 455.270292] .load_period : 0.000000 [ 455.271911] .load_contrib : 0 [ 455.273530] .load_tg : 0 [ 455.275163] [ 455.275165] runnable tasks: [ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.275169] ---------------------------------------------------------------------------------------------------------- [ 455.281810] [ 455.281811] cpu#3, 2491.994 MHz [ 455.285349] .nr_running : 4 [ 455.287149] .load : 2048 [ 455.288952] .nr_switches : 30358 [ 455.290787] .nr_load_updates : 223302 [ 455.292599] .nr_uninterruptible : 1 [ 455.294419] .next_balance : 4295.123506 [ 455.296250] .curr->pid : 46 [ 455.298090] .clock : 431792.394314 [ 455.299953] .cpu_load[0] : 2048 [ 455.301810] .cpu_load[1] : 2048 [ 455.303610] .cpu_load[2] : 2048 [ 455.305311] .cpu_load[3] : 2048 [ 455.306926] .cpu_load[4] : 2048 [ 455.308531] .yld_count : 63 [ 455.310130] .sched_switch : 0 [ 455.311723] .sched_count : 30824 [ 455.313315] .sched_goidle : 13491 [ 455.314904] .avg_idle : 1000000 [ 455.316481] .ttwu_count : 13858 [ 455.318057] .ttwu_local : 12507 [ 455.319629] .bkl_count : 0 [ 455.321187] [ 455.321188] cfs_rq[3]:/ [ 455.324213] .exec_clock : 5334.144946 [ 455.325795] .MIN_vruntime : 13295.262803 [ 455.327401] .min_vruntime : 13302.523317 [ 455.329046] .max_vruntime : 13295.262803 [ 455.330656] .spread : 0.000000 [ 455.332297] .spread0 : -15598.276773 [ 455.333925] .nr_spread_over : 117 [ 455.335577] .nr_running : 2 [ 455.337204] .load : 2048 [ 455.338823] .load_avg : 0.000000 [ 455.340476] .load_period : 0.000000 [ 455.342094] .load_contrib : 0 [ 455.343702] .load_tg : 0 [ 455.345302] [ 455.345303] rt_rq[3]:/ [ 455.348416] .rt_nr_running : 1 [ 455.350006] .rt_throttled : 0 [ 455.351587] .rt_time : 0.000000 [ 455.353173] .rt_runtime : 950.000000 [ 455.354759] [ 455.354760] runnable tasks: [ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.354764] ---------------------------------------------------------------------------------------------------------- [ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 / [ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 / [ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 / [ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 / [ 455.369317] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 17:21 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 17:21 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > case better. Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? This is the trace with sysrq-l and sysrq-w The repro this time doesn't have a soft lockup, just the tar is hung and one of my CPUs is in 99% system. James --- [ 453.351255] SysRq : Show backtrace of all active CPUs [ 453.352601] sending NMI to all CPUs: [ 453.353849] NMI backtrace for cpu 3 [ 453.355545] CPU 3 [ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.363188] [ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38 [ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246 [ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001 [ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820 [ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad [ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000 [ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820 [ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 [ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000) [ 453.392649] Stack: [ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5 [ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000 [ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89 [ 453.400015] Call Trace: [ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] [ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 [ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c [ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17 [ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f [ 453.421534] Call Trace: [ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] [ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 [ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c [ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 453.439654] NMI backtrace for cpu 2 [ 453.441508] CPU 2 [ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.449517] [ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046 [ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 [ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640 [ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5 [ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030 [ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002 [ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700) [ 453.479024] Stack: [ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290 [ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170 [ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868 [ 453.486518] Call Trace: [ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 453.499958] Call Trace: [ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.509598] NMI backtrace for cpu 1 [ 453.511390] CPU 1 [ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.519288] [ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 [ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 [ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 [ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 [ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 [ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001 [ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 [ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) [ 453.549274] Stack: [ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 [ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 [ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 [ 453.557156] Call Trace: [ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 453.570292] Call Trace: [ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.579811] NMI backtrace for cpu 0 [ 453.581279] CPU 0 [ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.587576] [ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14 [ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883 [ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040 [ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d [ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000 [ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d [ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001 [ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 [ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0 [ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) [ 453.611134] Stack: [ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb [ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002 [ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77 [ 453.617101] Call Trace: [ 453.618492] <IRQ> [ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 [ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 [ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11 [ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e [ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 453.649695] <EOI> [ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55 [ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2 [ 453.663728] Call Trace: [ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 [ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 [ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11 [ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e [ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 454.680802] SysRq : Show Blocked State [ 454.683427] task PC stack pid father [ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000 [ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000 [ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40 [ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db [ 454.696969] Call Trace: [ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde [ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c [ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17 [ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101 [ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 [ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f [ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c [ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 [ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3 [ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353 [ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121 [ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea [ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe [ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a [ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18 [ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 [ 454.753048] Call Trace: [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 [ 454.839453] Call Trace: [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 [ 454.899530] Call Trace: [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000 [ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000 [ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40 [ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8 [ 454.975323] Call Trace: [ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12 [ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b [ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68 [ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43 [ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640 [ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff [ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63 [ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 [ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f [ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8 [ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277 [ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a [ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0 [ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 [ 455.027487] ktime : 455879.226285 [ 455.030142] sched_clk : 455024.886257 [ 455.032786] cpu_clk : 455024.886397 [ 455.035352] jiffies : 4295123185 [ 455.037904] sched_clock_stable : 1 [ 455.040413] [ 455.042892] sysctl_sched [ 455.045306] .sysctl_sched_latency : 18.000000 [ 455.047775] .sysctl_sched_min_granularity : 2.250000 [ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000 [ 455.052643] .sysctl_sched_child_runs_first : 0 [ 455.055034] .sysctl_sched_features : 7279 [ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic) [ 455.059829] [ 455.059830] cpu#0, 2491.994 MHz [ 455.064443] .nr_running : 0 [ 455.066757] .load : 0 [ 455.069054] .nr_switches : 146510 [ 455.071353] .nr_load_updates : 233084 [ 455.073642] .nr_uninterruptible : 2 [ 455.075894] .next_balance : 4295.122831 [ 455.078152] .curr->pid : 0 [ 455.080396] .clock : 454680.481348 [ 455.082634] .cpu_load[0] : 0 [ 455.084867] .cpu_load[1] : 0 [ 455.087065] .cpu_load[2] : 0 [ 455.089233] .cpu_load[3] : 0 [ 455.091390] .cpu_load[4] : 0 [ 455.093499] .yld_count : 0 [ 455.095605] .sched_switch : 0 [ 455.097667] .sched_count : 149062 [ 455.099765] .sched_goidle : 62756 [ 455.101781] .avg_idle : 1000000 [ 455.103807] .ttwu_count : 77219 [ 455.105958] .ttwu_local : 74144 [ 455.107957] .bkl_count : 0 [ 455.109914] [ 455.109915] cfs_rq[0]:/ [ 455.113642] .exec_clock : 20017.048374 [ 455.115515] .MIN_vruntime : 0.000001 [ 455.117353] .min_vruntime : 28900.800090 [ 455.119185] .max_vruntime : 0.000001 [ 455.121028] .spread : 0.000000 [ 455.122820] .spread0 : 0.000000 [ 455.124581] .nr_spread_over : 54 [ 455.126318] .nr_running : 0 [ 455.128045] .load : 0 [ 455.129743] .load_avg : 0.000000 [ 455.131414] .load_period : 0.000000 [ 455.133052] .load_contrib : 0 [ 455.134692] .load_tg : 0 [ 455.136305] [ 455.136306] runnable tasks: [ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.136310] ---------------------------------------------------------------------------------------------------------- [ 455.142856] [ 455.142857] cpu#1, 2491.994 MHz [ 455.146215] .nr_running : 0 [ 455.147931] .load : 0 [ 455.149648] .nr_switches : 62175 [ 455.151348] .nr_load_updates : 217324 [ 455.153045] .nr_uninterruptible : 2 [ 455.154747] .next_balance : 4295.123306 [ 455.156457] .curr->pid : 0 [ 455.158149] .clock : 455157.364642 [ 455.159867] .cpu_load[0] : 0 [ 455.161594] .cpu_load[1] : 0 [ 455.163283] .cpu_load[2] : 0 [ 455.164954] .cpu_load[3] : 0 [ 455.166575] .cpu_load[4] : 0 [ 455.168185] .yld_count : 60 [ 455.169791] .sched_switch : 0 [ 455.171401] .sched_count : 62899 [ 455.172984] .sched_goidle : 27394 [ 455.174580] .avg_idle : 1000000 [ 455.176171] .ttwu_count : 30510 [ 455.177739] .ttwu_local : 25277 [ 455.179292] .bkl_count : 0 [ 455.180882] [ 455.180883] cfs_rq[1]:/ [ 455.183954] .exec_clock : 10655.021809 [ 455.185550] .MIN_vruntime : 0.000001 [ 455.187141] .min_vruntime : 19718.135550 [ 455.188771] .max_vruntime : 0.000001 [ 455.190407] .spread : 0.000000 [ 455.192016] .spread0 : -9182.664540 [ 455.193634] .nr_spread_over : 80 [ 455.195242] .nr_running : 0 [ 455.196848] .load : 0 [ 455.198441] .load_avg : 0.000000 [ 455.200059] .load_period : 0.000000 [ 455.201654] .load_contrib : 0 [ 455.203240] .load_tg : 0 [ 455.204836] [ 455.204837] runnable tasks: [ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.204841] ---------------------------------------------------------------------------------------------------------- [ 455.211403] [ 455.211404] cpu#2, 2491.994 MHz [ 455.214958] .nr_running : 0 [ 455.216771] .load : 0 [ 455.218564] .nr_switches : 37444 [ 455.220394] .nr_load_updates : 219042 [ 455.222202] .nr_uninterruptible : 0 [ 455.224039] .next_balance : 4295.123375 [ 455.225897] .curr->pid : 0 [ 455.227769] .clock : 455227.204963 [ 455.229681] .cpu_load[0] : 0 [ 455.231548] .cpu_load[1] : 0 [ 455.233353] .cpu_load[2] : 0 [ 455.235059] .cpu_load[3] : 0 [ 455.236676] .cpu_load[4] : 0 [ 455.238280] .yld_count : 0 [ 455.239879] .sched_switch : 0 [ 455.241471] .sched_count : 37815 [ 455.243075] .sched_goidle : 16831 [ 455.244674] .avg_idle : 1000000 [ 455.246270] .ttwu_count : 18348 [ 455.247849] .ttwu_local : 16899 [ 455.249430] .bkl_count : 0 [ 455.250992] [ 455.250993] cfs_rq[2]:/ [ 455.254040] .exec_clock : 6758.351942 [ 455.255630] .MIN_vruntime : 0.000001 [ 455.257236] .min_vruntime : 13719.942866 [ 455.258861] .max_vruntime : 0.000001 [ 455.260497] .spread : 0.000000 [ 455.262122] .spread0 : -15180.857224 [ 455.263753] .nr_spread_over : 21 [ 455.265389] .nr_running : 0 [ 455.267018] .load : 0 [ 455.268637] .load_avg : 0.000000 [ 455.270292] .load_period : 0.000000 [ 455.271911] .load_contrib : 0 [ 455.273530] .load_tg : 0 [ 455.275163] [ 455.275165] runnable tasks: [ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.275169] ---------------------------------------------------------------------------------------------------------- [ 455.281810] [ 455.281811] cpu#3, 2491.994 MHz [ 455.285349] .nr_running : 4 [ 455.287149] .load : 2048 [ 455.288952] .nr_switches : 30358 [ 455.290787] .nr_load_updates : 223302 [ 455.292599] .nr_uninterruptible : 1 [ 455.294419] .next_balance : 4295.123506 [ 455.296250] .curr->pid : 46 [ 455.298090] .clock : 431792.394314 [ 455.299953] .cpu_load[0] : 2048 [ 455.301810] .cpu_load[1] : 2048 [ 455.303610] .cpu_load[2] : 2048 [ 455.305311] .cpu_load[3] : 2048 [ 455.306926] .cpu_load[4] : 2048 [ 455.308531] .yld_count : 63 [ 455.310130] .sched_switch : 0 [ 455.311723] .sched_count : 30824 [ 455.313315] .sched_goidle : 13491 [ 455.314904] .avg_idle : 1000000 [ 455.316481] .ttwu_count : 13858 [ 455.318057] .ttwu_local : 12507 [ 455.319629] .bkl_count : 0 [ 455.321187] [ 455.321188] cfs_rq[3]:/ [ 455.324213] .exec_clock : 5334.144946 [ 455.325795] .MIN_vruntime : 13295.262803 [ 455.327401] .min_vruntime : 13302.523317 [ 455.329046] .max_vruntime : 13295.262803 [ 455.330656] .spread : 0.000000 [ 455.332297] .spread0 : -15598.276773 [ 455.333925] .nr_spread_over : 117 [ 455.335577] .nr_running : 2 [ 455.337204] .load : 2048 [ 455.338823] .load_avg : 0.000000 [ 455.340476] .load_period : 0.000000 [ 455.342094] .load_contrib : 0 [ 455.343702] .load_tg : 0 [ 455.345302] [ 455.345303] rt_rq[3]:/ [ 455.348416] .rt_nr_running : 1 [ 455.350006] .rt_throttled : 0 [ 455.351587] .rt_time : 0.000000 [ 455.353173] .rt_runtime : 950.000000 [ 455.354759] [ 455.354760] runnable tasks: [ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.354764] ---------------------------------------------------------------------------------------------------------- [ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 / [ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 / [ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 / [ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 / [ 455.369317] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 17:21 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 17:21 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > case better. Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? This is the trace with sysrq-l and sysrq-w The repro this time doesn't have a soft lockup, just the tar is hung and one of my CPUs is in 99% system. James --- [ 453.351255] SysRq : Show backtrace of all active CPUs [ 453.352601] sending NMI to all CPUs: [ 453.353849] NMI backtrace for cpu 3 [ 453.355545] CPU 3 [ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.363188] [ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38 [ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246 [ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001 [ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820 [ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad [ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000 [ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820 [ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 [ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000) [ 453.392649] Stack: [ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5 [ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000 [ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89 [ 453.400015] Call Trace: [ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] [ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 [ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c [ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17 [ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f [ 453.421534] Call Trace: [ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] [ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 [ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 [ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c [ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 453.439654] NMI backtrace for cpu 2 [ 453.441508] CPU 2 [ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.449517] [ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046 [ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 [ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640 [ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5 [ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030 [ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002 [ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700) [ 453.479024] Stack: [ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290 [ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170 [ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868 [ 453.486518] Call Trace: [ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 453.499958] Call Trace: [ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.509598] NMI backtrace for cpu 1 [ 453.511390] CPU 1 [ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.519288] [ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 [ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 [ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 [ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 [ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 [ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001 [ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 [ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) [ 453.549274] Stack: [ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 [ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 [ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 [ 453.557156] Call Trace: [ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 453.570292] Call Trace: [ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 453.579811] NMI backtrace for cpu 0 [ 453.581279] CPU 0 [ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 453.587576] [ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14 [ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883 [ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040 [ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d [ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000 [ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d [ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001 [ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 [ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0 [ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) [ 453.611134] Stack: [ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb [ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002 [ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77 [ 453.617101] Call Trace: [ 453.618492] <IRQ> [ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 [ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 [ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11 [ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e [ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 453.649695] <EOI> [ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55 [ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2 [ 453.663728] Call Trace: [ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 [ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 [ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11 [ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e [ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 454.680802] SysRq : Show Blocked State [ 454.683427] task PC stack pid father [ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000 [ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000 [ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40 [ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db [ 454.696969] Call Trace: [ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde [ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c [ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17 [ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101 [ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 [ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f [ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c [ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 [ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3 [ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353 [ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121 [ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea [ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe [ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a [ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18 [ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 [ 454.753048] Call Trace: [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 [ 454.839453] Call Trace: [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 [ 454.899530] Call Trace: [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000 [ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000 [ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40 [ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8 [ 454.975323] Call Trace: [ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12 [ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b [ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68 [ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43 [ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640 [ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff [ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63 [ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 [ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f [ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8 [ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277 [ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a [ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0 [ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 [ 455.027487] ktime : 455879.226285 [ 455.030142] sched_clk : 455024.886257 [ 455.032786] cpu_clk : 455024.886397 [ 455.035352] jiffies : 4295123185 [ 455.037904] sched_clock_stable : 1 [ 455.040413] [ 455.042892] sysctl_sched [ 455.045306] .sysctl_sched_latency : 18.000000 [ 455.047775] .sysctl_sched_min_granularity : 2.250000 [ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000 [ 455.052643] .sysctl_sched_child_runs_first : 0 [ 455.055034] .sysctl_sched_features : 7279 [ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic) [ 455.059829] [ 455.059830] cpu#0, 2491.994 MHz [ 455.064443] .nr_running : 0 [ 455.066757] .load : 0 [ 455.069054] .nr_switches : 146510 [ 455.071353] .nr_load_updates : 233084 [ 455.073642] .nr_uninterruptible : 2 [ 455.075894] .next_balance : 4295.122831 [ 455.078152] .curr->pid : 0 [ 455.080396] .clock : 454680.481348 [ 455.082634] .cpu_load[0] : 0 [ 455.084867] .cpu_load[1] : 0 [ 455.087065] .cpu_load[2] : 0 [ 455.089233] .cpu_load[3] : 0 [ 455.091390] .cpu_load[4] : 0 [ 455.093499] .yld_count : 0 [ 455.095605] .sched_switch : 0 [ 455.097667] .sched_count : 149062 [ 455.099765] .sched_goidle : 62756 [ 455.101781] .avg_idle : 1000000 [ 455.103807] .ttwu_count : 77219 [ 455.105958] .ttwu_local : 74144 [ 455.107957] .bkl_count : 0 [ 455.109914] [ 455.109915] cfs_rq[0]:/ [ 455.113642] .exec_clock : 20017.048374 [ 455.115515] .MIN_vruntime : 0.000001 [ 455.117353] .min_vruntime : 28900.800090 [ 455.119185] .max_vruntime : 0.000001 [ 455.121028] .spread : 0.000000 [ 455.122820] .spread0 : 0.000000 [ 455.124581] .nr_spread_over : 54 [ 455.126318] .nr_running : 0 [ 455.128045] .load : 0 [ 455.129743] .load_avg : 0.000000 [ 455.131414] .load_period : 0.000000 [ 455.133052] .load_contrib : 0 [ 455.134692] .load_tg : 0 [ 455.136305] [ 455.136306] runnable tasks: [ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.136310] ---------------------------------------------------------------------------------------------------------- [ 455.142856] [ 455.142857] cpu#1, 2491.994 MHz [ 455.146215] .nr_running : 0 [ 455.147931] .load : 0 [ 455.149648] .nr_switches : 62175 [ 455.151348] .nr_load_updates : 217324 [ 455.153045] .nr_uninterruptible : 2 [ 455.154747] .next_balance : 4295.123306 [ 455.156457] .curr->pid : 0 [ 455.158149] .clock : 455157.364642 [ 455.159867] .cpu_load[0] : 0 [ 455.161594] .cpu_load[1] : 0 [ 455.163283] .cpu_load[2] : 0 [ 455.164954] .cpu_load[3] : 0 [ 455.166575] .cpu_load[4] : 0 [ 455.168185] .yld_count : 60 [ 455.169791] .sched_switch : 0 [ 455.171401] .sched_count : 62899 [ 455.172984] .sched_goidle : 27394 [ 455.174580] .avg_idle : 1000000 [ 455.176171] .ttwu_count : 30510 [ 455.177739] .ttwu_local : 25277 [ 455.179292] .bkl_count : 0 [ 455.180882] [ 455.180883] cfs_rq[1]:/ [ 455.183954] .exec_clock : 10655.021809 [ 455.185550] .MIN_vruntime : 0.000001 [ 455.187141] .min_vruntime : 19718.135550 [ 455.188771] .max_vruntime : 0.000001 [ 455.190407] .spread : 0.000000 [ 455.192016] .spread0 : -9182.664540 [ 455.193634] .nr_spread_over : 80 [ 455.195242] .nr_running : 0 [ 455.196848] .load : 0 [ 455.198441] .load_avg : 0.000000 [ 455.200059] .load_period : 0.000000 [ 455.201654] .load_contrib : 0 [ 455.203240] .load_tg : 0 [ 455.204836] [ 455.204837] runnable tasks: [ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.204841] ---------------------------------------------------------------------------------------------------------- [ 455.211403] [ 455.211404] cpu#2, 2491.994 MHz [ 455.214958] .nr_running : 0 [ 455.216771] .load : 0 [ 455.218564] .nr_switches : 37444 [ 455.220394] .nr_load_updates : 219042 [ 455.222202] .nr_uninterruptible : 0 [ 455.224039] .next_balance : 4295.123375 [ 455.225897] .curr->pid : 0 [ 455.227769] .clock : 455227.204963 [ 455.229681] .cpu_load[0] : 0 [ 455.231548] .cpu_load[1] : 0 [ 455.233353] .cpu_load[2] : 0 [ 455.235059] .cpu_load[3] : 0 [ 455.236676] .cpu_load[4] : 0 [ 455.238280] .yld_count : 0 [ 455.239879] .sched_switch : 0 [ 455.241471] .sched_count : 37815 [ 455.243075] .sched_goidle : 16831 [ 455.244674] .avg_idle : 1000000 [ 455.246270] .ttwu_count : 18348 [ 455.247849] .ttwu_local : 16899 [ 455.249430] .bkl_count : 0 [ 455.250992] [ 455.250993] cfs_rq[2]:/ [ 455.254040] .exec_clock : 6758.351942 [ 455.255630] .MIN_vruntime : 0.000001 [ 455.257236] .min_vruntime : 13719.942866 [ 455.258861] .max_vruntime : 0.000001 [ 455.260497] .spread : 0.000000 [ 455.262122] .spread0 : -15180.857224 [ 455.263753] .nr_spread_over : 21 [ 455.265389] .nr_running : 0 [ 455.267018] .load : 0 [ 455.268637] .load_avg : 0.000000 [ 455.270292] .load_period : 0.000000 [ 455.271911] .load_contrib : 0 [ 455.273530] .load_tg : 0 [ 455.275163] [ 455.275165] runnable tasks: [ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.275169] ---------------------------------------------------------------------------------------------------------- [ 455.281810] [ 455.281811] cpu#3, 2491.994 MHz [ 455.285349] .nr_running : 4 [ 455.287149] .load : 2048 [ 455.288952] .nr_switches : 30358 [ 455.290787] .nr_load_updates : 223302 [ 455.292599] .nr_uninterruptible : 1 [ 455.294419] .next_balance : 4295.123506 [ 455.296250] .curr->pid : 46 [ 455.298090] .clock : 431792.394314 [ 455.299953] .cpu_load[0] : 2048 [ 455.301810] .cpu_load[1] : 2048 [ 455.303610] .cpu_load[2] : 2048 [ 455.305311] .cpu_load[3] : 2048 [ 455.306926] .cpu_load[4] : 2048 [ 455.308531] .yld_count : 63 [ 455.310130] .sched_switch : 0 [ 455.311723] .sched_count : 30824 [ 455.313315] .sched_goidle : 13491 [ 455.314904] .avg_idle : 1000000 [ 455.316481] .ttwu_count : 13858 [ 455.318057] .ttwu_local : 12507 [ 455.319629] .bkl_count : 0 [ 455.321187] [ 455.321188] cfs_rq[3]:/ [ 455.324213] .exec_clock : 5334.144946 [ 455.325795] .MIN_vruntime : 13295.262803 [ 455.327401] .min_vruntime : 13302.523317 [ 455.329046] .max_vruntime : 13295.262803 [ 455.330656] .spread : 0.000000 [ 455.332297] .spread0 : -15598.276773 [ 455.333925] .nr_spread_over : 117 [ 455.335577] .nr_running : 2 [ 455.337204] .load : 2048 [ 455.338823] .load_avg : 0.000000 [ 455.340476] .load_period : 0.000000 [ 455.342094] .load_contrib : 0 [ 455.343702] .load_tg : 0 [ 455.345302] [ 455.345303] rt_rq[3]:/ [ 455.348416] .rt_nr_running : 1 [ 455.350006] .rt_throttled : 0 [ 455.351587] .rt_time : 0.000000 [ 455.353173] .rt_runtime : 950.000000 [ 455.354759] [ 455.354760] runnable tasks: [ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 455.354764] ---------------------------------------------------------------------------------------------------------- [ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 / [ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 / [ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 / [ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 / [ 455.369317] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 17:21 ` James Bottomley @ 2011-04-27 17:34 ` Chris Mason -1 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-27 17:34 UTC (permalink / raw) To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [ cc'd linux-ext4 ] James is hitting softlockups in kswapd while doing writes to a large ext4 file. Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400: > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > case better. > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > This is the trace with sysrq-l and sysrq-w > > The repro this time doesn't have a soft lockup, just the tar is hung and > one of my CPUs is in 99% system. > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > [ 454.753048] Call Trace: > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a So our flusher threads are stuck waiting in jbd2_journal_get_write_access, which means they aren't cleaning dirty pages. In order to get write access, they probably need the transaction to commit: > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > [ 454.839453] Call Trace: > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 But that seems to be waiting for a page lock. Probably the same page lock held by the flusher thread? Looks like tar is stuck in the same boat down below. > [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 > [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 > [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 > [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 > [ 454.899530] Call Trace: > [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e > [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 > [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a > [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b -chris ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 17:34 ` Chris Mason 0 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-27 17:34 UTC (permalink / raw) To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [ cc'd linux-ext4 ] James is hitting softlockups in kswapd while doing writes to a large ext4 file. Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400: > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > case better. > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > This is the trace with sysrq-l and sysrq-w > > The repro this time doesn't have a soft lockup, just the tar is hung and > one of my CPUs is in 99% system. > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > [ 454.753048] Call Trace: > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a So our flusher threads are stuck waiting in jbd2_journal_get_write_access, which means they aren't cleaning dirty pages. In order to get write access, they probably need the transaction to commit: > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > [ 454.839453] Call Trace: > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 But that seems to be waiting for a page lock. Probably the same page lock held by the flusher thread? Looks like tar is stuck in the same boat down below. > [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 > [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 > [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 > [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 > [ 454.899530] Call Trace: > [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e > [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 > [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a > [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b -chris -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 17:34 ` Chris Mason @ 2011-04-27 17:50 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 17:50 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote: > [ cc'd linux-ext4 ] > > James is hitting softlockups in kswapd while doing writes to a large > ext4 file. > > Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400: > > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > > case better. > > > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > > > This is the trace with sysrq-l and sysrq-w > > > > > The repro this time doesn't have a soft lockup, just the tar is hung and > > one of my CPUs is in 99% system. > > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > > [ 454.753048] Call Trace: > > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > > So our flusher threads are stuck waiting in > jbd2_journal_get_write_access, which means they aren't cleaning dirty > pages. > > In order to get write access, they probably need the transaction to > commit: > > > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > > [ 454.839453] Call Trace: > > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > > But that seems to be waiting for a page lock. Probably the same page > lock held by the flusher thread? Looks like tar is stuck in the same > boat down below. To test the theory, Chris asked me to try with data=ordered. Unfortunately, the deadlock still shows up. This is what I get. James --- [ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 263.756081] CPU 2 [ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 263.763033] [ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166 [ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206 [ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002 [ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50 [ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2 [ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e [ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9 [ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) [ 263.788491] Stack: [ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700 [ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c [ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000 [ 263.795038] Call Trace: [ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c [ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7 [ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7 [ 263.809185] Call Trace: [ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c [ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 272.421176] SysRq : Show backtrace of all active CPUs [ 272.422748] sending NMI to all CPUs: [ 272.424304] NMI backtrace for cpu 2 [ 272.426364] CPU 2 [ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.435128] [ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae [ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282 [ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000 [ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0 [ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000 [ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000 [ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) [ 272.465234] Stack: [ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a [ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000 [ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000 [ 272.472144] Call Trace: [ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 [ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 [ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798 [ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c [ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c [ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8 [ 272.492911] Call Trace: [ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 [ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 [ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798 [ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c [ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 272.510355] NMI backtrace for cpu 1 [ 272.511996] CPU 1 [ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.519303] [ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 [ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 [ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 [ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 [ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 [ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001 [ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 [ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) [ 272.546756] Stack: [ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 [ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 [ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 [ 272.553780] Call Trace: [ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 272.566380] Call Trace: [ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 272.575463] NMI backtrace for cpu 0 [ 272.576865] CPU 0 [ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.582932] [ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e [ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887 [ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040 [ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958 [ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000 [ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008 [ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001 [ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 [ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0 [ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) [ 272.605715] Stack: [ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690 [ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310 [ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001 [ 272.611653] Call Trace: [ 272.613026] <IRQ> [ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 272.639155] <EOI> [ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2 [ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48 [ 272.654210] Call Trace: [ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 272.683442] NMI backtrace for cpu 3 [ 272.684688] CPU 3 [ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.690650] [ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046 [ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 [ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640 [ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4 [ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030 [ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003 [ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 [ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700) [ 272.715001] Stack: [ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290 [ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170 [ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868 [ 272.721848] Call Trace: [ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 272.733594] Call Trace: [ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 274.865072] SysRq : Show Blocked State [ 274.866350] task PC stack pid father [ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000 [ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000 [ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40 [ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0 [ 274.872869] Call Trace: [ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b [ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d [ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28 [ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39 [ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194 [ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a [ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 [ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000 [ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000 [ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40 [ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720 [ 274.906327] Call Trace: [ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b [ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8 [ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 [ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 [ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd [ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 [ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 [ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105 [ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 [ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 [ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17 [ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a [ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e [ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 [ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 [ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9 [ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d [ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56 [ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a [ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc [ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 [ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b [ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 [ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34 [ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1 [ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d [ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 [ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d [ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000 [ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000 [ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40 [ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3 [ 274.936966] Call Trace: [ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 [ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 [ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a [ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 [ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000 [ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246 [ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40 [ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720 [ 274.937011] Call Trace: [ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b [ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8 [ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 [ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 [ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd [ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 [ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 [ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105 [ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 [ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 [ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17 [ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a [ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e [ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 [ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 [ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56 [ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a [ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc [ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 [ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b [ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 [ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d [ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde [ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 [ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d [ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.937082] tar D 0000000000000000 0 980 826 0x00000000 [ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000 [ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40 [ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282 [ 274.937090] Call Trace: [ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde [ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a [ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98 [ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6 [ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a [ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 [ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba [ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 [ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff [ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 [ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 [ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3 [ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e [ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004 [ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000 [ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40 [ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908 [ 274.937135] Call Trace: [ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d [ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c [ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d [ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d [ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d [ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c [ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 [ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 [ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30 [ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c [ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 [ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131 [ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 [ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff [ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 [ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e [ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004 [ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000 [ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40 [ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908 [ 274.937195] Call Trace: [ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d [ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c [ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d [ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d [ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d [ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c [ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 [ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 [ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30 [ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c [ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 [ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131 [ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 [ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff [ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 [ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e [ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004 [ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000 [ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40 [ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0 [ 274.937252] Call Trace: [ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d [ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c [ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d [ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d [ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153 [ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf [ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33 [ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a [ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67 [ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e [ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e [ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab [ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c [ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf [ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86 [ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23 [ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d [ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100 [ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22 [ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004 [ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000 [ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40 [ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3 [ 274.937311] Call Trace: [ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465 [ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda [ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15 [ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120 [ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212 [ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74 [ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef [ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33 [ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18 [ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 [ 274.937336] ktime : 275382.978028 [ 274.937337] sched_clk : 274937.334155 [ 274.937338] cpu_clk : 274937.334189 [ 274.937339] jiffies : 4294942678 [ 274.937340] sched_clock_stable : 1 [ 274.937341] [ 274.937342] sysctl_sched [ 274.937342] .sysctl_sched_latency : 18.000000 [ 274.937344] .sysctl_sched_min_granularity : 2.250000 [ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000 [ 274.937346] .sysctl_sched_child_runs_first : 0 [ 274.937347] .sysctl_sched_features : 7279 [ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic) [ 274.937349] [ 274.937350] cpu#0, 2492.220 MHz [ 274.937350] .nr_running : 0 [ 274.937351] .load : 0 [ 274.937352] .nr_switches : 105039 [ 274.937353] .nr_load_updates : 121182 [ 274.937354] .nr_uninterruptible : 8 [ 274.937355] .next_balance : 4294.942607 [ 274.937356] .curr->pid : 0 [ 274.937357] .clock : 274864.523786 [ 274.937359] .cpu_load[0] : 0 [ 274.937360] .cpu_load[1] : 0 [ 274.937361] .cpu_load[2] : 0 [ 274.937362] .cpu_load[3] : 0 [ 274.937363] .cpu_load[4] : 0 [ 274.937364] .yld_count : 0 [ 274.937364] .sched_switch : 0 [ 274.937365] .sched_count : 107487 [ 274.937366] .sched_goidle : 43368 [ 274.937367] .avg_idle : 1000000 [ 274.937368] .ttwu_count : 56534 [ 274.937369] .ttwu_local : 53925 [ 274.937370] .bkl_count : 0 [ 274.937371] [ 274.937372] cfs_rq[0]:/system [ 274.937372] .exec_clock : 5906.036519 [ 274.937374] .MIN_vruntime : 0.000001 [ 274.937375] .min_vruntime : 7138.254657 [ 274.937376] .max_vruntime : 0.000001 [ 274.937377] .spread : 0.000000 [ 274.937378] .spread0 : -14302.149798 [ 274.937379] .nr_spread_over : 0 [ 274.937379] .nr_running : 0 [ 274.937380] .load : 0 [ 274.937381] .load_avg : 159.999997 [ 274.937382] .load_period : 5.496191 [ 274.937383] .load_contrib : 29 [ 274.937384] .load_tg : 29 [ 274.937385] .se->exec_start : 272446.423795 [ 274.937386] .se->vruntime : 21431.398168 [ 274.937387] .se->sum_exec_runtime : 5906.111805 [ 274.937388] .se->statistics.wait_start : 0.000000 [ 274.937389] .se->statistics.sleep_start : 0.000000 [ 274.937390] .se->statistics.block_start : 0.000000 [ 274.937391] .se->statistics.sleep_max : 0.000000 [ 274.937392] .se->statistics.block_max : 0.000000 [ 274.937393] .se->statistics.exec_max : 15.266617 [ 274.937394] .se->statistics.slice_max : 5.258344 [ 274.937395] .se->statistics.wait_max : 7.305679 [ 274.937396] .se->statistics.wait_sum : 190.506686 [ 274.937397] .se->statistics.wait_count : 18946 [ 274.937398] .se->load.weight : 2 [ 274.937399] [ 274.937399] cfs_rq[0]:/ [ 274.937400] .exec_clock : 17409.691339 [ 274.937401] .MIN_vruntime : 0.000001 [ 274.937402] .min_vruntime : 21440.404455 [ 274.937403] .max_vruntime : 0.000001 [ 274.937403] .spread : 0.000000 [ 274.937404] .spread0 : 0.000000 [ 274.937405] .nr_spread_over : 58 [ 274.937406] .nr_running : 0 [ 274.937407] .load : 0 [ 274.937408] .load_avg : 0.000000 [ 274.937409] .load_period : 0.000000 [ 274.937409] .load_contrib : 0 [ 274.937410] .load_tg : 0 [ 274.937411] [ 274.937412] cfs_rq[0]:/system/rescue.service [ 274.937413] .exec_clock : 4319.868509 [ 274.937414] .MIN_vruntime : 0.000001 [ 274.937414] .min_vruntime : 4255.887524 [ 274.937415] .max_vruntime : 0.000001 [ 274.937416] .spread : 0.000000 [ 274.937417] .spread0 : -17184.516931 [ 274.937418] .nr_spread_over : 24 [ 274.937419] .nr_running : 0 [ 274.937420] .load : 0 [ 274.937421] .load_avg : 160.004112 [ 274.937421] .load_period : 5.496149 [ 274.937422] .load_contrib : 29 [ 274.937423] .load_tg : 29 [ 274.937424] .se->exec_start : 272446.423795 [ 274.937425] .se->vruntime : 7138.254657 [ 274.937426] .se->sum_exec_runtime : 4319.796162 [ 274.937427] .se->statistics.wait_start : 0.000000 [ 274.937428] .se->statistics.sleep_start : 0.000000 [ 274.937429] .se->statistics.block_start : 0.000000 [ 274.937430] .se->statistics.sleep_max : 0.000000 [ 274.937431] .se->statistics.block_max : 0.000000 [ 274.937432] .se->statistics.exec_max : 6.213083 [ 274.937432] .se->statistics.slice_max : 7.894226 [ 274.937433] .se->statistics.wait_max : 4.870567 [ 274.937434] .se->statistics.wait_sum : 91.737803 [ 274.937435] .se->statistics.wait_count : 15504 [ 274.937436] .se->load.weight : 2 [ 274.937437] [ 274.937438] runnable tasks: [ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937439] ---------------------------------------------------------------------------------------------------------- [ 274.937444] [ 274.937445] cpu#1, 2492.220 MHz [ 274.937445] .nr_running : 0 [ 274.937446] .load : 0 [ 274.937447] .nr_switches : 35274 [ 274.937448] .nr_load_updates : 105259 [ 274.937449] .nr_uninterruptible : 1 [ 274.937449] .next_balance : 4294.942681 [ 274.937450] .curr->pid : 0 [ 274.937451] .clock : 274937.357922 [ 274.937452] .cpu_load[0] : 0 [ 274.937453] .cpu_load[1] : 0 [ 274.937454] .cpu_load[2] : 0 [ 274.937454] .cpu_load[3] : 0 [ 274.937455] .cpu_load[4] : 0 [ 274.937456] .yld_count : 0 [ 274.937457] .sched_switch : 0 [ 274.937458] .sched_count : 35560 [ 274.937458] .sched_goidle : 14561 [ 274.937459] .avg_idle : 1000000 [ 274.937460] .ttwu_count : 16938 [ 274.937461] .ttwu_local : 14491 [ 274.937462] .bkl_count : 0 [ 274.937463] [ 274.937463] cfs_rq[1]:/ [ 274.937464] .exec_clock : 5221.725565 [ 274.937465] .MIN_vruntime : 0.000001 [ 274.937466] .min_vruntime : 9077.562937 [ 274.937466] .max_vruntime : 0.000001 [ 274.937467] .spread : 0.000000 [ 274.937468] .spread0 : -12362.841518 [ 274.937469] .nr_spread_over : 69 [ 274.937470] .nr_running : 0 [ 274.937471] .load : 0 [ 274.937472] .load_avg : 0.000000 [ 274.937473] .load_period : 0.000000 [ 274.937473] .load_contrib : 0 [ 274.937474] .load_tg : 0 [ 274.937475] [ 274.937476] runnable tasks: [ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937477] ---------------------------------------------------------------------------------------------------------- [ 274.937481] [ 274.937482] cpu#2, 2492.220 MHz [ 274.937482] .nr_running : 4 [ 274.937483] .load : 2048 [ 274.937484] .nr_switches : 7092 [ 274.937485] .nr_load_updates : 100502 [ 274.937486] .nr_uninterruptible : 0 [ 274.937487] .next_balance : 4294.942806 [ 274.937487] .curr->pid : 46 [ 274.937488] .clock : 183206.600407 [ 274.937489] .cpu_load[0] : 2048 [ 274.937490] .cpu_load[1] : 2048 [ 274.937491] .cpu_load[2] : 2048 [ 274.937492] .cpu_load[3] : 2048 [ 274.937492] .cpu_load[4] : 2048 [ 274.937493] .yld_count : 0 [ 274.937494] .sched_switch : 0 [ 274.937495] .sched_count : 7323 [ 274.937496] .sched_goidle : 2831 [ 274.937496] .avg_idle : 1000000 [ 274.937497] .ttwu_count : 3275 [ 274.937498] .ttwu_local : 2028 [ 274.937499] .bkl_count : 0 [ 274.937500] [ 274.937500] cfs_rq[2]:/ [ 274.937501] .exec_clock : 2187.192139 [ 274.937502] .MIN_vruntime : 5109.505143 [ 274.937503] .min_vruntime : 5118.505143 [ 274.937504] .max_vruntime : 5109.505143 [ 274.937505] .spread : 0.000000 [ 274.937506] .spread0 : -16321.899312 [ 274.937506] .nr_spread_over : 31 [ 274.937507] .nr_running : 2 [ 274.937508] .load : 2048 [ 274.937509] .load_avg : 0.000000 [ 274.937510] .load_period : 0.000000 [ 274.937511] .load_contrib : 0 [ 274.937511] .load_tg : 0 [ 274.937512] [ 274.937513] rt_rq[2]:/ [ 274.937513] .rt_nr_running : 1 [ 274.937514] .rt_throttled : 0 [ 274.937515] .rt_time : 0.000000 [ 274.937516] .rt_runtime : 950.000000 [ 274.937517] [ 274.937517] runnable tasks: [ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937519] ---------------------------------------------------------------------------------------------------------- [ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 / [ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 / [ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 / [ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 / [ 274.937535] [ 274.937536] cpu#3, 2492.220 MHz [ 274.937536] .nr_running : 1 [ 274.937537] .load : 1024 [ 274.937538] .nr_switches : 22324 [ 274.937539] .nr_load_updates : 59855 [ 274.937540] .nr_uninterruptible : 0 [ 274.937541] .next_balance : 4294.942679 [ 274.937542] .curr->pid : 73 [ 274.937543] .clock : 274936.938702 [ 274.937543] .cpu_load[0] : 0 [ 274.937544] .cpu_load[1] : 0 [ 274.937545] .cpu_load[2] : 0 [ 274.937546] .cpu_load[3] : 0 [ 274.937546] .cpu_load[4] : 0 [ 274.937547] .yld_count : 123 [ 274.937548] .sched_switch : 0 [ 274.937549] .sched_count : 22774 [ 274.937550] .sched_goidle : 9952 [ 274.937551] .avg_idle : 946660 [ 274.937551] .ttwu_count : 10075 [ 274.937552] .ttwu_local : 8964 [ 274.937553] .bkl_count : 0 [ 274.937554] [ 274.937554] cfs_rq[3]:/ [ 274.937555] .exec_clock : 3244.912177 [ 274.937556] .MIN_vruntime : 0.000001 [ 274.937557] .min_vruntime : 7517.482154 [ 274.937558] .max_vruntime : 0.000001 [ 274.937559] .spread : 0.000000 [ 274.937559] .spread0 : -13922.922301 [ 274.937560] .nr_spread_over : 128 [ 274.937561] .nr_running : 1 [ 274.937562] .load : 1024 [ 274.937563] .load_avg : 0.000000 [ 274.937564] .load_period : 0.000000 [ 274.937565] .load_contrib : 0 [ 274.937565] .load_tg : 0 [ 274.937567] [ 274.937567] runnable tasks: [ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937568] ---------------------------------------------------------------------------------------------------------- [ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 / [ 274.937575] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 17:50 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 17:50 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote: > [ cc'd linux-ext4 ] > > James is hitting softlockups in kswapd while doing writes to a large > ext4 file. > > Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400: > > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > > case better. > > > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > > > This is the trace with sysrq-l and sysrq-w > > > > > The repro this time doesn't have a soft lockup, just the tar is hung and > > one of my CPUs is in 99% system. > > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > > [ 454.753048] Call Trace: > > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > > So our flusher threads are stuck waiting in > jbd2_journal_get_write_access, which means they aren't cleaning dirty > pages. > > In order to get write access, they probably need the transaction to > commit: > > > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > > [ 454.839453] Call Trace: > > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > > But that seems to be waiting for a page lock. Probably the same page > lock held by the flusher thread? Looks like tar is stuck in the same > boat down below. To test the theory, Chris asked me to try with data=ordered. Unfortunately, the deadlock still shows up. This is what I get. James --- [ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 263.756081] CPU 2 [ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 263.763033] [ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166 [ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206 [ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002 [ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50 [ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2 [ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e [ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9 [ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) [ 263.788491] Stack: [ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700 [ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c [ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000 [ 263.795038] Call Trace: [ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c [ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7 [ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7 [ 263.809185] Call Trace: [ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798 [ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c [ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 272.421176] SysRq : Show backtrace of all active CPUs [ 272.422748] sending NMI to all CPUs: [ 272.424304] NMI backtrace for cpu 2 [ 272.426364] CPU 2 [ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.435128] [ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae [ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282 [ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000 [ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0 [ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000 [ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000 [ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) [ 272.465234] Stack: [ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a [ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000 [ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000 [ 272.472144] Call Trace: [ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 [ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 [ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798 [ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c [ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c [ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8 [ 272.492911] Call Trace: [ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 [ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 [ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798 [ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c [ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 272.510355] NMI backtrace for cpu 1 [ 272.511996] CPU 1 [ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.519303] [ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 [ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 [ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 [ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 [ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 [ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001 [ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 [ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) [ 272.546756] Stack: [ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 [ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 [ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 [ 272.553780] Call Trace: [ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 272.566380] Call Trace: [ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 272.575463] NMI backtrace for cpu 0 [ 272.576865] CPU 0 [ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.582932] [ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e [ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887 [ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040 [ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958 [ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000 [ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008 [ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001 [ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 [ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0 [ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) [ 272.605715] Stack: [ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690 [ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310 [ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001 [ 272.611653] Call Trace: [ 272.613026] <IRQ> [ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 272.639155] <EOI> [ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2 [ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48 [ 272.654210] Call Trace: [ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 [ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 [ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c [ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e [ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc [ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f [ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 [ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87 [ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e [ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f [ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab [ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 [ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 [ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 [ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 [ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 [ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e [ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 [ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13 [ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd [ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 [ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 [ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74 [ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 [ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 [ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 [ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 [ 272.683442] NMI backtrace for cpu 3 [ 272.684688] CPU 3 [ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 272.690650] [ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO [ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 [ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046 [ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 [ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640 [ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4 [ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030 [ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003 [ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 [ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700) [ 272.715001] Stack: [ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290 [ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170 [ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868 [ 272.721848] Call Trace: [ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 [ 272.733594] Call Trace: [ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 [ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225 [ 274.865072] SysRq : Show Blocked State [ 274.866350] task PC stack pid father [ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000 [ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000 [ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40 [ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0 [ 274.872869] Call Trace: [ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b [ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d [ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28 [ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39 [ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194 [ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a [ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 [ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000 [ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000 [ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40 [ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720 [ 274.906327] Call Trace: [ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b [ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8 [ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 [ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 [ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd [ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 [ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 [ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105 [ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 [ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 [ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17 [ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a [ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e [ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 [ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 [ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9 [ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d [ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56 [ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a [ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc [ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 [ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b [ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 [ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34 [ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1 [ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d [ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 [ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d [ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000 [ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000 [ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40 [ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3 [ 274.936966] Call Trace: [ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 [ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 [ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a [ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 [ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000 [ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246 [ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40 [ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720 [ 274.937011] Call Trace: [ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b [ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8 [ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 [ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 [ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd [ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 [ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 [ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105 [ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 [ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c [ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 [ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 [ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17 [ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a [ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e [ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 [ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 [ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56 [ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a [ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc [ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 [ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b [ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 [ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d [ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde [ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 [ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d [ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c [ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 [ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13 [ 274.937082] tar D 0000000000000000 0 980 826 0x00000000 [ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000 [ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40 [ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282 [ 274.937090] Call Trace: [ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde [ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a [ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98 [ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6 [ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a [ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 [ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba [ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 [ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff [ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 [ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 [ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3 [ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e [ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004 [ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000 [ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40 [ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908 [ 274.937135] Call Trace: [ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d [ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c [ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d [ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d [ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d [ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c [ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 [ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 [ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30 [ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c [ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 [ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131 [ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 [ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff [ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 [ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e [ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004 [ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000 [ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40 [ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908 [ 274.937195] Call Trace: [ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d [ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c [ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d [ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d [ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d [ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c [ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 [ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 [ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30 [ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c [ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 [ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131 [ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 [ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff [ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 [ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd [ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 [ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e [ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004 [ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000 [ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40 [ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0 [ 274.937252] Call Trace: [ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e [ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 [ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f [ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d [ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a [ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d [ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c [ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d [ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d [ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153 [ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 [ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 [ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf [ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33 [ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a [ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67 [ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e [ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e [ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab [ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c [ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf [ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86 [ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23 [ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d [ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100 [ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22 [ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004 [ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000 [ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40 [ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3 [ 274.937311] Call Trace: [ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465 [ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda [ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15 [ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120 [ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212 [ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74 [ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef [ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33 [ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18 [ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b [ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 [ 274.937336] ktime : 275382.978028 [ 274.937337] sched_clk : 274937.334155 [ 274.937338] cpu_clk : 274937.334189 [ 274.937339] jiffies : 4294942678 [ 274.937340] sched_clock_stable : 1 [ 274.937341] [ 274.937342] sysctl_sched [ 274.937342] .sysctl_sched_latency : 18.000000 [ 274.937344] .sysctl_sched_min_granularity : 2.250000 [ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000 [ 274.937346] .sysctl_sched_child_runs_first : 0 [ 274.937347] .sysctl_sched_features : 7279 [ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic) [ 274.937349] [ 274.937350] cpu#0, 2492.220 MHz [ 274.937350] .nr_running : 0 [ 274.937351] .load : 0 [ 274.937352] .nr_switches : 105039 [ 274.937353] .nr_load_updates : 121182 [ 274.937354] .nr_uninterruptible : 8 [ 274.937355] .next_balance : 4294.942607 [ 274.937356] .curr->pid : 0 [ 274.937357] .clock : 274864.523786 [ 274.937359] .cpu_load[0] : 0 [ 274.937360] .cpu_load[1] : 0 [ 274.937361] .cpu_load[2] : 0 [ 274.937362] .cpu_load[3] : 0 [ 274.937363] .cpu_load[4] : 0 [ 274.937364] .yld_count : 0 [ 274.937364] .sched_switch : 0 [ 274.937365] .sched_count : 107487 [ 274.937366] .sched_goidle : 43368 [ 274.937367] .avg_idle : 1000000 [ 274.937368] .ttwu_count : 56534 [ 274.937369] .ttwu_local : 53925 [ 274.937370] .bkl_count : 0 [ 274.937371] [ 274.937372] cfs_rq[0]:/system [ 274.937372] .exec_clock : 5906.036519 [ 274.937374] .MIN_vruntime : 0.000001 [ 274.937375] .min_vruntime : 7138.254657 [ 274.937376] .max_vruntime : 0.000001 [ 274.937377] .spread : 0.000000 [ 274.937378] .spread0 : -14302.149798 [ 274.937379] .nr_spread_over : 0 [ 274.937379] .nr_running : 0 [ 274.937380] .load : 0 [ 274.937381] .load_avg : 159.999997 [ 274.937382] .load_period : 5.496191 [ 274.937383] .load_contrib : 29 [ 274.937384] .load_tg : 29 [ 274.937385] .se->exec_start : 272446.423795 [ 274.937386] .se->vruntime : 21431.398168 [ 274.937387] .se->sum_exec_runtime : 5906.111805 [ 274.937388] .se->statistics.wait_start : 0.000000 [ 274.937389] .se->statistics.sleep_start : 0.000000 [ 274.937390] .se->statistics.block_start : 0.000000 [ 274.937391] .se->statistics.sleep_max : 0.000000 [ 274.937392] .se->statistics.block_max : 0.000000 [ 274.937393] .se->statistics.exec_max : 15.266617 [ 274.937394] .se->statistics.slice_max : 5.258344 [ 274.937395] .se->statistics.wait_max : 7.305679 [ 274.937396] .se->statistics.wait_sum : 190.506686 [ 274.937397] .se->statistics.wait_count : 18946 [ 274.937398] .se->load.weight : 2 [ 274.937399] [ 274.937399] cfs_rq[0]:/ [ 274.937400] .exec_clock : 17409.691339 [ 274.937401] .MIN_vruntime : 0.000001 [ 274.937402] .min_vruntime : 21440.404455 [ 274.937403] .max_vruntime : 0.000001 [ 274.937403] .spread : 0.000000 [ 274.937404] .spread0 : 0.000000 [ 274.937405] .nr_spread_over : 58 [ 274.937406] .nr_running : 0 [ 274.937407] .load : 0 [ 274.937408] .load_avg : 0.000000 [ 274.937409] .load_period : 0.000000 [ 274.937409] .load_contrib : 0 [ 274.937410] .load_tg : 0 [ 274.937411] [ 274.937412] cfs_rq[0]:/system/rescue.service [ 274.937413] .exec_clock : 4319.868509 [ 274.937414] .MIN_vruntime : 0.000001 [ 274.937414] .min_vruntime : 4255.887524 [ 274.937415] .max_vruntime : 0.000001 [ 274.937416] .spread : 0.000000 [ 274.937417] .spread0 : -17184.516931 [ 274.937418] .nr_spread_over : 24 [ 274.937419] .nr_running : 0 [ 274.937420] .load : 0 [ 274.937421] .load_avg : 160.004112 [ 274.937421] .load_period : 5.496149 [ 274.937422] .load_contrib : 29 [ 274.937423] .load_tg : 29 [ 274.937424] .se->exec_start : 272446.423795 [ 274.937425] .se->vruntime : 7138.254657 [ 274.937426] .se->sum_exec_runtime : 4319.796162 [ 274.937427] .se->statistics.wait_start : 0.000000 [ 274.937428] .se->statistics.sleep_start : 0.000000 [ 274.937429] .se->statistics.block_start : 0.000000 [ 274.937430] .se->statistics.sleep_max : 0.000000 [ 274.937431] .se->statistics.block_max : 0.000000 [ 274.937432] .se->statistics.exec_max : 6.213083 [ 274.937432] .se->statistics.slice_max : 7.894226 [ 274.937433] .se->statistics.wait_max : 4.870567 [ 274.937434] .se->statistics.wait_sum : 91.737803 [ 274.937435] .se->statistics.wait_count : 15504 [ 274.937436] .se->load.weight : 2 [ 274.937437] [ 274.937438] runnable tasks: [ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937439] ---------------------------------------------------------------------------------------------------------- [ 274.937444] [ 274.937445] cpu#1, 2492.220 MHz [ 274.937445] .nr_running : 0 [ 274.937446] .load : 0 [ 274.937447] .nr_switches : 35274 [ 274.937448] .nr_load_updates : 105259 [ 274.937449] .nr_uninterruptible : 1 [ 274.937449] .next_balance : 4294.942681 [ 274.937450] .curr->pid : 0 [ 274.937451] .clock : 274937.357922 [ 274.937452] .cpu_load[0] : 0 [ 274.937453] .cpu_load[1] : 0 [ 274.937454] .cpu_load[2] : 0 [ 274.937454] .cpu_load[3] : 0 [ 274.937455] .cpu_load[4] : 0 [ 274.937456] .yld_count : 0 [ 274.937457] .sched_switch : 0 [ 274.937458] .sched_count : 35560 [ 274.937458] .sched_goidle : 14561 [ 274.937459] .avg_idle : 1000000 [ 274.937460] .ttwu_count : 16938 [ 274.937461] .ttwu_local : 14491 [ 274.937462] .bkl_count : 0 [ 274.937463] [ 274.937463] cfs_rq[1]:/ [ 274.937464] .exec_clock : 5221.725565 [ 274.937465] .MIN_vruntime : 0.000001 [ 274.937466] .min_vruntime : 9077.562937 [ 274.937466] .max_vruntime : 0.000001 [ 274.937467] .spread : 0.000000 [ 274.937468] .spread0 : -12362.841518 [ 274.937469] .nr_spread_over : 69 [ 274.937470] .nr_running : 0 [ 274.937471] .load : 0 [ 274.937472] .load_avg : 0.000000 [ 274.937473] .load_period : 0.000000 [ 274.937473] .load_contrib : 0 [ 274.937474] .load_tg : 0 [ 274.937475] [ 274.937476] runnable tasks: [ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937477] ---------------------------------------------------------------------------------------------------------- [ 274.937481] [ 274.937482] cpu#2, 2492.220 MHz [ 274.937482] .nr_running : 4 [ 274.937483] .load : 2048 [ 274.937484] .nr_switches : 7092 [ 274.937485] .nr_load_updates : 100502 [ 274.937486] .nr_uninterruptible : 0 [ 274.937487] .next_balance : 4294.942806 [ 274.937487] .curr->pid : 46 [ 274.937488] .clock : 183206.600407 [ 274.937489] .cpu_load[0] : 2048 [ 274.937490] .cpu_load[1] : 2048 [ 274.937491] .cpu_load[2] : 2048 [ 274.937492] .cpu_load[3] : 2048 [ 274.937492] .cpu_load[4] : 2048 [ 274.937493] .yld_count : 0 [ 274.937494] .sched_switch : 0 [ 274.937495] .sched_count : 7323 [ 274.937496] .sched_goidle : 2831 [ 274.937496] .avg_idle : 1000000 [ 274.937497] .ttwu_count : 3275 [ 274.937498] .ttwu_local : 2028 [ 274.937499] .bkl_count : 0 [ 274.937500] [ 274.937500] cfs_rq[2]:/ [ 274.937501] .exec_clock : 2187.192139 [ 274.937502] .MIN_vruntime : 5109.505143 [ 274.937503] .min_vruntime : 5118.505143 [ 274.937504] .max_vruntime : 5109.505143 [ 274.937505] .spread : 0.000000 [ 274.937506] .spread0 : -16321.899312 [ 274.937506] .nr_spread_over : 31 [ 274.937507] .nr_running : 2 [ 274.937508] .load : 2048 [ 274.937509] .load_avg : 0.000000 [ 274.937510] .load_period : 0.000000 [ 274.937511] .load_contrib : 0 [ 274.937511] .load_tg : 0 [ 274.937512] [ 274.937513] rt_rq[2]:/ [ 274.937513] .rt_nr_running : 1 [ 274.937514] .rt_throttled : 0 [ 274.937515] .rt_time : 0.000000 [ 274.937516] .rt_runtime : 950.000000 [ 274.937517] [ 274.937517] runnable tasks: [ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937519] ---------------------------------------------------------------------------------------------------------- [ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 / [ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 / [ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 / [ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 / [ 274.937535] [ 274.937536] cpu#3, 2492.220 MHz [ 274.937536] .nr_running : 1 [ 274.937537] .load : 1024 [ 274.937538] .nr_switches : 22324 [ 274.937539] .nr_load_updates : 59855 [ 274.937540] .nr_uninterruptible : 0 [ 274.937541] .next_balance : 4294.942679 [ 274.937542] .curr->pid : 73 [ 274.937543] .clock : 274936.938702 [ 274.937543] .cpu_load[0] : 0 [ 274.937544] .cpu_load[1] : 0 [ 274.937545] .cpu_load[2] : 0 [ 274.937546] .cpu_load[3] : 0 [ 274.937546] .cpu_load[4] : 0 [ 274.937547] .yld_count : 123 [ 274.937548] .sched_switch : 0 [ 274.937549] .sched_count : 22774 [ 274.937550] .sched_goidle : 9952 [ 274.937551] .avg_idle : 946660 [ 274.937551] .ttwu_count : 10075 [ 274.937552] .ttwu_local : 8964 [ 274.937553] .bkl_count : 0 [ 274.937554] [ 274.937554] cfs_rq[3]:/ [ 274.937555] .exec_clock : 3244.912177 [ 274.937556] .MIN_vruntime : 0.000001 [ 274.937557] .min_vruntime : 7517.482154 [ 274.937558] .max_vruntime : 0.000001 [ 274.937559] .spread : 0.000000 [ 274.937559] .spread0 : -13922.922301 [ 274.937560] .nr_spread_over : 128 [ 274.937561] .nr_running : 1 [ 274.937562] .load : 1024 [ 274.937563] .load_avg : 0.000000 [ 274.937564] .load_period : 0.000000 [ 274.937565] .load_contrib : 0 [ 274.937565] .load_tg : 0 [ 274.937567] [ 274.937567] runnable tasks: [ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 274.937568] ---------------------------------------------------------------------------------------------------------- [ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 / [ 274.937575] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 17:50 ` James Bottomley @ 2011-04-27 18:25 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-27 18:25 UTC (permalink / raw) To: James Bottomley Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Hi, Just like to add that I've seen almost identical issues with 2.6.38 copying large amounts of data to an ext4 filesystem with systems with small amounts of memory. I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked around the issue. Colin On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote: > > [ cc'd linux-ext4 ] > > > > James is hitting softlockups in kswapd while doing writes to a large > > ext4 file. > > > > Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400: > > > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > > > case better. > > > > > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > > > > > This is the trace with sysrq-l and sysrq-w > > > > > > > > The repro this time doesn't have a soft lockup, just the tar is hung and > > > one of my CPUs is in 99% system. > > > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > > > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > > > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > > > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > > > [ 454.753048] Call Trace: > > > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > > > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > > > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > > > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > > > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > > > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > > > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > > > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > > > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > > > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > > > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > > > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > > > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > > > > So our flusher threads are stuck waiting in > > jbd2_journal_get_write_access, which means they aren't cleaning dirty > > pages. > > > > In order to get write access, they probably need the transaction to > > commit: > > > > > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > > > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > > > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > > > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > > > [ 454.839453] Call Trace: > > > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > > > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > > > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > > > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > > > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > > > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > > > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > > > > But that seems to be waiting for a page lock. Probably the same page > > lock held by the flusher thread? Looks like tar is stuck in the same > > boat down below. > > To test the theory, Chris asked me to try with data=ordered. > Unfortunately, the deadlock still shows up. This is what I get. > > James > > --- > > [ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] > [ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 263.756081] CPU 2 > [ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 263.763033] > [ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166 > [ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206 > [ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002 > [ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50 > [ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2 > [ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e > [ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9 > [ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 > [ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) > [ 263.788491] Stack: > [ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700 > [ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c > [ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000 > [ 263.795038] Call Trace: > [ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7 > [ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7 > [ 263.809185] Call Trace: > [ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 272.421176] SysRq : Show backtrace of all active CPUs > [ 272.422748] sending NMI to all CPUs: > [ 272.424304] NMI backtrace for cpu 2 > [ 272.426364] CPU 2 > [ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.435128] > [ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae > [ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282 > [ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 > [ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000 > [ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0 > [ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000 > [ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000 > [ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 > [ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) > [ 272.465234] Stack: > [ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a > [ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000 > [ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000 > [ 272.472144] Call Trace: > [ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 > [ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 > [ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798 > [ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c > [ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8 > [ 272.492911] Call Trace: > [ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 > [ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 > [ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798 > [ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 272.510355] NMI backtrace for cpu 1 > [ 272.511996] CPU 1 > [ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.519303] > [ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 > [ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 > [ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 > [ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 > [ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 > [ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001 > [ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 > [ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) > [ 272.546756] Stack: > [ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 > [ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 > [ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 > [ 272.553780] Call Trace: > [ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 272.566380] Call Trace: > [ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 272.575463] NMI backtrace for cpu 0 > [ 272.576865] CPU 0 > [ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.582932] > [ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e > [ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887 > [ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040 > [ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958 > [ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000 > [ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008 > [ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001 > [ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 > [ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0 > [ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) > [ 272.605715] Stack: > [ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690 > [ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310 > [ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001 > [ 272.611653] Call Trace: > [ 272.613026] <IRQ> > [ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 272.639155] <EOI> > [ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2 > [ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48 > [ 272.654210] Call Trace: > [ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 272.683442] NMI backtrace for cpu 3 > [ 272.684688] CPU 3 > [ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.690650] > [ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046 > [ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 > [ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640 > [ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4 > [ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030 > [ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003 > [ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 > [ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700) > [ 272.715001] Stack: > [ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290 > [ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170 > [ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868 > [ 272.721848] Call Trace: > [ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 272.733594] Call Trace: > [ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 274.865072] SysRq : Show Blocked State > [ 274.866350] task PC stack pid father > [ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000 > [ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000 > [ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40 > [ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0 > [ 274.872869] Call Trace: > [ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d > [ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28 > [ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39 > [ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194 > [ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 > [ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000 > [ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000 > [ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40 > [ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720 > [ 274.906327] Call Trace: > [ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b > [ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8 > [ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 > [ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 > [ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd > [ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 > [ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 > [ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105 > [ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 > [ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 > [ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17 > [ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a > [ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e > [ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 > [ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 > [ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9 > [ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d > [ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56 > [ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34 > [ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1 > [ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d > [ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000 > [ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000 > [ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40 > [ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3 > [ 274.936966] Call Trace: > [ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194 > [ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 > [ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd > [ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 > [ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 > [ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 > [ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 > [ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 > [ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000 > [ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246 > [ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40 > [ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720 > [ 274.937011] Call Trace: > [ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b > [ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8 > [ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 > [ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 > [ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd > [ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 > [ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 > [ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105 > [ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 > [ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 > [ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17 > [ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a > [ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e > [ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 > [ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 > [ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56 > [ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d > [ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde > [ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.937082] tar D 0000000000000000 0 980 826 0x00000000 > [ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000 > [ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40 > [ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282 > [ 274.937090] Call Trace: > [ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde > [ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a > [ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98 > [ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6 > [ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a > [ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004 > [ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000 > [ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40 > [ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908 > [ 274.937135] Call Trace: > [ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d > [ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c > [ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d > [ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d > [ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d > [ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30 > [ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131 > [ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 > [ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff > [ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 > [ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e > [ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004 > [ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000 > [ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40 > [ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908 > [ 274.937195] Call Trace: > [ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d > [ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c > [ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d > [ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d > [ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d > [ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30 > [ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131 > [ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 > [ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff > [ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 > [ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e > [ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004 > [ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000 > [ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40 > [ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0 > [ 274.937252] Call Trace: > [ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d > [ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c > [ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d > [ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d > [ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153 > [ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf > [ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33 > [ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a > [ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67 > [ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e > [ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e > [ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab > [ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c > [ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf > [ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86 > [ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23 > [ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d > [ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100 > [ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22 > [ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004 > [ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000 > [ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40 > [ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3 > [ 274.937311] Call Trace: > [ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465 > [ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda > [ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15 > [ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120 > [ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212 > [ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74 > [ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef > [ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33 > [ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18 > [ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 > [ 274.937336] ktime : 275382.978028 > [ 274.937337] sched_clk : 274937.334155 > [ 274.937338] cpu_clk : 274937.334189 > [ 274.937339] jiffies : 4294942678 > [ 274.937340] sched_clock_stable : 1 > [ 274.937341] > [ 274.937342] sysctl_sched > [ 274.937342] .sysctl_sched_latency : 18.000000 > [ 274.937344] .sysctl_sched_min_granularity : 2.250000 > [ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000 > [ 274.937346] .sysctl_sched_child_runs_first : 0 > [ 274.937347] .sysctl_sched_features : 7279 > [ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic) > [ 274.937349] > [ 274.937350] cpu#0, 2492.220 MHz > [ 274.937350] .nr_running : 0 > [ 274.937351] .load : 0 > [ 274.937352] .nr_switches : 105039 > [ 274.937353] .nr_load_updates : 121182 > [ 274.937354] .nr_uninterruptible : 8 > [ 274.937355] .next_balance : 4294.942607 > [ 274.937356] .curr->pid : 0 > [ 274.937357] .clock : 274864.523786 > [ 274.937359] .cpu_load[0] : 0 > [ 274.937360] .cpu_load[1] : 0 > [ 274.937361] .cpu_load[2] : 0 > [ 274.937362] .cpu_load[3] : 0 > [ 274.937363] .cpu_load[4] : 0 > [ 274.937364] .yld_count : 0 > [ 274.937364] .sched_switch : 0 > [ 274.937365] .sched_count : 107487 > [ 274.937366] .sched_goidle : 43368 > [ 274.937367] .avg_idle : 1000000 > [ 274.937368] .ttwu_count : 56534 > [ 274.937369] .ttwu_local : 53925 > [ 274.937370] .bkl_count : 0 > [ 274.937371] > [ 274.937372] cfs_rq[0]:/system > [ 274.937372] .exec_clock : 5906.036519 > [ 274.937374] .MIN_vruntime : 0.000001 > [ 274.937375] .min_vruntime : 7138.254657 > [ 274.937376] .max_vruntime : 0.000001 > [ 274.937377] .spread : 0.000000 > [ 274.937378] .spread0 : -14302.149798 > [ 274.937379] .nr_spread_over : 0 > [ 274.937379] .nr_running : 0 > [ 274.937380] .load : 0 > [ 274.937381] .load_avg : 159.999997 > [ 274.937382] .load_period : 5.496191 > [ 274.937383] .load_contrib : 29 > [ 274.937384] .load_tg : 29 > [ 274.937385] .se->exec_start : 272446.423795 > [ 274.937386] .se->vruntime : 21431.398168 > [ 274.937387] .se->sum_exec_runtime : 5906.111805 > [ 274.937388] .se->statistics.wait_start : 0.000000 > [ 274.937389] .se->statistics.sleep_start : 0.000000 > [ 274.937390] .se->statistics.block_start : 0.000000 > [ 274.937391] .se->statistics.sleep_max : 0.000000 > [ 274.937392] .se->statistics.block_max : 0.000000 > [ 274.937393] .se->statistics.exec_max : 15.266617 > [ 274.937394] .se->statistics.slice_max : 5.258344 > [ 274.937395] .se->statistics.wait_max : 7.305679 > [ 274.937396] .se->statistics.wait_sum : 190.506686 > [ 274.937397] .se->statistics.wait_count : 18946 > [ 274.937398] .se->load.weight : 2 > [ 274.937399] > [ 274.937399] cfs_rq[0]:/ > [ 274.937400] .exec_clock : 17409.691339 > [ 274.937401] .MIN_vruntime : 0.000001 > [ 274.937402] .min_vruntime : 21440.404455 > [ 274.937403] .max_vruntime : 0.000001 > [ 274.937403] .spread : 0.000000 > [ 274.937404] .spread0 : 0.000000 > [ 274.937405] .nr_spread_over : 58 > [ 274.937406] .nr_running : 0 > [ 274.937407] .load : 0 > [ 274.937408] .load_avg : 0.000000 > [ 274.937409] .load_period : 0.000000 > [ 274.937409] .load_contrib : 0 > [ 274.937410] .load_tg : 0 > [ 274.937411] > [ 274.937412] cfs_rq[0]:/system/rescue.service > [ 274.937413] .exec_clock : 4319.868509 > [ 274.937414] .MIN_vruntime : 0.000001 > [ 274.937414] .min_vruntime : 4255.887524 > [ 274.937415] .max_vruntime : 0.000001 > [ 274.937416] .spread : 0.000000 > [ 274.937417] .spread0 : -17184.516931 > [ 274.937418] .nr_spread_over : 24 > [ 274.937419] .nr_running : 0 > [ 274.937420] .load : 0 > [ 274.937421] .load_avg : 160.004112 > [ 274.937421] .load_period : 5.496149 > [ 274.937422] .load_contrib : 29 > [ 274.937423] .load_tg : 29 > [ 274.937424] .se->exec_start : 272446.423795 > [ 274.937425] .se->vruntime : 7138.254657 > [ 274.937426] .se->sum_exec_runtime : 4319.796162 > [ 274.937427] .se->statistics.wait_start : 0.000000 > [ 274.937428] .se->statistics.sleep_start : 0.000000 > [ 274.937429] .se->statistics.block_start : 0.000000 > [ 274.937430] .se->statistics.sleep_max : 0.000000 > [ 274.937431] .se->statistics.block_max : 0.000000 > [ 274.937432] .se->statistics.exec_max : 6.213083 > [ 274.937432] .se->statistics.slice_max : 7.894226 > [ 274.937433] .se->statistics.wait_max : 4.870567 > [ 274.937434] .se->statistics.wait_sum : 91.737803 > [ 274.937435] .se->statistics.wait_count : 15504 > [ 274.937436] .se->load.weight : 2 > [ 274.937437] > [ 274.937438] runnable tasks: > [ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937439] ---------------------------------------------------------------------------------------------------------- > [ 274.937444] > [ 274.937445] cpu#1, 2492.220 MHz > [ 274.937445] .nr_running : 0 > [ 274.937446] .load : 0 > [ 274.937447] .nr_switches : 35274 > [ 274.937448] .nr_load_updates : 105259 > [ 274.937449] .nr_uninterruptible : 1 > [ 274.937449] .next_balance : 4294.942681 > [ 274.937450] .curr->pid : 0 > [ 274.937451] .clock : 274937.357922 > [ 274.937452] .cpu_load[0] : 0 > [ 274.937453] .cpu_load[1] : 0 > [ 274.937454] .cpu_load[2] : 0 > [ 274.937454] .cpu_load[3] : 0 > [ 274.937455] .cpu_load[4] : 0 > [ 274.937456] .yld_count : 0 > [ 274.937457] .sched_switch : 0 > [ 274.937458] .sched_count : 35560 > [ 274.937458] .sched_goidle : 14561 > [ 274.937459] .avg_idle : 1000000 > [ 274.937460] .ttwu_count : 16938 > [ 274.937461] .ttwu_local : 14491 > [ 274.937462] .bkl_count : 0 > [ 274.937463] > [ 274.937463] cfs_rq[1]:/ > [ 274.937464] .exec_clock : 5221.725565 > [ 274.937465] .MIN_vruntime : 0.000001 > [ 274.937466] .min_vruntime : 9077.562937 > [ 274.937466] .max_vruntime : 0.000001 > [ 274.937467] .spread : 0.000000 > [ 274.937468] .spread0 : -12362.841518 > [ 274.937469] .nr_spread_over : 69 > [ 274.937470] .nr_running : 0 > [ 274.937471] .load : 0 > [ 274.937472] .load_avg : 0.000000 > [ 274.937473] .load_period : 0.000000 > [ 274.937473] .load_contrib : 0 > [ 274.937474] .load_tg : 0 > [ 274.937475] > [ 274.937476] runnable tasks: > [ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937477] ---------------------------------------------------------------------------------------------------------- > [ 274.937481] > [ 274.937482] cpu#2, 2492.220 MHz > [ 274.937482] .nr_running : 4 > [ 274.937483] .load : 2048 > [ 274.937484] .nr_switches : 7092 > [ 274.937485] .nr_load_updates : 100502 > [ 274.937486] .nr_uninterruptible : 0 > [ 274.937487] .next_balance : 4294.942806 > [ 274.937487] .curr->pid : 46 > [ 274.937488] .clock : 183206.600407 > [ 274.937489] .cpu_load[0] : 2048 > [ 274.937490] .cpu_load[1] : 2048 > [ 274.937491] .cpu_load[2] : 2048 > [ 274.937492] .cpu_load[3] : 2048 > [ 274.937492] .cpu_load[4] : 2048 > [ 274.937493] .yld_count : 0 > [ 274.937494] .sched_switch : 0 > [ 274.937495] .sched_count : 7323 > [ 274.937496] .sched_goidle : 2831 > [ 274.937496] .avg_idle : 1000000 > [ 274.937497] .ttwu_count : 3275 > [ 274.937498] .ttwu_local : 2028 > [ 274.937499] .bkl_count : 0 > [ 274.937500] > [ 274.937500] cfs_rq[2]:/ > [ 274.937501] .exec_clock : 2187.192139 > [ 274.937502] .MIN_vruntime : 5109.505143 > [ 274.937503] .min_vruntime : 5118.505143 > [ 274.937504] .max_vruntime : 5109.505143 > [ 274.937505] .spread : 0.000000 > [ 274.937506] .spread0 : -16321.899312 > [ 274.937506] .nr_spread_over : 31 > [ 274.937507] .nr_running : 2 > [ 274.937508] .load : 2048 > [ 274.937509] .load_avg : 0.000000 > [ 274.937510] .load_period : 0.000000 > [ 274.937511] .load_contrib : 0 > [ 274.937511] .load_tg : 0 > [ 274.937512] > [ 274.937513] rt_rq[2]:/ > [ 274.937513] .rt_nr_running : 1 > [ 274.937514] .rt_throttled : 0 > [ 274.937515] .rt_time : 0.000000 > [ 274.937516] .rt_runtime : 950.000000 > [ 274.937517] > [ 274.937517] runnable tasks: > [ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937519] ---------------------------------------------------------------------------------------------------------- > [ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 / > [ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 / > [ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 / > [ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 / > [ 274.937535] > [ 274.937536] cpu#3, 2492.220 MHz > [ 274.937536] .nr_running : 1 > [ 274.937537] .load : 1024 > [ 274.937538] .nr_switches : 22324 > [ 274.937539] .nr_load_updates : 59855 > [ 274.937540] .nr_uninterruptible : 0 > [ 274.937541] .next_balance : 4294.942679 > [ 274.937542] .curr->pid : 73 > [ 274.937543] .clock : 274936.938702 > [ 274.937543] .cpu_load[0] : 0 > [ 274.937544] .cpu_load[1] : 0 > [ 274.937545] .cpu_load[2] : 0 > [ 274.937546] .cpu_load[3] : 0 > [ 274.937546] .cpu_load[4] : 0 > [ 274.937547] .yld_count : 123 > [ 274.937548] .sched_switch : 0 > [ 274.937549] .sched_count : 22774 > [ 274.937550] .sched_goidle : 9952 > [ 274.937551] .avg_idle : 946660 > [ 274.937551] .ttwu_count : 10075 > [ 274.937552] .ttwu_local : 8964 > [ 274.937553] .bkl_count : 0 > [ 274.937554] > [ 274.937554] cfs_rq[3]:/ > [ 274.937555] .exec_clock : 3244.912177 > [ 274.937556] .MIN_vruntime : 0.000001 > [ 274.937557] .min_vruntime : 7517.482154 > [ 274.937558] .max_vruntime : 0.000001 > [ 274.937559] .spread : 0.000000 > [ 274.937559] .spread0 : -13922.922301 > [ 274.937560] .nr_spread_over : 128 > [ 274.937561] .nr_running : 1 > [ 274.937562] .load : 1024 > [ 274.937563] .load_avg : 0.000000 > [ 274.937564] .load_period : 0.000000 > [ 274.937565] .load_contrib : 0 > [ 274.937565] .load_tg : 0 > [ 274.937567] > [ 274.937567] runnable tasks: > [ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937568] ---------------------------------------------------------------------------------------------------------- > [ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 / > [ 274.937575] > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 18:25 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-27 18:25 UTC (permalink / raw) To: James Bottomley Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Hi, Just like to add that I've seen almost identical issues with 2.6.38 copying large amounts of data to an ext4 filesystem with systems with small amounts of memory. I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked around the issue. Colin On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote: > > [ cc'd linux-ext4 ] > > > > James is hitting softlockups in kswapd while doing writes to a large > > ext4 file. > > > > Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400: > > > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > > > case better. > > > > > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > > > > > This is the trace with sysrq-l and sysrq-w > > > > > > > > The repro this time doesn't have a soft lockup, just the tar is hung and > > > one of my CPUs is in 99% system. > > > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > > > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > > > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > > > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > > > [ 454.753048] Call Trace: > > > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > > > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > > > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > > > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > > > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > > > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > > > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > > > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > > > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > > > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > > > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > > > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > > > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > > > > So our flusher threads are stuck waiting in > > jbd2_journal_get_write_access, which means they aren't cleaning dirty > > pages. > > > > In order to get write access, they probably need the transaction to > > commit: > > > > > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > > > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > > > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > > > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > > > [ 454.839453] Call Trace: > > > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > > > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > > > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > > > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > > > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > > > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > > > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > > > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > > > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > > > > But that seems to be waiting for a page lock. Probably the same page > > lock held by the flusher thread? Looks like tar is stuck in the same > > boat down below. > > To test the theory, Chris asked me to try with data=ordered. > Unfortunately, the deadlock still shows up. This is what I get. > > James > > --- > > [ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] > [ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 263.756081] CPU 2 > [ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 263.763033] > [ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166 > [ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206 > [ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002 > [ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50 > [ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2 > [ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e > [ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9 > [ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 > [ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) > [ 263.788491] Stack: > [ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700 > [ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c > [ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000 > [ 263.795038] Call Trace: > [ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7 > [ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7 > [ 263.809185] Call Trace: > [ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 272.421176] SysRq : Show backtrace of all active CPUs > [ 272.422748] sending NMI to all CPUs: > [ 272.424304] NMI backtrace for cpu 2 > [ 272.426364] CPU 2 > [ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.435128] > [ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae > [ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282 > [ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 > [ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000 > [ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0 > [ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000 > [ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000 > [ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 > [ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000) > [ 272.465234] Stack: > [ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a > [ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000 > [ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000 > [ 272.472144] Call Trace: > [ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 > [ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 > [ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798 > [ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c > [ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8 > [ 272.492911] Call Trace: > [ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2 > [ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77 > [ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798 > [ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 272.510355] NMI backtrace for cpu 1 > [ 272.511996] CPU 1 > [ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.519303] > [ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 > [ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 > [ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 > [ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 > [ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 > [ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001 > [ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 > [ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) > [ 272.546756] Stack: > [ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 > [ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 > [ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 > [ 272.553780] Call Trace: > [ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 272.566380] Call Trace: > [ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 272.575463] NMI backtrace for cpu 0 > [ 272.576865] CPU 0 > [ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.582932] > [ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e > [ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887 > [ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040 > [ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958 > [ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000 > [ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008 > [ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001 > [ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 > [ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0 > [ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) > [ 272.605715] Stack: > [ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690 > [ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310 > [ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001 > [ 272.611653] Call Trace: > [ 272.613026] <IRQ> > [ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 272.639155] <EOI> > [ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2 > [ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48 > [ 272.654210] Call Trace: > [ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 272.683442] NMI backtrace for cpu 3 > [ 272.684688] CPU 3 > [ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 272.690650] > [ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046 > [ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 > [ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640 > [ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4 > [ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030 > [ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003 > [ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 > [ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700) > [ 272.715001] Stack: > [ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290 > [ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170 > [ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868 > [ 272.721848] Call Trace: > [ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 272.733594] Call Trace: > [ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 274.865072] SysRq : Show Blocked State > [ 274.866350] task PC stack pid father > [ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000 > [ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000 > [ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40 > [ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0 > [ 274.872869] Call Trace: > [ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d > [ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28 > [ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39 > [ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194 > [ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 > [ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000 > [ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000 > [ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40 > [ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720 > [ 274.906327] Call Trace: > [ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b > [ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8 > [ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 > [ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 > [ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd > [ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 > [ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 > [ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105 > [ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 > [ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 > [ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17 > [ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a > [ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e > [ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 > [ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 > [ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9 > [ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d > [ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56 > [ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34 > [ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1 > [ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d > [ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000 > [ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000 > [ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40 > [ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3 > [ 274.936966] Call Trace: > [ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194 > [ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 > [ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd > [ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 > [ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 > [ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 > [ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 > [ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 > [ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000 > [ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246 > [ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40 > [ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720 > [ 274.937011] Call Trace: > [ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b > [ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8 > [ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323 > [ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17 > [ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd > [ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3 > [ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29 > [ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105 > [ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7 > [ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c > [ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79 > [ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97 > [ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17 > [ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a > [ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e > [ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330 > [ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33 > [ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56 > [ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d > [ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde > [ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 274.937082] tar D 0000000000000000 0 980 826 0x00000000 > [ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000 > [ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40 > [ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282 > [ 274.937090] Call Trace: > [ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde > [ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a > [ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98 > [ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6 > [ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a > [ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004 > [ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000 > [ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40 > [ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908 > [ 274.937135] Call Trace: > [ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d > [ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c > [ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d > [ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d > [ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d > [ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30 > [ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131 > [ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 > [ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff > [ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 > [ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e > [ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004 > [ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000 > [ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40 > [ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908 > [ 274.937195] Call Trace: > [ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d > [ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c > [ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d > [ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d > [ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d > [ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30 > [ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131 > [ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640 > [ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff > [ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 > [ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e > [ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004 > [ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000 > [ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40 > [ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0 > [ 274.937252] Call Trace: > [ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12 > [ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d > [ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a > [ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c > [ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d > [ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d > [ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153 > [ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf > [ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33 > [ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a > [ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67 > [ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e > [ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e > [ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab > [ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c > [ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf > [ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86 > [ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23 > [ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d > [ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100 > [ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22 > [ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004 > [ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000 > [ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40 > [ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3 > [ 274.937311] Call Trace: > [ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465 > [ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda > [ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15 > [ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120 > [ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212 > [ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74 > [ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef > [ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33 > [ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18 > [ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 > [ 274.937336] ktime : 275382.978028 > [ 274.937337] sched_clk : 274937.334155 > [ 274.937338] cpu_clk : 274937.334189 > [ 274.937339] jiffies : 4294942678 > [ 274.937340] sched_clock_stable : 1 > [ 274.937341] > [ 274.937342] sysctl_sched > [ 274.937342] .sysctl_sched_latency : 18.000000 > [ 274.937344] .sysctl_sched_min_granularity : 2.250000 > [ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000 > [ 274.937346] .sysctl_sched_child_runs_first : 0 > [ 274.937347] .sysctl_sched_features : 7279 > [ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic) > [ 274.937349] > [ 274.937350] cpu#0, 2492.220 MHz > [ 274.937350] .nr_running : 0 > [ 274.937351] .load : 0 > [ 274.937352] .nr_switches : 105039 > [ 274.937353] .nr_load_updates : 121182 > [ 274.937354] .nr_uninterruptible : 8 > [ 274.937355] .next_balance : 4294.942607 > [ 274.937356] .curr->pid : 0 > [ 274.937357] .clock : 274864.523786 > [ 274.937359] .cpu_load[0] : 0 > [ 274.937360] .cpu_load[1] : 0 > [ 274.937361] .cpu_load[2] : 0 > [ 274.937362] .cpu_load[3] : 0 > [ 274.937363] .cpu_load[4] : 0 > [ 274.937364] .yld_count : 0 > [ 274.937364] .sched_switch : 0 > [ 274.937365] .sched_count : 107487 > [ 274.937366] .sched_goidle : 43368 > [ 274.937367] .avg_idle : 1000000 > [ 274.937368] .ttwu_count : 56534 > [ 274.937369] .ttwu_local : 53925 > [ 274.937370] .bkl_count : 0 > [ 274.937371] > [ 274.937372] cfs_rq[0]:/system > [ 274.937372] .exec_clock : 5906.036519 > [ 274.937374] .MIN_vruntime : 0.000001 > [ 274.937375] .min_vruntime : 7138.254657 > [ 274.937376] .max_vruntime : 0.000001 > [ 274.937377] .spread : 0.000000 > [ 274.937378] .spread0 : -14302.149798 > [ 274.937379] .nr_spread_over : 0 > [ 274.937379] .nr_running : 0 > [ 274.937380] .load : 0 > [ 274.937381] .load_avg : 159.999997 > [ 274.937382] .load_period : 5.496191 > [ 274.937383] .load_contrib : 29 > [ 274.937384] .load_tg : 29 > [ 274.937385] .se->exec_start : 272446.423795 > [ 274.937386] .se->vruntime : 21431.398168 > [ 274.937387] .se->sum_exec_runtime : 5906.111805 > [ 274.937388] .se->statistics.wait_start : 0.000000 > [ 274.937389] .se->statistics.sleep_start : 0.000000 > [ 274.937390] .se->statistics.block_start : 0.000000 > [ 274.937391] .se->statistics.sleep_max : 0.000000 > [ 274.937392] .se->statistics.block_max : 0.000000 > [ 274.937393] .se->statistics.exec_max : 15.266617 > [ 274.937394] .se->statistics.slice_max : 5.258344 > [ 274.937395] .se->statistics.wait_max : 7.305679 > [ 274.937396] .se->statistics.wait_sum : 190.506686 > [ 274.937397] .se->statistics.wait_count : 18946 > [ 274.937398] .se->load.weight : 2 > [ 274.937399] > [ 274.937399] cfs_rq[0]:/ > [ 274.937400] .exec_clock : 17409.691339 > [ 274.937401] .MIN_vruntime : 0.000001 > [ 274.937402] .min_vruntime : 21440.404455 > [ 274.937403] .max_vruntime : 0.000001 > [ 274.937403] .spread : 0.000000 > [ 274.937404] .spread0 : 0.000000 > [ 274.937405] .nr_spread_over : 58 > [ 274.937406] .nr_running : 0 > [ 274.937407] .load : 0 > [ 274.937408] .load_avg : 0.000000 > [ 274.937409] .load_period : 0.000000 > [ 274.937409] .load_contrib : 0 > [ 274.937410] .load_tg : 0 > [ 274.937411] > [ 274.937412] cfs_rq[0]:/system/rescue.service > [ 274.937413] .exec_clock : 4319.868509 > [ 274.937414] .MIN_vruntime : 0.000001 > [ 274.937414] .min_vruntime : 4255.887524 > [ 274.937415] .max_vruntime : 0.000001 > [ 274.937416] .spread : 0.000000 > [ 274.937417] .spread0 : -17184.516931 > [ 274.937418] .nr_spread_over : 24 > [ 274.937419] .nr_running : 0 > [ 274.937420] .load : 0 > [ 274.937421] .load_avg : 160.004112 > [ 274.937421] .load_period : 5.496149 > [ 274.937422] .load_contrib : 29 > [ 274.937423] .load_tg : 29 > [ 274.937424] .se->exec_start : 272446.423795 > [ 274.937425] .se->vruntime : 7138.254657 > [ 274.937426] .se->sum_exec_runtime : 4319.796162 > [ 274.937427] .se->statistics.wait_start : 0.000000 > [ 274.937428] .se->statistics.sleep_start : 0.000000 > [ 274.937429] .se->statistics.block_start : 0.000000 > [ 274.937430] .se->statistics.sleep_max : 0.000000 > [ 274.937431] .se->statistics.block_max : 0.000000 > [ 274.937432] .se->statistics.exec_max : 6.213083 > [ 274.937432] .se->statistics.slice_max : 7.894226 > [ 274.937433] .se->statistics.wait_max : 4.870567 > [ 274.937434] .se->statistics.wait_sum : 91.737803 > [ 274.937435] .se->statistics.wait_count : 15504 > [ 274.937436] .se->load.weight : 2 > [ 274.937437] > [ 274.937438] runnable tasks: > [ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937439] ---------------------------------------------------------------------------------------------------------- > [ 274.937444] > [ 274.937445] cpu#1, 2492.220 MHz > [ 274.937445] .nr_running : 0 > [ 274.937446] .load : 0 > [ 274.937447] .nr_switches : 35274 > [ 274.937448] .nr_load_updates : 105259 > [ 274.937449] .nr_uninterruptible : 1 > [ 274.937449] .next_balance : 4294.942681 > [ 274.937450] .curr->pid : 0 > [ 274.937451] .clock : 274937.357922 > [ 274.937452] .cpu_load[0] : 0 > [ 274.937453] .cpu_load[1] : 0 > [ 274.937454] .cpu_load[2] : 0 > [ 274.937454] .cpu_load[3] : 0 > [ 274.937455] .cpu_load[4] : 0 > [ 274.937456] .yld_count : 0 > [ 274.937457] .sched_switch : 0 > [ 274.937458] .sched_count : 35560 > [ 274.937458] .sched_goidle : 14561 > [ 274.937459] .avg_idle : 1000000 > [ 274.937460] .ttwu_count : 16938 > [ 274.937461] .ttwu_local : 14491 > [ 274.937462] .bkl_count : 0 > [ 274.937463] > [ 274.937463] cfs_rq[1]:/ > [ 274.937464] .exec_clock : 5221.725565 > [ 274.937465] .MIN_vruntime : 0.000001 > [ 274.937466] .min_vruntime : 9077.562937 > [ 274.937466] .max_vruntime : 0.000001 > [ 274.937467] .spread : 0.000000 > [ 274.937468] .spread0 : -12362.841518 > [ 274.937469] .nr_spread_over : 69 > [ 274.937470] .nr_running : 0 > [ 274.937471] .load : 0 > [ 274.937472] .load_avg : 0.000000 > [ 274.937473] .load_period : 0.000000 > [ 274.937473] .load_contrib : 0 > [ 274.937474] .load_tg : 0 > [ 274.937475] > [ 274.937476] runnable tasks: > [ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937477] ---------------------------------------------------------------------------------------------------------- > [ 274.937481] > [ 274.937482] cpu#2, 2492.220 MHz > [ 274.937482] .nr_running : 4 > [ 274.937483] .load : 2048 > [ 274.937484] .nr_switches : 7092 > [ 274.937485] .nr_load_updates : 100502 > [ 274.937486] .nr_uninterruptible : 0 > [ 274.937487] .next_balance : 4294.942806 > [ 274.937487] .curr->pid : 46 > [ 274.937488] .clock : 183206.600407 > [ 274.937489] .cpu_load[0] : 2048 > [ 274.937490] .cpu_load[1] : 2048 > [ 274.937491] .cpu_load[2] : 2048 > [ 274.937492] .cpu_load[3] : 2048 > [ 274.937492] .cpu_load[4] : 2048 > [ 274.937493] .yld_count : 0 > [ 274.937494] .sched_switch : 0 > [ 274.937495] .sched_count : 7323 > [ 274.937496] .sched_goidle : 2831 > [ 274.937496] .avg_idle : 1000000 > [ 274.937497] .ttwu_count : 3275 > [ 274.937498] .ttwu_local : 2028 > [ 274.937499] .bkl_count : 0 > [ 274.937500] > [ 274.937500] cfs_rq[2]:/ > [ 274.937501] .exec_clock : 2187.192139 > [ 274.937502] .MIN_vruntime : 5109.505143 > [ 274.937503] .min_vruntime : 5118.505143 > [ 274.937504] .max_vruntime : 5109.505143 > [ 274.937505] .spread : 0.000000 > [ 274.937506] .spread0 : -16321.899312 > [ 274.937506] .nr_spread_over : 31 > [ 274.937507] .nr_running : 2 > [ 274.937508] .load : 2048 > [ 274.937509] .load_avg : 0.000000 > [ 274.937510] .load_period : 0.000000 > [ 274.937511] .load_contrib : 0 > [ 274.937511] .load_tg : 0 > [ 274.937512] > [ 274.937513] rt_rq[2]:/ > [ 274.937513] .rt_nr_running : 1 > [ 274.937514] .rt_throttled : 0 > [ 274.937515] .rt_time : 0.000000 > [ 274.937516] .rt_runtime : 950.000000 > [ 274.937517] > [ 274.937517] runnable tasks: > [ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937519] ---------------------------------------------------------------------------------------------------------- > [ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 / > [ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 / > [ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 / > [ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 / > [ 274.937535] > [ 274.937536] cpu#3, 2492.220 MHz > [ 274.937536] .nr_running : 1 > [ 274.937537] .load : 1024 > [ 274.937538] .nr_switches : 22324 > [ 274.937539] .nr_load_updates : 59855 > [ 274.937540] .nr_uninterruptible : 0 > [ 274.937541] .next_balance : 4294.942679 > [ 274.937542] .curr->pid : 73 > [ 274.937543] .clock : 274936.938702 > [ 274.937543] .cpu_load[0] : 0 > [ 274.937544] .cpu_load[1] : 0 > [ 274.937545] .cpu_load[2] : 0 > [ 274.937546] .cpu_load[3] : 0 > [ 274.937546] .cpu_load[4] : 0 > [ 274.937547] .yld_count : 123 > [ 274.937548] .sched_switch : 0 > [ 274.937549] .sched_count : 22774 > [ 274.937550] .sched_goidle : 9952 > [ 274.937551] .avg_idle : 946660 > [ 274.937551] .ttwu_count : 10075 > [ 274.937552] .ttwu_local : 8964 > [ 274.937553] .bkl_count : 0 > [ 274.937554] > [ 274.937554] cfs_rq[3]:/ > [ 274.937555] .exec_clock : 3244.912177 > [ 274.937556] .MIN_vruntime : 0.000001 > [ 274.937557] .min_vruntime : 7517.482154 > [ 274.937558] .max_vruntime : 0.000001 > [ 274.937559] .spread : 0.000000 > [ 274.937559] .spread0 : -13922.922301 > [ 274.937560] .nr_spread_over : 128 > [ 274.937561] .nr_running : 1 > [ 274.937562] .load : 1024 > [ 274.937563] .load_avg : 0.000000 > [ 274.937564] .load_period : 0.000000 > [ 274.937565] .load_contrib : 0 > [ 274.937565] .load_tg : 0 > [ 274.937567] > [ 274.937567] runnable tasks: > [ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 274.937568] ---------------------------------------------------------------------------------------------------------- > [ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 / > [ 274.937575] > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 18:25 ` Colin Ian King @ 2011-04-28 15:57 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 15:57 UTC (permalink / raw) To: Colin Ian King Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, 2011-04-27 at 19:25 +0100, Colin Ian King wrote: > Just like to add that I've seen almost identical issues with 2.6.38 > copying large amounts of data to an ext4 filesystem with systems with > small amounts of memory. > > I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked > around the issue. With the PREEMPT kernel, values of 256 and 512 don't prevent kswapd spinning up to 99% and staying there. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 15:57 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 15:57 UTC (permalink / raw) To: Colin Ian King Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, 2011-04-27 at 19:25 +0100, Colin Ian King wrote: > Just like to add that I've seen almost identical issues with 2.6.38 > copying large amounts of data to an ext4 filesystem with systems with > small amounts of memory. > > I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked > around the issue. With the PREEMPT kernel, values of 256 and 512 don't prevent kswapd spinning up to 99% and staying there. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 17:50 ` James Bottomley @ 2011-04-27 20:05 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 20:05 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > To test the theory, Chris asked me to try with data=ordered. > Unfortunately, the deadlock still shows up. This is what I get. As another data point: I'm trying the same kernel with CONFIG_PREEMPT enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets pegged at 99% CPU for much of the untar, but it does eventually complete. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-27 20:05 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-27 20:05 UTC (permalink / raw) To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > To test the theory, Chris asked me to try with data=ordered. > Unfortunately, the deadlock still shows up. This is what I get. As another data point: I'm trying the same kernel with CONFIG_PREEMPT enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets pegged at 99% CPU for much of the untar, but it does eventually complete. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 20:05 ` James Bottomley @ 2011-04-28 11:36 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 11:36 UTC (permalink / raw) To: James Bottomley Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 One more data point to add, I've been looking at an identical issue when copying large amounts of data. I bisected this - and the lockups occur with commit 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the issue. With this commit, my file copy test locks up after ~8-10 iterations, before this commit I can copy > 100 times and don't see the lockup. On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote: > On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > > To test the theory, Chris asked me to try with data=ordered. > > Unfortunately, the deadlock still shows up. This is what I get. > > As another data point: I'm trying the same kernel with CONFIG_PREEMPT > enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets > pegged at 99% CPU for much of the untar, but it does eventually > complete. > > James > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 11:36 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 11:36 UTC (permalink / raw) To: James Bottomley Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 One more data point to add, I've been looking at an identical issue when copying large amounts of data. I bisected this - and the lockups occur with commit 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the issue. With this commit, my file copy test locks up after ~8-10 iterations, before this commit I can copy > 100 times and don't see the lockup. On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote: > On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > > To test the theory, Chris asked me to try with data=ordered. > > Unfortunately, the deadlock still shows up. This is what I get. > > As another data point: I'm trying the same kernel with CONFIG_PREEMPT > enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets > pegged at 99% CPU for much of the untar, but it does eventually > complete. > > James > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 11:36 ` Colin Ian King @ 2011-04-28 12:29 ` Chris Mason -1 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-28 12:29 UTC (permalink / raw) To: Colin Ian King Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > One more data point to add, I've been looking at an identical issue when > copying large amounts of data. I bisected this - and the lockups occur > with commit > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > issue. With this commit, my file copy test locks up after ~8-10 > iterations, before this commit I can copy > 100 times and don't see the > lockup. Well, that's really interesting. I tried with compaction on here and couldn't trigger it, but this (very very lightly) tested patch might help. It moves the writeout throttle before the goto restart, and also makes sure we do at least one cond_resched before we loop. diff --git a/mm/vmscan.c b/mm/vmscan.c index 6771ea7..cb08b41 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1934,12 +1934,14 @@ restart: if (inactive_anon_is_low(zone, sc)) shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0); + throttle_vm_writeout(sc->gfp_mask); + /* reclaim/compaction might need reclaim to continue */ if (should_continue_reclaim(zone, nr_reclaimed, - sc->nr_scanned - nr_scanned, sc)) + sc->nr_scanned - nr_scanned, sc)) { + cond_resched(); goto restart; - - throttle_vm_writeout(sc->gfp_mask); + } } /* ^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 12:29 ` Chris Mason 0 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-28 12:29 UTC (permalink / raw) To: Colin Ian King Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > One more data point to add, I've been looking at an identical issue when > copying large amounts of data. I bisected this - and the lockups occur > with commit > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > issue. With this commit, my file copy test locks up after ~8-10 > iterations, before this commit I can copy > 100 times and don't see the > lockup. Well, that's really interesting. I tried with compaction on here and couldn't trigger it, but this (very very lightly) tested patch might help. It moves the writeout throttle before the goto restart, and also makes sure we do at least one cond_resched before we loop. diff --git a/mm/vmscan.c b/mm/vmscan.c index 6771ea7..cb08b41 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1934,12 +1934,14 @@ restart: if (inactive_anon_is_low(zone, sc)) shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0); + throttle_vm_writeout(sc->gfp_mask); + /* reclaim/compaction might need reclaim to continue */ if (should_continue_reclaim(zone, nr_reclaimed, - sc->nr_scanned - nr_scanned, sc)) + sc->nr_scanned - nr_scanned, sc)) { + cond_resched(); goto restart; - - throttle_vm_writeout(sc->gfp_mask); + } } /* -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 12:29 ` Chris Mason @ 2011-04-28 13:42 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 13:42 UTC (permalink / raw) To: Chris Mason Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote: > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > > One more data point to add, I've been looking at an identical issue when > > copying large amounts of data. I bisected this - and the lockups occur > > with commit > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > issue. With this commit, my file copy test locks up after ~8-10 > > iterations, before this commit I can copy > 100 times and don't see the > > lockup. > > Well, that's really interesting. I tried with compaction on here and > couldn't trigger it, but this (very very lightly) tested patch might > help. > Thanks Chris, I've given this a soak test but I still see the same lockup. > It moves the writeout throttle before the goto restart, and also makes > sure we do at least one cond_resched before we loop. > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 6771ea7..cb08b41 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1934,12 +1934,14 @@ restart: > if (inactive_anon_is_low(zone, sc)) > shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0); > > + throttle_vm_writeout(sc->gfp_mask); > + > /* reclaim/compaction might need reclaim to continue */ > if (should_continue_reclaim(zone, nr_reclaimed, > - sc->nr_scanned - nr_scanned, sc)) > + sc->nr_scanned - nr_scanned, sc)) { > + cond_resched(); > goto restart; > - > - throttle_vm_writeout(sc->gfp_mask); > + } > } > > /* ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 13:42 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 13:42 UTC (permalink / raw) To: Chris Mason Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote: > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > > One more data point to add, I've been looking at an identical issue when > > copying large amounts of data. I bisected this - and the lockups occur > > with commit > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > issue. With this commit, my file copy test locks up after ~8-10 > > iterations, before this commit I can copy > 100 times and don't see the > > lockup. > > Well, that's really interesting. I tried with compaction on here and > couldn't trigger it, but this (very very lightly) tested patch might > help. > Thanks Chris, I've given this a soak test but I still see the same lockup. > It moves the writeout throttle before the goto restart, and also makes > sure we do at least one cond_resched before we loop. > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 6771ea7..cb08b41 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1934,12 +1934,14 @@ restart: > if (inactive_anon_is_low(zone, sc)) > shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0); > > + throttle_vm_writeout(sc->gfp_mask); > + > /* reclaim/compaction might need reclaim to continue */ > if (should_continue_reclaim(zone, nr_reclaimed, > - sc->nr_scanned - nr_scanned, sc)) > + sc->nr_scanned - nr_scanned, sc)) { > + cond_resched(); > goto restart; > - > - throttle_vm_writeout(sc->gfp_mask); > + } > } > > /* -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 13:42 ` Colin Ian King @ 2011-04-28 13:45 ` Chris Mason -1 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-28 13:45 UTC (permalink / raw) To: Colin Ian King Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Excerpts from Colin Ian King's message of 2011-04-28 09:42:20 -0400: > > On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote: > > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > > > One more data point to add, I've been looking at an identical issue when > > > copying large amounts of data. I bisected this - and the lockups occur > > > with commit > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > issue. With this commit, my file copy test locks up after ~8-10 > > > iterations, before this commit I can copy > 100 times and don't see the > > > lockup. > > > > Well, that's really interesting. I tried with compaction on here and > > couldn't trigger it, but this (very very lightly) tested patch might > > help. > > > Thanks Chris, > > I've given this a soak test but I still see the same lockup. Could you post the soft lockups you're seeing? -chris ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 13:45 ` Chris Mason 0 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-28 13:45 UTC (permalink / raw) To: Colin Ian King Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Excerpts from Colin Ian King's message of 2011-04-28 09:42:20 -0400: > > On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote: > > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > > > One more data point to add, I've been looking at an identical issue when > > > copying large amounts of data. I bisected this - and the lockups occur > > > with commit > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > issue. With this commit, my file copy test locks up after ~8-10 > > > iterations, before this commit I can copy > 100 times and don't see the > > > lockup. > > > > Well, that's really interesting. I tried with compaction on here and > > couldn't trigger it, but this (very very lightly) tested patch might > > help. > > > Thanks Chris, > > I've given this a soak test but I still see the same lockup. Could you post the soft lockups you're seeing? -chris -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 13:45 ` Chris Mason (?) @ 2011-04-28 14:01 ` Colin Ian King 2011-04-28 14:04 ` Chris Mason 2011-04-28 14:25 ` Jan Kara -1 siblings, 2 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 14:01 UTC (permalink / raw) To: Chris Mason Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 349 bytes --] > Could you post the soft lockups you're seeing? As requested, attached Colin > > -chris > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ [-- Attachment #2: call-trace.log --] [-- Type: text/x-log, Size: 26550 bytes --] Apr 28 14:35:35 ubuntu kernel: [ 257.112169] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 257.112179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 257.112182] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 257.112186] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 Apr 28 14:46:15 ubuntu kernel: [ 257.112190] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 257.112194] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 Apr 28 14:46:15 ubuntu kernel: [ 257.112198] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 257.112205] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112210] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 257.112213] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112216] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 257.112220] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 257.112223] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112227] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 257.112230] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 257.112234] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 257.112237] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 257.112240] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 Apr 28 14:46:15 ubuntu kernel: [ 257.112242] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 Apr 28 14:46:15 ubuntu kernel: [ 257.112245] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 257.112249] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 257.112252] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 Apr 28 14:46:15 ubuntu kernel: [ 257.112255] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 14:46:15 ubuntu kernel: [ 257.112260] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 257.112262] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 257.112264] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 257.112267] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 Apr 28 14:46:15 ubuntu kernel: [ 257.112271] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 257.112275] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 Apr 28 14:46:15 ubuntu kernel: [ 257.112279] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 257.112282] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112285] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 257.112288] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112291] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 257.112294] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 257.112297] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112300] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 Apr 28 14:46:15 ubuntu kernel: [ 257.112304] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 Apr 28 14:46:15 ubuntu kernel: [ 257.112307] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 Apr 28 14:46:15 ubuntu kernel: [ 257.112311] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 Apr 28 14:46:15 ubuntu kernel: [ 257.112315] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 Apr 28 14:46:15 ubuntu kernel: [ 257.112319] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 Apr 28 14:46:15 ubuntu kernel: [ 257.112322] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 14:46:15 ubuntu kernel: [ 257.112325] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 Apr 28 14:46:15 ubuntu kernel: [ 257.112328] [<ffffffff810877e6>] kthread+0x96/0xa0 Apr 28 14:46:15 ubuntu kernel: [ 257.112332] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 Apr 28 14:46:15 ubuntu kernel: [ 257.112335] [<ffffffff81087750>] ? kthread+0x0/0xa0 Apr 28 14:46:15 ubuntu kernel: [ 257.112337] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 Apr 28 14:46:15 ubuntu kernel: [ 257.112340] INFO: task dirname:5969 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 257.112341] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 257.112343] dirname D 0000000000000002 0 5969 5214 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 257.112347] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 Apr 28 14:46:15 ubuntu kernel: [ 257.112351] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 257.112354] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 Apr 28 14:46:15 ubuntu kernel: [ 257.112358] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 257.112361] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112364] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 257.112367] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112370] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 257.112373] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 257.112376] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112380] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 257.112383] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 257.112386] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 257.112389] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 257.112392] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 Apr 28 14:46:15 ubuntu kernel: [ 257.112395] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 Apr 28 14:46:15 ubuntu kernel: [ 257.112398] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 Apr 28 14:46:15 ubuntu kernel: [ 257.112400] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 257.112403] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 257.112407] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 Apr 28 14:46:15 ubuntu kernel: [ 257.112409] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 257.112412] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 14:46:15 ubuntu kernel: [ 257.112416] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 Apr 28 14:46:15 ubuntu kernel: [ 257.112419] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 Apr 28 14:46:15 ubuntu kernel: [ 257.112422] [<ffffffff812e6dc6>] clear_user+0x36/0x40 Apr 28 14:46:15 ubuntu kernel: [ 257.112426] [<ffffffff811b0b6d>] padzero+0x2d/0x40 Apr 28 14:46:15 ubuntu kernel: [ 257.112428] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 Apr 28 14:46:15 ubuntu kernel: [ 257.112432] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 Apr 28 14:46:15 ubuntu kernel: [ 257.112435] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 Apr 28 14:46:15 ubuntu kernel: [ 257.112438] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 Apr 28 14:46:15 ubuntu kernel: [ 257.112441] [<ffffffff8101521a>] sys_execve+0x4a/0x80 Apr 28 14:46:15 ubuntu kernel: [ 257.112444] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 Apr 28 14:46:15 ubuntu kernel: [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088400] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 Apr 28 14:46:15 ubuntu kernel: [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 Apr 28 14:46:15 ubuntu kernel: [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 Apr 28 14:46:15 ubuntu kernel: [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 Apr 28 14:46:15 ubuntu kernel: [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 Apr 28 14:46:15 ubuntu kernel: [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 Apr 28 14:46:15 ubuntu kernel: [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 14:46:15 ubuntu kernel: [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 Apr 28 14:46:15 ubuntu kernel: [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088484] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 Apr 28 14:46:15 ubuntu kernel: [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 Apr 28 14:46:15 ubuntu kernel: [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 Apr 28 14:46:15 ubuntu kernel: [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 Apr 28 14:46:15 ubuntu kernel: [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 Apr 28 14:46:15 ubuntu kernel: [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 Apr 28 14:46:15 ubuntu kernel: [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 Apr 28 14:46:15 ubuntu kernel: [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 14:46:15 ubuntu kernel: [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 Apr 28 14:46:15 ubuntu kernel: [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088577] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 Apr 28 14:46:15 ubuntu kernel: [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 Apr 28 14:46:15 ubuntu kernel: [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 Apr 28 14:46:15 ubuntu kernel: [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 14:46:15 ubuntu kernel: [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 Apr 28 14:46:15 ubuntu kernel: [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088647] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 Apr 28 14:46:15 ubuntu kernel: [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 Apr 28 14:46:15 ubuntu kernel: [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 Apr 28 14:46:15 ubuntu kernel: [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 Apr 28 14:46:15 ubuntu kernel: [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 Apr 28 14:46:15 ubuntu kernel: [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 Apr 28 14:46:15 ubuntu kernel: [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 14:46:15 ubuntu kernel: [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 Apr 28 14:46:15 ubuntu kernel: [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 Apr 28 14:46:15 ubuntu kernel: [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 Apr 28 14:46:15 ubuntu kernel: [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 Apr 28 14:46:15 ubuntu kernel: [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 Apr 28 14:46:15 ubuntu kernel: [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 Apr 28 14:46:15 ubuntu kernel: [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 Apr 28 14:46:15 ubuntu kernel: [ 287.088723] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 Apr 28 14:46:15 ubuntu kernel: [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 Apr 28 14:46:15 ubuntu kernel: [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 Apr 28 14:46:15 ubuntu kernel: [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 Apr 28 14:46:15 ubuntu kernel: [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 14:46:15 ubuntu kernel: [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 Apr 28 14:46:15 ubuntu kernel: [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 Apr 28 14:46:15 ubuntu kernel: [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 Apr 28 14:46:15 ubuntu kernel: [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 Apr 28 14:46:15 ubuntu kernel: [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 Apr 28 14:46:15 ubuntu kernel: [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 Apr 28 14:46:15 ubuntu kernel: [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 Apr 28 14:46:15 ubuntu kernel: [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 Apr 28 14:46:15 ubuntu kernel: [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 Apr 28 14:46:15 ubuntu kernel: [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 317.064582] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 317.064585] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 317.064587] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 317.064591] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 Apr 28 14:46:15 ubuntu kernel: [ 317.064596] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 317.064607] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 Apr 28 14:46:15 ubuntu kernel: [ 317.064611] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 317.064616] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 317.064620] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 317.064623] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 317.064627] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 317.064630] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 317.064633] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 317.064637] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 317.064640] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 317.064643] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 317.064646] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 Apr 28 14:46:15 ubuntu kernel: [ 317.064649] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 Apr 28 14:46:15 ubuntu kernel: [ 317.064652] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 317.064655] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 Apr 28 14:46:15 ubuntu kernel: [ 317.064658] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 Apr 28 14:46:15 ubuntu kernel: [ 317.064661] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 Apr 28 14:46:15 ubuntu kernel: [ 317.064664] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 317.064667] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 317.064670] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 Apr 28 14:46:15 ubuntu kernel: [ 317.064673] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 Apr 28 14:46:15 ubuntu kernel: [ 317.064676] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 14:46:15 ubuntu kernel: [ 317.064679] INFO: task NetworkManager:764 blocked for more than 30 seconds. Apr 28 14:46:15 ubuntu kernel: [ 317.064681] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 14:46:15 ubuntu kernel: [ 317.064683] NetworkManager D 0000000000000002 0 764 1 0x00000000 Apr 28 14:46:15 ubuntu kernel: [ 317.064686] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 Apr 28 14:46:15 ubuntu kernel: [ 317.064690] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 Apr 28 14:46:15 ubuntu kernel: [ 317.064693] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 Apr 28 14:46:15 ubuntu kernel: [ 317.064697] Call Trace: Apr 28 14:46:15 ubuntu kernel: [ 317.064701] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 317.064704] [<ffffffff815c0990>] io_schedule+0x70/0xc0 Apr 28 14:46:15 ubuntu kernel: [ 317.064707] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 14:46:15 ubuntu kernel: [ 317.064710] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 Apr 28 14:46:15 ubuntu kernel: [ 317.064713] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 14:46:15 ubuntu kernel: [ 317.064716] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 14:46:15 ubuntu kernel: [ 317.064719] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 14:46:15 ubuntu kernel: [ 317.064722] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 14:46:15 ubuntu kernel: [ 317.064726] [<ffffffff81177110>] ? pollwake+0x0/0x60 Apr 28 14:46:15 ubuntu kernel: [ 317.064728] [<ffffffff8112d144>] __do_fault+0x54/0x520 Apr 28 14:46:15 ubuntu kernel: [ 317.064731] [<ffffffff81177110>] ? pollwake+0x0/0x60 Apr 28 14:46:15 ubuntu kernel: [ 317.064734] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 Apr 28 14:46:15 ubuntu kernel: [ 317.064737] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 Apr 28 14:46:15 ubuntu kernel: [ 317.064739] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 Apr 28 14:46:15 ubuntu kernel: [ 317.064742] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 Apr 28 14:46:15 ubuntu kernel: [ 317.064745] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 14:46:15 ubuntu kernel: [ 317.064749] [<ffffffff81013859>] ? read_tsc+0x9/0x20 Apr 28 14:46:15 ubuntu kernel: [ 317.064752] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 Apr 28 14:46:15 ubuntu kernel: [ 317.064755] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 Apr 28 14:46:15 ubuntu kernel: [ 317.064758] [<ffffffff815c34d5>] page_fault+0x25/0x30 ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:01 ` Colin Ian King @ 2011-04-28 14:04 ` Chris Mason 2011-04-28 14:25 ` Jan Kara 1 sibling, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-28 14:04 UTC (permalink / raw) To: Colin Ian King Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400: > > > Could you post the soft lockups you're seeing? > > As requested, attached These are not good, but they aren't the lockup James was seeing. Were these messages with my patch? If yes, please post the messages from without my patch. -chris ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 14:04 ` Chris Mason 0 siblings, 0 replies; 138+ messages in thread From: Chris Mason @ 2011-04-28 14:04 UTC (permalink / raw) To: Colin Ian King Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400: > > > Could you post the soft lockups you're seeing? > > As requested, attached These are not good, but they aren't the lockup James was seeing. Were these messages with my patch? If yes, please post the messages from without my patch. -chris -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:04 ` Chris Mason (?) @ 2011-04-28 15:23 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 15:23 UTC (permalink / raw) To: Chris Mason Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 449 bytes --] On Thu, 2011-04-28 at 10:04 -0400, Chris Mason wrote: > Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400: > > > > > Could you post the soft lockups you're seeing? > > > > As requested, attached > > These are not good, but they aren't the lockup James was seeing. Were > these messages with my patch? If yes, please post the messages from > without my patch. Attached are the messages without your patch. Colin > > -chris [-- Attachment #2: kern.log --] [-- Type: text/x-log, Size: 24228 bytes --] Apr 28 15:47:41 ubuntu kernel: [ 493.876427] INFO: task jbd2/sda1-8:290 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 493.876447] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 493.876455] jbd2/sda1-8 D 0000000000000000 0 290 2 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 493.876468] ffff8800363e5c20 0000000000000046 ffff8800363e5fd8 ffff8800363e4000 Apr 28 15:49:04 ubuntu kernel: [ 493.876481] 0000000000013d00 ffff8800720b3178 ffff8800363e5fd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 493.876507] ffffffff81a0b020 ffff8800720b2dc0 ffff88001fffb1d0 ffff88001fc13d00 Apr 28 15:49:04 ubuntu kernel: [ 493.876522] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 493.876542] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 493.876557] [<ffffffff815c0980>] io_schedule+0x70/0xc0 Apr 28 15:49:04 ubuntu kernel: [ 493.876570] [<ffffffff81191df0>] sync_buffer+0x40/0x50 Apr 28 15:49:04 ubuntu kernel: [ 493.876583] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90 Apr 28 15:49:04 ubuntu kernel: [ 493.876596] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 493.876609] [<ffffffff815c13ac>] out_of_line_wait_on_bit+0x7c/0x90 Apr 28 15:49:04 ubuntu kernel: [ 493.876623] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 493.876636] [<ffffffff81191dae>] __wait_on_buffer+0x2e/0x30 Apr 28 15:49:04 ubuntu kernel: [ 493.876649] [<ffffffff81243c3e>] jbd2_journal_commit_transaction+0x10ae/0x1190 Apr 28 15:49:04 ubuntu kernel: [ 493.876663] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.876677] [<ffffffff81247e8b>] kjournald2+0xbb/0x220 Apr 28 15:49:04 ubuntu kernel: [ 493.876688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.876700] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220 Apr 28 15:49:04 ubuntu kernel: [ 493.876711] [<ffffffff810877e6>] kthread+0x96/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 493.876725] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 Apr 28 15:49:04 ubuntu kernel: [ 493.876737] [<ffffffff81087750>] ? kthread+0x0/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 493.876747] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 Apr 28 15:49:04 ubuntu kernel: [ 493.876777] INFO: task flush-8:0:1137 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 493.876783] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 493.876792] flush-8:0 D 0000000000000002 0 1137 2 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 493.876806] ffff88007495f420 0000000000000046 ffff88007495ffd8 ffff88007495e000 Apr 28 15:49:04 ubuntu kernel: [ 493.876820] 0000000000013d00 ffff88007552b178 ffff88007495ffd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 493.876836] ffff8801005e5b80 ffff88007552adc0 ffff88005dd9a840 ffff88001fc53d00 Apr 28 15:49:04 ubuntu kernel: [ 493.876850] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 493.876863] [<ffffffff815c0980>] io_schedule+0x70/0xc0 Apr 28 15:49:04 ubuntu kernel: [ 493.876878] [<ffffffff812c3499>] get_request_wait+0xc9/0x1a0 Apr 28 15:49:04 ubuntu kernel: [ 493.876890] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.876902] [<ffffffff812bc680>] ? elv_merge+0x50/0x120 Apr 28 15:49:04 ubuntu kernel: [ 493.876915] [<ffffffff812c3c56>] __make_request+0x76/0x4c0 Apr 28 15:49:04 ubuntu kernel: [ 493.876929] [<ffffffff812c1148>] generic_make_request+0x2d8/0x5c0 Apr 28 15:49:04 ubuntu kernel: [ 493.876946] [<ffffffff8110e725>] ? mempool_alloc_slab+0x15/0x20 Apr 28 15:49:04 ubuntu kernel: [ 493.876957] [<ffffffff8110ea69>] ? mempool_alloc+0x59/0x140 Apr 28 15:49:04 ubuntu kernel: [ 493.876970] [<ffffffff812c14b9>] submit_bio+0x89/0x120 Apr 28 15:49:04 ubuntu kernel: [ 493.876982] [<ffffffff811976fb>] ? bio_alloc_bioset+0x5b/0xf0 Apr 28 15:49:04 ubuntu kernel: [ 493.876995] [<ffffffff811918eb>] submit_bh+0xeb/0x120 Apr 28 15:49:04 ubuntu kernel: [ 493.877007] [<ffffffff81193630>] __block_write_full_page+0x210/0x3a0 Apr 28 15:49:04 ubuntu kernel: [ 493.877018] [<ffffffff81192720>] ? end_buffer_async_write+0x0/0x170 Apr 28 15:49:04 ubuntu kernel: [ 493.877032] [<ffffffff81206aa0>] ? noalloc_get_block_write+0x0/0x30 Apr 28 15:49:04 ubuntu kernel: [ 493.877044] [<ffffffff81206aa0>] ? noalloc_get_block_write+0x0/0x30 Apr 28 15:49:04 ubuntu kernel: [ 493.877056] [<ffffffff811944d3>] block_write_full_page_endio+0xe3/0x120 Apr 28 15:49:04 ubuntu kernel: [ 493.877067] [<ffffffff81194525>] block_write_full_page+0x15/0x20 Apr 28 15:49:04 ubuntu kernel: [ 493.877079] [<ffffffff81204e9a>] mpage_da_submit_io+0x43a/0x4c0 Apr 28 15:49:04 ubuntu kernel: [ 493.877094] [<ffffffff8122929b>] ? __ext4_handle_dirty_metadata+0x7b/0x120 Apr 28 15:49:04 ubuntu kernel: [ 493.877110] [<ffffffff81208c8e>] mpage_da_map_and_submit+0x1ae/0x440 Apr 28 15:49:04 ubuntu kernel: [ 493.877122] [<ffffffff812dfe2d>] ? radix_tree_gang_lookup_tag_slot+0x8d/0xd0 Apr 28 15:49:04 ubuntu kernel: [ 493.877136] [<ffffffff81208f8d>] mpage_add_bh_to_extent+0x6d/0xf0 Apr 28 15:49:04 ubuntu kernel: [ 493.877149] [<ffffffff81209102>] __mpage_da_writepage+0xf2/0x190 Apr 28 15:49:04 ubuntu kernel: [ 493.877162] [<ffffffff8120935b>] write_cache_pages_da+0x1bb/0x2d0 Apr 28 15:49:04 ubuntu kernel: [ 493.877176] [<ffffffff81209786>] ext4_da_writepages+0x316/0x630 Apr 28 15:49:04 ubuntu kernel: [ 493.877191] [<ffffffff81117151>] do_writepages+0x21/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.877202] [<ffffffff8118b5df>] writeback_single_inode+0x9f/0x240 Apr 28 15:49:04 ubuntu kernel: [ 493.877215] [<ffffffff8118b9cb>] writeback_sb_inodes+0xcb/0x160 Apr 28 15:49:04 ubuntu kernel: [ 493.877229] [<ffffffff8118bc1b>] writeback_inodes_wb+0x10b/0x1c0 Apr 28 15:49:04 ubuntu kernel: [ 493.877240] [<ffffffff8118c04e>] wb_writeback+0x37e/0x490 Apr 28 15:49:04 ubuntu kernel: [ 493.877254] [<ffffffff815c2e6f>] ? _raw_spin_lock_irqsave+0x2f/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.877269] [<ffffffff81074fdb>] ? lock_timer_base.clone.20+0x3b/0x70 Apr 28 15:49:04 ubuntu kernel: [ 493.877283] [<ffffffff811166aa>] ? determine_dirtyable_memory+0x1a/0x30 Apr 28 15:49:04 ubuntu kernel: [ 493.877296] [<ffffffff8118c2ef>] wb_do_writeback+0x18f/0x230 Apr 28 15:49:04 ubuntu kernel: [ 493.877309] [<ffffffff8118c412>] bdi_writeback_thread+0x82/0x260 Apr 28 15:49:04 ubuntu kernel: [ 493.877321] [<ffffffff8118c390>] ? bdi_writeback_thread+0x0/0x260 Apr 28 15:49:04 ubuntu kernel: [ 493.877333] [<ffffffff810877e6>] kthread+0x96/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 493.877345] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 Apr 28 15:49:04 ubuntu kernel: [ 493.877357] [<ffffffff81087750>] ? kthread+0x0/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 493.877368] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 Apr 28 15:49:04 ubuntu kernel: [ 493.877386] INFO: task unity-panel-ser:1548 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 493.877393] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 493.877400] unity-panel-ser D 0000000000000002 0 1548 1 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 493.877414] ffff88006837fb68 0000000000000086 ffff88006837ffd8 ffff88006837e000 Apr 28 15:49:04 ubuntu kernel: [ 493.877429] 0000000000013d00 ffff8800723ac858 ffff88006837ffd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 493.877444] ffff8801005e5b80 ffff8800723ac4a0 ffff88001fff41c8 ffff88001fc53d00 Apr 28 15:49:04 ubuntu kernel: [ 493.877459] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 493.877473] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 493.877484] [<ffffffff815c0980>] io_schedule+0x70/0xc0 Apr 28 15:49:04 ubuntu kernel: [ 493.877497] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 15:49:04 ubuntu kernel: [ 493.877509] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90 Apr 28 15:49:04 ubuntu kernel: [ 493.877522] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 15:49:04 ubuntu kernel: [ 493.877534] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 493.877548] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 15:49:04 ubuntu kernel: [ 493.877561] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 Apr 28 15:49:04 ubuntu kernel: [ 493.877573] [<ffffffff8112d134>] __do_fault+0x54/0x520 Apr 28 15:49:04 ubuntu kernel: [ 493.877585] [<ffffffff811309ca>] handle_pte_fault+0xfa/0x210 Apr 28 15:49:04 ubuntu kernel: [ 493.877597] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.877608] [<ffffffff8112de3f>] ? __pte_alloc+0xdf/0x100 Apr 28 15:49:04 ubuntu kernel: [ 493.877619] [<ffffffff81131d4d>] handle_mm_fault+0x16d/0x250 Apr 28 15:49:04 ubuntu kernel: [ 493.877632] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 15:49:04 ubuntu kernel: [ 493.877644] [<ffffffff81136f75>] ? do_mmap_pgoff+0x335/0x370 Apr 28 15:49:04 ubuntu kernel: [ 493.877656] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 15:49:04 ubuntu kernel: [ 493.877672] INFO: task jbd2/sda4-8:2990 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 493.877678] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 493.877686] jbd2/sda4-8 D 0000000000000000 0 2990 2 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 493.877699] ffff880074f4fd20 0000000000000046 ffff880074f4ffd8 ffff880074f4e000 Apr 28 15:49:04 ubuntu kernel: [ 493.877714] 0000000000013d00 ffff88005c2103b8 ffff880074f4ffd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 493.877729] ffffffff81a0b020 ffff88005c210000 0000000000000002 ffff8800755ae5a0 Apr 28 15:49:04 ubuntu kernel: [ 493.877744] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 493.877756] [<ffffffff81242d40>] jbd2_journal_commit_transaction+0x1b0/0x1190 Apr 28 15:49:04 ubuntu kernel: [ 493.877772] [<ffffffff8100a6e0>] ? __switch_to+0xc0/0x2f0 Apr 28 15:49:04 ubuntu kernel: [ 493.877786] [<ffffffff815c2cbe>] ? _raw_spin_lock+0xe/0x20 Apr 28 15:49:04 ubuntu kernel: [ 493.877799] [<ffffffff81074fdb>] ? lock_timer_base.clone.20+0x3b/0x70 Apr 28 15:49:04 ubuntu kernel: [ 493.877812] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.877824] [<ffffffff81247e8b>] kjournald2+0xbb/0x220 Apr 28 15:49:04 ubuntu kernel: [ 493.877836] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 493.877847] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220 Apr 28 15:49:04 ubuntu kernel: [ 493.877858] [<ffffffff810877e6>] kthread+0x96/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 493.877869] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 Apr 28 15:49:04 ubuntu kernel: [ 493.877882] [<ffffffff81087750>] ? kthread+0x0/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 493.877892] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 Apr 28 15:49:04 ubuntu kernel: [ 553.778917] INFO: task jbd2/sda1-8:290 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 553.778936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 553.778944] jbd2/sda1-8 D 0000000000000002 0 290 2 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 553.778956] ffff8800363e5c20 0000000000000046 ffff8800363e5fd8 ffff8800363e4000 Apr 28 15:49:04 ubuntu kernel: [ 553.778969] 0000000000013d00 ffff8800720b3178 ffff8800363e5fd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 553.778981] ffff8801005e2dc0 ffff8800720b2dc0 ffff88001ffec230 ffff88001fc53d00 Apr 28 15:49:04 ubuntu kernel: [ 553.779007] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 553.779027] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779043] [<ffffffff815c0980>] io_schedule+0x70/0xc0 Apr 28 15:49:04 ubuntu kernel: [ 553.779055] [<ffffffff81191df0>] sync_buffer+0x40/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779067] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90 Apr 28 15:49:04 ubuntu kernel: [ 553.779080] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779093] [<ffffffff815c13ac>] out_of_line_wait_on_bit+0x7c/0x90 Apr 28 15:49:04 ubuntu kernel: [ 553.779107] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779121] [<ffffffff81191dae>] __wait_on_buffer+0x2e/0x30 Apr 28 15:49:04 ubuntu kernel: [ 553.779134] [<ffffffff812432ef>] jbd2_journal_commit_transaction+0x75f/0x1190 Apr 28 15:49:04 ubuntu kernel: [ 553.779148] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 553.779162] [<ffffffff81247e8b>] kjournald2+0xbb/0x220 Apr 28 15:49:04 ubuntu kernel: [ 553.779174] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 553.779185] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220 Apr 28 15:49:04 ubuntu kernel: [ 553.779196] [<ffffffff810877e6>] kthread+0x96/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 553.779210] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 Apr 28 15:49:04 ubuntu kernel: [ 553.779222] [<ffffffff81087750>] ? kthread+0x0/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 553.779233] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 Apr 28 15:49:04 ubuntu kernel: [ 553.779245] INFO: task rs:main Q:Reg:775 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 553.779252] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 553.779260] rs:main Q:Reg D 0000000000000000 0 775 1 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 553.779274] ffff880036317b98 0000000000000086 ffff880036317fd8 ffff880036316000 Apr 28 15:49:04 ubuntu kernel: [ 553.779289] 0000000000013d00 ffff8800759983b8 ffff880036317fd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 553.779304] ffff88003632adc0 ffff880075998000 ffff88001ffe6f68 ffff88001fc13d00 Apr 28 15:49:04 ubuntu kernel: [ 553.779318] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 553.779333] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779345] [<ffffffff815c0980>] io_schedule+0x70/0xc0 Apr 28 15:49:04 ubuntu kernel: [ 553.779357] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779369] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90 Apr 28 15:49:04 ubuntu kernel: [ 553.779381] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 15:49:04 ubuntu kernel: [ 553.779393] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779405] [<ffffffff812df74e>] ? radix_tree_lookup_slot+0xe/0x10 Apr 28 15:49:04 ubuntu kernel: [ 553.779418] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 15:49:04 ubuntu kernel: [ 553.779431] [<ffffffff8112f090>] do_swap_page.clone.50+0x5e0/0x640 Apr 28 15:49:04 ubuntu kernel: [ 553.779446] [<ffffffff8108bb90>] ? lock_hrtimer_base.clone.25+0x30/0x60 Apr 28 15:49:04 ubuntu kernel: [ 553.779459] [<ffffffff8108bc5c>] ? hrtimer_try_to_cancel+0x4c/0xe0 Apr 28 15:49:04 ubuntu kernel: [ 553.779472] [<ffffffff81130a89>] handle_pte_fault+0x1b9/0x210 Apr 28 15:49:04 ubuntu kernel: [ 553.779484] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 Apr 28 15:49:04 ubuntu kernel: [ 553.779494] [<ffffffff8112de3f>] ? __pte_alloc+0xdf/0x100 Apr 28 15:49:04 ubuntu kernel: [ 553.779505] [<ffffffff81131d4d>] handle_mm_fault+0x16d/0x250 Apr 28 15:49:04 ubuntu kernel: [ 553.779518] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 15:49:04 ubuntu kernel: [ 553.779531] [<ffffffff8109ce7b>] ? do_futex+0xbb/0x210 Apr 28 15:49:04 ubuntu kernel: [ 553.779543] [<ffffffff8109d04b>] ? sys_futex+0x7b/0x180 Apr 28 15:49:04 ubuntu kernel: [ 553.779555] [<ffffffff8116f985>] ? putname+0x35/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779567] [<ffffffff81164200>] ? do_sys_open+0x10/0x150 Apr 28 15:49:04 ubuntu kernel: [ 553.779579] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 15:49:04 ubuntu kernel: [ 553.779589] INFO: task rsyslogd:795 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 553.779595] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 553.779603] rsyslogd D 0000000000000000 0 795 1 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 553.779616] ffff880075c0db98 0000000000000086 ffff880075c0dfd8 ffff880075c0c000 Apr 28 15:49:04 ubuntu kernel: [ 553.779631] 0000000000013d00 ffff88003623b178 ffff880075c0dfd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 553.779646] ffff880075998000 ffff88003623adc0 ffff88001ffe6f68 ffff88001fc13d00 Apr 28 15:49:04 ubuntu kernel: [ 553.779660] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 553.779672] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779684] [<ffffffff815c0980>] io_schedule+0x70/0xc0 Apr 28 15:49:04 ubuntu kernel: [ 553.779696] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779709] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90 Apr 28 15:49:04 ubuntu kernel: [ 553.779720] [<ffffffff8114077f>] ? read_swap_cache_async+0x4f/0x160 Apr 28 15:49:04 ubuntu kernel: [ 553.779734] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 15:49:04 ubuntu kernel: [ 553.779746] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.779759] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 Apr 28 15:49:04 ubuntu kernel: [ 553.779770] [<ffffffff8112f090>] do_swap_page.clone.50+0x5e0/0x640 Apr 28 15:49:04 ubuntu kernel: [ 553.779783] [<ffffffff81130a89>] handle_pte_fault+0x1b9/0x210 Apr 28 15:49:04 ubuntu kernel: [ 553.779794] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 Apr 28 15:49:04 ubuntu kernel: [ 553.779805] [<ffffffff8112de3f>] ? __pte_alloc+0xdf/0x100 Apr 28 15:49:04 ubuntu kernel: [ 553.779816] [<ffffffff81131d4d>] handle_mm_fault+0x16d/0x250 Apr 28 15:49:04 ubuntu kernel: [ 553.779828] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 Apr 28 15:49:04 ubuntu kernel: [ 553.779841] [<ffffffff81164fd0>] ? vfs_read+0x120/0x180 Apr 28 15:49:04 ubuntu kernel: [ 553.779851] [<ffffffff815c34d5>] page_fault+0x25/0x30 Apr 28 15:49:04 ubuntu kernel: [ 553.779894] INFO: task tee:2977 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 553.779900] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 553.779908] tee D 0000000000000002 0 2977 2875 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 553.779921] ffff88005c1e39e8 0000000000000082 ffff88005c1e3fd8 ffff88005c1e2000 Apr 28 15:49:04 ubuntu kernel: [ 553.779935] 0000000000013d00 ffff8800755283b8 ffff88005c1e3fd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 553.779950] ffff8801005e5b80 ffff880075528000 ffff88005c1e39d8 ffff88005c1e3a38 Apr 28 15:49:04 ubuntu kernel: [ 553.779965] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 553.779980] [<ffffffff81241db5>] do_get_write_access+0x255/0x490 Apr 28 15:49:04 ubuntu kernel: [ 553.779992] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.780006] [<ffffffff81209e2d>] ? ext4_dirty_inode+0x3d/0x60 Apr 28 15:49:04 ubuntu kernel: [ 553.780019] [<ffffffff81242131>] jbd2_journal_get_write_access+0x31/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.780034] [<ffffffff81228f9e>] __ext4_journal_get_write_access+0x3e/0x80 Apr 28 15:49:04 ubuntu kernel: [ 553.780047] [<ffffffff81205268>] ext4_reserve_inode_write+0x78/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 553.780059] [<ffffffff812052df>] ext4_mark_inode_dirty+0x4f/0x220 Apr 28 15:49:04 ubuntu kernel: [ 553.780072] [<ffffffff8121dc03>] ? ext4_journal_start_sb+0xb3/0x140 Apr 28 15:49:04 ubuntu kernel: [ 553.780084] [<ffffffff81209e2d>] ext4_dirty_inode+0x3d/0x60 Apr 28 15:49:04 ubuntu kernel: [ 553.780096] [<ffffffff8118b31f>] __mark_inode_dirty+0x3f/0x260 Apr 28 15:49:04 ubuntu kernel: [ 553.780109] [<ffffffff8117f0b5>] file_update_time+0xf5/0x170 Apr 28 15:49:04 ubuntu kernel: [ 553.780124] [<ffffffff8110d928>] __generic_file_aio_write+0x1f8/0x440 Apr 28 15:49:04 ubuntu kernel: [ 553.780137] [<ffffffff8105f652>] ? default_wake_function+0x12/0x20 Apr 28 15:49:04 ubuntu kernel: [ 553.780150] [<ffffffff8110dbd6>] generic_file_aio_write+0x66/0xd0 Apr 28 15:49:04 ubuntu kernel: [ 553.780162] [<ffffffff811fe2a9>] ext4_file_write+0x69/0x280 Apr 28 15:49:04 ubuntu kernel: [ 553.780176] [<ffffffff8138b0c3>] ? pty_write+0x73/0x80 Apr 28 15:49:04 ubuntu kernel: [ 553.780188] [<ffffffff81038c79>] ? default_spin_lock_flags+0x9/0x10 Apr 28 15:49:04 ubuntu kernel: [ 553.780201] [<ffffffff81164682>] do_sync_write+0xd2/0x110 Apr 28 15:49:04 ubuntu kernel: [ 553.780217] [<ffffffff812ae7e8>] ? apparmor_file_permission+0x18/0x20 Apr 28 15:49:04 ubuntu kernel: [ 553.780231] [<ffffffff81279bac>] ? security_file_permission+0x2c/0xb0 Apr 28 15:49:04 ubuntu kernel: [ 553.780244] [<ffffffff81384d70>] ? n_tty_write+0x0/0x280 Apr 28 15:49:04 ubuntu kernel: [ 553.780256] [<ffffffff81164ab1>] ? rw_verify_area+0x61/0xf0 Apr 28 15:49:04 ubuntu kernel: [ 553.780268] [<ffffffff81164df6>] vfs_write+0xc6/0x180 Apr 28 15:49:04 ubuntu kernel: [ 553.780278] [<ffffffff81165111>] sys_write+0x51/0x90 Apr 28 15:49:04 ubuntu kernel: [ 553.780290] [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b Apr 28 15:49:04 ubuntu kernel: [ 553.780299] INFO: task jbd2/sda4-8:2990 blocked for more than 30 seconds. Apr 28 15:49:04 ubuntu kernel: [ 553.780307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 28 15:49:04 ubuntu kernel: [ 553.780314] jbd2/sda4-8 D 0000000000000000 0 2990 2 0x00000000 Apr 28 15:49:04 ubuntu kernel: [ 553.780327] ffff880074f4faf0 0000000000000046 ffff880074f4ffd8 ffff880074f4e000 Apr 28 15:49:04 ubuntu kernel: [ 553.780342] 0000000000013d00 ffff88005c2103b8 ffff880074f4ffd8 0000000000013d00 Apr 28 15:49:04 ubuntu kernel: [ 553.780357] ffffffff81a0b020 ffff88005c210000 ffff88001ffee048 ffff88001fc13d00 Apr 28 15:49:04 ubuntu kernel: [ 553.780372] Call Trace: Apr 28 15:49:04 ubuntu kernel: [ 553.780384] [<ffffffff8110c070>] ? sync_page+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.780396] [<ffffffff815c0980>] io_schedule+0x70/0xc0 Apr 28 15:49:04 ubuntu kernel: [ 553.780407] [<ffffffff8110c0b0>] sync_page+0x40/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.780419] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90 Apr 28 15:49:04 ubuntu kernel: [ 553.780432] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 Apr 28 15:49:04 ubuntu kernel: [ 553.780444] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 Apr 28 15:49:04 ubuntu kernel: [ 553.780457] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 Apr 28 15:49:04 ubuntu kernel: [ 553.780472] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 Apr 28 15:49:04 ubuntu kernel: [ 553.780484] [<ffffffff81242a83>] journal_finish_inode_data_buffers+0x63/0x170 Apr 28 15:49:04 ubuntu kernel: [ 553.780496] [<ffffffff81243274>] jbd2_journal_commit_transaction+0x6e4/0x1190 Apr 28 15:49:04 ubuntu kernel: [ 553.780510] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 553.780522] [<ffffffff81247e8b>] kjournald2+0xbb/0x220 Apr 28 15:49:04 ubuntu kernel: [ 553.780534] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 Apr 28 15:49:04 ubuntu kernel: [ 553.780546] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220 Apr 28 15:49:04 ubuntu kernel: [ 553.780556] [<ffffffff810877e6>] kthread+0x96/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 553.780568] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 Apr 28 15:49:04 ubuntu kernel: [ 553.780580] [<ffffffff81087750>] ? kthread+0x0/0xa0 Apr 28 15:49:04 ubuntu kernel: [ 553.780591] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:01 ` Colin Ian King @ 2011-04-28 14:25 ` Jan Kara 2011-04-28 14:25 ` Jan Kara 1 sibling, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 14:25 UTC (permalink / raw) To: Colin Ian King Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > Could you post the soft lockups you're seeing? > > As requested, attached Hum, what keeps puzzling me is that in all the cases of hangs I've seen so far, we are stuck waiting for IO to finish for a long time - e.g. in the traces below kjournald waits for PageWriteback bit to get cleared. Also we are stuck waiting for page locks which might be because those pages are being read in? All in all it seems that the IO is just incredibly slow. But it's not clear to me what pushes us into that situation (especially since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the underlying blocks are not already allocated. Honza [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 [ 287.088400] Call Trace: [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 [ 287.088484] Call Trace: [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 [ 287.088577] Call Trace: [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 [ 287.088647] Call Trace: [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 [ 287.088723] Call Trace: [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 14:25 ` Jan Kara 0 siblings, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 14:25 UTC (permalink / raw) To: Colin Ian King Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > Could you post the soft lockups you're seeing? > > As requested, attached Hum, what keeps puzzling me is that in all the cases of hangs I've seen so far, we are stuck waiting for IO to finish for a long time - e.g. in the traces below kjournald waits for PageWriteback bit to get cleared. Also we are stuck waiting for page locks which might be because those pages are being read in? All in all it seems that the IO is just incredibly slow. But it's not clear to me what pushes us into that situation (especially since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the underlying blocks are not already allocated. Honza [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 [ 287.088400] Call Trace: [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 [ 287.088484] Call Trace: [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 [ 287.088577] Call Trace: [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 [ 287.088647] Call Trace: [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 [ 287.088723] Call Trace: [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 -- Jan Kara <jack@suse.cz> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:25 ` Jan Kara @ 2011-04-28 14:33 ` Jan Kara -1 siblings, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 14:33 UTC (permalink / raw) To: Colin Ian King Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu 28-04-11 16:25:51, Jan Kara wrote: > On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > > > Could you post the soft lockups you're seeing? > > > > As requested, attached > Hum, what keeps puzzling me is that in all the cases of hangs I've seen > so far, we are stuck waiting for IO to finish for a long time - e.g. in the > traces below kjournald waits for PageWriteback bit to get cleared. Also we > are stuck waiting for page locks which might be because those pages are > being read in? All in all it seems that the IO is just incredibly slow. > > But it's not clear to me what pushes us into that situation (especially > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the > underlying blocks are not already allocated. Hmm, maybe because the system is under memory pressure (and kswapd is not able to get rid of dirty pages), we page out clean pages. Thus also pages of executables which need to be paged in soon anyway thus putting heavy read load on the system which makes writes crawl? I'm not sure why compaction should make this any worse but maybe it can. James, Colin, can you capture output of 'vmstat 1' while you do the copying? Thanks. Honza > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 > [ 287.088400] Call Trace: > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 > [ 287.088484] Call Trace: > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 > [ 287.088577] Call Trace: > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 > [ 287.088647] Call Trace: > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 > [ 287.088723] Call Trace: > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 > -- > Jan Kara <jack@suse.cz> > SUSE Labs, CR > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 14:33 ` Jan Kara 0 siblings, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 14:33 UTC (permalink / raw) To: Colin Ian King Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu 28-04-11 16:25:51, Jan Kara wrote: > On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > > > Could you post the soft lockups you're seeing? > > > > As requested, attached > Hum, what keeps puzzling me is that in all the cases of hangs I've seen > so far, we are stuck waiting for IO to finish for a long time - e.g. in the > traces below kjournald waits for PageWriteback bit to get cleared. Also we > are stuck waiting for page locks which might be because those pages are > being read in? All in all it seems that the IO is just incredibly slow. > > But it's not clear to me what pushes us into that situation (especially > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the > underlying blocks are not already allocated. Hmm, maybe because the system is under memory pressure (and kswapd is not able to get rid of dirty pages), we page out clean pages. Thus also pages of executables which need to be paged in soon anyway thus putting heavy read load on the system which makes writes crawl? I'm not sure why compaction should make this any worse but maybe it can. James, Colin, can you capture output of 'vmstat 1' while you do the copying? Thanks. Honza > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 > [ 287.088400] Call Trace: > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 > [ 287.088484] Call Trace: > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 > [ 287.088577] Call Trace: > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 > [ 287.088647] Call Trace: > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 > [ 287.088723] Call Trace: > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 > -- > Jan Kara <jack@suse.cz> > SUSE Labs, CR > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Jan Kara <jack@suse.cz> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:33 ` Jan Kara (?) @ 2011-04-28 14:58 ` Colin Ian King 2011-04-28 22:40 ` Jan Kara -1 siblings, 1 reply; 138+ messages in thread From: Colin Ian King @ 2011-04-28 14:58 UTC (permalink / raw) To: Jan Kara Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 11166 bytes --] On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote: > On Thu 28-04-11 16:25:51, Jan Kara wrote: > > On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > > > > > Could you post the soft lockups you're seeing? > > > > > > As requested, attached > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the > > traces below kjournald waits for PageWriteback bit to get cleared. Also we > > are stuck waiting for page locks which might be because those pages are > > being read in? All in all it seems that the IO is just incredibly slow. > > > > But it's not clear to me what pushes us into that situation (especially > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the > > underlying blocks are not already allocated. > Hmm, maybe because the system is under memory pressure (and kswapd is not > able to get rid of dirty pages), we page out clean pages. Thus also pages > of executables which need to be paged in soon anyway thus putting heavy > read load on the system which makes writes crawl? I'm not sure why > compaction should make this any worse but maybe it can. > > James, Colin, can you capture output of 'vmstat 1' while you do the > copying? Thanks. Attached. > > Honza > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 > > [ 287.088400] Call Trace: > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 > > [ 287.088484] Call Trace: > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 > > [ 287.088577] Call Trace: > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 > > [ 287.088647] Call Trace: > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 > > [ 287.088723] Call Trace: > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 > > -- > > Jan Kara <jack@suse.cz> > > SUSE Labs, CR > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: vmstat.log.gz --] [-- Type: application/x-gzip, Size: 14197 bytes --] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:58 ` Colin Ian King @ 2011-04-28 22:40 ` Jan Kara 0 siblings, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 22:40 UTC (permalink / raw) To: Colin Ian King Cc: Jan Kara, Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu 28-04-11 15:58:21, Colin Ian King wrote: > On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote: > > On Thu 28-04-11 16:25:51, Jan Kara wrote: > > > On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > > > > > > > Could you post the soft lockups you're seeing? > > > > > > > > As requested, attached > > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen > > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the > > > traces below kjournald waits for PageWriteback bit to get cleared. Also we > > > are stuck waiting for page locks which might be because those pages are > > > being read in? All in all it seems that the IO is just incredibly slow. > > > > > > But it's not clear to me what pushes us into that situation (especially > > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the > > > underlying blocks are not already allocated. > > Hmm, maybe because the system is under memory pressure (and kswapd is not > > able to get rid of dirty pages), we page out clean pages. Thus also pages > > of executables which need to be paged in soon anyway thus putting heavy > > read load on the system which makes writes crawl? I'm not sure why > > compaction should make this any worse but maybe it can. > > > > James, Colin, can you capture output of 'vmstat 1' while you do the > > copying? Thanks. > > Attached. Thanks. So I there are a few interesting points in the vmstat output: For first 30 seconds, we are happily copying data - relatively steady read throughput (about 20-40 MB/s) and occasional peak from flusher thread flushing dirty data. During this time free memory drops from about 1.4 GB to about 22!!! MB - mm seems to like to really use the machine ;). Then things get interesting: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22 0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18 2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21 0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24 1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3 1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0 1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0 1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0 4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3 0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28 0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32 0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34 0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22 So when free memory reached about 20 MB, both read and write activity almost stalled for 7 s (probably everybody waits for free memory). Then mm manages to free 100 MB from page cache, things move on for two seconds, then we swap out! about 36 MB and page reclaim also finally decides it maybe has too much of page cache and reaps most of it (1.3 GB in one go). Then things get going again, although there are still occasional stalls such as this (about 30s later): 1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33 0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39 0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28 1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38 1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1 1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3 1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0 1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2 1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1 0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15 0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27 0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23 ... I'm not sure what we were blocked on here since there is still plenty of free memory (750 MB). These stalls repeat once in a while but things go on. Then at about 350s, things just stop: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32 3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49 3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58 3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44 3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49 3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50 3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49 3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50 and nothing really happens for 150 s, except more and more tasks blocking in D state (second column). 3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50 0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55 Then suddently things get going again: 0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59 0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27 1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11 I guess this 150 s stall is when kernel barfs the "task blocked for more than 30 seconds" messages. And from the traces we know that everyone is waiting for PageWriteback or page lock during this time. Also James's vmstat report shows that IO is stalled when kswapd is spinning. Really strange. James in the meantime identified that cgroups are somehow involved. Are you using systemd by any chance? Maybe cgroup IO throttling screws us? Honza > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. > > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 > > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 > > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 > > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 > > > [ 287.088400] Call Trace: > > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 > > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 > > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 > > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. > > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 > > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 > > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 > > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 > > > [ 287.088484] Call Trace: > > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 > > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 > > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 > > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. > > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 > > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 > > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 > > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 > > > [ 287.088577] Call Trace: > > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. > > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 > > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 > > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 > > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 > > > [ 287.088647] Call Trace: > > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 > > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 > > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 > > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 > > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 > > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 > > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 > > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 > > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 > > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 > > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 > > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 > > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. > > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 > > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 > > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 > > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 > > > [ 287.088723] Call Trace: > > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 > > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 > > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 > > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 > > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 > > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 > > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 > > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 > > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 > > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 > > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 > > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 > > > -- > > > Jan Kara <jack@suse.cz> > > > SUSE Labs, CR > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 22:40 ` Jan Kara 0 siblings, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 22:40 UTC (permalink / raw) To: Colin Ian King Cc: Jan Kara, Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu 28-04-11 15:58:21, Colin Ian King wrote: > On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote: > > On Thu 28-04-11 16:25:51, Jan Kara wrote: > > > On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > > > > > > > Could you post the soft lockups you're seeing? > > > > > > > > As requested, attached > > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen > > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the > > > traces below kjournald waits for PageWriteback bit to get cleared. Also we > > > are stuck waiting for page locks which might be because those pages are > > > being read in? All in all it seems that the IO is just incredibly slow. > > > > > > But it's not clear to me what pushes us into that situation (especially > > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the > > > underlying blocks are not already allocated. > > Hmm, maybe because the system is under memory pressure (and kswapd is not > > able to get rid of dirty pages), we page out clean pages. Thus also pages > > of executables which need to be paged in soon anyway thus putting heavy > > read load on the system which makes writes crawl? I'm not sure why > > compaction should make this any worse but maybe it can. > > > > James, Colin, can you capture output of 'vmstat 1' while you do the > > copying? Thanks. > > Attached. Thanks. So I there are a few interesting points in the vmstat output: For first 30 seconds, we are happily copying data - relatively steady read throughput (about 20-40 MB/s) and occasional peak from flusher thread flushing dirty data. During this time free memory drops from about 1.4 GB to about 22!!! MB - mm seems to like to really use the machine ;). Then things get interesting: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22 0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18 2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21 0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24 1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3 1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0 1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0 1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0 4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3 0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28 0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32 0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34 0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22 So when free memory reached about 20 MB, both read and write activity almost stalled for 7 s (probably everybody waits for free memory). Then mm manages to free 100 MB from page cache, things move on for two seconds, then we swap out! about 36 MB and page reclaim also finally decides it maybe has too much of page cache and reaps most of it (1.3 GB in one go). Then things get going again, although there are still occasional stalls such as this (about 30s later): 1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33 0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39 0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28 1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38 1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1 1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3 1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0 1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2 1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1 0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15 0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27 0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23 ... I'm not sure what we were blocked on here since there is still plenty of free memory (750 MB). These stalls repeat once in a while but things go on. Then at about 350s, things just stop: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32 3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49 3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58 3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44 3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49 3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50 3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49 3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50 and nothing really happens for 150 s, except more and more tasks blocking in D state (second column). 3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50 0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55 Then suddently things get going again: 0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59 0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27 1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11 I guess this 150 s stall is when kernel barfs the "task blocked for more than 30 seconds" messages. And from the traces we know that everyone is waiting for PageWriteback or page lock during this time. Also James's vmstat report shows that IO is stalled when kswapd is spinning. Really strange. James in the meantime identified that cgroups are somehow involved. Are you using systemd by any chance? Maybe cgroup IO throttling screws us? Honza > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. > > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 > > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 > > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 > > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 > > > [ 287.088400] Call Trace: > > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 > > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 > > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 > > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. > > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 > > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 > > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 > > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 > > > [ 287.088484] Call Trace: > > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 > > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 > > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 > > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. > > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 > > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 > > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 > > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 > > > [ 287.088577] Call Trace: > > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. > > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 > > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 > > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 > > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 > > > [ 287.088647] Call Trace: > > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 > > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 > > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 > > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 > > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 > > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 > > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 > > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 > > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 > > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 > > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 > > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 > > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. > > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 > > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 > > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 > > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 > > > [ 287.088723] Call Trace: > > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 > > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 > > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 > > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 > > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 > > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 > > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 > > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 > > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 > > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 > > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 > > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 > > > -- > > > Jan Kara <jack@suse.cz> > > > SUSE Labs, CR > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Jan Kara <jack@suse.cz> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 22:40 ` Jan Kara @ 2011-04-28 22:44 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 22:44 UTC (permalink / raw) To: Jan Kara Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote: > James in the meantime identified that cgroups are somehow involved. > Are you > using systemd by any chance? Maybe cgroup IO throttling screws us? Yes, this is a FC15 system with systemd as the default. See related post: it looks like disabling the memory controller is what makes this go away. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 22:44 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 22:44 UTC (permalink / raw) To: Jan Kara Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote: > James in the meantime identified that cgroups are somehow involved. > Are you > using systemd by any chance? Maybe cgroup IO throttling screws us? Yes, this is a FC15 system with systemd as the default. See related post: it looks like disabling the memory controller is what makes this go away. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 22:40 ` Jan Kara @ 2011-05-03 18:55 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-05-03 18:55 UTC (permalink / raw) To: Jan Kara Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote: > On Thu 28-04-11 15:58:21, Colin Ian King wrote: > > On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote: > > > On Thu 28-04-11 16:25:51, Jan Kara wrote: > > > > On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > > > > > > > > > Could you post the soft lockups you're seeing? > > > > > > > > > > As requested, attached > > > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen > > > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the > > > > traces below kjournald waits for PageWriteback bit to get cleared. Also we > > > > are stuck waiting for page locks which might be because those pages are > > > > being read in? All in all it seems that the IO is just incredibly slow. > > > > > > > > But it's not clear to me what pushes us into that situation (especially > > > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the > > > > underlying blocks are not already allocated. > > > Hmm, maybe because the system is under memory pressure (and kswapd is not > > > able to get rid of dirty pages), we page out clean pages. Thus also pages > > > of executables which need to be paged in soon anyway thus putting heavy > > > read load on the system which makes writes crawl? I'm not sure why > > > compaction should make this any worse but maybe it can. > > > > > > James, Colin, can you capture output of 'vmstat 1' while you do the > > > copying? Thanks. > > > > Attached. > Thanks. So I there are a few interesting points in the vmstat output: > For first 30 seconds, we are happily copying data - relatively steady read > throughput (about 20-40 MB/s) and occasional peak from flusher thread > flushing dirty data. During this time free memory drops from about 1.4 GB > to about 22!!! MB - mm seems to like to really use the machine ;). Then > things get interesting: > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22 > 0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18 > 2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21 > 0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24 > 1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3 > 1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0 > 1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0 > 1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0 > 4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3 > 0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28 > 0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32 > 0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34 > 0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22 > > So when free memory reached about 20 MB, both read and write activity > almost stalled for 7 s (probably everybody waits for free memory). Then > mm manages to free 100 MB from page cache, things move on for two seconds, > then we swap out! about 36 MB and page reclaim also finally decides it > maybe has too much of page cache and reaps most of it (1.3 GB in one go). > Then things get going again, although there are still occasional stalls > such as this (about 30s later): > 1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33 > 0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39 > 0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28 > 1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38 > 1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1 > 1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3 > 1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0 > 1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2 > 1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1 > 0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15 > 0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27 > 0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23 > ... > I'm not sure what we were blocked on here since there is still plenty of > free memory (750 MB). These stalls repeat once in a while but things go on. > Then at about 350s, things just stop: > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32 > 3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49 > 3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58 > 3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44 > 3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49 > 3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50 > 3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49 > 3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50 > > and nothing really happens for 150 s, except more and more tasks blocking > in D state (second column). > 3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50 > 0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55 > Then suddently things get going again: > 0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59 > 0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27 > 1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11 > > I guess this 150 s stall is when kernel barfs the "task blocked for more > than 30 seconds" messages. And from the traces we know that everyone is > waiting for PageWriteback or page lock during this time. Also James's vmstat > report shows that IO is stalled when kswapd is spinning. Really strange. Just to add, this machine has relatively low amount of memory (1GB). I've re-run the tests today with cgroups disabled and it ran for 47 'copy' cycles, 27 'copy' cycles and then 35 'copy' cycles. One extra data point, with maxcpus=1 I get a lockup after 2 'copy' cycles every time, so it's more predictable than the default 4 processor configuration. > > James in the meantime identified that cgroups are somehow involved. Are you > using systemd by any chance? No, I'm using upstart. > Maybe cgroup IO throttling screws us? > > Honza > > > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. > > > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 > > > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 > > > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 > > > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 > > > > [ 287.088400] Call Trace: > > > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 > > > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 > > > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 > > > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. > > > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 > > > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 > > > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 > > > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 > > > > [ 287.088484] Call Trace: > > > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 > > > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 > > > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 > > > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. > > > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 > > > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 > > > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 > > > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 > > > > [ 287.088577] Call Trace: > > > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. > > > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 > > > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 > > > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 > > > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 > > > > [ 287.088647] Call Trace: > > > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 > > > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 > > > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 > > > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 > > > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 > > > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 > > > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 > > > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 > > > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 > > > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 > > > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 > > > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 > > > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. > > > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 > > > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 > > > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 > > > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 > > > > [ 287.088723] Call Trace: > > > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 > > > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 > > > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 > > > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 > > > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 > > > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 > > > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 > > > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 > > > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 > > > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 > > > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 > > > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 > > > > -- > > > > Jan Kara <jack@suse.cz> > > > > SUSE Labs, CR > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-03 18:55 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-05-03 18:55 UTC (permalink / raw) To: Jan Kara Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote: > On Thu 28-04-11 15:58:21, Colin Ian King wrote: > > On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote: > > > On Thu 28-04-11 16:25:51, Jan Kara wrote: > > > > On Thu 28-04-11 15:01:22, Colin Ian King wrote: > > > > > > > > > > > Could you post the soft lockups you're seeing? > > > > > > > > > > As requested, attached > > > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen > > > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the > > > > traces below kjournald waits for PageWriteback bit to get cleared. Also we > > > > are stuck waiting for page locks which might be because those pages are > > > > being read in? All in all it seems that the IO is just incredibly slow. > > > > > > > > But it's not clear to me what pushes us into that situation (especially > > > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the > > > > underlying blocks are not already allocated. > > > Hmm, maybe because the system is under memory pressure (and kswapd is not > > > able to get rid of dirty pages), we page out clean pages. Thus also pages > > > of executables which need to be paged in soon anyway thus putting heavy > > > read load on the system which makes writes crawl? I'm not sure why > > > compaction should make this any worse but maybe it can. > > > > > > James, Colin, can you capture output of 'vmstat 1' while you do the > > > copying? Thanks. > > > > Attached. > Thanks. So I there are a few interesting points in the vmstat output: > For first 30 seconds, we are happily copying data - relatively steady read > throughput (about 20-40 MB/s) and occasional peak from flusher thread > flushing dirty data. During this time free memory drops from about 1.4 GB > to about 22!!! MB - mm seems to like to really use the machine ;). Then > things get interesting: > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22 > 0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18 > 2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21 > 0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24 > 1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3 > 1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0 > 1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0 > 1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0 > 4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3 > 0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28 > 0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32 > 0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34 > 0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22 > > So when free memory reached about 20 MB, both read and write activity > almost stalled for 7 s (probably everybody waits for free memory). Then > mm manages to free 100 MB from page cache, things move on for two seconds, > then we swap out! about 36 MB and page reclaim also finally decides it > maybe has too much of page cache and reaps most of it (1.3 GB in one go). > Then things get going again, although there are still occasional stalls > such as this (about 30s later): > 1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33 > 0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39 > 0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28 > 1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38 > 1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1 > 1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3 > 1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0 > 1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2 > 1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1 > 0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15 > 0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27 > 0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23 > ... > I'm not sure what we were blocked on here since there is still plenty of > free memory (750 MB). These stalls repeat once in a while but things go on. > Then at about 350s, things just stop: > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32 > 3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49 > 3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58 > 3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44 > 3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49 > 3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50 > 3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49 > 3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50 > > and nothing really happens for 150 s, except more and more tasks blocking > in D state (second column). > 3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50 > 0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55 > Then suddently things get going again: > 0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59 > 0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27 > 1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11 > > I guess this 150 s stall is when kernel barfs the "task blocked for more > than 30 seconds" messages. And from the traces we know that everyone is > waiting for PageWriteback or page lock during this time. Also James's vmstat > report shows that IO is stalled when kswapd is spinning. Really strange. Just to add, this machine has relatively low amount of memory (1GB). I've re-run the tests today with cgroups disabled and it ran for 47 'copy' cycles, 27 'copy' cycles and then 35 'copy' cycles. One extra data point, with maxcpus=1 I get a lockup after 2 'copy' cycles every time, so it's more predictable than the default 4 processor configuration. > > James in the meantime identified that cgroups are somehow involved. Are you > using systemd by any chance? No, I'm using upstart. > Maybe cgroup IO throttling screws us? > > Honza > > > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds. > > > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000 > > > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000 > > > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00 > > > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00 > > > > [ 287.088400] Call Trace: > > > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170 > > > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0 > > > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230 > > > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds. > > > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000 > > > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000 > > > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00 > > > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00 > > > > [ 287.088484] Call Trace: > > > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60 > > > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20 > > > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0 > > > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90 > > > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds. > > > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000 > > > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000 > > > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00 > > > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00 > > > > [ 287.088577] Call Trace: > > > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40 > > > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100 > > > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370 > > > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds. > > > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000 > > > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000 > > > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00 > > > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00 > > > > [ 287.088647] Call Trace: > > > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190 > > > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30 > > > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170 > > > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190 > > > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0 > > > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220 > > > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40 > > > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220 > > > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0 > > > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 > > > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0 > > > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 > > > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds. > > > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000 > > > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000 > > > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00 > > > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00 > > > > [ 287.088723] Call Trace: > > > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50 > > > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0 > > > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50 > > > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90 > > > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80 > > > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50 > > > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70 > > > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0 > > > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520 > > > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210 > > > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50 > > > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20 > > > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100 > > > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250 > > > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540 > > > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500 > > > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0 > > > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30 > > > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70 > > > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70 > > > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40 > > > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40 > > > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00 > > > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300 > > > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00 > > > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0 > > > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80 > > > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0 > > > > -- > > > > Jan Kara <jack@suse.cz> > > > > SUSE Labs, CR > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:33 ` Jan Kara @ 2011-04-28 16:11 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:11 UTC (permalink / raw) To: Jan Kara Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote: > James, Colin, can you capture output of 'vmstat 1' while you do the > copying? Thanks. Sure; this is with a PREEMPT kernel so the tar goes to completion without locking the system. I started using a memory balloon to clear out the page cache. procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 98708 1676100 4112 39556 0 45 1566 1879 392 168 14 17 59 10 0 0 0 98708 1676100 4112 39604 0 0 0 0 37 35 0 0 100 0 0 0 0 98708 1676100 4112 39652 0 0 0 0 31 35 0 0 100 0 0 0 0 98708 1676100 4112 39652 0 0 0 0 26 27 0 0 100 0 0 0 0 98708 1676100 4112 39652 32 0 32 0 50 48 0 0 100 0 0 1 1 98708 1673984 4616 40576 192 0 1812 0 286 117 0 0 92 8 0 0 1 98708 1590540 4792 124236 0 0 41972 0 901 822 0 4 67 29 0 0 1 98708 1499524 4808 214820 0 0 45328 0 917 890 0 4 77 20 0 0 1 98708 1404532 4808 308316 64 0 46784 0 965 1033 0 3 77 20 0 0 1 98708 1312020 5612 399008 0 0 44836 0 1274 1075 0 6 77 18 0 0 1 98708 1304540 5628 406324 0 0 3600 47260 289 115 0 2 68 30 0 0 1 98708 1230760 5672 479020 0 0 36260 712 914 854 0 3 79 18 0 0 1 98708 1208696 5676 500016 0 0 10500 93888 421 251 0 3 67 31 0 0 1 98708 1157632 5676 551264 0 0 25600 0 658 518 0 2 75 23 0 0 1 98708 1052356 5680 655344 0 0 51972 0 1112 1112 0 4 78 19 0 0 2 98708 1024960 5680 679080 0 0 11904 87304 477 277 0 3 54 43 0 1 1 98708 1027288 5684 679432 0 0 128 2896 195 42 0 0 85 14 0 0 1 98708 947124 5688 758528 0 0 39552 4 932 899 0 4 68 28 0 0 1 98708 849792 5692 854272 0 0 47876 4468 1045 936 0 4 77 20 0 0 1 98708 813000 5820 890492 0 0 18176 18300 512 386 0 3 66 32 0 1 2 98708 794304 5820 908132 0 0 8832 95640 379 324 0 2 77 21 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 1 98708 794272 5824 908128 0 0 0 39456 212 58 0 1 73 26 0 0 1 98708 792344 5832 910452 0 0 1156 44440 193 62 0 1 74 25 0 0 1 98708 764392 5836 938288 0 0 13956 0 441 374 0 1 79 19 0 0 1 98708 660828 5840 1040232 0 0 50948 0 1141 992 0 6 65 29 0 0 1 98708 555156 5840 1144216 0 0 51968 0 1072 1008 0 4 76 20 0 0 1 98708 458692 5856 1239212 0 0 47496 120 995 1046 0 3 78 19 0 0 1 98708 411336 5856 1285896 0 0 23296 17764 536 463 0 3 71 26 0 1 1 98708 367716 5856 1328904 0 0 21504 22272 541 463 0 3 78 20 0 0 1 98708 330048 5864 1366044 0 0 18568 17524 529 515 0 2 78 20 0 0 1 98708 301620 5864 1393912 0 0 13952 28148 382 309 0 2 68 30 0 1 1 98708 245140 5872 1449576 0 0 27776 13224 610 567 0 3 73 24 0 0 2 98708 226868 5872 1467000 0 0 8704 114804 375 322 0 2 77 21 0 0 2 98708 226268 5876 1467552 0 0 260 51720 204 40 0 1 75 24 0 0 1 98708 226548 5880 1467536 0 0 4 13180 149 29 0 1 70 30 0 2 0 98708 159792 5880 1533844 0 0 33152 0 721 747 0 2 81 17 0 0 1 98708 55252 5888 1636692 0 0 51456 104 1021 977 0 5 70 25 0 1 1 98708 22804 2680 1671344 0 0 53632 0 1806 1188 0 20 59 21 0 2 1 98708 22832 2672 1671372 0 0 40704 11196 1967 1126 0 22 60 17 0 1 1 98708 22908 2676 1671196 0 0 24580 23040 1621 705 0 35 36 30 0 1 1 98708 22968 2676 1670796 0 0 21504 22212 1576 674 0 22 60 18 0 1 1 98708 22672 2684 1670996 0 0 17920 16968 1450 687 0 27 51 23 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 1 98708 22904 2452 1671344 0 0 24832 25780 1632 714 0 27 50 23 0 1 2 98708 22764 2452 1670724 0 0 15488 96304 1459 531 0 37 33 30 0 2 2 98708 22640 2452 1670724 0 0 0 60300 1219 356 0 20 45 35 0 1 1 98708 22744 2452 1670688 0 0 768 29312 1200 253 0 29 36 34 0 2 0 98708 22964 2460 1671040 0 0 10752 68 1345 445 0 23 56 21 0 1 1 98708 22652 2536 1671560 0 0 51540 0 2113 1246 0 28 49 23 0 1 1 98708 22676 2528 1671768 0 0 118276 14604 4875 2681 0 27 51 22 0 1 1 98708 22644 2528 1671804 0 0 23680 18172 1539 774 0 29 44 27 0 1 1 98708 22720 2536 1671464 0 0 21888 24664 1494 651 0 24 55 22 0 2 1 98708 22672 2496 1671592 0 0 22528 18144 1493 643 0 30 42 28 0 1 1 98708 22732 2488 1671508 0 0 22528 25192 1537 726 0 23 55 21 0 1 2 98708 22884 2488 1670504 0 0 18816 89440 1464 581 0 31 37 32 0 1 2 98708 22740 2488 1670764 0 0 768 75012 1196 254 0 22 55 23 0 1 2 98708 22848 2492 1670600 0 0 256 15512 1225 358 0 29 27 44 0 1 1 98708 22620 2500 1671628 0 0 19076 11408 1490 575 0 23 54 23 0 1 1 98708 22988 2500 1671108 0 0 55040 0 2118 1203 0 27 51 22 0 1 1 98708 22876 2488 1671044 0 0 53760 0 2138 1323 0 28 49 23 0 1 1 98708 22904 2500 1671012 0 0 48012 5528 2049 1097 0 32 43 25 0 1 1 98708 22856 2508 1670876 0 0 21504 18536 1500 646 0 23 55 22 0 2 0 98708 22684 2504 1670972 0 0 23680 23552 1537 784 0 29 44 26 0 1 1 98708 22960 2500 1670520 0 0 23808 25552 1529 677 0 24 55 21 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 1 98708 22820 2504 1670804 0 0 21764 25104 1507 647 0 30 43 26 0 1 1 98708 22880 2504 1670812 0 0 24192 18900 1541 768 0 23 55 22 0 2 3 98708 22716 2512 1670068 0 0 2816 132592 1257 303 0 30 30 40 0 1 1 98708 22944 2512 1669940 0 0 512 40400 1165 247 0 23 57 20 0 1 1 98708 22672 2512 1670624 0 0 512 0 1186 349 0 28 44 29 0 1 1 98708 22788 2504 1670416 0 0 51200 0 2130 1158 0 25 53 22 0 1 1 98708 22660 2508 1670688 0 0 50564 0 1972 1161 0 31 43 26 0 1 1 98708 22780 2516 1670088 0 0 50176 64 2028 1204 0 24 56 20 0 1 1 98708 22856 2508 1669884 0 0 32640 19040 1723 819 0 31 43 26 0 3 0 98708 22808 2512 1669812 0 0 23812 19176 1535 670 0 23 55 22 0 1 1 98708 22744 2504 1669728 0 0 21888 24872 1535 653 0 30 43 27 0 1 1 98708 22696 2504 1669636 0 0 22784 25032 1529 647 0 23 55 21 0 1 1 98708 22632 2512 1669596 0 0 21896 19228 1515 647 0 30 43 27 0 1 2 98708 22676 2512 1668992 0 0 9600 74484 1366 418 0 24 43 33 0 1 2 98708 22780 2512 1668772 0 0 256 107120 1211 245 0 29 42 30 0 1 1 98708 23196 2512 1668920 0 0 256 40 1158 228 0 23 57 20 0 2 1 98708 22968 2512 1669092 0 0 47360 0 2023 1097 0 26 49 25 0 1 1 98708 22864 2704 1668784 0 0 39992 116 1878 994 0 27 49 24 0 1 1 98708 22704 2708 1668872 0 0 53388 0 2166 1206 0 28 49 23 0 1 1 98708 22796 2712 1668472 0 0 39940 12784 1873 966 0 27 52 21 0 2 1 98708 22624 2712 1668508 0 0 21888 21904 1498 645 0 30 43 27 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 1 98708 22700 2704 1668688 0 0 24320 26240 1548 685 0 23 55 22 0 1 1 98708 22776 2712 1668284 0 0 18048 17532 1438 586 0 30 43 27 0 1 1 98708 22868 2712 1668336 0 0 24192 21764 1552 661 0 23 55 23 0 2 2 98708 22772 2716 1667696 0 0 12420 88772 1433 494 0 30 41 28 0 1 2 98708 23424 2720 1667584 0 0 4 9092 1181 246 0 23 40 37 0 1 1 98708 23120 2724 1667584 0 0 4 86380 1262 269 0 29 42 29 0 1 0 98708 23408 2984 1667708 0 0 364 0 1187 249 0 22 60 18 0 2 0 98708 23416 2992 1667772 0 0 0 120 1045 246 0 25 74 1 0 1 0 98708 23408 2992 1667720 0 0 0 0 1025 221 0 26 74 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1030 225 0 28 72 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1024 224 0 23 77 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1034 226 0 28 72 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1022 221 0 23 77 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1032 227 0 28 72 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1024 221 0 23 77 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1034 228 0 28 72 0 0 kswapd goes up to 99% at nearly the end of the second page. I ^C on the tar but kswapd stays at 99% for the end. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 16:11 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:11 UTC (permalink / raw) To: Jan Kara Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote: > James, Colin, can you capture output of 'vmstat 1' while you do the > copying? Thanks. Sure; this is with a PREEMPT kernel so the tar goes to completion without locking the system. I started using a memory balloon to clear out the page cache. procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 98708 1676100 4112 39556 0 45 1566 1879 392 168 14 17 59 10 0 0 0 98708 1676100 4112 39604 0 0 0 0 37 35 0 0 100 0 0 0 0 98708 1676100 4112 39652 0 0 0 0 31 35 0 0 100 0 0 0 0 98708 1676100 4112 39652 0 0 0 0 26 27 0 0 100 0 0 0 0 98708 1676100 4112 39652 32 0 32 0 50 48 0 0 100 0 0 1 1 98708 1673984 4616 40576 192 0 1812 0 286 117 0 0 92 8 0 0 1 98708 1590540 4792 124236 0 0 41972 0 901 822 0 4 67 29 0 0 1 98708 1499524 4808 214820 0 0 45328 0 917 890 0 4 77 20 0 0 1 98708 1404532 4808 308316 64 0 46784 0 965 1033 0 3 77 20 0 0 1 98708 1312020 5612 399008 0 0 44836 0 1274 1075 0 6 77 18 0 0 1 98708 1304540 5628 406324 0 0 3600 47260 289 115 0 2 68 30 0 0 1 98708 1230760 5672 479020 0 0 36260 712 914 854 0 3 79 18 0 0 1 98708 1208696 5676 500016 0 0 10500 93888 421 251 0 3 67 31 0 0 1 98708 1157632 5676 551264 0 0 25600 0 658 518 0 2 75 23 0 0 1 98708 1052356 5680 655344 0 0 51972 0 1112 1112 0 4 78 19 0 0 2 98708 1024960 5680 679080 0 0 11904 87304 477 277 0 3 54 43 0 1 1 98708 1027288 5684 679432 0 0 128 2896 195 42 0 0 85 14 0 0 1 98708 947124 5688 758528 0 0 39552 4 932 899 0 4 68 28 0 0 1 98708 849792 5692 854272 0 0 47876 4468 1045 936 0 4 77 20 0 0 1 98708 813000 5820 890492 0 0 18176 18300 512 386 0 3 66 32 0 1 2 98708 794304 5820 908132 0 0 8832 95640 379 324 0 2 77 21 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 1 98708 794272 5824 908128 0 0 0 39456 212 58 0 1 73 26 0 0 1 98708 792344 5832 910452 0 0 1156 44440 193 62 0 1 74 25 0 0 1 98708 764392 5836 938288 0 0 13956 0 441 374 0 1 79 19 0 0 1 98708 660828 5840 1040232 0 0 50948 0 1141 992 0 6 65 29 0 0 1 98708 555156 5840 1144216 0 0 51968 0 1072 1008 0 4 76 20 0 0 1 98708 458692 5856 1239212 0 0 47496 120 995 1046 0 3 78 19 0 0 1 98708 411336 5856 1285896 0 0 23296 17764 536 463 0 3 71 26 0 1 1 98708 367716 5856 1328904 0 0 21504 22272 541 463 0 3 78 20 0 0 1 98708 330048 5864 1366044 0 0 18568 17524 529 515 0 2 78 20 0 0 1 98708 301620 5864 1393912 0 0 13952 28148 382 309 0 2 68 30 0 1 1 98708 245140 5872 1449576 0 0 27776 13224 610 567 0 3 73 24 0 0 2 98708 226868 5872 1467000 0 0 8704 114804 375 322 0 2 77 21 0 0 2 98708 226268 5876 1467552 0 0 260 51720 204 40 0 1 75 24 0 0 1 98708 226548 5880 1467536 0 0 4 13180 149 29 0 1 70 30 0 2 0 98708 159792 5880 1533844 0 0 33152 0 721 747 0 2 81 17 0 0 1 98708 55252 5888 1636692 0 0 51456 104 1021 977 0 5 70 25 0 1 1 98708 22804 2680 1671344 0 0 53632 0 1806 1188 0 20 59 21 0 2 1 98708 22832 2672 1671372 0 0 40704 11196 1967 1126 0 22 60 17 0 1 1 98708 22908 2676 1671196 0 0 24580 23040 1621 705 0 35 36 30 0 1 1 98708 22968 2676 1670796 0 0 21504 22212 1576 674 0 22 60 18 0 1 1 98708 22672 2684 1670996 0 0 17920 16968 1450 687 0 27 51 23 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 1 98708 22904 2452 1671344 0 0 24832 25780 1632 714 0 27 50 23 0 1 2 98708 22764 2452 1670724 0 0 15488 96304 1459 531 0 37 33 30 0 2 2 98708 22640 2452 1670724 0 0 0 60300 1219 356 0 20 45 35 0 1 1 98708 22744 2452 1670688 0 0 768 29312 1200 253 0 29 36 34 0 2 0 98708 22964 2460 1671040 0 0 10752 68 1345 445 0 23 56 21 0 1 1 98708 22652 2536 1671560 0 0 51540 0 2113 1246 0 28 49 23 0 1 1 98708 22676 2528 1671768 0 0 118276 14604 4875 2681 0 27 51 22 0 1 1 98708 22644 2528 1671804 0 0 23680 18172 1539 774 0 29 44 27 0 1 1 98708 22720 2536 1671464 0 0 21888 24664 1494 651 0 24 55 22 0 2 1 98708 22672 2496 1671592 0 0 22528 18144 1493 643 0 30 42 28 0 1 1 98708 22732 2488 1671508 0 0 22528 25192 1537 726 0 23 55 21 0 1 2 98708 22884 2488 1670504 0 0 18816 89440 1464 581 0 31 37 32 0 1 2 98708 22740 2488 1670764 0 0 768 75012 1196 254 0 22 55 23 0 1 2 98708 22848 2492 1670600 0 0 256 15512 1225 358 0 29 27 44 0 1 1 98708 22620 2500 1671628 0 0 19076 11408 1490 575 0 23 54 23 0 1 1 98708 22988 2500 1671108 0 0 55040 0 2118 1203 0 27 51 22 0 1 1 98708 22876 2488 1671044 0 0 53760 0 2138 1323 0 28 49 23 0 1 1 98708 22904 2500 1671012 0 0 48012 5528 2049 1097 0 32 43 25 0 1 1 98708 22856 2508 1670876 0 0 21504 18536 1500 646 0 23 55 22 0 2 0 98708 22684 2504 1670972 0 0 23680 23552 1537 784 0 29 44 26 0 1 1 98708 22960 2500 1670520 0 0 23808 25552 1529 677 0 24 55 21 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 1 98708 22820 2504 1670804 0 0 21764 25104 1507 647 0 30 43 26 0 1 1 98708 22880 2504 1670812 0 0 24192 18900 1541 768 0 23 55 22 0 2 3 98708 22716 2512 1670068 0 0 2816 132592 1257 303 0 30 30 40 0 1 1 98708 22944 2512 1669940 0 0 512 40400 1165 247 0 23 57 20 0 1 1 98708 22672 2512 1670624 0 0 512 0 1186 349 0 28 44 29 0 1 1 98708 22788 2504 1670416 0 0 51200 0 2130 1158 0 25 53 22 0 1 1 98708 22660 2508 1670688 0 0 50564 0 1972 1161 0 31 43 26 0 1 1 98708 22780 2516 1670088 0 0 50176 64 2028 1204 0 24 56 20 0 1 1 98708 22856 2508 1669884 0 0 32640 19040 1723 819 0 31 43 26 0 3 0 98708 22808 2512 1669812 0 0 23812 19176 1535 670 0 23 55 22 0 1 1 98708 22744 2504 1669728 0 0 21888 24872 1535 653 0 30 43 27 0 1 1 98708 22696 2504 1669636 0 0 22784 25032 1529 647 0 23 55 21 0 1 1 98708 22632 2512 1669596 0 0 21896 19228 1515 647 0 30 43 27 0 1 2 98708 22676 2512 1668992 0 0 9600 74484 1366 418 0 24 43 33 0 1 2 98708 22780 2512 1668772 0 0 256 107120 1211 245 0 29 42 30 0 1 1 98708 23196 2512 1668920 0 0 256 40 1158 228 0 23 57 20 0 2 1 98708 22968 2512 1669092 0 0 47360 0 2023 1097 0 26 49 25 0 1 1 98708 22864 2704 1668784 0 0 39992 116 1878 994 0 27 49 24 0 1 1 98708 22704 2708 1668872 0 0 53388 0 2166 1206 0 28 49 23 0 1 1 98708 22796 2712 1668472 0 0 39940 12784 1873 966 0 27 52 21 0 2 1 98708 22624 2712 1668508 0 0 21888 21904 1498 645 0 30 43 27 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 1 98708 22700 2704 1668688 0 0 24320 26240 1548 685 0 23 55 22 0 1 1 98708 22776 2712 1668284 0 0 18048 17532 1438 586 0 30 43 27 0 1 1 98708 22868 2712 1668336 0 0 24192 21764 1552 661 0 23 55 23 0 2 2 98708 22772 2716 1667696 0 0 12420 88772 1433 494 0 30 41 28 0 1 2 98708 23424 2720 1667584 0 0 4 9092 1181 246 0 23 40 37 0 1 1 98708 23120 2724 1667584 0 0 4 86380 1262 269 0 29 42 29 0 1 0 98708 23408 2984 1667708 0 0 364 0 1187 249 0 22 60 18 0 2 0 98708 23416 2992 1667772 0 0 0 120 1045 246 0 25 74 1 0 1 0 98708 23408 2992 1667720 0 0 0 0 1025 221 0 26 74 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1030 225 0 28 72 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1024 224 0 23 77 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1034 226 0 28 72 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1022 221 0 23 77 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1032 227 0 28 72 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1024 221 0 23 77 0 0 1 0 98708 23408 2992 1667720 0 0 0 0 1034 228 0 28 72 0 0 kswapd goes up to 99% at nearly the end of the second page. I ^C on the tar but kswapd stays at 99% for the end. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 12:29 ` Chris Mason @ 2011-04-28 14:49 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 14:49 UTC (permalink / raw) To: Chris Mason Cc: Colin Ian King, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote: > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > > One more data point to add, I've been looking at an identical issue when > > copying large amounts of data. I bisected this - and the lockups occur > > with commit > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > issue. With this commit, my file copy test locks up after ~8-10 > > iterations, before this commit I can copy > 100 times and don't see the > > lockup. > > Well, that's really interesting. I tried with compaction on here and > couldn't trigger it, but this (very very lightly) tested patch might > help. > > It moves the writeout throttle before the goto restart, and also makes > sure we do at least one cond_resched before we loop. It seems to take longer, but with a PREEMPT kernel, kswapd eventually shoots up to 99% during the tar. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 14:49 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 14:49 UTC (permalink / raw) To: Chris Mason Cc: Colin Ian King, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote: > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400: > > One more data point to add, I've been looking at an identical issue when > > copying large amounts of data. I bisected this - and the lockups occur > > with commit > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > issue. With this commit, my file copy test locks up after ~8-10 > > iterations, before this commit I can copy > 100 times and don't see the > > lockup. > > Well, that's really interesting. I tried with compaction on here and > couldn't trigger it, but this (very very lightly) tested patch might > help. > > It moves the writeout throttle before the goto restart, and also makes > sure we do at least one cond_resched before we loop. It seems to take longer, but with a PREEMPT kernel, kswapd eventually shoots up to 99% during the tar. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 11:36 ` Colin Ian King @ 2011-04-28 13:52 ` Jan Kara -1 siblings, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 13:52 UTC (permalink / raw) To: colin.king Cc: James Bottomley, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu 28-04-11 12:36:30, Colin Ian King wrote: > One more data point to add, I've been looking at an identical issue when > copying large amounts of data. I bisected this - and the lockups occur > with commit > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > issue. With this commit, my file copy test locks up after ~8-10 > iterations, before this commit I can copy > 100 times and don't see the > lockup. Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit of yours causes kswapd on non-preempt kernels spin for a *long* time... Honza > On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote: > > On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > > > To test the theory, Chris asked me to try with data=ordered. > > > Unfortunately, the deadlock still shows up. This is what I get. > > > > As another data point: I'm trying the same kernel with CONFIG_PREEMPT > > enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets > > pegged at 99% CPU for much of the untar, but it does eventually > > complete. > > > > James > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 13:52 ` Jan Kara 0 siblings, 0 replies; 138+ messages in thread From: Jan Kara @ 2011-04-28 13:52 UTC (permalink / raw) To: colin.king Cc: James Bottomley, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu 28-04-11 12:36:30, Colin Ian King wrote: > One more data point to add, I've been looking at an identical issue when > copying large amounts of data. I bisected this - and the lockups occur > with commit > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > issue. With this commit, my file copy test locks up after ~8-10 > iterations, before this commit I can copy > 100 times and don't see the > lockup. Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit of yours causes kswapd on non-preempt kernels spin for a *long* time... Honza > On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote: > > On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote: > > > To test the theory, Chris asked me to try with data=ordered. > > > Unfortunately, the deadlock still shows up. This is what I get. > > > > As another data point: I'm trying the same kernel with CONFIG_PREEMPT > > enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets > > pegged at 99% CPU for much of the untar, but it does eventually > > complete. > > > > James > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Jan Kara <jack@suse.cz> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 13:52 ` Jan Kara @ 2011-04-28 14:07 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 14:07 UTC (permalink / raw) To: Jan Kara Cc: colin.king, James Bottomley, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > One more data point to add, I've been looking at an identical issue when > > copying large amounts of data. I bisected this - and the lockups occur > > with commit > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > issue. With this commit, my file copy test locks up after ~8-10 > > iterations, before this commit I can copy > 100 times and don't see the > > lockup. > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > of yours causes kswapd on non-preempt kernels spin for a *long* time... > I'm still thinking about the traces which do not point the finger directly at compaction per-se but it's possible that the change means kswapd is not reclaiming like it should be. To test this theory, does applying [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce CPU usage] help? -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 14:07 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 14:07 UTC (permalink / raw) To: Jan Kara Cc: colin.king, James Bottomley, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > One more data point to add, I've been looking at an identical issue when > > copying large amounts of data. I bisected this - and the lockups occur > > with commit > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > issue. With this commit, my file copy test locks up after ~8-10 > > iterations, before this commit I can copy > 100 times and don't see the > > lockup. > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > of yours causes kswapd on non-preempt kernels spin for a *long* time... > I'm still thinking about the traces which do not point the finger directly at compaction per-se but it's possible that the change means kswapd is not reclaiming like it should be. To test this theory, does applying [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce CPU usage] help? -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:07 ` Mel Gorman @ 2011-04-28 14:25 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 14:25 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > > One more data point to add, I've been looking at an identical issue when > > > copying large amounts of data. I bisected this - and the lockups occur > > > with commit > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > issue. With this commit, my file copy test locks up after ~8-10 > > > iterations, before this commit I can copy > 100 times and don't see the > > > lockup. > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > > of yours causes kswapd on non-preempt kernels spin for a *long* time... > > > > I'm still thinking about the traces which do not point the finger > directly at compaction per-se but it's possible that the change means > kswapd is not reclaiming like it should be. > > To test this theory, does applying > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce > CPU usage] help? I can answer definitively no to this. The upstream kernel I reproduced this on has that patch included. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 14:25 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 14:25 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > > One more data point to add, I've been looking at an identical issue when > > > copying large amounts of data. I bisected this - and the lockups occur > > > with commit > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > issue. With this commit, my file copy test locks up after ~8-10 > > > iterations, before this commit I can copy > 100 times and don't see the > > > lockup. > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > > of yours causes kswapd on non-preempt kernels spin for a *long* time... > > > > I'm still thinking about the traces which do not point the finger > directly at compaction per-se but it's possible that the change means > kswapd is not reclaiming like it should be. > > To test this theory, does applying > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce > CPU usage] help? I can answer definitively no to this. The upstream kernel I reproduced this on has that patch included. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 14:25 ` James Bottomley @ 2011-04-28 15:08 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 15:08 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote: > > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > > > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > > > One more data point to add, I've been looking at an identical issue when > > > > copying large amounts of data. I bisected this - and the lockups occur > > > > with commit > > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > > issue. With this commit, my file copy test locks up after ~8-10 > > > > iterations, before this commit I can copy > 100 times and don't see the > > > > lockup. > > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > > > of yours causes kswapd on non-preempt kernels spin for a *long* time... > > > > > > > I'm still thinking about the traces which do not point the finger > > directly at compaction per-se but it's possible that the change means > > kswapd is not reclaiming like it should be. > > > > To test this theory, does applying > > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce > > CPU usage] help? > > I can answer definitively no to this. The upstream kernel I reproduced > this on has that patch included. > So it is. Another consequence of this patch is that when high order allocations are in progress (is the test case fork heavy in any way for example? alternatively, it might be something in the storage stack that requires high-order allocs) we are no longer necessarily going to sleep because of should_reclaim_continue() check. This could explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is set (does unsetting CONFIG_COMPACTION help). If the bug only triggers for CONFIG_COMPACTION, does the following *untested* patch help any? (as a warning, I'm offline Friday until Tuesday morning. I'll try check mail over the weekend but it's unlikely I'll find a terminal or be allowed to use it without an ass kicking) diff --git a/mm/vmscan.c b/mm/vmscan.c index 148c6e6..c74a501 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone, return false; /* - * If we failed to reclaim and have scanned the full list, stop. - * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far - * faster but obviously would be less likely to succeed - * allocation. If this is desirable, use GFP_REPEAT to decide - * if both reclaimed and scanned should be checked or just - * reclaimed + * For direct reclaimers + * If we failed to reclaim and have scanned the full list, stop. + * The caller will check congestion and sleep if necessary until + * some IO completes. + * For kswapd + * Check just nr_reclaimed. If we are failing to reclaim, we + * want to stop this reclaim loop, increase the priority and + * go to sleep if necessary to allow IO a change to complete. + * This avoids kswapd going into a busy loop in shrink_zone() */ - if (!nr_reclaimed && !nr_scanned) - return false; + if (!nr_reclaimed) { + if (current_is_kswapd()) + return false; + else if (!nr_scanned) + return false; + } /* * If we have not reclaimed enough pages for compaction and the @@ -1924,8 +1931,13 @@ restart: /* reclaim/compaction might need reclaim to continue */ if (should_continue_reclaim(zone, nr_reclaimed, - sc->nr_scanned - nr_scanned, sc)) + sc->nr_scanned - nr_scanned, sc)) { + /* Throttle direct reclaimers if congested */ + if (!current_is_kswapd()) + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10); + goto restart; + } throttle_vm_writeout(sc->gfp_mask); } ^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 15:08 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 15:08 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote: > > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > > > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > > > One more data point to add, I've been looking at an identical issue when > > > > copying large amounts of data. I bisected this - and the lockups occur > > > > with commit > > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > > issue. With this commit, my file copy test locks up after ~8-10 > > > > iterations, before this commit I can copy > 100 times and don't see the > > > > lockup. > > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > > > of yours causes kswapd on non-preempt kernels spin for a *long* time... > > > > > > > I'm still thinking about the traces which do not point the finger > > directly at compaction per-se but it's possible that the change means > > kswapd is not reclaiming like it should be. > > > > To test this theory, does applying > > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce > > CPU usage] help? > > I can answer definitively no to this. The upstream kernel I reproduced > this on has that patch included. > So it is. Another consequence of this patch is that when high order allocations are in progress (is the test case fork heavy in any way for example? alternatively, it might be something in the storage stack that requires high-order allocs) we are no longer necessarily going to sleep because of should_reclaim_continue() check. This could explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is set (does unsetting CONFIG_COMPACTION help). If the bug only triggers for CONFIG_COMPACTION, does the following *untested* patch help any? (as a warning, I'm offline Friday until Tuesday morning. I'll try check mail over the weekend but it's unlikely I'll find a terminal or be allowed to use it without an ass kicking) diff --git a/mm/vmscan.c b/mm/vmscan.c index 148c6e6..c74a501 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone, return false; /* - * If we failed to reclaim and have scanned the full list, stop. - * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far - * faster but obviously would be less likely to succeed - * allocation. If this is desirable, use GFP_REPEAT to decide - * if both reclaimed and scanned should be checked or just - * reclaimed + * For direct reclaimers + * If we failed to reclaim and have scanned the full list, stop. + * The caller will check congestion and sleep if necessary until + * some IO completes. + * For kswapd + * Check just nr_reclaimed. If we are failing to reclaim, we + * want to stop this reclaim loop, increase the priority and + * go to sleep if necessary to allow IO a change to complete. + * This avoids kswapd going into a busy loop in shrink_zone() */ - if (!nr_reclaimed && !nr_scanned) - return false; + if (!nr_reclaimed) { + if (current_is_kswapd()) + return false; + else if (!nr_scanned) + return false; + } /* * If we have not reclaimed enough pages for compaction and the @@ -1924,8 +1931,13 @@ restart: /* reclaim/compaction might need reclaim to continue */ if (should_continue_reclaim(zone, nr_reclaimed, - sc->nr_scanned - nr_scanned, sc)) + sc->nr_scanned - nr_scanned, sc)) { + /* Throttle direct reclaimers if congested */ + if (!current_is_kswapd()) + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10); + goto restart; + } throttle_vm_writeout(sc->gfp_mask); } -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 15:08 ` Mel Gorman @ 2011-04-28 16:01 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:01 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote: > > > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > > > > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > > > > One more data point to add, I've been looking at an identical issue when > > > > > copying large amounts of data. I bisected this - and the lockups occur > > > > > with commit > > > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > > > issue. With this commit, my file copy test locks up after ~8-10 > > > > > iterations, before this commit I can copy > 100 times and don't see the > > > > > lockup. > > > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > > > > of yours causes kswapd on non-preempt kernels spin for a *long* time... > > > > > > > > > > I'm still thinking about the traces which do not point the finger > > > directly at compaction per-se but it's possible that the change means > > > kswapd is not reclaiming like it should be. > > > > > > To test this theory, does applying > > > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce > > > CPU usage] help? > > > > I can answer definitively no to this. The upstream kernel I reproduced > > this on has that patch included. > > > > So it is. > > Another consequence of this patch is that when high order allocations > are in progress (is the test case fork heavy in any way for > example? It's a simple huge untar, so it shouldn't fork. > alternatively, it might be something in the storage stack > that requires high-order allocs) I've tried switching from CFQ to deadline with no apparent changes > we are no longer necessarily going > to sleep because of should_reclaim_continue() check. This could > explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is > set (does unsetting CONFIG_COMPACTION help). If the bug only triggers > for CONFIG_COMPACTION, does the following *untested* patch help any? Turning off COMPACTION and HUGEPAGES doesn't help ... kswapd still goes to 99% on the PREEMPT kernel, so it doesn't seem to be related James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 16:01 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:01 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote: > > > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote: > > > > On Thu 28-04-11 12:36:30, Colin Ian King wrote: > > > > > One more data point to add, I've been looking at an identical issue when > > > > > copying large amounts of data. I bisected this - and the lockups occur > > > > > with commit > > > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the > > > > > issue. With this commit, my file copy test locks up after ~8-10 > > > > > iterations, before this commit I can copy > 100 times and don't see the > > > > > lockup. > > > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit > > > > of yours causes kswapd on non-preempt kernels spin for a *long* time... > > > > > > > > > > I'm still thinking about the traces which do not point the finger > > > directly at compaction per-se but it's possible that the change means > > > kswapd is not reclaiming like it should be. > > > > > > To test this theory, does applying > > > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce > > > CPU usage] help? > > > > I can answer definitively no to this. The upstream kernel I reproduced > > this on has that patch included. > > > > So it is. > > Another consequence of this patch is that when high order allocations > are in progress (is the test case fork heavy in any way for > example? It's a simple huge untar, so it shouldn't fork. > alternatively, it might be something in the storage stack > that requires high-order allocs) I've tried switching from CFQ to deadline with no apparent changes > we are no longer necessarily going > to sleep because of should_reclaim_continue() check. This could > explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is > set (does unsetting CONFIG_COMPACTION help). If the bug only triggers > for CONFIG_COMPACTION, does the following *untested* patch help any? Turning off COMPACTION and HUGEPAGES doesn't help ... kswapd still goes to 99% on the PREEMPT kernel, so it doesn't seem to be related James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 16:01 ` James Bottomley @ 2011-04-28 16:50 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:50 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman This is the output of perf record -g -a -f sleep 5 (hopefully the list won't choke) James --- # Events: 1K cycles # # Overhead Command Shared Object Symbol # ........ ........... ................. ....................................... # 9.03% tar [kernel.kallsyms] [k] copy_user_generic_string | --- copy_user_generic_string | |--60.55%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --39.45%-- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 3.45% swapper [kernel.kallsyms] [k] intel_idle | --- intel_idle cpuidle_idle_call cpu_idle | |--76.47%-- rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --23.53%-- start_secondary 3.39% tar [kernel.kallsyms] [k] zero_user_segments | --- zero_user_segments __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.78% tar [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--16.72%-- __percpu_counter_add | | | |--59.73%-- __prop_inc_single | | task_dirty_inc | | account_page_dirtied | | __set_page_dirty | | mark_buffer_dirty | | __block_commit_write | | block_write_end | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --40.27%-- ext4_claim_free_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--14.83%-- _raw_spin_lock_irq | | | |--63.66%-- add_to_page_cache_locked | | add_to_page_cache_lru | | | | | |--51.51%-- mpage_readpages | | | ext4_readpages | | | __do_page_cache_readahead | | | ra_submit | | | ondemand_readahead | | | page_cache_async_readahead | | | generic_file_aio_read | | | do_sync_read | | | vfs_read | | | sys_read | | | system_call_fastpath | | | __GI___libc_read | | | | | --48.49%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --36.34%-- blk_throtl_bio | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--13.85%-- bit_spin_lock | jbd2_journal_add_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.19%-- __lru_cache_add | add_to_page_cache_lru | | | |--70.86%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --29.14%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--8.30%-- bit_spin_lock | lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--58.88%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --41.12%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.17%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.90%-- __srcu_read_lock | fsnotify | fsnotify_modify | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.61%-- __srcu_read_unlock | fsnotify | fsnotify_modify | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.14%-- _raw_read_lock_irqsave | dm_get_live_table | dm_merge_bvec | __bio_add_page.part.2 | bio_add_page | do_mpage_readpage | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.81%-- _raw_read_lock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.77%-- balance_dirty_pages_ratelimited_nr | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.55%-- __add_bdi_stat | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --2.18%-- _raw_spin_lock ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.70% tar [kernel.kallsyms] [k] __list_del_entry | --- __list_del_entry | |--72.43%-- list_del | | | |--76.01%-- __rmqueue | | get_page_from_freelist | | __alloc_pages_nodemask | | alloc_pages_current | | | | | |--92.23%-- __page_cache_alloc | | | | | | | |--60.54%-- __do_page_cache_readahead | | | | ra_submit | | | | ondemand_readahead | | | | page_cache_async_readahead | | | | generic_file_aio_read | | | | do_sync_read | | | | vfs_read | | | | sys_read | | | | system_call_fastpath | | | | __GI___libc_read | | | | | | | --39.46%-- grab_cache_page_write_begin | | | ext4_da_write_begin | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --7.77%-- alloc_slab_page | | new_slab | | __slab_alloc | | kmem_cache_alloc | | alloc_buffer_head | | alloc_page_buffers | | create_empty_buffers | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--18.55%-- get_page_from_freelist | | __alloc_pages_nodemask | | alloc_pages_current | | __page_cache_alloc | | | | | |--60.32%-- __do_page_cache_readahead | | | ra_submit | | | ondemand_readahead | | | page_cache_async_readahead | | | generic_file_aio_read | | | do_sync_read | | | vfs_read | | | sys_read | | | system_call_fastpath | | | __GI___libc_read | | | | | --39.68%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --5.44%-- activate_page | mark_page_accessed | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.37%-- __rmqueue | get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--63.38%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --36.62%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --10.20%-- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--65.70%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --34.30%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.53% tar tar [.] 0x75ff | --- 0x407546 | --- 0x42a9c2 | --- 0x40753f | --- 0x40e790 | --- 0x407586 | --- 0x40fe1a | --- 0x424b50 | --- 0x412972 | --- 0x407501 | --- 0x433336 | --- 0x4075ff | --- 0x403b40 | --- 0x4055c7 | --- 0x4332d9 | --- 0x405efa | --- 0x42a9fe | --- 0x405a43 | --- 0x4332d6 1.44% tar [kernel.kallsyms] [k] __ext4_get_inode_loc | --- __ext4_get_inode_loc ext4_get_inode_loc | |--62.07%-- ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--94.67%-- generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --5.33%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.93%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.43% tar [kernel.kallsyms] [k] kmem_cache_alloc | --- kmem_cache_alloc | |--66.77%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--22.22%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | | | |--65.27%-- ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --34.73%-- ext4_dirty_inode | __mark_inode_dirty | file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.29%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --4.72%-- scsi_pool_alloc_command scsi_host_alloc_command __scsi_get_command scsi_get_command scsi_get_cmd_from_req scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 1.25% tar [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--24.47%-- _raw_spin_unlock_irqrestore | | | |--30.14%-- scsi_get_command | | scsi_get_cmd_from_req | | scsi_setup_fs_cmnd | | sd_prep_fn | | blk_peek_request | | scsi_request_fn | | __blk_run_queue | | queue_unplugged | | blk_flush_plug_list | | io_schedule | | sleep_on_page_killable | | __wait_on_bit_lock | | __lock_page_killable | | lock_page_killable | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | |--28.74%-- prepare_to_wait_exclusive | | __wait_on_bit_lock | | __lock_page_killable | | lock_page_killable | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | |--23.48%-- __wake_up | | jbd2_journal_stop | | __ext4_journal_stop | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --17.65%-- blkiocg_update_dispatch_stats | throtl_charge_bio | blk_throtl_bio | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--12.68%-- radix_tree_preload_end | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.51%-- pagefault_enable | iov_iter_copy_from_user_atomic | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.90%-- _raw_spin_unlock_irq | | | |--52.31%-- add_to_page_cache_locked | | add_to_page_cache_lru | | mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --47.69%-- __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.37%-- mem_cgroup_charge_statistics | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--9.99%-- _raw_spin_unlock | | | |--60.73%-- inode_add_rsv_space | | __dquot_alloc_space | | ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --39.27%-- ext4_ext_map_blocks | ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.89%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.86%-- balance_dirty_pages_ratelimited_nr | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.52%-- add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --3.81%-- avc_has_perm_noaudit task_has_capability selinux_capable security_capable ns_capable generic_permission inode_permission may_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 1.24% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock | |--94.43%-- lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--56.27%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --43.73%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --5.57%-- __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.13% swapper [kernel.kallsyms] [k] poll_idle | --- poll_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 1.00% tar [kernel.kallsyms] [k] ext4_da_get_block_prep | --- ext4_da_get_block_prep | |--82.10%-- __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --17.90%-- ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.95% tar [kernel.kallsyms] [k] __find_get_block | --- __find_get_block | |--85.78%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | | | |--84.94%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --15.06%-- ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.86%-- ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --6.36%-- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.92% tar [kernel.kallsyms] [k] radix_tree_lookup_element | --- radix_tree_lookup_element | |--79.19%-- radix_tree_lookup_slot | find_get_page | | | |--39.67%-- __find_get_block_slow | | unmap_underlying_metadata | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--37.78%-- generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --22.55%-- find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--13.79%-- radix_tree_lookup | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --7.02%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.86% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty | --- ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.84% tar [kernel.kallsyms] [k] __alloc_pages_nodemask | --- __alloc_pages_nodemask | |--88.77%-- alloc_pages_current | __page_cache_alloc | | | |--53.12%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --46.88%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --11.23%-- __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.80% tar [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip | |--27.09%-- add_preempt_count | | | |--44.58%-- jbd_lock_bh_state | | do_get_write_access | | jbd2_journal_get_write_access | | __ext4_journal_get_write_access | | ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--32.43%-- bit_spin_lock | | lock_page_cgroup | | __mem_cgroup_commit_charge.constprop.28 | | __mem_cgroup_commit_charge_lrucare | | mem_cgroup_cache_charge | | add_to_page_cache_locked | | add_to_page_cache_lru | | grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --22.99%-- _raw_spin_lock_irqsave | __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--26.54%-- sub_preempt_count | | | |--43.60%-- __percpu_counter_add | | ext4_claim_free_blocks | | ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--29.86%-- bit_spin_unlock | | unlock_page_cgroup | | __mem_cgroup_commit_charge.constprop.28 | | __mem_cgroup_commit_charge_lrucare | | mem_cgroup_cache_charge | | add_to_page_cache_locked | | add_to_page_cache_lru | | mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --26.53%-- __lru_cache_add | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--13.83%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--9.64%-- jbd_lock_bh_state | jbd2_journal_dirty_metadata | __ext4_handle_dirty_metadata | ext4_mark_iloc_dirty | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--8.02%-- bit_spin_unlock | jbd2_journal_add_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.60%-- _raw_spin_lock | inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.27%-- balance_dirty_pages_ratelimited_nr generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.74% tar [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock | |--55.40%-- _raw_spin_lock_irq | | | |--86.22%-- __set_page_dirty | | mark_buffer_dirty | | __block_commit_write | | block_write_end | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --13.78%-- blk_throtl_bio | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --44.60%-- _raw_spin_lock | |--76.80%-- inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --23.20%-- ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.71% tar [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero put_page | |--49.42%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--26.92%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --23.66%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.69% tar [kernel.kallsyms] [k] get_page_from_freelist | --- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--63.44%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --36.56%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.67% tar [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit | |--90.71%-- unlock_page | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --9.29%-- wake_up_bit unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.65% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed | |--92.20%-- alloc_pages_current | __page_cache_alloc | | | |--54.97%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --45.03%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.80%-- __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.65% tar [kernel.kallsyms] [k] __rmqueue | --- __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--70.26%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --29.74%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.62% tar [kernel.kallsyms] [k] start_this_handle | --- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--71.12%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --28.88%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.58% tar [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock | |--35.45%-- __prop_inc_single | task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--14.77%-- find_get_page | find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--11.81%-- __find_get_block_slow | __find_get_block | ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | |--11.28%-- avc_has_perm_noaudit | avc_has_perm | inode_has_perm | selinux_inode_permission | security_inode_permission | inode_permission | may_delete | vfs_unlink | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | |--10.21%-- __mem_cgroup_try_charge.constprop.27 | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--8.34%-- generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --8.15%-- dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.58% tar [kernel.kallsyms] [k] __mark_inode_dirty | --- __mark_inode_dirty | |--54.06%-- __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--33.43%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.51%-- ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.55% tar [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock | |--55.58%-- ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.42%-- create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.53% tar [kernel.kallsyms] [k] next_zones_zonelist | --- next_zones_zonelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--55.47%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --44.53%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.52% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock | |--55.87%-- jbd2_journal_put_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | | | |--71.83%-- ext4_dirty_inode | | __mark_inode_dirty | | | | | |--57.59%-- generic_write_end | | | ext4_da_write_end | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --42.41%-- file_update_time | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --28.17%-- add_dirent_to_buf | ext4_add_entry | ext4_add_nondir | ext4_create | vfs_create | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --44.13%-- jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.51% tar [kernel.kallsyms] [k] jbd_unlock_bh_state | --- jbd_unlock_bh_state | |--56.56%-- do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | | | |--85.42%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | | | | |--77.84%-- generic_write_end | | | ext4_da_write_end | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --22.16%-- file_update_time | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --14.58%-- ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --43.44%-- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata | |--53.02%-- ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --46.98%-- ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.50% tar [kernel.kallsyms] [k] __block_write_begin | --- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.49% tar [kernel.kallsyms] [k] __might_sleep | --- __might_sleep | |--30.54%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.09%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--14.20%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.60%-- mutex_lock | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.58%-- kmem_cache_alloc | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--9.67%-- lock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.31%-- ext4_map_blocks ext4_getblk ext4_bread dx_probe ext4_add_entry ext4_add_nondir ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.49% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head | --- jbd2_journal_add_journal_head | |--69.06%-- __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--77.62%-- generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --22.38%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --30.94%-- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.48% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28 | --- __mem_cgroup_commit_charge.constprop.28 | |--83.14%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--56.52%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --43.48%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --16.86%-- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.46% tar [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock | |--30.47%-- avc_has_perm_noaudit | | | |--54.86%-- task_has_capability | | selinux_capable | | security_capable | | ns_capable | | exec_permission | | link_path_walk | | path_lookupat | | do_path_lookup | | user_path_parent | | do_unlinkat | | sys_unlinkat | | system_call_fastpath | | unlinkat | | | --45.14%-- avc_has_perm | may_create | selinux_inode_create | security_inode_create | vfs_create | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | |--20.91%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--18.25%-- find_get_page | find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.06%-- __account_system_time | irqtime_account_process_tick | account_process_tick | update_process_times | tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | rw_verify_area | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --13.31%-- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.44% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke | --- jbd2_journal_cancel_revoke do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.44% tar [kernel.kallsyms] [k] mark_page_accessed | --- mark_page_accessed | |--56.95%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--15.53%-- do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--14.54%-- __find_get_block | __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.98%-- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.44% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed | |--55.41%-- alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.59%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--68.42%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --31.58%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.43% tar [kernel.kallsyms] [k] in_lock_functions | --- in_lock_functions | |--76.43%-- get_parent_ip | | | |--80.00%-- add_preempt_count | | | | | |--39.87%-- balance_dirty_pages_ratelimited_nr | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | |--30.07%-- radix_tree_preload | | | add_to_page_cache_locked | | | add_to_page_cache_lru | | | grab_cache_page_write_begin | | | ext4_da_write_begin | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --30.07%-- _raw_read_lock | | start_this_handle | | jbd2__journal_start | | jbd2_journal_start | | ext4_journal_start_sb | | ext4_dirty_inode | | __mark_inode_dirty | | ext4_setattr | | notify_change | | chown_common | | sys_fchown | | system_call_fastpath | | __GI___fchown | | | --20.00%-- sub_preempt_count | avc_has_perm_noaudit | avc_has_perm | inode_has_perm | dentry_has_perm | selinux_inode_setattr | security_inode_setattr | notify_change | chown_common | sys_fchown | system_call_fastpath | __GI___fchown | |--12.38%-- add_preempt_count | jbd_lock_bh_state | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --11.19%-- sub_preempt_count _raw_spin_unlock_irqrestore __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.43% swapper [kernel.kallsyms] [k] ext4_end_bio | --- ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.42% tar [kernel.kallsyms] [k] ext4_get_group_desc | --- ext4_get_group_desc | |--88.05%-- __ext4_get_inode_loc | ext4_get_inode_loc | | | |--64.96%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --35.04%-- ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --11.95%-- ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.42% tar [kernel.kallsyms] [k] __percpu_counter_add | --- __percpu_counter_add | |--39.75%-- ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--25.32%-- ext4_claim_free_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--19.28%-- ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --15.64%-- __add_bdi_stat account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.42% tar [kernel.kallsyms] [k] do_get_write_access | --- do_get_write_access | |--80.57%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --19.43%-- __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.42% tar [kernel.kallsyms] [k] generic_file_buffered_write | --- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.41% tar [kernel.kallsyms] [k] jbd2_journal_stop | --- jbd2_journal_stop __ext4_journal_stop | |--79.71%-- ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --20.29%-- ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.41% tar [kernel.kallsyms] [k] jbd2__journal_start | --- jbd2__journal_start | |--82.99%-- jbd2_journal_start | ext4_journal_start_sb | | | |--83.87%-- ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --16.13%-- ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --17.01%-- ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.40% tar [kernel.kallsyms] [k] do_mpage_readpage | --- do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.39% tar [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep | |--55.35%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--61.18%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --38.82%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--28.25%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --16.40%-- __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.38% tar [kernel.kallsyms] [k] alloc_pages_current | --- alloc_pages_current | |--82.52%-- __page_cache_alloc | | | |--74.42%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --25.58%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --17.48%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.37% tar [kernel.kallsyms] [k] __nr_to_section | --- __nr_to_section lookup_page_cgroup | |--70.03%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.38%-- mem_cgroup_get_reclaim_stat_from_page | update_page_reclaim_stat | ____pagevec_lru_add_fn | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.60%-- mem_cgroup_del_lru_list activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.37% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore | |--52.73%-- _raw_spin_unlock_irqrestore | | | |--51.73%-- __wake_up | | jbd2_journal_stop | | __ext4_journal_stop | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --48.27%-- pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.27%-- __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.36% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27 | --- __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--67.66%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --32.34%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.36% swapper [kernel.kallsyms] [k] unlock_page | --- unlock_page | |--85.03%-- mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --14.97%-- bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.35% tar [kernel.kallsyms] [k] ext4_xattr_get | --- ext4_xattr_get | |--68.95%-- ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --31.05%-- generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.35% perf [kernel.kallsyms] [k] seq_puts | --- seq_puts render_sigset_t proc_pid_status proc_single_show seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.35% tar [kernel.kallsyms] [k] grab_cache_page_write_begin | --- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.34% tar [kernel.kallsyms] [k] alloc_page_buffers | --- alloc_page_buffers | |--72.20%-- create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --27.80%-- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.34% tar [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--58.75%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--70.45%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --29.55%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.25%-- __lru_cache_add add_to_page_cache_lru | |--61.47%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --38.53%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.34% perf [kernel.kallsyms] [k] clear_page_c | --- clear_page_c __alloc_pages_nodemask perf_mmap_alloc_page perf_mmap mmap_region do_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap 0x418523 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.34% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic | --- iov_iter_copy_from_user_atomic generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.33% tar [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_page | |--84.05%-- find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --15.95%-- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.33% top [kernel.kallsyms] [k] follow_managed | --- follow_managed walk_component link_path_walk path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.33% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages | --- ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.32% swapper [kernel.kallsyms] [k] do_timer | --- do_timer tick_do_update_jiffies64 | |--61.57%-- tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --38.43%-- tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.32% tar [kernel.kallsyms] [k] __generic_file_aio_write | --- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.32% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list | --- mem_cgroup_add_lru_list add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--87.33%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.67%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.32% tar [kernel.kallsyms] [k] ext4_da_write_end | --- ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.32% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page | --- ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.32% swapper [kernel.kallsyms] [k] tick_nohz_stop_sched_tick | --- tick_nohz_stop_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.31% tar [kernel.kallsyms] [k] down_read | --- down_read | |--78.35%-- ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --21.65%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.31% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--83.21%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --16.79%-- force_qs_rnp force_quiescent_state __rcu_process_callbacks rcu_process_callbacks __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.30% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--59.37%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --40.63%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.30% perf [kernel.kallsyms] [k] file_update_time | --- file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0.30% tar [kernel.kallsyms] [k] mark_buffer_dirty | --- mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.29% swapper [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--54.58%-- _raw_spin_lock_irqsave | | | |--57.58%-- scsi_device_unbusy | | scsi_finish_command | | scsi_softirq_done | | blk_done_softirq | | __do_softirq | | call_softirq | | do_softirq | | irq_exit | | do_IRQ | | common_interrupt | | cpuidle_idle_call | | cpu_idle | | rest_init | | start_kernel | | x86_64_start_reservations | | x86_64_start_kernel | | | --42.42%-- test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--20.90%-- __percpu_counter_add | __add_bdi_stat | test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--15.95%-- irq_enter | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --8.56%-- scsi_device_unbusy scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.29% tar [kernel.kallsyms] [k] mnt_want_write_file | --- mnt_want_write_file | |--76.04%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --23.96%-- system_call_fastpath __GI___fchown 0.27% tar [kernel.kallsyms] [k] find_busiest_group | --- find_busiest_group load_balance schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.26% perf [kernel.kallsyms] [k] perf_event_mmap_output | --- perf_event_mmap_output perf_event_mmap_ctx perf_event_mmap mmap_region do_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap 0.26% swapper [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free | |--59.24%-- mempool_free | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --40.76%-- scsi_pool_free_command __scsi_put_command scsi_put_command scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.26% perf [kernel.kallsyms] [k] fsnotify | --- fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __write_nocancel 0x4293b8 0x429c0a 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.26% perf [kernel.kallsyms] [k] put_bh | --- put_bh ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4293b8 0x429c0a 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.26% perf [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock prepend_path path_with_deleted d_path seq_path show_map_vma show_map seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.26% tar [kernel.kallsyms] [k] get_page | --- get_page __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.26% tar [kernel.kallsyms] [k] selinux_file_permission | --- selinux_file_permission security_file_permission rw_verify_area vfs_write sys_write system_call_fastpath __GI___libc_write 0.26% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag need_resched should_resched _cond_resched | |--64.24%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --35.76%-- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.26% perf [kernel.kallsyms] [k] number | --- number vsnprintf seq_printf show_map_vma show_map seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.25% tar libc-2.13.90.so [.] __GI___libc_write | --- __GI___libc_write 0.25% tar [kernel.kallsyms] [k] test_and_set_bit | --- test_and_set_bit mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.25% perf [kernel.kallsyms] [k] vsnprintf | --- vsnprintf seq_printf show_map_vma show_map seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.25% tar [kernel.kallsyms] [k] run_timer_softirq | --- run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.25% perf [kernel.kallsyms] [k] setup_object | --- setup_object new_slab __slab_alloc kmem_cache_alloc d_alloc proc_fill_cache proc_task_readdir vfs_readdir sys_getdents system_call_fastpath __getdents64 0x429beb 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.24% swapper [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--40.64%-- __bit_spin_unlock.constprop.24 | __slab_free | kmem_cache_free | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--33.53%-- _raw_spin_unlock_irqrestore | rcu_report_qs_rnp | __rcu_process_callbacks | rcu_process_callbacks | __do_softirq | call_softirq | do_softirq | irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --25.82%-- __slab_free kmem_cache_free mempool_free_slab mempool_free bio_free bio_fs_destructor bio_put ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.24% tar [kernel.kallsyms] [k] add_to_page_cache_locked | --- add_to_page_cache_locked | |--64.11%-- add_to_page_cache_lru | | | |--63.51%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --36.49%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --35.89%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.24% tar [kernel.kallsyms] [k] file_remove_suid | --- file_remove_suid | |--81.19%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --18.81%-- generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.24% tar [kernel.kallsyms] [k] clear_bit_unlock | --- clear_bit_unlock | |--71.97%-- unlock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--58.50%-- file_update_time | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --41.50%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --28.03%-- do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.23% tar [kernel.kallsyms] [k] sys_write | --- sys_write system_call_fastpath __GI___libc_write 0.23% swapper [kernel.kallsyms] [k] SetPageUptodate | --- SetPageUptodate mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.23% tar [kernel.kallsyms] [k] __bio_add_page.part.2 | --- __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.23% swapper [kernel.kallsyms] [k] rcu_irq_enter | --- rcu_irq_enter irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.23% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq | --- _raw_spin_unlock_irq | |--45.40%-- generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--29.98%-- mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --24.62%-- add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.23% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock | |--51.02%-- jbd2_journal_put_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.98%-- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% swapper [kernel.kallsyms] [k] nr_iowait_cpu | --- nr_iowait_cpu tick_nohz_stop_idle tick_check_idle irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.22% flush-253:2 [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue | |--51.36%-- unlock_page | ext4_bio_write_page | mpage_da_submit_io | mpage_da_map_and_submit | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --48.64%-- ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.22% tar [kernel.kallsyms] [k] __jbd2_log_space_left | --- __jbd2_log_space_left start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--50.77%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.23%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% tar [kernel.kallsyms] [k] mb_test_bit | --- mb_test_bit mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.22% tar [kernel.kallsyms] [k] __blk_recalc_rq_segments | --- __blk_recalc_rq_segments blk_recount_segments bio_phys_segments blk_rq_bio_prep init_request_from_bio __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.22% tar [kernel.kallsyms] [k] generic_write_end | --- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata | --- jbd2_journal_dirty_metadata | |--62.74%-- __ext4_handle_dirty_metadata | ext4_mark_iloc_dirty | ext4_mark_inode_dirty | | | |--59.79%-- ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --40.21%-- ext4_unlink | vfs_unlink | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --37.26%-- ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% tar [kernel.kallsyms] [k] __mem_cgroup_threshold | --- __mem_cgroup_threshold memcg_check_events __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.21% tar [kernel.kallsyms] [k] kfree | --- kfree start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--79.14%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --20.86%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.21% tar [kernel.kallsyms] [k] ext4_get_inode_flags | --- ext4_get_inode_flags ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--63.71%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --36.29%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.21% flush-253:2 [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero release_pages __pagevec_release pagevec_release | |--57.93%-- ext4_num_dirty_pages | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --42.07%-- write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.21% flush-253:2 [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock __find_get_block_slow unmap_underlying_metadata mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.21% tar [kernel.kallsyms] [k] avtab_search | --- avtab_search security_compute_sid security_transition_sid may_create selinux_inode_create security_inode_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.21% tar [kernel.kallsyms] [k] search_dirblock | --- search_dirblock ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.21% tar [kernel.kallsyms] [k] __sg_alloc_table | --- __sg_alloc_table scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.21% tar [kernel.kallsyms] [k] ext4_journal_start_sb | --- ext4_journal_start_sb ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.21% top libc-2.13.90.so [.] __GI_____strtoll_l_internal | --- __GI_____strtoll_l_internal 0.20% swapper [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock | |--57.86%-- handle_edge_irq | handle_irq | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --42.14%-- tick_do_update_jiffies64 tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.20% tar [kernel.kallsyms] [k] generic_write_sync | --- generic_write_sync generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.20% tar [kernel.kallsyms] [k] put_page | --- put_page | |--64.65%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --35.35%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.20% flush-253:2 [kernel.kallsyms] [k] tag_get | --- tag_get | |--58.46%-- __lookup_tag | radix_tree_gang_lookup_tag_slot | find_get_pages_tag | pagevec_lookup_tag | ext4_num_dirty_pages | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --41.54%-- test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.20% tar [kernel.kallsyms] [k] dm_merge_bvec | --- dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.19% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata | --- __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.19% tar [kernel.kallsyms] [k] __block_commit_write | --- __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.19% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore | --- _raw_spin_unlock_irqrestore | |--54.81%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --45.19%-- __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.19% tar [kernel.kallsyms] [k] ext4_writepage_trans_blocks | --- ext4_writepage_trans_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.19% tar [kernel.kallsyms] [k] jbd_lock_bh_state | --- jbd_lock_bh_state | |--62.13%-- do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.87%-- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.19% tar [kernel.kallsyms] [k] __list_add | --- __list_add | |--70.74%-- __make_request | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --29.26%-- expand __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.19% tar [kernel.kallsyms] [k] fsnotify_modify | --- fsnotify_modify | |--74.13%-- vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --25.87%-- sys_write system_call_fastpath __GI___libc_write 0.19% flush-253:2 [kernel.kallsyms] [k] lock_page | --- lock_page write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.19% flush-253:2 [kernel.kallsyms] [k] rb_next | --- rb_next set_next_entity pick_next_task_fair pick_next_task schedule preempt_schedule_irq retint_kernel ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.19% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount | --- ext4_ext_calc_metadata_amount ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.18% swapper [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit | |--61.24%-- unlock_page | mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --38.76%-- end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.18% swapper [kernel.kallsyms] [k] select_nohz_load_balancer | --- select_nohz_load_balancer | |--75.59%-- tick_nohz_restart_sched_tick | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --24.41%-- tick_nohz_stop_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.18% swapper [kernel.kallsyms] [k] menu_select | --- menu_select cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.18% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_pages pagevec_lookup mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.18% tar [kernel.kallsyms] [k] generic_file_aio_read | --- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.18% tar [kernel.kallsyms] [k] __prop_inc_single | --- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.18% swapper [kernel.kallsyms] [k] page_index | --- page_index test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.17% tar [kernel.kallsyms] [k] arch_read_lock | --- arch_read_lock _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] mutex_unlock | --- mutex_unlock | |--73.48%-- generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --26.52%-- do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.17% tar [kernel.kallsyms] [k] blk_rq_map_sg | --- blk_rq_map_sg scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] bit_waitqueue | --- bit_waitqueue | |--51.20%-- wake_up_bit | unlock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.80%-- unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] SetPageLRU | --- SetPageLRU ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--71.47%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --28.53%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 | |--64.90%-- _raw_spin_unlock_irq | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --35.10%-- inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] lookup_page_cgroup | --- lookup_page_cgroup | |--58.52%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.48%-- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] generic_make_request | --- generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] lock_page_cgroup | --- lock_page_cgroup __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% swapper [kernel.kallsyms] [k] mempool_free | --- mempool_free clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.17% tar [kernel.kallsyms] [k] put_prev_task_fair | --- put_prev_task_fair io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] perf_event_task_tick | --- perf_event_task_tick scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--51.07%-- scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.93%-- _ext4_get_block ext4_get_block do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] arch_read_unlock | --- arch_read_unlock | |--66.07%-- _raw_read_unlock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --33.93%-- _raw_read_unlock_irqrestore dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.16% swapper [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--38.89%-- __cycles_2_ns | native_sched_clock | sched_clock | sched_clock_cpu | account_system_vtime | irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--32.56%-- tick_check_idle | irq_enter | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--14.27%-- account_idle_time | irqtime_account_process_tick | account_idle_ticks | tick_nohz_restart_sched_tick | cpu_idle | start_secondary | --14.27%-- irqtime_account_process_tick account_idle_ticks tick_nohz_restart_sched_tick cpu_idle start_secondary 0.16% tar [kernel.kallsyms] [k] fget_light | --- fget_light | |--51.07%-- sys_read | system_call_fastpath | __GI___libc_read | --48.93%-- sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] SetPageUptodate | --- SetPageUptodate __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] __ext4_journal_stop | --- __ext4_journal_stop | |--50.22%-- ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.78%-- ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] ext4_clear_inode_state | --- ext4_clear_inode_state ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--51.89%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.11%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge | --- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--65.55%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --34.45%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] ext4_map_blocks | --- ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] ata_dev_enabled | --- ata_dev_enabled ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.16% tar [kernel.kallsyms] [k] new_slab | --- new_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% swapper [kernel.kallsyms] [k] hrtimer_cancel | --- hrtimer_cancel tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.15% tar [kernel.kallsyms] [k] lock_buffer | --- lock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock unlock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--72.67%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --27.33%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] up_read | --- up_read | |--62.56%-- ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.44%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] preempt_schedule | --- preempt_schedule try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] fsnotify_access | --- fsnotify_access sys_read system_call_fastpath __GI___libc_read 0.15% swapper [kernel.kallsyms] [k] native_read_tsc | --- native_read_tsc paravirt_read_tsc native_sched_clock sched_clock sched_clock_cpu account_system_vtime irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.14% tar [kernel.kallsyms] [k] dequeue_task_fair | --- dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.14% tar [kernel.kallsyms] [k] ata_scsi_queuecmd | --- ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.14% tar [kernel.kallsyms] [k] file_read_actor | --- file_read_actor generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.14% tar [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock | |--52.50%-- ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.50%-- ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.14% top [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count _raw_spin_lock nameidata_drop_rcu link_path_walk path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.14% tar [kernel.kallsyms] [k] generic_file_aio_write | --- generic_file_aio_write | |--52.11%-- do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.89%-- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.14% tar [kernel.kallsyms] [k] block_write_end | --- block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.14% tar [kernel.kallsyms] [k] hrtimer_interrupt | --- hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--52.67%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.33%-- string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.14% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count __percpu_counter_add __add_bdi_stat clear_page_dirty_for_io mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.14% swapper [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock | |--50.04%-- __prop_inc_percpu_max | test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --49.96%-- __atomic_notifier_call_chain atomic_notifier_call_chain exit_idle do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.13% tar [kernel.kallsyms] [k] ktime_get_ts | --- ktime_get_ts __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.13% tar [kernel.kallsyms] [k] page_zone | --- page_zone pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] __read_lock_failed | --- __read_lock_failed _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] prop_norm_single | --- prop_norm_single __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] put_bh | --- put_bh __brelse brelse ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] radix_tree_insert | --- radix_tree_insert add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] policy_zonelist | --- policy_zonelist alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.13% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% swapper [kernel.kallsyms] [k] irq_entries_start | --- irq_entries_start cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.13% top libproc-3.2.8.so [.] 0x44c2 | --- 0x37d94044c2 0.13% flush-253:2 [kernel.kallsyms] [k] attempt_back_merge | --- attempt_back_merge generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.13% tar [kernel.kallsyms] [k] virt_to_head_page | --- virt_to_head_page kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] mnt_clone_write | --- mnt_clone_write mnt_want_write_file file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] rb_erase | --- rb_erase deadline_remove_request deadline_move_request deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.13% tar [kernel.kallsyms] [k] __set_page_dirty | --- __set_page_dirty | |--51.11%-- __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.89%-- mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] do_sync_write | --- do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable _raw_spin_lock_irq | |--55.62%-- add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.38%-- __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] ext4_da_write_begin | --- ext4_da_write_begin __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% swapper [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore | |--69.63%-- account_system_vtime | irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --30.37%-- irqtime_account_process_tick account_idle_ticks tick_nohz_restart_sched_tick cpu_idle start_secondary 0.12% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.12% tar [kernel.kallsyms] [k] inode_reserved_space | --- inode_reserved_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% swapper [kernel.kallsyms] [k] update_cfs_shares | --- update_cfs_shares enqueue_task_fair enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.12% tar [kernel.kallsyms] [k] schedule | --- schedule | |--59.95%-- io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --40.05%-- __GI___libc_read 0.12% tar [kernel.kallsyms] [k] security_sid_to_context_force | --- security_sid_to_context_force security_inode_init_security ext4_init_security ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.12% tar [kernel.kallsyms] [k] radix_tree_preload | --- radix_tree_preload add_to_page_cache_locked add_to_page_cache_lru | |--54.07%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --45.93%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags _local_bh_enable irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.12% flush-253:2 [kernel.kallsyms] [k] release_pages | --- release_pages __pagevec_release pagevec_release write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.12% flush-253:2 [kernel.kallsyms] [k] page_mkclean | --- page_mkclean mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.12% tar [kernel.kallsyms] [k] set_bh_page | --- set_bh_page create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] account_entity_dequeue | --- account_entity_dequeue dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% tar [kernel.kallsyms] [k] ata_qc_issue | --- ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% tar libc-2.13.90.so [.] __cfree | --- __cfree 0.12% tar [kernel.kallsyms] [k] radix_tree_lookup_slot | --- radix_tree_lookup_slot | |--62.63%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --37.37%-- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% top libc-2.13.90.so [.] _IO_vfscanf_internal | --- _IO_vfscanf_internal _IO_vsscanf 0x7fffef292b20 0.12% tar [kernel.kallsyms] [k] ext4_inode_table | --- ext4_inode_table __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] xattr_resolve_name | --- xattr_resolve_name generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] add_to_page_cache_lru | --- add_to_page_cache_lru | |--67.10%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --32.90%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% migration/0 [kernel.kallsyms] [k] resched_task | --- resched_task check_preempt_curr __migrate_task migration_cpu_stop cpu_stopper_thread kthread kernel_thread_helper 0.12% tar [kernel.kallsyms] [k] __count_vm_events | --- __count_vm_events __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--50.90%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --49.10%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] system_call_fastpath | --- system_call_fastpath __GI___libc_write 0.11% flush-253:2 [kernel.kallsyms] [k] __percpu_counter_add | --- __percpu_counter_add __add_bdi_stat clear_page_dirty_for_io mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% swapper [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.11% kworker/0:2 [kernel.kallsyms] [k] worker_thread | --- worker_thread kthread kernel_thread_helper 0.11% tar [kernel.kallsyms] [k] fsnotify | --- fsnotify vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] is_handle_aborted | --- is_handle_aborted jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] blk_finish_plug | --- blk_finish_plug ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access | --- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] vfs_write | --- vfs_write system_call_fastpath __GI___libc_write 0.11% swapper [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.11% tar [kernel.kallsyms] [k] blk_flush_plug_list | --- blk_flush_plug_list blk_finish_plug generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] deadline_remove_request | --- deadline_remove_request deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] mb_find_buddy | --- mb_find_buddy mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.11% tar [kernel.kallsyms] [k] sys_read | --- sys_read | |--58.45%-- system_call_fastpath | __GI___libc_read | --41.55%-- __GI___libc_read 0.11% flush-253:2 [kernel.kallsyms] [k] arch_read_unlock | --- arch_read_unlock _raw_read_unlock_irqrestore dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% swapper [kernel.kallsyms] [k] __slab_free | --- __slab_free kmem_cache_free mempool_free_slab mempool_free scsi_sg_free __sg_free_table __scsi_release_buffers scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.11% tar [kernel.kallsyms] [k] sidtab_search_core | --- sidtab_search_core sidtab_search security_compute_sid security_transition_sid may_create selinux_inode_create security_inode_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.11% swapper [kernel.kallsyms] [k] switch_mm | --- switch_mm schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.11% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io | --- mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics | --- mem_cgroup_charge_statistics __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--56.66%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.34%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] zone_watermark_ok | --- zone_watermark_ok | |--50.80%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.20%-- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] ext4_free_blocks | --- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.11% tar [kernel.kallsyms] [k] ext4_ext_map_blocks | --- ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] add_dirent_to_buf | --- add_dirent_to_buf ext4_add_entry ext4_add_nondir ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.11% tar [kernel.kallsyms] [k] strcmp | --- strcmp symcmp hashtab_search string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.11% tar [kernel.kallsyms] [k] select_idle_sibling | --- select_idle_sibling select_task_rq_fair select_task_rq try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% flush-253:2 [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit unlock_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% tar [kernel.kallsyms] [k] radix_tree_node_alloc | --- radix_tree_node_alloc radix_tree_insert add_to_page_cache_locked add_to_page_cache_lru | |--60.14%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --39.86%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% swapper [kernel.kallsyms] [k] rcu_irq_exit | --- rcu_irq_exit | |--54.97%-- smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --45.03%-- irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.10% tar [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] check_irqs_on | --- check_irqs_on __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% flush-253:2 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count __add_bdi_stat test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% flush-253:2 [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep unmap_underlying_metadata mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] alloc_buffer_head | --- alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] jiffies_to_timeval | --- jiffies_to_timeval __account_system_time irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.10% tar [kernel.kallsyms] [k] ext4_get_reserved_space | --- ext4_get_reserved_space inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.10% tar [kernel.kallsyms] [k] blk_start_plug | --- blk_start_plug mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue __make_request generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] ata_qc_new_init | --- ata_qc_new_init | |--51.58%-- ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.42%-- __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% tar [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_unlock_irq scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] inc_zone_page_state | --- inc_zone_page_state account_page_writeback test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] system_call | --- system_call __GI___libc_write 0.10% flush-253:2 [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio ext4_io_submit ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% flush-253:2 [kernel.kallsyms] [k] mod_state | --- mod_state inc_zone_page_state account_page_writeback test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save _raw_spin_lock_irqsave __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] bio_get_nr_vecs | --- bio_get_nr_vecs mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% tar [kernel.kallsyms] [k] inode_init_once | --- inode_init_once init_once setup_object new_slab __slab_alloc kmem_cache_alloc ext4_alloc_inode alloc_inode new_inode ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--61.31%-- _raw_spin_lock | jbd2_journal_file_buffer | jbd2_journal_write_metadata_buffer | jbd2_journal_commit_transaction | kjournald2 | kthread | kernel_thread_helper | --38.69%-- jbd_trylock_bh_state journal_clean_one_cp_list __jbd2_journal_clean_checkpoint_list jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.09% swapper [kernel.kallsyms] [k] kobject_put | --- kobject_put __scsi_put_command scsi_put_command scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% flush-253:2 [kernel.kallsyms] [k] __elv_add_request | --- __elv_add_request __make_request generic_make_request submit_bio ext4_io_submit ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% swapper [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore rcu_process_gp_end __rcu_process_callbacks rcu_process_callbacks __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% tar [kernel.kallsyms] [k] round_jiffies_up | --- round_jiffies_up blk_start_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] attach_page_buffers | --- attach_page_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% tar [kernel.kallsyms] [k] hrtick_update | --- hrtick_update dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] ext4_clear_inode_flag | --- ext4_clear_inode_flag ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock_irq blk_throtl_bio generic_make_request submit_bio submit_bh jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.09% top [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock __d_lookup walk_component link_path_walk path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.09% tar [kernel.kallsyms] [k] __brelse | --- __brelse brelse ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% flush-253:2 [kernel.kallsyms] [k] page_zone | --- page_zone inc_zone_page_state account_page_writeback test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% tar [kernel.kallsyms] [k] bio_integrity_enabled | --- bio_integrity_enabled generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] dm_table_find_target | --- dm_table_find_target __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] __mod_zone_page_state | --- __mod_zone_page_state add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] ahci_fill_cmd_slot | --- ahci_fill_cmd_slot ahci_qc_prep ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% swapper [kernel.kallsyms] [k] blk_unprep_request | --- blk_unprep_request blk_finish_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% top libc-2.13.90.so [.] __GI___close | --- __GI___close 0.09% tar [kernel.kallsyms] [k] put_bh | --- put_bh ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% swapper [kernel.kallsyms] [k] finish_task_switch | --- finish_task_switch schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% flush-253:2 [kernel.kallsyms] [k] page_mapping | --- page_mapping mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% tar [kernel.kallsyms] [k] alloc_tio | --- alloc_tio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear | --- radix_tree_tag_clear test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.08% tar [kernel.kallsyms] [k] timespec_trunc | --- timespec_trunc touch_atime generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] ext4_claim_free_blocks | --- ext4_claim_free_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% tar libc-2.13.90.so [.] __GI___errno_location | --- __GI___errno_location 0.08% tar [kernel.kallsyms] [k] path_to_nameidata | --- path_to_nameidata walk_component link_path_walk path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.08% tar [kernel.kallsyms] [k] current_sid | --- current_sid selinux_file_permission security_file_permission rw_verify_area vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% top [kernel.kallsyms] [k] path_openat | --- path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.08% tar [kernel.kallsyms] [k] __do_page_cache_readahead | --- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] _cond_resched | --- _cond_resched generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% swapper [kernel.kallsyms] [k] run_timer_softirq | --- run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% swapper [kernel.kallsyms] [k] note_interrupt | --- note_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% swapper [kernel.kallsyms] [k] native_apic_mem_write | --- native_apic_mem_write apic_write ack_APIC_irq smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.08% tar [kernel.kallsyms] [k] bio_data | --- bio_data init_request_from_bio __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] string_to_context_struct | --- string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.08% tar [kernel.kallsyms] [k] mb_clear_bit | --- mb_clear_bit mb_clear_bits ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.08% tar [kernel.kallsyms] [k] ext4_mb_load_buddy | --- ext4_mb_load_buddy ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.08% top [kernel.kallsyms] [k] __fsnotify_parent | --- __fsnotify_parent fsnotify_perm security_file_permission rw_verify_area vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% sleep [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock | |--60.83%-- __do_fault | handle_pte_fault | handle_mm_fault | do_page_fault | page_fault | 0x37d9082b80 | --39.17%-- deny_write_access mmap_region do_mmap_pgoff do_mmap elf_map load_elf_binary search_binary_handler do_execve sys_execve stub_execve 0x37d90ae097 0.08% swapper [kernel.kallsyms] [k] ahci_interrupt | --- ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% tar [kernel.kallsyms] [k] __srcu_read_unlock | --- __srcu_read_unlock fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% tar [kernel.kallsyms] [k] signal_pending | --- signal_pending fatal_signal_pending __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] __read_seqcount_begin | --- __read_seqcount_begin path_init path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.08% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag mnt_clone_write mnt_want_write_file file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% tar [kernel.kallsyms] [k] jbd2_journal_start | --- jbd2_journal_start ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] __srcu_read_lock | --- __srcu_read_lock fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] handle_irq | --- handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% swapper [kernel.kallsyms] [k] __ata_scsi_queuecmd | --- __ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc drive_stat_acct __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __switch_to | --- __switch_to __GI___libc_read 0.07% tar [kernel.kallsyms] [k] ext4_file_write | --- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] wake_bit_function | --- wake_bit_function __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] _raw_spin_lock_irq | --- _raw_spin_lock_irq add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] drive_stat_acct | --- drive_stat_acct __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22 | --- test_ti_thread_flag.constprop.22 zero_user_segments __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] sd_prep_fn | --- sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% top libc-2.13.90.so [.] __strchrnul | --- __strchrnul ___vsnprintf_chk 0x39335b1b6d5b1b42 0.07% swapper [kernel.kallsyms] [k] sd_prep_fn | --- sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] try_to_wake_up | --- try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_unlock_irqrestore ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __zone_watermark_ok | --- __zone_watermark_ok zone_watermark_ok get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] __remove_hrtimer | --- __remove_hrtimer remove_hrtimer.part.4 hrtimer_try_to_cancel hrtimer_cancel tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% swapper [kernel.kallsyms] [k] scsi_device_unbusy | --- scsi_device_unbusy scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] scsi_prep_return | --- scsi_prep_return sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] prepare_to_wait_exclusive | --- prepare_to_wait_exclusive __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] set_buffer_uptodate | --- set_buffer_uptodate __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] ata_get_xlat_func | --- ata_get_xlat_func __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __find_get_block_slow | --- __find_get_block_slow __find_get_block ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.07% tar [kernel.kallsyms] [k] generic_getxattr | --- generic_getxattr security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] bit_spin_lock.constprop.22 | --- bit_spin_lock.constprop.22 __slab_free kmem_cache_free put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% swapper [kernel.kallsyms] [k] is_swiotlb_buffer | --- is_swiotlb_buffer unmap_single swiotlb_unmap_sg_attrs ata_sg_clean __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] native_read_tsc | --- native_read_tsc paravirt_read_tsc native_sched_clock sched_clock blk_rq_init get_request get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] ext4_has_free_blocks | --- ext4_has_free_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar libc-2.13.90.so [.] __GI___fxstat64 | --- __GI___fxstat64 0.07% swapper [kernel.kallsyms] [k] perf_pmu_disable | --- perf_pmu_disable scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] sidtab_context_to_sid | --- sidtab_context_to_sid security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.07% tar [kernel.kallsyms] [k] scsi_pool_alloc_command | --- scsi_pool_alloc_command scsi_host_alloc_command __scsi_get_command scsi_get_command scsi_get_cmd_from_req scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] list_del | --- list_del activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] pagefault_enable | --- pagefault_enable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] security_file_permission | --- security_file_permission rw_verify_area vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% swapper [kernel.kallsyms] [k] scsi_next_command | --- scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] __phys_addr | --- __phys_addr virt_to_head_page kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] page_cache_async_readahead | --- page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] truncate_inode_pages_range | --- truncate_inode_pages_range ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.07% tar [kernel.kallsyms] [k] mem_cgroup_del_lru_list | --- mem_cgroup_del_lru_list activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] ext4_xattr_check_names | --- ext4_xattr_check_names ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] arch_local_irq_enable | --- arch_local_irq_enable __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] get_request | --- get_request get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs | --- swiotlb_map_sg_attrs ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] dm_get_live_table | --- dm_get_live_table page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page | --- mem_cgroup_get_reclaim_stat_from_page update_page_reclaim_stat ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] check_preempt_curr | --- check_preempt_curr ttwu_post_activation try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] atomic_long_add | --- atomic_long_add zone_page_state_add __mod_zone_page_state add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] pm_qos_request | --- pm_qos_request cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] __kmalloc | --- __kmalloc kzalloc.constprop.9 ext4_ext_find_extent ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] rcu_bh_qs | --- rcu_bh_qs __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] __blk_complete_request | --- __blk_complete_request scsi_done ata_scsi_qc_complete __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] __d_lookup | --- __d_lookup walk_component link_path_walk path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.06% tar [kernel.kallsyms] [k] context_struct_to_string | --- context_struct_to_string security_sid_to_context_core security_sid_to_context_force selinux_inode_init_security security_inode_init_security ext4_init_security ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.06% swapper [kernel.kallsyms] [k] tick_nohz_restart_sched_tick | --- tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] max_sane_readahead | --- max_sane_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] rw_verify_area | --- rw_verify_area vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] ahci_qc_prep | --- ahci_qc_prep ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] linear_map | --- linear_map __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% swapper [kernel.kallsyms] [k] sched_clock | --- sched_clock account_system_vtime irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] fput_light | --- fput_light system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] iov_iter_advance | --- iov_iter_advance generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] __queue_work | --- __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt lock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% swapper [kernel.kallsyms] [k] scsi_io_completion | --- scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] activate_page | --- activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] touch_atime | --- touch_atime generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] atomic_sub | --- atomic_sub jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] __inc_zone_page_state | --- __inc_zone_page_state add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] ext4_test_inode_state | --- ext4_test_inode_state ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] ext4_xattr_find_entry | --- ext4_xattr_find_entry ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] deadline_dispatch_requests | --- deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] hrtick_update | --- hrtick_update enqueue_task_fair enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] try_to_wake_up | --- try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] pick_next_task_fair | --- pick_next_task_fair schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] pick_next_task_idle | --- pick_next_task_idle schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% swapper [kernel.kallsyms] [k] ktime_get | --- ktime_get tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr | --- balance_dirty_pages_ratelimited_nr generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] mpage_end_io | --- mpage_end_io dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] radix_tree_preload_end | --- radix_tree_preload_end add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] submit_bio | --- submit_bio ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] __ata_scsi_queuecmd | --- __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] dm_table_put | --- dm_table_put dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% sleep [kernel.kallsyms] [k] __ext4_journal_get_write_access | --- __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty touch_atime ext4_file_mmap mmap_region do_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap _nl_find_locale __GI_setlocale 0x40124c 0.05% tar [kernel.kallsyms] [k] sysret_check | --- sysret_check __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] clockevents_program_event | --- clockevents_program_event tick_dev_program_event tick_program_event __hrtimer_start_range_ns hrtimer_start_range_ns hrtimer_start_expires.constprop.1 tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] pick_next_task_rt | --- pick_next_task_rt pick_next_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] account_entity_enqueue | --- account_entity_enqueue enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] need_resched | --- need_resched _cond_resched generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] find_lock_page | --- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] memcmp | --- memcmp search_dirblock ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.05% tar [kernel.kallsyms] [k] generic_segment_checks | --- generic_segment_checks generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% tar [kernel.kallsyms] [k] update_vsyscall | --- update_vsyscall do_timer tick_do_update_jiffies64 tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_lock ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% swapper [kernel.kallsyms] [k] hrtimer_start_expires.constprop.1 | --- hrtimer_start_expires.constprop.1 cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] __page_cache_alloc | --- __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] avc_has_perm_noaudit | --- avc_has_perm_noaudit avc_has_perm inode_has_perm selinux_inode_permission security_inode_exec_permission exec_permission link_path_walk path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.05% tar [kernel.kallsyms] [k] deactivate_slab | --- deactivate_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% swapper [kernel.kallsyms] [k] bio_free | --- bio_free bio_fs_destructor bio_put mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar libc-2.13.90.so [.] __GI___libc_read | --- __GI___libc_read 0.05% tar [kernel.kallsyms] [k] selinux_capable | --- selinux_capable ns_capable capable inode_change_ok ext4_setattr notify_change chown_common sys_fchown system_call_fastpath __GI___fchown 0.05% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave | --- _raw_read_lock_irqsave dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] scsi_handle_queue_ramp_up | --- scsi_handle_queue_ramp_up scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] dm_table_any_congested | --- dm_table_any_congested dm_any_congested page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% tar [kernel.kallsyms] [k] __wake_up_common | --- __wake_up_common __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] acct_update_integrals | --- acct_update_integrals __account_system_time irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt blk_finish_plug generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] ns_to_timespec | --- ns_to_timespec intel_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] _raw_read_unlock_irqrestore | --- _raw_read_unlock_irqrestore dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% tar [kernel.kallsyms] [k] __add_bdi_stat | --- __add_bdi_stat __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] getnstimeofday | --- getnstimeofday ktime_get_real intel_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.04% tar [kernel.kallsyms] [k] get_page | --- get_page add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] timekeeping_get_ns | --- timekeeping_get_ns delayacct_end __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] trylock_page | --- trylock_page lock_page find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% swapper [kernel.kallsyms] [k] native_sched_clock | --- native_sched_clock sched_clock_cpu account_system_vtime irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.04% tar [kernel.kallsyms] [k] native_load_tls | --- native_load_tls __GI___libc_read 0.04% tar [kernel.kallsyms] [k] files_lglock_local_unlock_cpu | --- files_lglock_local_unlock_cpu file_sb_list_del fput filp_close sys_close system_call_fastpath __GI___close 0.04% tar [kernel.kallsyms] [k] release_pages | --- release_pages pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% jbd2/dm-2-8 [kernel.kallsyms] [k] bit_spin_lock.constprop.22 | --- bit_spin_lock.constprop.22 __slab_free kmem_cache_free __journal_remove_journal_head jbd2_journal_remove_journal_head journal_clean_one_cp_list __jbd2_journal_clean_checkpoint_list jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] pick_next_task_fair | --- pick_next_task_fair schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% swapper [kernel.kallsyms] [k] ktime_get_real | --- ktime_get_real cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.04% tar [kernel.kallsyms] [k] tick_do_update_jiffies64 | --- tick_do_update_jiffies64 tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags arch_local_irq_save irqtime_account_process_tick account_idle_ticks tick_nohz_restart_sched_tick cpu_idle start_secondary 0.03% sleep [kernel.kallsyms] [k] mem_cgroup_uncharge_start | --- mem_cgroup_uncharge_start exit_mmap mmput flush_old_exec load_elf_binary search_binary_handler do_execve sys_execve stub_execve __execve 0.02% swapper [kernel.kallsyms] [k] irq_enter | --- irq_enter apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary # # (For a higher level overview, try: perf report --sort comm,dso) # ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 16:50 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:50 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman This is the output of perf record -g -a -f sleep 5 (hopefully the list won't choke) James --- # Events: 1K cycles # # Overhead Command Shared Object Symbol # ........ ........... ................. ....................................... # 9.03% tar [kernel.kallsyms] [k] copy_user_generic_string | --- copy_user_generic_string | |--60.55%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --39.45%-- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 3.45% swapper [kernel.kallsyms] [k] intel_idle | --- intel_idle cpuidle_idle_call cpu_idle | |--76.47%-- rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --23.53%-- start_secondary 3.39% tar [kernel.kallsyms] [k] zero_user_segments | --- zero_user_segments __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.78% tar [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--16.72%-- __percpu_counter_add | | | |--59.73%-- __prop_inc_single | | task_dirty_inc | | account_page_dirtied | | __set_page_dirty | | mark_buffer_dirty | | __block_commit_write | | block_write_end | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --40.27%-- ext4_claim_free_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--14.83%-- _raw_spin_lock_irq | | | |--63.66%-- add_to_page_cache_locked | | add_to_page_cache_lru | | | | | |--51.51%-- mpage_readpages | | | ext4_readpages | | | __do_page_cache_readahead | | | ra_submit | | | ondemand_readahead | | | page_cache_async_readahead | | | generic_file_aio_read | | | do_sync_read | | | vfs_read | | | sys_read | | | system_call_fastpath | | | __GI___libc_read | | | | | --48.49%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --36.34%-- blk_throtl_bio | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--13.85%-- bit_spin_lock | jbd2_journal_add_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.19%-- __lru_cache_add | add_to_page_cache_lru | | | |--70.86%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --29.14%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--8.30%-- bit_spin_lock | lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--58.88%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --41.12%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.17%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.90%-- __srcu_read_lock | fsnotify | fsnotify_modify | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.61%-- __srcu_read_unlock | fsnotify | fsnotify_modify | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.14%-- _raw_read_lock_irqsave | dm_get_live_table | dm_merge_bvec | __bio_add_page.part.2 | bio_add_page | do_mpage_readpage | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.81%-- _raw_read_lock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.77%-- balance_dirty_pages_ratelimited_nr | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.55%-- __add_bdi_stat | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --2.18%-- _raw_spin_lock ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.70% tar [kernel.kallsyms] [k] __list_del_entry | --- __list_del_entry | |--72.43%-- list_del | | | |--76.01%-- __rmqueue | | get_page_from_freelist | | __alloc_pages_nodemask | | alloc_pages_current | | | | | |--92.23%-- __page_cache_alloc | | | | | | | |--60.54%-- __do_page_cache_readahead | | | | ra_submit | | | | ondemand_readahead | | | | page_cache_async_readahead | | | | generic_file_aio_read | | | | do_sync_read | | | | vfs_read | | | | sys_read | | | | system_call_fastpath | | | | __GI___libc_read | | | | | | | --39.46%-- grab_cache_page_write_begin | | | ext4_da_write_begin | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --7.77%-- alloc_slab_page | | new_slab | | __slab_alloc | | kmem_cache_alloc | | alloc_buffer_head | | alloc_page_buffers | | create_empty_buffers | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--18.55%-- get_page_from_freelist | | __alloc_pages_nodemask | | alloc_pages_current | | __page_cache_alloc | | | | | |--60.32%-- __do_page_cache_readahead | | | ra_submit | | | ondemand_readahead | | | page_cache_async_readahead | | | generic_file_aio_read | | | do_sync_read | | | vfs_read | | | sys_read | | | system_call_fastpath | | | __GI___libc_read | | | | | --39.68%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --5.44%-- activate_page | mark_page_accessed | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.37%-- __rmqueue | get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--63.38%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --36.62%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --10.20%-- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--65.70%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --34.30%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.53% tar tar [.] 0x75ff | --- 0x407546 | --- 0x42a9c2 | --- 0x40753f | --- 0x40e790 | --- 0x407586 | --- 0x40fe1a | --- 0x424b50 | --- 0x412972 | --- 0x407501 | --- 0x433336 | --- 0x4075ff | --- 0x403b40 | --- 0x4055c7 | --- 0x4332d9 | --- 0x405efa | --- 0x42a9fe | --- 0x405a43 | --- 0x4332d6 1.44% tar [kernel.kallsyms] [k] __ext4_get_inode_loc | --- __ext4_get_inode_loc ext4_get_inode_loc | |--62.07%-- ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--94.67%-- generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --5.33%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.93%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.43% tar [kernel.kallsyms] [k] kmem_cache_alloc | --- kmem_cache_alloc | |--66.77%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--22.22%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | | | |--65.27%-- ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --34.73%-- ext4_dirty_inode | __mark_inode_dirty | file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.29%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --4.72%-- scsi_pool_alloc_command scsi_host_alloc_command __scsi_get_command scsi_get_command scsi_get_cmd_from_req scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 1.25% tar [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--24.47%-- _raw_spin_unlock_irqrestore | | | |--30.14%-- scsi_get_command | | scsi_get_cmd_from_req | | scsi_setup_fs_cmnd | | sd_prep_fn | | blk_peek_request | | scsi_request_fn | | __blk_run_queue | | queue_unplugged | | blk_flush_plug_list | | io_schedule | | sleep_on_page_killable | | __wait_on_bit_lock | | __lock_page_killable | | lock_page_killable | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | |--28.74%-- prepare_to_wait_exclusive | | __wait_on_bit_lock | | __lock_page_killable | | lock_page_killable | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | |--23.48%-- __wake_up | | jbd2_journal_stop | | __ext4_journal_stop | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --17.65%-- blkiocg_update_dispatch_stats | throtl_charge_bio | blk_throtl_bio | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--12.68%-- radix_tree_preload_end | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.51%-- pagefault_enable | iov_iter_copy_from_user_atomic | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.90%-- _raw_spin_unlock_irq | | | |--52.31%-- add_to_page_cache_locked | | add_to_page_cache_lru | | mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --47.69%-- __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.37%-- mem_cgroup_charge_statistics | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--9.99%-- _raw_spin_unlock | | | |--60.73%-- inode_add_rsv_space | | __dquot_alloc_space | | ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --39.27%-- ext4_ext_map_blocks | ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.89%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.86%-- balance_dirty_pages_ratelimited_nr | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.52%-- add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --3.81%-- avc_has_perm_noaudit task_has_capability selinux_capable security_capable ns_capable generic_permission inode_permission may_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 1.24% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock | |--94.43%-- lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--56.27%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --43.73%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --5.57%-- __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 1.13% swapper [kernel.kallsyms] [k] poll_idle | --- poll_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 1.00% tar [kernel.kallsyms] [k] ext4_da_get_block_prep | --- ext4_da_get_block_prep | |--82.10%-- __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --17.90%-- ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.95% tar [kernel.kallsyms] [k] __find_get_block | --- __find_get_block | |--85.78%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | | | |--84.94%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --15.06%-- ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.86%-- ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --6.36%-- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.92% tar [kernel.kallsyms] [k] radix_tree_lookup_element | --- radix_tree_lookup_element | |--79.19%-- radix_tree_lookup_slot | find_get_page | | | |--39.67%-- __find_get_block_slow | | unmap_underlying_metadata | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--37.78%-- generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --22.55%-- find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--13.79%-- radix_tree_lookup | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --7.02%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.86% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty | --- ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.84% tar [kernel.kallsyms] [k] __alloc_pages_nodemask | --- __alloc_pages_nodemask | |--88.77%-- alloc_pages_current | __page_cache_alloc | | | |--53.12%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --46.88%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --11.23%-- __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.80% tar [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip | |--27.09%-- add_preempt_count | | | |--44.58%-- jbd_lock_bh_state | | do_get_write_access | | jbd2_journal_get_write_access | | __ext4_journal_get_write_access | | ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--32.43%-- bit_spin_lock | | lock_page_cgroup | | __mem_cgroup_commit_charge.constprop.28 | | __mem_cgroup_commit_charge_lrucare | | mem_cgroup_cache_charge | | add_to_page_cache_locked | | add_to_page_cache_lru | | grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --22.99%-- _raw_spin_lock_irqsave | __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--26.54%-- sub_preempt_count | | | |--43.60%-- __percpu_counter_add | | ext4_claim_free_blocks | | ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--29.86%-- bit_spin_unlock | | unlock_page_cgroup | | __mem_cgroup_commit_charge.constprop.28 | | __mem_cgroup_commit_charge_lrucare | | mem_cgroup_cache_charge | | add_to_page_cache_locked | | add_to_page_cache_lru | | mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --26.53%-- __lru_cache_add | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--13.83%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--9.64%-- jbd_lock_bh_state | jbd2_journal_dirty_metadata | __ext4_handle_dirty_metadata | ext4_mark_iloc_dirty | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--8.02%-- bit_spin_unlock | jbd2_journal_add_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.60%-- _raw_spin_lock | inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.27%-- balance_dirty_pages_ratelimited_nr generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.74% tar [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock | |--55.40%-- _raw_spin_lock_irq | | | |--86.22%-- __set_page_dirty | | mark_buffer_dirty | | __block_commit_write | | block_write_end | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --13.78%-- blk_throtl_bio | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --44.60%-- _raw_spin_lock | |--76.80%-- inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --23.20%-- ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.71% tar [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero put_page | |--49.42%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--26.92%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --23.66%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.69% tar [kernel.kallsyms] [k] get_page_from_freelist | --- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--63.44%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --36.56%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.67% tar [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit | |--90.71%-- unlock_page | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --9.29%-- wake_up_bit unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.65% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed | |--92.20%-- alloc_pages_current | __page_cache_alloc | | | |--54.97%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --45.03%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.80%-- __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.65% tar [kernel.kallsyms] [k] __rmqueue | --- __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--70.26%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --29.74%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.62% tar [kernel.kallsyms] [k] start_this_handle | --- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--71.12%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --28.88%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.58% tar [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock | |--35.45%-- __prop_inc_single | task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--14.77%-- find_get_page | find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--11.81%-- __find_get_block_slow | __find_get_block | ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | |--11.28%-- avc_has_perm_noaudit | avc_has_perm | inode_has_perm | selinux_inode_permission | security_inode_permission | inode_permission | may_delete | vfs_unlink | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | |--10.21%-- __mem_cgroup_try_charge.constprop.27 | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--8.34%-- generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --8.15%-- dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.58% tar [kernel.kallsyms] [k] __mark_inode_dirty | --- __mark_inode_dirty | |--54.06%-- __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--33.43%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.51%-- ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.55% tar [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock | |--55.58%-- ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.42%-- create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.53% tar [kernel.kallsyms] [k] next_zones_zonelist | --- next_zones_zonelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--55.47%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --44.53%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.52% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock | |--55.87%-- jbd2_journal_put_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | | | |--71.83%-- ext4_dirty_inode | | __mark_inode_dirty | | | | | |--57.59%-- generic_write_end | | | ext4_da_write_end | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --42.41%-- file_update_time | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --28.17%-- add_dirent_to_buf | ext4_add_entry | ext4_add_nondir | ext4_create | vfs_create | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --44.13%-- jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.51% tar [kernel.kallsyms] [k] jbd_unlock_bh_state | --- jbd_unlock_bh_state | |--56.56%-- do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | | | |--85.42%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | | | | |--77.84%-- generic_write_end | | | ext4_da_write_end | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --22.16%-- file_update_time | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --14.58%-- ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --43.44%-- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata | |--53.02%-- ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --46.98%-- ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.50% tar [kernel.kallsyms] [k] __block_write_begin | --- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.49% tar [kernel.kallsyms] [k] __might_sleep | --- __might_sleep | |--30.54%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.09%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--14.20%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.60%-- mutex_lock | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.58%-- kmem_cache_alloc | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--9.67%-- lock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.31%-- ext4_map_blocks ext4_getblk ext4_bread dx_probe ext4_add_entry ext4_add_nondir ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.49% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head | --- jbd2_journal_add_journal_head | |--69.06%-- __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--77.62%-- generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --22.38%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --30.94%-- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.48% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28 | --- __mem_cgroup_commit_charge.constprop.28 | |--83.14%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--56.52%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --43.48%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --16.86%-- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.46% tar [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock | |--30.47%-- avc_has_perm_noaudit | | | |--54.86%-- task_has_capability | | selinux_capable | | security_capable | | ns_capable | | exec_permission | | link_path_walk | | path_lookupat | | do_path_lookup | | user_path_parent | | do_unlinkat | | sys_unlinkat | | system_call_fastpath | | unlinkat | | | --45.14%-- avc_has_perm | may_create | selinux_inode_create | security_inode_create | vfs_create | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | |--20.91%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--18.25%-- find_get_page | find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.06%-- __account_system_time | irqtime_account_process_tick | account_process_tick | update_process_times | tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | rw_verify_area | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --13.31%-- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.44% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke | --- jbd2_journal_cancel_revoke do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.44% tar [kernel.kallsyms] [k] mark_page_accessed | --- mark_page_accessed | |--56.95%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--15.53%-- do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--14.54%-- __find_get_block | __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.98%-- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.44% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed | |--55.41%-- alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.59%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--68.42%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --31.58%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.43% tar [kernel.kallsyms] [k] in_lock_functions | --- in_lock_functions | |--76.43%-- get_parent_ip | | | |--80.00%-- add_preempt_count | | | | | |--39.87%-- balance_dirty_pages_ratelimited_nr | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | |--30.07%-- radix_tree_preload | | | add_to_page_cache_locked | | | add_to_page_cache_lru | | | grab_cache_page_write_begin | | | ext4_da_write_begin | | | generic_file_buffered_write | | | __generic_file_aio_write | | | generic_file_aio_write | | | ext4_file_write | | | do_sync_write | | | vfs_write | | | sys_write | | | system_call_fastpath | | | __GI___libc_write | | | | | --30.07%-- _raw_read_lock | | start_this_handle | | jbd2__journal_start | | jbd2_journal_start | | ext4_journal_start_sb | | ext4_dirty_inode | | __mark_inode_dirty | | ext4_setattr | | notify_change | | chown_common | | sys_fchown | | system_call_fastpath | | __GI___fchown | | | --20.00%-- sub_preempt_count | avc_has_perm_noaudit | avc_has_perm | inode_has_perm | dentry_has_perm | selinux_inode_setattr | security_inode_setattr | notify_change | chown_common | sys_fchown | system_call_fastpath | __GI___fchown | |--12.38%-- add_preempt_count | jbd_lock_bh_state | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --11.19%-- sub_preempt_count _raw_spin_unlock_irqrestore __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.43% swapper [kernel.kallsyms] [k] ext4_end_bio | --- ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.42% tar [kernel.kallsyms] [k] ext4_get_group_desc | --- ext4_get_group_desc | |--88.05%-- __ext4_get_inode_loc | ext4_get_inode_loc | | | |--64.96%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --35.04%-- ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --11.95%-- ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.42% tar [kernel.kallsyms] [k] __percpu_counter_add | --- __percpu_counter_add | |--39.75%-- ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--25.32%-- ext4_claim_free_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--19.28%-- ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --15.64%-- __add_bdi_stat account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.42% tar [kernel.kallsyms] [k] do_get_write_access | --- do_get_write_access | |--80.57%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --19.43%-- __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.42% tar [kernel.kallsyms] [k] generic_file_buffered_write | --- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.41% tar [kernel.kallsyms] [k] jbd2_journal_stop | --- jbd2_journal_stop __ext4_journal_stop | |--79.71%-- ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --20.29%-- ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.41% tar [kernel.kallsyms] [k] jbd2__journal_start | --- jbd2__journal_start | |--82.99%-- jbd2_journal_start | ext4_journal_start_sb | | | |--83.87%-- ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --16.13%-- ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --17.01%-- ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.40% tar [kernel.kallsyms] [k] do_mpage_readpage | --- do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.39% tar [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep | |--55.35%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--61.18%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --38.82%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--28.25%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --16.40%-- __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.38% tar [kernel.kallsyms] [k] alloc_pages_current | --- alloc_pages_current | |--82.52%-- __page_cache_alloc | | | |--74.42%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --25.58%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --17.48%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.37% tar [kernel.kallsyms] [k] __nr_to_section | --- __nr_to_section lookup_page_cgroup | |--70.03%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.38%-- mem_cgroup_get_reclaim_stat_from_page | update_page_reclaim_stat | ____pagevec_lru_add_fn | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.60%-- mem_cgroup_del_lru_list activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.37% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore | |--52.73%-- _raw_spin_unlock_irqrestore | | | |--51.73%-- __wake_up | | jbd2_journal_stop | | __ext4_journal_stop | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --48.27%-- pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.27%-- __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.36% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27 | --- __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--67.66%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --32.34%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.36% swapper [kernel.kallsyms] [k] unlock_page | --- unlock_page | |--85.03%-- mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --14.97%-- bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.35% tar [kernel.kallsyms] [k] ext4_xattr_get | --- ext4_xattr_get | |--68.95%-- ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --31.05%-- generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.35% perf [kernel.kallsyms] [k] seq_puts | --- seq_puts render_sigset_t proc_pid_status proc_single_show seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.35% tar [kernel.kallsyms] [k] grab_cache_page_write_begin | --- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.34% tar [kernel.kallsyms] [k] alloc_page_buffers | --- alloc_page_buffers | |--72.20%-- create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --27.80%-- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.34% tar [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--58.75%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--70.45%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --29.55%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.25%-- __lru_cache_add add_to_page_cache_lru | |--61.47%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --38.53%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.34% perf [kernel.kallsyms] [k] clear_page_c | --- clear_page_c __alloc_pages_nodemask perf_mmap_alloc_page perf_mmap mmap_region do_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap 0x418523 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.34% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic | --- iov_iter_copy_from_user_atomic generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.33% tar [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_page | |--84.05%-- find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --15.95%-- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.33% top [kernel.kallsyms] [k] follow_managed | --- follow_managed walk_component link_path_walk path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.33% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages | --- ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.32% swapper [kernel.kallsyms] [k] do_timer | --- do_timer tick_do_update_jiffies64 | |--61.57%-- tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --38.43%-- tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.32% tar [kernel.kallsyms] [k] __generic_file_aio_write | --- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.32% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list | --- mem_cgroup_add_lru_list add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--87.33%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --12.67%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.32% tar [kernel.kallsyms] [k] ext4_da_write_end | --- ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.32% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page | --- ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.32% swapper [kernel.kallsyms] [k] tick_nohz_stop_sched_tick | --- tick_nohz_stop_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.31% tar [kernel.kallsyms] [k] down_read | --- down_read | |--78.35%-- ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --21.65%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.31% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--83.21%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --16.79%-- force_qs_rnp force_quiescent_state __rcu_process_callbacks rcu_process_callbacks __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.30% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--59.37%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --40.63%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.30% perf [kernel.kallsyms] [k] file_update_time | --- file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0.30% tar [kernel.kallsyms] [k] mark_buffer_dirty | --- mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.29% swapper [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--54.58%-- _raw_spin_lock_irqsave | | | |--57.58%-- scsi_device_unbusy | | scsi_finish_command | | scsi_softirq_done | | blk_done_softirq | | __do_softirq | | call_softirq | | do_softirq | | irq_exit | | do_IRQ | | common_interrupt | | cpuidle_idle_call | | cpu_idle | | rest_init | | start_kernel | | x86_64_start_reservations | | x86_64_start_kernel | | | --42.42%-- test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--20.90%-- __percpu_counter_add | __add_bdi_stat | test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--15.95%-- irq_enter | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --8.56%-- scsi_device_unbusy scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.29% tar [kernel.kallsyms] [k] mnt_want_write_file | --- mnt_want_write_file | |--76.04%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --23.96%-- system_call_fastpath __GI___fchown 0.27% tar [kernel.kallsyms] [k] find_busiest_group | --- find_busiest_group load_balance schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.26% perf [kernel.kallsyms] [k] perf_event_mmap_output | --- perf_event_mmap_output perf_event_mmap_ctx perf_event_mmap mmap_region do_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap 0.26% swapper [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free | |--59.24%-- mempool_free | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --40.76%-- scsi_pool_free_command __scsi_put_command scsi_put_command scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.26% perf [kernel.kallsyms] [k] fsnotify | --- fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __write_nocancel 0x4293b8 0x429c0a 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.26% perf [kernel.kallsyms] [k] put_bh | --- put_bh ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4293b8 0x429c0a 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.26% perf [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock prepend_path path_with_deleted d_path seq_path show_map_vma show_map seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.26% tar [kernel.kallsyms] [k] get_page | --- get_page __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.26% tar [kernel.kallsyms] [k] selinux_file_permission | --- selinux_file_permission security_file_permission rw_verify_area vfs_write sys_write system_call_fastpath __GI___libc_write 0.26% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag need_resched should_resched _cond_resched | |--64.24%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --35.76%-- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.26% perf [kernel.kallsyms] [k] number | --- number vsnprintf seq_printf show_map_vma show_map seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.25% tar libc-2.13.90.so [.] __GI___libc_write | --- __GI___libc_write 0.25% tar [kernel.kallsyms] [k] test_and_set_bit | --- test_and_set_bit mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.25% perf [kernel.kallsyms] [k] vsnprintf | --- vsnprintf seq_printf show_map_vma show_map seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.25% tar [kernel.kallsyms] [k] run_timer_softirq | --- run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.25% perf [kernel.kallsyms] [k] setup_object | --- setup_object new_slab __slab_alloc kmem_cache_alloc d_alloc proc_fill_cache proc_task_readdir vfs_readdir sys_getdents system_call_fastpath __getdents64 0x429beb 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.24% swapper [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--40.64%-- __bit_spin_unlock.constprop.24 | __slab_free | kmem_cache_free | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--33.53%-- _raw_spin_unlock_irqrestore | rcu_report_qs_rnp | __rcu_process_callbacks | rcu_process_callbacks | __do_softirq | call_softirq | do_softirq | irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --25.82%-- __slab_free kmem_cache_free mempool_free_slab mempool_free bio_free bio_fs_destructor bio_put ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.24% tar [kernel.kallsyms] [k] add_to_page_cache_locked | --- add_to_page_cache_locked | |--64.11%-- add_to_page_cache_lru | | | |--63.51%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --36.49%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --35.89%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.24% tar [kernel.kallsyms] [k] file_remove_suid | --- file_remove_suid | |--81.19%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --18.81%-- generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.24% tar [kernel.kallsyms] [k] clear_bit_unlock | --- clear_bit_unlock | |--71.97%-- unlock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--58.50%-- file_update_time | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --41.50%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --28.03%-- do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.23% tar [kernel.kallsyms] [k] sys_write | --- sys_write system_call_fastpath __GI___libc_write 0.23% swapper [kernel.kallsyms] [k] SetPageUptodate | --- SetPageUptodate mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.23% tar [kernel.kallsyms] [k] __bio_add_page.part.2 | --- __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.23% swapper [kernel.kallsyms] [k] rcu_irq_enter | --- rcu_irq_enter irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.23% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq | --- _raw_spin_unlock_irq | |--45.40%-- generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--29.98%-- mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --24.62%-- add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.23% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock | |--51.02%-- jbd2_journal_put_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.98%-- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% swapper [kernel.kallsyms] [k] nr_iowait_cpu | --- nr_iowait_cpu tick_nohz_stop_idle tick_check_idle irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.22% flush-253:2 [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue | |--51.36%-- unlock_page | ext4_bio_write_page | mpage_da_submit_io | mpage_da_map_and_submit | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --48.64%-- ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.22% tar [kernel.kallsyms] [k] __jbd2_log_space_left | --- __jbd2_log_space_left start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--50.77%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.23%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% tar [kernel.kallsyms] [k] mb_test_bit | --- mb_test_bit mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.22% tar [kernel.kallsyms] [k] __blk_recalc_rq_segments | --- __blk_recalc_rq_segments blk_recount_segments bio_phys_segments blk_rq_bio_prep init_request_from_bio __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.22% tar [kernel.kallsyms] [k] generic_write_end | --- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata | --- jbd2_journal_dirty_metadata | |--62.74%-- __ext4_handle_dirty_metadata | ext4_mark_iloc_dirty | ext4_mark_inode_dirty | | | |--59.79%-- ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --40.21%-- ext4_unlink | vfs_unlink | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --37.26%-- ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.22% tar [kernel.kallsyms] [k] __mem_cgroup_threshold | --- __mem_cgroup_threshold memcg_check_events __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.21% tar [kernel.kallsyms] [k] kfree | --- kfree start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--79.14%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --20.86%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.21% tar [kernel.kallsyms] [k] ext4_get_inode_flags | --- ext4_get_inode_flags ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--63.71%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --36.29%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.21% flush-253:2 [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero release_pages __pagevec_release pagevec_release | |--57.93%-- ext4_num_dirty_pages | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --42.07%-- write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.21% flush-253:2 [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock __find_get_block_slow unmap_underlying_metadata mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.21% tar [kernel.kallsyms] [k] avtab_search | --- avtab_search security_compute_sid security_transition_sid may_create selinux_inode_create security_inode_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.21% tar [kernel.kallsyms] [k] search_dirblock | --- search_dirblock ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.21% tar [kernel.kallsyms] [k] __sg_alloc_table | --- __sg_alloc_table scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.21% tar [kernel.kallsyms] [k] ext4_journal_start_sb | --- ext4_journal_start_sb ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.21% top libc-2.13.90.so [.] __GI_____strtoll_l_internal | --- __GI_____strtoll_l_internal 0.20% swapper [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock | |--57.86%-- handle_edge_irq | handle_irq | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --42.14%-- tick_do_update_jiffies64 tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.20% tar [kernel.kallsyms] [k] generic_write_sync | --- generic_write_sync generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.20% tar [kernel.kallsyms] [k] put_page | --- put_page | |--64.65%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --35.35%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.20% flush-253:2 [kernel.kallsyms] [k] tag_get | --- tag_get | |--58.46%-- __lookup_tag | radix_tree_gang_lookup_tag_slot | find_get_pages_tag | pagevec_lookup_tag | ext4_num_dirty_pages | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --41.54%-- test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.20% tar [kernel.kallsyms] [k] dm_merge_bvec | --- dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.19% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata | --- __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.19% tar [kernel.kallsyms] [k] __block_commit_write | --- __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.19% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore | --- _raw_spin_unlock_irqrestore | |--54.81%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --45.19%-- __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.19% tar [kernel.kallsyms] [k] ext4_writepage_trans_blocks | --- ext4_writepage_trans_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.19% tar [kernel.kallsyms] [k] jbd_lock_bh_state | --- jbd_lock_bh_state | |--62.13%-- do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.87%-- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.19% tar [kernel.kallsyms] [k] __list_add | --- __list_add | |--70.74%-- __make_request | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --29.26%-- expand __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.19% tar [kernel.kallsyms] [k] fsnotify_modify | --- fsnotify_modify | |--74.13%-- vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --25.87%-- sys_write system_call_fastpath __GI___libc_write 0.19% flush-253:2 [kernel.kallsyms] [k] lock_page | --- lock_page write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.19% flush-253:2 [kernel.kallsyms] [k] rb_next | --- rb_next set_next_entity pick_next_task_fair pick_next_task schedule preempt_schedule_irq retint_kernel ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.19% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount | --- ext4_ext_calc_metadata_amount ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.18% swapper [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit | |--61.24%-- unlock_page | mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --38.76%-- end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.18% swapper [kernel.kallsyms] [k] select_nohz_load_balancer | --- select_nohz_load_balancer | |--75.59%-- tick_nohz_restart_sched_tick | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --24.41%-- tick_nohz_stop_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.18% swapper [kernel.kallsyms] [k] menu_select | --- menu_select cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.18% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_pages pagevec_lookup mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.18% tar [kernel.kallsyms] [k] generic_file_aio_read | --- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.18% tar [kernel.kallsyms] [k] __prop_inc_single | --- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.18% swapper [kernel.kallsyms] [k] page_index | --- page_index test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.17% tar [kernel.kallsyms] [k] arch_read_lock | --- arch_read_lock _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] mutex_unlock | --- mutex_unlock | |--73.48%-- generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --26.52%-- do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.17% tar [kernel.kallsyms] [k] blk_rq_map_sg | --- blk_rq_map_sg scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] bit_waitqueue | --- bit_waitqueue | |--51.20%-- wake_up_bit | unlock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.80%-- unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] SetPageLRU | --- SetPageLRU ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--71.47%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --28.53%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 | |--64.90%-- _raw_spin_unlock_irq | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --35.10%-- inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] lookup_page_cgroup | --- lookup_page_cgroup | |--58.52%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.48%-- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% tar [kernel.kallsyms] [k] generic_make_request | --- generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] lock_page_cgroup | --- lock_page_cgroup __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.17% swapper [kernel.kallsyms] [k] mempool_free | --- mempool_free clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.17% tar [kernel.kallsyms] [k] put_prev_task_fair | --- put_prev_task_fair io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] perf_event_task_tick | --- perf_event_task_tick scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--51.07%-- scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.93%-- _ext4_get_block ext4_get_block do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.17% tar [kernel.kallsyms] [k] arch_read_unlock | --- arch_read_unlock | |--66.07%-- _raw_read_unlock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --33.93%-- _raw_read_unlock_irqrestore dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.16% swapper [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--38.89%-- __cycles_2_ns | native_sched_clock | sched_clock | sched_clock_cpu | account_system_vtime | irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--32.56%-- tick_check_idle | irq_enter | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--14.27%-- account_idle_time | irqtime_account_process_tick | account_idle_ticks | tick_nohz_restart_sched_tick | cpu_idle | start_secondary | --14.27%-- irqtime_account_process_tick account_idle_ticks tick_nohz_restart_sched_tick cpu_idle start_secondary 0.16% tar [kernel.kallsyms] [k] fget_light | --- fget_light | |--51.07%-- sys_read | system_call_fastpath | __GI___libc_read | --48.93%-- sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] SetPageUptodate | --- SetPageUptodate __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] __ext4_journal_stop | --- __ext4_journal_stop | |--50.22%-- ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.78%-- ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] ext4_clear_inode_state | --- ext4_clear_inode_state ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--51.89%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.11%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge | --- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--65.55%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --34.45%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] ext4_map_blocks | --- ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% tar [kernel.kallsyms] [k] ata_dev_enabled | --- ata_dev_enabled ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.16% tar [kernel.kallsyms] [k] new_slab | --- new_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.16% swapper [kernel.kallsyms] [k] hrtimer_cancel | --- hrtimer_cancel tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.15% tar [kernel.kallsyms] [k] lock_buffer | --- lock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock unlock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--72.67%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --27.33%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] up_read | --- up_read | |--62.56%-- ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.44%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] preempt_schedule | --- preempt_schedule try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% tar [kernel.kallsyms] [k] fsnotify_access | --- fsnotify_access sys_read system_call_fastpath __GI___libc_read 0.15% swapper [kernel.kallsyms] [k] native_read_tsc | --- native_read_tsc paravirt_read_tsc native_sched_clock sched_clock sched_clock_cpu account_system_vtime irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.14% tar [kernel.kallsyms] [k] dequeue_task_fair | --- dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.14% tar [kernel.kallsyms] [k] ata_scsi_queuecmd | --- ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.14% tar [kernel.kallsyms] [k] file_read_actor | --- file_read_actor generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.14% tar [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock | |--52.50%-- ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.50%-- ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.14% top [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count _raw_spin_lock nameidata_drop_rcu link_path_walk path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.14% tar [kernel.kallsyms] [k] generic_file_aio_write | --- generic_file_aio_write | |--52.11%-- do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.89%-- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.14% tar [kernel.kallsyms] [k] block_write_end | --- block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.14% tar [kernel.kallsyms] [k] hrtimer_interrupt | --- hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--52.67%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.33%-- string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.14% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count __percpu_counter_add __add_bdi_stat clear_page_dirty_for_io mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.14% swapper [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock | |--50.04%-- __prop_inc_percpu_max | test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --49.96%-- __atomic_notifier_call_chain atomic_notifier_call_chain exit_idle do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.13% tar [kernel.kallsyms] [k] ktime_get_ts | --- ktime_get_ts __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.13% tar [kernel.kallsyms] [k] page_zone | --- page_zone pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] __read_lock_failed | --- __read_lock_failed _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] prop_norm_single | --- prop_norm_single __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] put_bh | --- put_bh __brelse brelse ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] radix_tree_insert | --- radix_tree_insert add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] policy_zonelist | --- policy_zonelist alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.13% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% swapper [kernel.kallsyms] [k] irq_entries_start | --- irq_entries_start cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.13% top libproc-3.2.8.so [.] 0x44c2 | --- 0x37d94044c2 0.13% flush-253:2 [kernel.kallsyms] [k] attempt_back_merge | --- attempt_back_merge generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.13% tar [kernel.kallsyms] [k] virt_to_head_page | --- virt_to_head_page kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] mnt_clone_write | --- mnt_clone_write mnt_want_write_file file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] rb_erase | --- rb_erase deadline_remove_request deadline_move_request deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.13% tar [kernel.kallsyms] [k] __set_page_dirty | --- __set_page_dirty | |--51.11%-- __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.89%-- mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] do_sync_write | --- do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable _raw_spin_lock_irq | |--55.62%-- add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.38%-- __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.13% tar [kernel.kallsyms] [k] ext4_da_write_begin | --- ext4_da_write_begin __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% swapper [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore | |--69.63%-- account_system_vtime | irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --30.37%-- irqtime_account_process_tick account_idle_ticks tick_nohz_restart_sched_tick cpu_idle start_secondary 0.12% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.12% tar [kernel.kallsyms] [k] inode_reserved_space | --- inode_reserved_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% swapper [kernel.kallsyms] [k] update_cfs_shares | --- update_cfs_shares enqueue_task_fair enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.12% tar [kernel.kallsyms] [k] schedule | --- schedule | |--59.95%-- io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --40.05%-- __GI___libc_read 0.12% tar [kernel.kallsyms] [k] security_sid_to_context_force | --- security_sid_to_context_force security_inode_init_security ext4_init_security ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.12% tar [kernel.kallsyms] [k] radix_tree_preload | --- radix_tree_preload add_to_page_cache_locked add_to_page_cache_lru | |--54.07%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --45.93%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags _local_bh_enable irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.12% flush-253:2 [kernel.kallsyms] [k] release_pages | --- release_pages __pagevec_release pagevec_release write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.12% flush-253:2 [kernel.kallsyms] [k] page_mkclean | --- page_mkclean mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.12% tar [kernel.kallsyms] [k] set_bh_page | --- set_bh_page create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] account_entity_dequeue | --- account_entity_dequeue dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% tar [kernel.kallsyms] [k] ata_qc_issue | --- ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% tar libc-2.13.90.so [.] __cfree | --- __cfree 0.12% tar [kernel.kallsyms] [k] radix_tree_lookup_slot | --- radix_tree_lookup_slot | |--62.63%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --37.37%-- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% top libc-2.13.90.so [.] _IO_vfscanf_internal | --- _IO_vfscanf_internal _IO_vsscanf 0x7fffef292b20 0.12% tar [kernel.kallsyms] [k] ext4_inode_table | --- ext4_inode_table __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] xattr_resolve_name | --- xattr_resolve_name generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] add_to_page_cache_lru | --- add_to_page_cache_lru | |--67.10%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --32.90%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.12% migration/0 [kernel.kallsyms] [k] resched_task | --- resched_task check_preempt_curr __migrate_task migration_cpu_stop cpu_stopper_thread kthread kernel_thread_helper 0.12% tar [kernel.kallsyms] [k] __count_vm_events | --- __count_vm_events __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--50.90%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --49.10%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.12% tar [kernel.kallsyms] [k] system_call_fastpath | --- system_call_fastpath __GI___libc_write 0.11% flush-253:2 [kernel.kallsyms] [k] __percpu_counter_add | --- __percpu_counter_add __add_bdi_stat clear_page_dirty_for_io mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% swapper [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.11% kworker/0:2 [kernel.kallsyms] [k] worker_thread | --- worker_thread kthread kernel_thread_helper 0.11% tar [kernel.kallsyms] [k] fsnotify | --- fsnotify vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] is_handle_aborted | --- is_handle_aborted jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] blk_finish_plug | --- blk_finish_plug ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access | --- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] vfs_write | --- vfs_write system_call_fastpath __GI___libc_write 0.11% swapper [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.11% tar [kernel.kallsyms] [k] blk_flush_plug_list | --- blk_flush_plug_list blk_finish_plug generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] deadline_remove_request | --- deadline_remove_request deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] mb_find_buddy | --- mb_find_buddy mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.11% tar [kernel.kallsyms] [k] sys_read | --- sys_read | |--58.45%-- system_call_fastpath | __GI___libc_read | --41.55%-- __GI___libc_read 0.11% flush-253:2 [kernel.kallsyms] [k] arch_read_unlock | --- arch_read_unlock _raw_read_unlock_irqrestore dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% swapper [kernel.kallsyms] [k] __slab_free | --- __slab_free kmem_cache_free mempool_free_slab mempool_free scsi_sg_free __sg_free_table __scsi_release_buffers scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.11% tar [kernel.kallsyms] [k] sidtab_search_core | --- sidtab_search_core sidtab_search security_compute_sid security_transition_sid may_create selinux_inode_create security_inode_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.11% swapper [kernel.kallsyms] [k] switch_mm | --- switch_mm schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.11% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io | --- mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics | --- mem_cgroup_charge_statistics __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--56.66%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.34%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% tar [kernel.kallsyms] [k] zone_watermark_ok | --- zone_watermark_ok | |--50.80%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.20%-- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] ext4_free_blocks | --- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.11% tar [kernel.kallsyms] [k] ext4_ext_map_blocks | --- ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.11% tar [kernel.kallsyms] [k] add_dirent_to_buf | --- add_dirent_to_buf ext4_add_entry ext4_add_nondir ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.11% tar [kernel.kallsyms] [k] strcmp | --- strcmp symcmp hashtab_search string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.11% tar [kernel.kallsyms] [k] select_idle_sibling | --- select_idle_sibling select_task_rq_fair select_task_rq try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.11% flush-253:2 [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit unlock_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.11% tar [kernel.kallsyms] [k] radix_tree_node_alloc | --- radix_tree_node_alloc radix_tree_insert add_to_page_cache_locked add_to_page_cache_lru | |--60.14%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --39.86%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% swapper [kernel.kallsyms] [k] rcu_irq_exit | --- rcu_irq_exit | |--54.97%-- smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --45.03%-- irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.10% tar [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] check_irqs_on | --- check_irqs_on __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% flush-253:2 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count __add_bdi_stat test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% flush-253:2 [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep unmap_underlying_metadata mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] alloc_buffer_head | --- alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] jiffies_to_timeval | --- jiffies_to_timeval __account_system_time irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.10% tar [kernel.kallsyms] [k] ext4_get_reserved_space | --- ext4_get_reserved_space inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.10% tar [kernel.kallsyms] [k] blk_start_plug | --- blk_start_plug mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue __make_request generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] ata_qc_new_init | --- ata_qc_new_init | |--51.58%-- ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.42%-- __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% tar [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_unlock_irq scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] inc_zone_page_state | --- inc_zone_page_state account_page_writeback test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] system_call | --- system_call __GI___libc_write 0.10% flush-253:2 [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio ext4_io_submit ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% flush-253:2 [kernel.kallsyms] [k] mod_state | --- mod_state inc_zone_page_state account_page_writeback test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save _raw_spin_lock_irqsave __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.10% tar [kernel.kallsyms] [k] bio_get_nr_vecs | --- bio_get_nr_vecs mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.10% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.10% flush-253:2 [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% tar [kernel.kallsyms] [k] inode_init_once | --- inode_init_once init_once setup_object new_slab __slab_alloc kmem_cache_alloc ext4_alloc_inode alloc_inode new_inode ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--61.31%-- _raw_spin_lock | jbd2_journal_file_buffer | jbd2_journal_write_metadata_buffer | jbd2_journal_commit_transaction | kjournald2 | kthread | kernel_thread_helper | --38.69%-- jbd_trylock_bh_state journal_clean_one_cp_list __jbd2_journal_clean_checkpoint_list jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.09% swapper [kernel.kallsyms] [k] kobject_put | --- kobject_put __scsi_put_command scsi_put_command scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% flush-253:2 [kernel.kallsyms] [k] __elv_add_request | --- __elv_add_request __make_request generic_make_request submit_bio ext4_io_submit ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% swapper [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore rcu_process_gp_end __rcu_process_callbacks rcu_process_callbacks __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% tar [kernel.kallsyms] [k] round_jiffies_up | --- round_jiffies_up blk_start_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] attach_page_buffers | --- attach_page_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% tar [kernel.kallsyms] [k] hrtick_update | --- hrtick_update dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] ext4_clear_inode_flag | --- ext4_clear_inode_flag ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock_irq blk_throtl_bio generic_make_request submit_bio submit_bh jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.09% top [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock __d_lookup walk_component link_path_walk path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.09% tar [kernel.kallsyms] [k] __brelse | --- __brelse brelse ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% flush-253:2 [kernel.kallsyms] [k] page_zone | --- page_zone inc_zone_page_state account_page_writeback test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% tar [kernel.kallsyms] [k] bio_integrity_enabled | --- bio_integrity_enabled generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] dm_table_find_target | --- dm_table_find_target __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] __mod_zone_page_state | --- __mod_zone_page_state add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% tar [kernel.kallsyms] [k] ahci_fill_cmd_slot | --- ahci_fill_cmd_slot ahci_qc_prep ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.09% swapper [kernel.kallsyms] [k] blk_unprep_request | --- blk_unprep_request blk_finish_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% top libc-2.13.90.so [.] __GI___close | --- __GI___close 0.09% tar [kernel.kallsyms] [k] put_bh | --- put_bh ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.09% swapper [kernel.kallsyms] [k] finish_task_switch | --- finish_task_switch schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% flush-253:2 [kernel.kallsyms] [k] page_mapping | --- page_mapping mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.09% tar [kernel.kallsyms] [k] alloc_tio | --- alloc_tio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear | --- radix_tree_tag_clear test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.08% tar [kernel.kallsyms] [k] timespec_trunc | --- timespec_trunc touch_atime generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] ext4_claim_free_blocks | --- ext4_claim_free_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% tar libc-2.13.90.so [.] __GI___errno_location | --- __GI___errno_location 0.08% tar [kernel.kallsyms] [k] path_to_nameidata | --- path_to_nameidata walk_component link_path_walk path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.08% tar [kernel.kallsyms] [k] current_sid | --- current_sid selinux_file_permission security_file_permission rw_verify_area vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% top [kernel.kallsyms] [k] path_openat | --- path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0.08% tar [kernel.kallsyms] [k] __do_page_cache_readahead | --- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] _cond_resched | --- _cond_resched generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% swapper [kernel.kallsyms] [k] run_timer_softirq | --- run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% swapper [kernel.kallsyms] [k] note_interrupt | --- note_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% swapper [kernel.kallsyms] [k] native_apic_mem_write | --- native_apic_mem_write apic_write ack_APIC_irq smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.08% tar [kernel.kallsyms] [k] bio_data | --- bio_data init_request_from_bio __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] string_to_context_struct | --- string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.08% tar [kernel.kallsyms] [k] mb_clear_bit | --- mb_clear_bit mb_clear_bits ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.08% tar [kernel.kallsyms] [k] ext4_mb_load_buddy | --- ext4_mb_load_buddy ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.08% top [kernel.kallsyms] [k] __fsnotify_parent | --- __fsnotify_parent fsnotify_perm security_file_permission rw_verify_area vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% sleep [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock | |--60.83%-- __do_fault | handle_pte_fault | handle_mm_fault | do_page_fault | page_fault | 0x37d9082b80 | --39.17%-- deny_write_access mmap_region do_mmap_pgoff do_mmap elf_map load_elf_binary search_binary_handler do_execve sys_execve stub_execve 0x37d90ae097 0.08% swapper [kernel.kallsyms] [k] ahci_interrupt | --- ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% tar [kernel.kallsyms] [k] __srcu_read_unlock | --- __srcu_read_unlock fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% tar [kernel.kallsyms] [k] signal_pending | --- signal_pending fatal_signal_pending __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.08% tar [kernel.kallsyms] [k] __read_seqcount_begin | --- __read_seqcount_begin path_init path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.08% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag mnt_clone_write mnt_want_write_file file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% tar [kernel.kallsyms] [k] jbd2_journal_start | --- jbd2_journal_start ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] __srcu_read_lock | --- __srcu_read_lock fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] handle_irq | --- handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% swapper [kernel.kallsyms] [k] __ata_scsi_queuecmd | --- __ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc drive_stat_acct __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __switch_to | --- __switch_to __GI___libc_read 0.07% tar [kernel.kallsyms] [k] ext4_file_write | --- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] wake_bit_function | --- wake_bit_function __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] _raw_spin_lock_irq | --- _raw_spin_lock_irq add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] drive_stat_acct | --- drive_stat_acct __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22 | --- test_ti_thread_flag.constprop.22 zero_user_segments __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] sd_prep_fn | --- sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% top libc-2.13.90.so [.] __strchrnul | --- __strchrnul ___vsnprintf_chk 0x39335b1b6d5b1b42 0.07% swapper [kernel.kallsyms] [k] sd_prep_fn | --- sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] try_to_wake_up | --- try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_unlock_irqrestore ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __zone_watermark_ok | --- __zone_watermark_ok zone_watermark_ok get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] __remove_hrtimer | --- __remove_hrtimer remove_hrtimer.part.4 hrtimer_try_to_cancel hrtimer_cancel tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% swapper [kernel.kallsyms] [k] scsi_device_unbusy | --- scsi_device_unbusy scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] scsi_prep_return | --- scsi_prep_return sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] prepare_to_wait_exclusive | --- prepare_to_wait_exclusive __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] set_buffer_uptodate | --- set_buffer_uptodate __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] ata_get_xlat_func | --- ata_get_xlat_func __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] __find_get_block_slow | --- __find_get_block_slow __find_get_block ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.07% tar [kernel.kallsyms] [k] generic_getxattr | --- generic_getxattr security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% swapper [kernel.kallsyms] [k] bit_spin_lock.constprop.22 | --- bit_spin_lock.constprop.22 __slab_free kmem_cache_free put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% swapper [kernel.kallsyms] [k] is_swiotlb_buffer | --- is_swiotlb_buffer unmap_single swiotlb_unmap_sg_attrs ata_sg_clean __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] native_read_tsc | --- native_read_tsc paravirt_read_tsc native_sched_clock sched_clock blk_rq_init get_request get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] ext4_has_free_blocks | --- ext4_has_free_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar libc-2.13.90.so [.] __GI___fxstat64 | --- __GI___fxstat64 0.07% swapper [kernel.kallsyms] [k] perf_pmu_disable | --- perf_pmu_disable scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] sidtab_context_to_sid | --- sidtab_context_to_sid security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.07% tar [kernel.kallsyms] [k] scsi_pool_alloc_command | --- scsi_pool_alloc_command scsi_host_alloc_command __scsi_get_command scsi_get_command scsi_get_cmd_from_req scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] list_del | --- list_del activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] pagefault_enable | --- pagefault_enable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] security_file_permission | --- security_file_permission rw_verify_area vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% swapper [kernel.kallsyms] [k] scsi_next_command | --- scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.07% tar [kernel.kallsyms] [k] __phys_addr | --- __phys_addr virt_to_head_page kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] page_cache_async_readahead | --- page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.07% tar [kernel.kallsyms] [k] truncate_inode_pages_range | --- truncate_inode_pages_range ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.07% tar [kernel.kallsyms] [k] mem_cgroup_del_lru_list | --- mem_cgroup_del_lru_list activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] ext4_xattr_check_names | --- ext4_xattr_check_names ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] arch_local_irq_enable | --- arch_local_irq_enable __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.07% tar [kernel.kallsyms] [k] get_request | --- get_request get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs | --- swiotlb_map_sg_attrs ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] dm_get_live_table | --- dm_get_live_table page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page | --- mem_cgroup_get_reclaim_stat_from_page update_page_reclaim_stat ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] check_preempt_curr | --- check_preempt_curr ttwu_post_activation try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] atomic_long_add | --- atomic_long_add zone_page_state_add __mod_zone_page_state add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] pm_qos_request | --- pm_qos_request cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] __kmalloc | --- __kmalloc kzalloc.constprop.9 ext4_ext_find_extent ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] rcu_bh_qs | --- rcu_bh_qs __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] __blk_complete_request | --- __blk_complete_request scsi_done ata_scsi_qc_complete __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] __d_lookup | --- __d_lookup walk_component link_path_walk path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.06% tar [kernel.kallsyms] [k] context_struct_to_string | --- context_struct_to_string security_sid_to_context_core security_sid_to_context_force selinux_inode_init_security security_inode_init_security ext4_init_security ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.06% swapper [kernel.kallsyms] [k] tick_nohz_restart_sched_tick | --- tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] max_sane_readahead | --- max_sane_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] rw_verify_area | --- rw_verify_area vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] ahci_qc_prep | --- ahci_qc_prep ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] linear_map | --- linear_map __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% swapper [kernel.kallsyms] [k] sched_clock | --- sched_clock account_system_vtime irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] fput_light | --- fput_light system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] iov_iter_advance | --- iov_iter_advance generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] __queue_work | --- __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt lock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% swapper [kernel.kallsyms] [k] scsi_io_completion | --- scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] activate_page | --- activate_page mark_page_accessed generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] touch_atime | --- touch_atime generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] atomic_sub | --- atomic_sub jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] __inc_zone_page_state | --- __inc_zone_page_state add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] ext4_test_inode_state | --- ext4_test_inode_state ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] ext4_xattr_find_entry | --- ext4_xattr_find_entry ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] deadline_dispatch_requests | --- deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] hrtick_update | --- hrtick_update enqueue_task_fair enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] try_to_wake_up | --- try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% swapper [kernel.kallsyms] [k] pick_next_task_fair | --- pick_next_task_fair schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] pick_next_task_idle | --- pick_next_task_idle schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% swapper [kernel.kallsyms] [k] ktime_get | --- ktime_get tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr | --- balance_dirty_pages_ratelimited_nr generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% swapper [kernel.kallsyms] [k] mpage_end_io | --- mpage_end_io dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] radix_tree_preload_end | --- radix_tree_preload_end add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] submit_bio | --- submit_bio ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] __ata_scsi_queuecmd | --- __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% tar [kernel.kallsyms] [k] dm_table_put | --- dm_table_put dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.06% sleep [kernel.kallsyms] [k] __ext4_journal_get_write_access | --- __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty touch_atime ext4_file_mmap mmap_region do_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap _nl_find_locale __GI_setlocale 0x40124c 0.05% tar [kernel.kallsyms] [k] sysret_check | --- sysret_check __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] clockevents_program_event | --- clockevents_program_event tick_dev_program_event tick_program_event __hrtimer_start_range_ns hrtimer_start_range_ns hrtimer_start_expires.constprop.1 tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] pick_next_task_rt | --- pick_next_task_rt pick_next_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] account_entity_enqueue | --- account_entity_enqueue enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] need_resched | --- need_resched _cond_resched generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] find_lock_page | --- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] memcmp | --- memcmp search_dirblock ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.05% tar [kernel.kallsyms] [k] generic_segment_checks | --- generic_segment_checks generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% tar [kernel.kallsyms] [k] update_vsyscall | --- update_vsyscall do_timer tick_do_update_jiffies64 tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_lock ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% swapper [kernel.kallsyms] [k] hrtimer_start_expires.constprop.1 | --- hrtimer_start_expires.constprop.1 cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] __page_cache_alloc | --- __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] avc_has_perm_noaudit | --- avc_has_perm_noaudit avc_has_perm inode_has_perm selinux_inode_permission security_inode_exec_permission exec_permission link_path_walk path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.05% tar [kernel.kallsyms] [k] deactivate_slab | --- deactivate_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% swapper [kernel.kallsyms] [k] bio_free | --- bio_free bio_fs_destructor bio_put mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar libc-2.13.90.so [.] __GI___libc_read | --- __GI___libc_read 0.05% tar [kernel.kallsyms] [k] selinux_capable | --- selinux_capable ns_capable capable inode_change_ok ext4_setattr notify_change chown_common sys_fchown system_call_fastpath __GI___fchown 0.05% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave | --- _raw_read_lock_irqsave dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] scsi_handle_queue_ramp_up | --- scsi_handle_queue_ramp_up scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] dm_table_any_congested | --- dm_table_any_congested dm_any_congested page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% tar [kernel.kallsyms] [k] __wake_up_common | --- __wake_up_common __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] acct_update_integrals | --- acct_update_integrals __account_system_time irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt blk_finish_plug generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] ns_to_timespec | --- ns_to_timespec intel_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.05% tar [kernel.kallsyms] [k] _raw_read_unlock_irqrestore | --- _raw_read_unlock_irqrestore dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% tar [kernel.kallsyms] [k] __add_bdi_stat | --- __add_bdi_stat __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.05% swapper [kernel.kallsyms] [k] getnstimeofday | --- getnstimeofday ktime_get_real intel_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.04% tar [kernel.kallsyms] [k] get_page | --- get_page add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] timekeeping_get_ns | --- timekeeping_get_ns delayacct_end __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] trylock_page | --- trylock_page lock_page find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% swapper [kernel.kallsyms] [k] native_sched_clock | --- native_sched_clock sched_clock_cpu account_system_vtime irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.04% tar [kernel.kallsyms] [k] native_load_tls | --- native_load_tls __GI___libc_read 0.04% tar [kernel.kallsyms] [k] files_lglock_local_unlock_cpu | --- files_lglock_local_unlock_cpu file_sb_list_del fput filp_close sys_close system_call_fastpath __GI___close 0.04% tar [kernel.kallsyms] [k] release_pages | --- release_pages pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% jbd2/dm-2-8 [kernel.kallsyms] [k] bit_spin_lock.constprop.22 | --- bit_spin_lock.constprop.22 __slab_free kmem_cache_free __journal_remove_journal_head jbd2_journal_remove_journal_head journal_clean_one_cp_list __jbd2_journal_clean_checkpoint_list jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] pick_next_task_fair | --- pick_next_task_fair schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% swapper [kernel.kallsyms] [k] ktime_get_real | --- ktime_get_real cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.04% tar [kernel.kallsyms] [k] tick_do_update_jiffies64 | --- tick_do_update_jiffies64 tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags arch_local_irq_save irqtime_account_process_tick account_idle_ticks tick_nohz_restart_sched_tick cpu_idle start_secondary 0.03% sleep [kernel.kallsyms] [k] mem_cgroup_uncharge_start | --- mem_cgroup_uncharge_start exit_mmap mmput flush_old_exec load_elf_binary search_binary_handler do_execve sys_execve stub_execve __execve 0.02% swapper [kernel.kallsyms] [k] irq_enter | --- irq_enter apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary # # (For a higher level overview, try: perf report --sort comm,dso) # -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 16:50 ` James Bottomley @ 2011-04-28 16:56 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:56 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote: > This is the output of perf record -g -a -f sleep 5 > > (hopefully the list won't choke) Um, this one actually shows kswapd James --- # Events: 6K cycles # # Overhead Command Shared Object Symbol # ........ ........... ................... ....................................... # 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab | --- shrink_slab | |--99.91%-- kswapd | kthread | kernel_thread_helper --0.09%-- [...] 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone | --- shrink_zone | |--99.46%-- kswapd | kthread | kernel_thread_helper | --0.54%-- kthread kernel_thread_helper 7.70% kswapd0 [kernel.kallsyms] [k] kswapd | --- kswapd kthread kernel_thread_helper 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe | --- zone_watermark_ok_safe | |--72.66%-- kswapd | kthread | kernel_thread_helper | |--20.88%-- sleeping_prematurely.part.12 | kswapd | kthread | kernel_thread_helper | --6.46%-- kthread kernel_thread_helper 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock | |--77.49%-- _raw_spin_lock | | | |--51.85%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | --48.15%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--19.47%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--1.73%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--0.88%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper --0.44%-- [...] 3.34% kswapd0 [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--48.58%-- _raw_spin_unlock | | | |--80.77%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --19.23%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--17.63%-- _raw_spin_unlock_irqrestore | | | |--52.96%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --47.04%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--13.83%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--6.67%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--5.54%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--2.76%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--2.24%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--1.68%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--0.55%-- irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | --0.54%-- mem_cgroup_charge_statistics __mem_cgroup_uncharge_common mem_cgroup_uncharge_cache_page __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 2.96% kswapd0 [kernel.kallsyms] [k] __zone_watermark_ok | --- __zone_watermark_ok | |--85.77%-- zone_watermark_ok_safe | | | |--74.67%-- kswapd | | kthread | | kernel_thread_helper | | | --25.33%-- sleeping_prematurely.part.12 | kswapd | kthread | kernel_thread_helper | |--11.78%-- kswapd | kthread | kernel_thread_helper | --2.45%-- sleeping_prematurely.part.12 kswapd kthread kernel_thread_helper 2.67% kswapd0 [kernel.kallsyms] [k] global_dirty_limits | --- global_dirty_limits | |--97.28%-- throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | --2.72%-- shrink_zone kswapd kthread kernel_thread_helper 2.30% kswapd0 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--30.33%-- _raw_spin_lock | | | |--52.29%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --47.71%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--28.79%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--24.01%-- _raw_spin_lock_irqsave | | | |--63.28%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --36.72%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--6.31%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--4.82%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--3.28%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--1.64%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | --0.82%-- finish_wait kswapd kthread kernel_thread_helper 2.02% kswapd0 [kernel.kallsyms] [k] up_read | --- up_read | |--95.48%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --4.52%-- kswapd kthread kernel_thread_helper 2.01% kswapd0 [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip | |--37.59%-- sub_preempt_count | | | |--66.19%-- _raw_spin_unlock | | | | | |--55.77%-- mem_cgroup_soft_limit_reclaim | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --44.23%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--17.13%-- _raw_spin_unlock_irqrestore | | | | | |--57.38%-- finish_wait | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --42.62%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --16.68%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--32.16%-- add_preempt_count | | | |--71.32%-- _raw_spin_lock | | | | | |--56.02%-- mem_cgroup_soft_limit_reclaim | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --43.98%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--17.25%-- _raw_spin_lock_irqsave | | | | | |--83.09%-- finish_wait | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --16.91%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --11.43%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--8.20%-- _raw_spin_unlock | | | |--66.09%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --33.91%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--7.42%-- _raw_spin_lock | | | |--62.14%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | --37.86%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--5.55%-- _raw_spin_lock_irqsave | | | |--66.67%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --33.33%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--4.53%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--3.62%-- _raw_spin_unlock_irqrestore | | | |--51.20%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --48.80%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --0.94%-- _raw_spin_unlock_irq shrink_zone kswapd kthread kernel_thread_helper 1.88% kswapd0 [kernel.kallsyms] [k] zone_nr_lru_pages | --- zone_nr_lru_pages | |--85.21%-- shrink_zone | kswapd | kthread | kernel_thread_helper | --14.79%-- kswapd kthread kernel_thread_helper 1.79% kswapd0 [kernel.kallsyms] [k] down_read_trylock | --- down_read_trylock | |--92.80%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --7.20%-- kswapd kthread kernel_thread_helper 1.69% kswapd0 [kernel.kallsyms] [k] mutex_trylock | --- mutex_trylock i915_gem_inactive_shrink shrink_slab kswapd kthread kernel_thread_helper 1.66% kswapd0 [kernel.kallsyms] [k] sleeping_prematurely.part.12 | --- sleeping_prematurely.part.12 | |--93.41%-- kswapd | kthread | kernel_thread_helper | --6.59%-- kthread kernel_thread_helper 1.54% kswapd0 [kernel.kallsyms] [k] find_next_bit | --- find_next_bit | |--84.45%-- cpumask_next | zone_watermark_ok_safe | kswapd | kthread | kernel_thread_helper | --15.55%-- zone_watermark_ok_safe kswapd kthread kernel_thread_helper 1.51% kswapd0 [kernel.kallsyms] [k] throttle_vm_writeout | --- throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 1.48% kswapd0 [kernel.kallsyms] [k] mb_cache_shrink_fn | --- mb_cache_shrink_fn | |--96.31%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --3.69%-- kswapd kthread kernel_thread_helper 1.31% kswapd0 [kernel.kallsyms] [k] mem_cgroup_soft_limit_reclaim | --- mem_cgroup_soft_limit_reclaim | |--97.19%-- kswapd | kthread | kernel_thread_helper | --2.81%-- kthread kernel_thread_helper 1.26% kswapd0 [kernel.kallsyms] [k] mutex_unlock | --- mutex_unlock | |--98.57%-- i915_gem_inactive_shrink | shrink_slab | kswapd | kthread | kernel_thread_helper | --1.43%-- shrink_slab kswapd kthread kernel_thread_helper 1.19% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--47.70%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--47.65%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --4.65%-- kswapd kthread kernel_thread_helper 1.14% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock | |--59.71%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--38.63%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | --1.66%-- kswapd kthread kernel_thread_helper 0.96% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_largest_soft_limit_node | --- __mem_cgroup_largest_soft_limit_node | |--59.47%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | --40.53%-- kswapd kthread kernel_thread_helper 0.88% kswapd0 [kernel.kallsyms] [k] in_lock_functions | --- in_lock_functions | |--43.80%-- get_parent_ip | | | |--52.47%-- add_preempt_count | | | | | |--72.99%-- _raw_spin_lock | | | | | | | |--62.71%-- mem_cgroup_soft_limit_reclaim | | | | kswapd | | | | kthread | | | | kernel_thread_helper | | | | | | | --37.29%-- mb_cache_shrink_fn | | | shrink_slab | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | |--18.30%-- _raw_spin_lock_irq | | | shrink_zone | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --8.71%-- _raw_spin_lock_irqsave | | prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --47.53%-- sub_preempt_count | | | |--49.81%-- _raw_spin_unlock | | mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--30.32%-- _raw_spin_unlock_irqrestore | | finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --19.86%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--29.12%-- add_preempt_count | | | |--49.50%-- _raw_spin_lock | | | | | |--56.31%-- mem_cgroup_soft_limit_reclaim | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --43.69%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--29.19%-- _raw_spin_lock_irq | | shrink_zone | | kswapd | | kthread | | kernel_thread_helper | | | --21.32%-- _raw_spin_lock_irqsave | | | |--65.66%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --34.34%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --27.07%-- sub_preempt_count | |--62.26%-- _raw_spin_unlock | | | |--50.27%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | --49.73%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--22.85%-- _raw_spin_unlock_irqrestore | | | |--67.34%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --32.66%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --14.89%-- _raw_spin_unlock_irq shrink_zone kswapd kthread kernel_thread_helper 0.81% kswapd0 [kernel.kallsyms] [k] cpumask_next | --- cpumask_next | |--79.81%-- zone_watermark_ok_safe | kswapd | kthread | kernel_thread_helper | --20.19%-- kswapd kthread kernel_thread_helper 0.79% kswapd0 [kernel.kallsyms] [k] zone_reclaimable_pages | --- zone_reclaimable_pages | |--93.08%-- kswapd | kthread | kernel_thread_helper | --6.92%-- kthread kernel_thread_helper 0.77% kswapd0 [i915] [k] i915_gem_inactive_shrink | --- i915_gem_inactive_shrink | |--97.55%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --2.45%-- kswapd kthread kernel_thread_helper 0.76% kswapd0 [kernel.kallsyms] [k] prepare_to_wait | --- prepare_to_wait | |--97.54%-- kswapd | kthread | kernel_thread_helper | --2.46%-- kthread kernel_thread_helper 0.74% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore | |--79.88%-- _raw_spin_unlock_irqrestore | | | |--62.49%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --37.51%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--15.16%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --4.96%-- finish_wait kswapd kthread kernel_thread_helper 0.65% kswapd0 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 | |--20.12%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--20.09%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--14.28%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--14.16%-- _raw_spin_unlock | | | |--80.68%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --19.32%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--11.40%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--8.63%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--8.58%-- _raw_spin_unlock_irqrestore | | | |--66.13%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --33.87%-- finish_wait | kswapd | kthread | kernel_thread_helper | --2.74%-- _raw_spin_unlock_irq shrink_zone kswapd kthread kernel_thread_helper 0.63% tar [kernel.kallsyms] [k] copy_user_generic_string | --- copy_user_generic_string | |--56.11%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --43.89%-- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.59% kswapd0 [kernel.kallsyms] [k] test_tsk_thread_flag | --- test_tsk_thread_flag | |--50.16%-- kswapd | kthread | kernel_thread_helper | |--46.73%-- try_to_freeze | kswapd | kthread | kernel_thread_helper | --3.11%-- kthread kernel_thread_helper 0.59% kswapd0 [kernel.kallsyms] [k] shrink_icache_memory | --- shrink_icache_memory | |--81.19%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --18.81%-- kswapd kthread kernel_thread_helper 0.52% kswapd0 [kernel.kallsyms] [k] shrink_dqcache_memory | --- shrink_dqcache_memory | |--89.35%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --10.65%-- kswapd kthread kernel_thread_helper 0.50% kswapd0 [kernel.kallsyms] [k] zone_clear_flag | --- zone_clear_flag | |--77.78%-- kswapd | kthread | kernel_thread_helper | --22.22%-- kthread kernel_thread_helper 0.45% kswapd0 [kernel.kallsyms] [k] kthread_should_stop | --- kthread_should_stop | |--54.28%-- kswapd | kthread | kernel_thread_helper | --45.72%-- kthread kernel_thread_helper 0.44% kswapd0 [kernel.kallsyms] [k] global_page_state | --- global_page_state | |--33.60%-- throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | |--29.47%-- determine_dirtyable_memory | global_dirty_limits | throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | |--20.52%-- shrink_zone | kswapd | kthread | kernel_thread_helper | --16.41%-- global_dirty_limits throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 0.44% kswapd0 [kernel.kallsyms] [k] shrink_dcache_memory | --- shrink_dcache_memory shrink_slab kswapd kthread kernel_thread_helper 0.41% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore | --- _raw_spin_unlock_irqrestore | |--59.01%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --40.99%-- finish_wait kswapd kthread kernel_thread_helper 0.39% kswapd0 [kernel.kallsyms] [k] global_reclaimable_pages | --- global_reclaimable_pages determine_dirtyable_memory global_dirty_limits throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 0.37% kswapd0 [kernel.kallsyms] [k] finish_wait | --- finish_wait | |--95.03%-- kswapd | kthread | kernel_thread_helper | --4.97%-- kthread kernel_thread_helper 0.33% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock | |--60.70%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | --39.30%-- mb_cache_shrink_fn shrink_slab kswapd kthread kernel_thread_helper 0.31% kswapd0 [kernel.kallsyms] [k] arch_local_irq_enable | --- arch_local_irq_enable | |--76.94%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | --23.06%-- shrink_zone kswapd kthread kernel_thread_helper 0.28% kswapd0 [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save | |--93.18%-- _raw_spin_lock_irqsave | | | |--64.21%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --35.79%-- finish_wait | kswapd | kthread | kernel_thread_helper | --6.82%-- finish_wait kswapd kthread kernel_thread_helper 0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irq | --- _raw_spin_unlock_irq | |--92.00%-- shrink_zone | kswapd | kthread | kernel_thread_helper | --8.00%-- __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.22% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable | |--58.81%-- arch_local_irq_save | _raw_spin_lock_irqsave | | | |--70.97%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --29.03%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--33.06%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | --8.13%-- shrink_zone kswapd kthread kernel_thread_helper 0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irq | --- _raw_spin_lock_irq | |--58.62%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--33.19%-- kswapd | kthread | kernel_thread_helper | --8.19%-- __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.22% swapper [kernel.kallsyms] [k] poll_idle | --- poll_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.20% kswapd0 [kernel.kallsyms] [k] __list_add | --- __list_add | |--90.97%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --9.03%-- free_hot_cold_page __pagevec_free free_page_list shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.15% tar [kernel.kallsyms] [k] zero_user_segments | --- zero_user_segments __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% kswapd0 [kernel.kallsyms] [k] rb_last | --- rb_last mem_cgroup_soft_limit_reclaim kswapd kthread kernel_thread_helper 0.14% swapper [kernel.kallsyms] [k] intel_idle | --- intel_idle cpuidle_idle_call cpu_idle | |--65.05%-- rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --34.95%-- start_secondary 0.13% kswapd0 [kernel.kallsyms] [k] get_reclaim_stat | --- get_reclaim_stat | |--57.55%-- kswapd | kthread | kernel_thread_helper | --42.45%-- shrink_zone kswapd kthread kernel_thread_helper 0.12% perf [kernel.kallsyms] [k] selinux_file_alloc_security | --- selinux_file_alloc_security security_file_alloc get_empty_filp path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0x429beb 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.12% tar [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--18.90%-- _raw_spin_lock | | | |--63.01%-- ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--20.46%-- get_page_from_freelist | | __alloc_pages_nodemask | | alloc_pages_current | | __page_cache_alloc | | grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --16.53%-- ext4_ext_map_blocks | ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--13.68%-- __lru_cache_add | add_to_page_cache_lru | | | |--51.23%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --48.77%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--13.32%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--12.87%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--9.79%-- _raw_spin_lock_irq | | | |--62.37%-- add_to_page_cache_locked | | add_to_page_cache_lru | | grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --37.63%-- __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.52%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.32%-- lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.69%-- __percpu_counter_add | __add_bdi_stat | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.54%-- bit_spin_lock | lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.41%-- create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.32%-- _raw_read_lock_irqsave | dm_get_live_table | __split_and_process_bio | dm_request | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.29%-- _raw_read_lock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --2.33%-- dput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.11% kswapd0 [kernel.kallsyms] [k] try_to_freeze | --- try_to_freeze | |--83.76%-- kswapd | kthread | kernel_thread_helper | --16.24%-- kthread kernel_thread_helper 0.10% swapper [kernel.kallsyms] [k] ahci_interrupt | --- ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% kswapd0 [kernel.kallsyms] [k] determine_dirtyable_memory | --- determine_dirtyable_memory | |--80.46%-- global_dirty_limits | throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | --19.54%-- throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 0.08% tar [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--27.37%-- _raw_spin_unlock | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.90%-- blk_dequeue_request | blk_start_request | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--9.76%-- bit_spin_unlock | unlock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.85%-- _raw_spin_unlock_irq | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--6.81%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.86%-- _raw_spin_unlock_irqrestore | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.51%-- _raw_read_unlock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.37%-- jbd_unlock_bh_state | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | |--4.68%-- deactivate_slab | __slab_alloc | kmem_cache_alloc | alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.15%-- __srcu_read_unlock | fsnotify | fsnotify_modify | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.72%-- radix_tree_preload_end | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.62%-- bit_spin_unlock | jbd2_journal_add_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --3.41%-- pagefault_enable iov_iter_copy_from_user_atomic generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% swapper [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--52.45%-- _raw_spin_lock | get_next_timer_interrupt | tick_nohz_stop_sched_tick | irq_exit | smp_call_function_single_interrupt | call_function_single_interrupt | cpuidle_idle_call | cpu_idle | start_secondary | |--26.09%-- bit_spin_lock.constprop.22 | __slab_free | kmem_cache_free | ext4_free_io_end | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --21.46%-- __percpu_counter_add __prop_inc_percpu_max test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% tar [kernel.kallsyms] [k] kmem_cache_alloc | --- kmem_cache_alloc | |--74.26%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.98%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.88%-- mempool_alloc_slab | mempool_alloc | scsi_sg_alloc | __sg_alloc_table | scsi_alloc_sgtable | scsi_init_sgtable | scsi_init_io | scsi_setup_fs_cmnd | sd_prep_fn | blk_peek_request | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --5.87%-- selinux_inode_alloc_security security_inode_alloc inode_init_always alloc_inode new_inode ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.07% kswapd0 [kernel.kallsyms] [k] __free_one_page | --- __free_one_page free_pcppages_bulk free_hot_cold_page __pagevec_free free_page_list shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.07% swapper [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--56.43%-- __percpu_counter_add | __add_bdi_stat | test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--17.99%-- _raw_spin_unlock_irqrestore | __hrtimer_start_range_ns | hrtimer_start_range_ns | hrtimer_start_expires.constprop.1 | tick_nohz_restart_sched_tick | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--15.61%-- _raw_spin_unlock | | | |--55.21%-- handle_irq_event | | handle_edge_irq | | handle_irq | | do_IRQ | | common_interrupt | | cpuidle_idle_call | | cpu_idle | | rest_init | | start_kernel | | x86_64_start_reservations | | x86_64_start_kernel | | | --44.79%-- scheduler_tick | update_process_times | tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--5.91%-- try_to_wake_up | default_wake_function | autoremove_wake_function | wake_bit_function | __wake_up_common | __wake_up | __wake_up_bit | unlock_page | mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --4.06%-- default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] mark_page_accessed | --- mark_page_accessed | |--87.24%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.42%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --6.35%-- __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] get_page_from_freelist | --- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--54.79%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --45.21%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] __ext4_get_inode_loc | --- __ext4_get_inode_loc ext4_get_inode_loc | |--55.97%-- ext4_reserve_inode_write | ext4_mark_inode_dirty | | | |--76.52%-- ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --23.48%-- ext4_ext_dirty | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --44.03%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar tar [.] 0x2aa04 | --- 0x40753f | --- 0x4055c7 | --- 0x42aa04 | --- 0x4166d8 0x417081 | --- 0x40fdfb | --- 0x405f26 | --- 0x4332d6 | --- 0x40fe1a | --- 0x408ab1 | --- 0x40fdb8 | --- 0x408a87 | --- 0x4100c5 | --- 0x407564 0.06% perf [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4293b8 0x429c0a 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.05% tar [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock | |--79.42%-- _raw_spin_lock_irq | | | |--57.06%-- __set_page_dirty | | mark_buffer_dirty | | __block_commit_write | | block_write_end | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --42.94%-- add_to_page_cache_locked | add_to_page_cache_lru | | | |--52.51%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --47.49%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--15.99%-- _raw_spin_lock | | | |--60.63%-- ext4_ext_map_blocks | | ext4_map_blocks | | ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --39.37%-- inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --4.59%-- inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28 | --- __mem_cgroup_commit_charge.constprop.28 | |--91.58%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--51.67%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --48.33%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --8.42%-- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] radix_tree_lookup_element | --- radix_tree_lookup_element | |--69.74%-- radix_tree_lookup_slot | find_get_page | | | |--58.70%-- generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --41.30%-- find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.45%-- radix_tree_lookup | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --12.81%-- find_get_page generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] __alloc_pages_nodemask | --- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--71.39%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --28.61%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed alloc_pages_current __page_cache_alloc | |--70.79%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.21%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% swapper [kernel.kallsyms] [k] load_balance | --- load_balance rebalance_domains run_rebalance_domains __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.04% flush-253:2 [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit | |--89.40%-- unlock_page | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --10.60%-- wake_up_bit unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% tar [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip | |--29.28%-- sub_preempt_count | | | |--59.65%-- pagefault_enable | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --40.35%-- bit_spin_unlock | jbd2_journal_put_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--19.16%-- avc_has_perm_noaudit | avc_has_perm | inode_has_perm | selinux_inode_permission | security_inode_exec_permission | exec_permission | link_path_walk | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | |--19.01%-- _raw_spin_lock | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.20%-- bit_spin_lock | lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --15.36%-- add_preempt_count file_read_actor generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty | --- ext4_mark_iloc_dirty ext4_mark_inode_dirty | |--77.33%-- ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--13.35%-- ext4_new_inode | ext4_create | vfs_create | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --9.32%-- ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.04% kswapd0 [kernel.kallsyms] [k] native_write_msr_safe | --- native_write_msr_safe paravirt_write_msr intel_pmu_disable_all x86_pmu_disable perf_pmu_disable perf_event_task_tick scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--50.07%-- zone_watermark_ok_safe | sleeping_prematurely.part.12 | kswapd | kthread | kernel_thread_helper | --49.93%-- kswapd kthread kernel_thread_helper 0.04% swapper [kernel.kallsyms] [k] ext4_end_bio | --- ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.04% kswapd0 [kernel.kallsyms] [k] __list_del_entry | --- __list_del_entry finish_wait kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] mcount | --- mcount _raw_spin_lock_irq shrink_zone kswapd kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] __percpu_counter_add | --- __percpu_counter_add | |--87.00%-- ext4_claim_free_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --13.00%-- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% tar [kernel.kallsyms] [k] arch_read_unlock | --- arch_read_unlock | |--57.64%-- _raw_read_unlock_irqrestore | dm_get_live_table | dm_merge_bvec | __bio_add_page.part.2 | bio_add_page | do_mpage_readpage | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--31.54%-- _raw_read_unlock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --10.82%-- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% kswapd0 [kernel.kallsyms] [k] mem_cgroup_del_lru_list | --- mem_cgroup_del_lru_list mem_cgroup_del_lru isolate_lru_pages shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] test_and_set_bit | --- test_and_set_bit shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] __might_sleep | --- __might_sleep | |--22.45%-- kmem_cache_alloc | | | |--59.27%-- getname_flags | | getname | | do_sys_open | | sys_openat | | system_call_fastpath | | __GI___openat | | | --40.73%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--22.30%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--54.37%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --45.63%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--15.00%-- lock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--11.33%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--11.31%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.36%-- unmap_underlying_metadata | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.25%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% kswapd0 [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--50.69%-- __pagevec_free | free_page_list | shrink_page_list | shrink_inactive_list | shrink_zone | kswapd | kthread | kernel_thread_helper | --49.31%-- free_hot_cold_page __pagevec_free free_page_list shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_uncharge_common | --- __mem_cgroup_uncharge_common mem_cgroup_uncharge_cache_page __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] tag_get | --- tag_get | |--50.59%-- radix_tree_delete | __delete_from_page_cache | __remove_mapping | shrink_page_list | shrink_inactive_list | shrink_zone | kswapd | kthread | kernel_thread_helper | --49.41%-- __delete_from_page_cache __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.03% flush-253:2 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--51.77%-- _raw_read_lock_irqsave | dm_get_live_table | dm_merge_bvec | __bio_add_page.part.2 | bio_add_page | ext4_bio_write_page | mpage_da_submit_io | mpage_da_map_and_submit | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --48.23%-- blk_throtl_bio generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] __inc_zone_state | --- __inc_zone_state | |--49.98%-- __inc_zone_page_state | add_to_page_cache_locked | add_to_page_cache_lru | | | |--79.48%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --20.52%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--32.04%-- zone_statistics | get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --17.98%-- account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% tar [kernel.kallsyms] [k] __list_del_entry | --- __list_del_entry | |--56.86%-- list_del | __rmqueue | get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--78.10%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --21.90%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.14%-- __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% flush-253:2 [kernel.kallsyms] [k] dm_table_find_target | --- dm_table_find_target dm_merge_bvec __bio_add_page.part.2 bio_add_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] jbd_lock_bh_state | --- jbd_lock_bh_state | |--61.84%-- do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--72.93%-- generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --27.07%-- ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --38.16%-- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% flush-253:2 [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% flush-253:2 [kernel.kallsyms] [k] radix_tree_lookup_element | --- radix_tree_lookup_element radix_tree_lookup_slot find_get_page __find_get_block_slow unmap_underlying_metadata mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] fcheck_files | --- fcheck_files fget_light | |--71.87%-- sys_read | system_call_fastpath | __GI___libc_read | --28.13%-- sys_write system_call_fastpath __GI___libc_write 0.03% flush-253:2 [kernel.kallsyms] [k] atomic_inc | --- atomic_inc ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% flush-253:2 [kernel.kallsyms] [k] lock_page | --- lock_page ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero | |--70.70%-- release_pages | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | | | |--77.62%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --22.38%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.30%-- put_page mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] zone_watermark_ok | --- zone_watermark_ok get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--84.29%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --15.71%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] start_this_handle | --- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--86.30%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --13.70%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages | --- ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed | |--74.79%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--65.17%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --34.83%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --25.21%-- alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% tar [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--57.28%-- load_balance | schedule | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--25.57%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --17.15%-- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% swapper [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.03% tar [kernel.kallsyms] [k] avc_has_perm_noaudit | --- avc_has_perm_noaudit avc_has_perm | |--82.82%-- inode_has_perm | | | |--71.72%-- dentry_has_perm | | selinux_inode_getattr | | security_inode_getattr | | vfs_getattr | | vfs_fstat | | sys_newfstat | | system_call_fastpath | | __GI___fxstat64 | | | --28.28%-- selinux_inode_permission | security_inode_exec_permission | exec_permission | link_path_walk | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --17.18%-- may_create selinux_inode_create security_inode_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.03% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] do_mpage_readpage | --- do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] list_del | --- list_del | |--55.44%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--67.14%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --32.86%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--28.53%-- ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --16.02%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] __rmqueue | --- __rmqueue | |--77.01%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --22.99%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge_lrucare | --- __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--83.13%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --16.87%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] update_page_reclaim_stat | --- update_page_reclaim_stat ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--75.26%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --24.74%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% tar [kernel.kallsyms] [k] radix_tree_insert | --- radix_tree_insert add_to_page_cache_locked add_to_page_cache_lru | |--81.25%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --18.75%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] add_to_page_cache_locked | --- add_to_page_cache_locked add_to_page_cache_lru | |--62.06%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --37.94%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue | |--58.60%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.40%-- unlock_page generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] up_read | --- up_read | |--52.38%-- ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.62%-- ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% swapper [kernel.kallsyms] [k] elv_completed_request | --- elv_completed_request blk_finish_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save | |--84.09%-- task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --15.91%-- account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] ext4_map_blocks | --- ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mcount | --- mcount _raw_spin_unlock_irqrestore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] clear_buffer_dirty | --- clear_buffer_dirty ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list | --- mem_cgroup_add_lru_list add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% swapper [kernel.kallsyms] [k] blk_complete_request | --- blk_complete_request scsi_done ata_scsi_qc_complete __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% swapper [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] touch_atime | --- touch_atime | |--90.35%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --9.65%-- do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% perf ld-2.13.90.so [.] _dl_fixup | --- _dl_fixup 0.02% tar [kernel.kallsyms] [k] __block_write_begin | --- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% kswapd0 [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt i915_gem_inactive_shrink shrink_slab kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] apic_timer_interrupt | --- apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kworker/2:1 [kernel.kallsyms] [k] ns_to_timeval | --- ns_to_timeval get_cpu_iowait_time_us do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] task_rq_lock | --- task_rq_lock try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt zone_watermark_ok_safe kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] hrtimer_run_queues | --- hrtimer_run_queues run_local_timers update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] next_zones_zonelist | --- next_zones_zonelist | |--71.41%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --28.59%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable run_local_timers update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] __rcu_pending | --- __rcu_pending rcu_check_callbacks update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] write_cache_pages_da | --- write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] intel_pmu_disable_all | --- intel_pmu_disable_all x86_pmu_disable perf_pmu_disable perf_event_task_tick scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt sleeping_prematurely.part.12 kswapd kthread kernel_thread_helper 0.02% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags arch_local_irq_save irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore call_rcu d_free dentry_kill shrink_dentry_list __shrink_dcache_sb shrink_dcache_memory shrink_slab kswapd kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] SetPageLRU | --- SetPageLRU | |--66.95%-- ____pagevec_lru_add_fn | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | | | |--56.88%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --43.12%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --33.05%-- pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% kswapd0 [kernel.kallsyms] [k] run_timer_softirq | --- run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt i915_gem_inactive_shrink shrink_slab kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page | --- __isolate_lru_page isolate_lru_pages shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] radix_tree_delete | --- radix_tree_delete __delete_from_page_cache __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear | --- radix_tree_tag_clear test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_map_blocks | --- ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep | |--51.15%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | | | |--50.51%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_ext_truncate | | ext4_truncate | | ext4_evict_inode | | evict | | iput | | do_unlinkat | | sys_unlinkat | | system_call_fastpath | | unlinkat | | | --49.49%-- ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.85%-- kmem_cache_alloc jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% kswapd0 [kernel.kallsyms] [k] mem_cgroup_uncharge_cache_page | --- mem_cgroup_uncharge_cache_page shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] update_curr | --- update_curr task_tick_fair scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] timerqueue_del | --- timerqueue_del __remove_hrtimer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_unlock_irqrestore prepare_to_wait kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] ack_APIC_irq | --- ack_APIC_irq smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kworker/2:1 [kernel.kallsyms] [k] too_many_workers | --- too_many_workers worker_enter_idle worker_thread kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] update_isolated_counts | --- update_isolated_counts shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 _raw_spin_unlock_irqrestore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] jbd2_journal_get_write_access | --- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_ext_dirty ext4_ext_insert_extent ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] __lookup | --- __lookup radix_tree_gang_lookup_slot find_get_pages pagevec_lookup mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_pages_tag pagevec_lookup_tag ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore _raw_spin_unlock_irqrestore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_ext_map_blocks | --- ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] jbd_lock_bh_state | --- jbd_lock_bh_state jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mb_mark_diskspace_used ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count jbd_unlock_bh_state jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mb_mark_diskspace_used ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] alloc_pages_current | --- alloc_pages_current __page_cache_alloc | |--72.21%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --27.79%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page | --- ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_mb_complex_scan_group | --- ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% top [kernel.kallsyms] [k] __d_lookup | --- __d_lookup d_lookup proc_fill_cache proc_pid_readdir proc_root_readdir vfs_readdir sys_getdents system_call_fastpath __getdents64 0.02% flush-253:2 [kernel.kallsyms] [k] kmem_cache_alloc | --- kmem_cache_alloc ext4_init_io_end ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_page | |--51.46%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.54%-- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_extent.constprop.12 | --- mb_find_extent.constprop.12 ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] generic_file_buffered_write | --- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% top libc-2.13.90.so [.] __GI_vfprintf | --- __GI_vfprintf ___vsprintf_chk 0.02% tar [kernel.kallsyms] [k] ext4_da_get_block_prep | --- ext4_da_get_block_prep | |--75.27%-- __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --24.73%-- ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% top [kernel.kallsyms] [k] getname_flags | --- getname_flags getname do_sys_open sys_open system_call_fastpath __GI___libc_open 0.02% swapper [kernel.kallsyms] [k] __count_vm_events | --- __count_vm_events __free_pages __free_slab discard_slab __slab_free kmem_cache_free ext4_free_io_end ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% swapper [kernel.kallsyms] [k] tick_check_idle | --- tick_check_idle irq_enter | |--73.47%-- smp_call_function_single_interrupt | call_function_single_interrupt | cpuidle_idle_call | cpu_idle | start_secondary | --26.53%-- smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% flush-253:2 [kernel.kallsyms] [k] find_next_zero_bit | --- find_next_zero_bit mb_find_next_zero_bit ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] arch_read_lock | --- arch_read_lock _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_buddy | --- mb_find_buddy mb_mark_used ext4_mb_use_best_found ext4_mb_check_limits ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% swapper [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock __prop_inc_percpu_max test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] mark_buffer_dirty | --- mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] new_slab | --- new_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% swapper [kernel.kallsyms] [k] __do_softirq | --- __do_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% swapper [kernel.kallsyms] [k] select_task_rq_fair | --- select_task_rq_fair select_task_rq try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] do_get_write_access | --- do_get_write_access | |--70.88%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | | | |--54.37%-- ext4_new_inode | | ext4_create | | vfs_create | | do_last | | path_openat | | do_filp_open | | do_sys_open | | sys_openat | | system_call_fastpath | | __GI___openat | | | --45.63%-- ext4_reserve_inode_write | ext4_orphan_add | ext4_unlink | vfs_unlink | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --29.12%-- __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] in_lock_functions | --- in_lock_functions | |--58.27%-- get_parent_ip | add_preempt_count | | | |--56.37%-- jbd_lock_bh_state | | jbd2_journal_dirty_metadata | | __ext4_handle_dirty_metadata | | ext4_mark_iloc_dirty | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --43.63%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.73%-- sub_preempt_count bit_spin_unlock jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata | --- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] ext4_ext_map_blocks | --- ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] block_write_end | --- block_write_end | |--67.22%-- ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --32.78%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mb_test_bit | --- mb_test_bit mb_find_order_for_block mb_find_extent.constprop.12 ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] attach_page_buffers | --- attach_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __phys_addr | --- __phys_addr virt_to_head_page kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] do_sync_read | --- do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs | --- swiotlb_map_sg_attrs ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] jbd2__journal_start | --- jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--73.97%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --26.03%-- ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] iov_iter_advance | --- iov_iter_advance __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] generic_getxattr | --- generic_getxattr security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% jbd2/dm-2-8 [kernel.kallsyms] [k] elv_rq_merge_ok | --- elv_rq_merge_ok elv_try_merge __make_request generic_make_request submit_bio submit_bh jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] clear_bit_unlock | --- clear_bit_unlock unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue | |--52.26%-- unlock_page | mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --47.74%-- mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] ext4_xattr_find_entry | --- ext4_xattr_find_entry ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock lock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--76.11%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --23.89%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] writeback_single_inode | --- writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] find_get_page | --- find_get_page | |--70.62%-- __find_get_block_slow | __find_get_block | ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --29.38%-- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] __ext4_get_inode_loc | --- __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] fsnotify_modify | --- fsnotify_modify | |--73.71%-- vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --26.29%-- sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] blk_rq_map_sg | --- blk_rq_map_sg scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] get_page | --- get_page add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--44.23%-- blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--32.84%-- blk_run_queue | scsi_run_queue | scsi_next_command | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --22.93%-- pm_qos_request menu_select cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] sg_next | --- sg_next | |--36.30%-- ata_qc_issue | __ata_scsi_queuecmd | ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--32.06%-- swiotlb_map_sg_attrs | ata_qc_issue | __ata_scsi_queuecmd | ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --31.64%-- blk_rq_map_sg scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock | |--70.72%-- __prop_inc_single | task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.28%-- avc_has_perm_noaudit task_has_capability selinux_capable security_capable ns_capable capable setattr_copy ext4_setattr notify_change sys_fchmod system_call_fastpath __GI___fchmod 0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock | |--36.11%-- smp_apic_timer_interrupt | apic_timer_interrupt | __find_get_block_slow | unmap_underlying_metadata | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--34.75%-- scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --29.14%-- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] generic_file_aio_read | --- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] map_bh | --- map_bh ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] memcmp | --- memcmp | |--67.73%-- ext4_dx_find_entry | ext4_find_entry | ext4_lookup | d_alloc_and_lookup | __lookup_hash.part.3 | lookup_hash | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --32.27%-- search_dirblock ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] clone_endio | --- clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] __slab_free | --- __slab_free kmem_cache_free put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] nr_iowait_cpu | --- nr_iowait_cpu tick_nohz_stop_idle tick_check_idle irq_enter smp_call_function_single_interrupt call_function_single_interrupt cpuidle_idle_call cpu_idle start_secondary 0.01% swapper [kernel.kallsyms] [k] timekeeping_get_ns | --- timekeeping_get_ns getnstimeofday ktime_get_real intel_idle cpuidle_idle_call cpu_idle start_secondary 0.01% flush-253:2 [kernel.kallsyms] [k] crc16 | --- crc16 ext4_group_desc_csum ext4_mb_mark_diskspace_used ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] ata_scsi_queuecmd | --- ata_scsi_queuecmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] jbd2_journal_put_journal_head | --- jbd2_journal_put_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--60.89%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --39.11%-- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] ext4_da_write_end | --- ext4_da_write_end | |--68.79%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --31.21%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] acct_update_integrals | --- acct_update_integrals __account_system_time irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--51.45%-- get_parent_ip | add_preempt_count | __lru_cache_add | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.55%-- find_get_page __find_get_block_slow __find_get_block ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] __inc_zone_page_state | --- __inc_zone_page_state | |--61.16%-- account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --38.84%-- add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] account_idle_time | --- account_idle_time account_idle_ticks tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] recalc_bh_state | --- recalc_bh_state | |--62.12%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.88%-- alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero put_page put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] generic_segment_checks | --- generic_segment_checks | |--56.32%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --43.68%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save __free_pages __free_slab discard_slab __slab_free kmem_cache_free put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] rw_verify_area | --- rw_verify_area | |--65.20%-- vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --34.80%-- vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] file_update_time | --- file_update_time | |--70.97%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.03%-- generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% flush-253:2 [kernel.kallsyms] [k] alloc_tio | --- alloc_tio dm_request generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __mark_inode_dirty | --- __mark_inode_dirty __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore | --- _raw_spin_unlock_irqrestore __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke | --- jbd2_journal_cancel_revoke do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] lookup_page_cgroup | --- lookup_page_cgroup __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--53.32%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --46.68%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq | --- _raw_spin_unlock_irq add_to_page_cache_locked add_to_page_cache_lru | |--69.09%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --30.91%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __do_page_cache_readahead | --- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] ext4_acl_chmod | --- ext4_acl_chmod notify_change sys_fchmod system_call_fastpath __GI___fchmod 0.01% flush-253:2 [kernel.kallsyms] [k] start_this_handle | --- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] _raw_read_lock | --- _raw_read_lock | |--56.63%-- start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.37%-- jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] mix_pool_bytes_extract | --- mix_pool_bytes_extract add_timer_randomness add_disk_randomness blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] system_call_after_swapgs | --- system_call_after_swapgs __GI___libc_read 0.01% tar [kernel.kallsyms] [k] ext4_claim_free_blocks | --- ext4_claim_free_blocks | |--53.08%-- __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --46.92%-- ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] sys_write | --- sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] mcount | --- mcount | |--52.17%-- radix_tree_preload_end | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.83%-- _raw_spin_unlock ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] pagevec_lru_move_fn | --- pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head | --- jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __block_commit_write | --- __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] virt_to_head_page | --- virt_to_head_page | |--57.98%-- kmem_cache_free | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --42.02%-- jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] mb_test_bit | --- mb_test_bit | |--53.79%-- ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --46.21%-- mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc | |--57.00%-- ext4_mark_iloc_dirty | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.00%-- ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ahci_qc_prep | --- ahci_qc_prep | |--56.69%-- __ata_scsi_queuecmd | ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --43.31%-- ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] generic_write_end | --- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] iov_iter_fault_in_readable | --- iov_iter_fault_in_readable | |--51.45%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.55%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] inode_reserved_space | --- inode_reserved_space | |--54.74%-- inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --45.26%-- __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] update_curr | --- update_curr dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] atomic_sub | --- atomic_sub | |--56.22%-- __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.78%-- jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_get_inode_flags | --- ext4_get_inode_flags ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] prop_norm_single | --- prop_norm_single | |--55.85%-- task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.15%-- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] jbd_unlock_bh_state | --- jbd_unlock_bh_state | |--50.48%-- jbd2_journal_dirty_metadata | __ext4_handle_dirty_metadata | ext4_mark_iloc_dirty | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.52%-- do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable arch_local_irq_save _raw_spin_lock_irqsave | |--65.53%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --34.47%-- pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __generic_file_aio_write | --- __generic_file_aio_write | |--53.33%-- generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --46.67%-- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] test_and_set_bit | --- test_and_set_bit mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% flush-253:2 [kernel.kallsyms] [k] _raw_read_lock_irqsave | --- _raw_read_lock_irqsave dm_merge_bvec __bio_add_page.part.2 bio_add_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __find_get_block | --- __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--64.27%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --35.73%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __split_and_process_bio | --- __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] memcg_check_events | --- memcg_check_events __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--52.26%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.74%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] xattr_resolve_name | --- xattr_resolve_name generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% kworker/0:2 [kernel.kallsyms] [k] bit_cursor | --- bit_cursor fb_flashcursor process_one_work worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __set_page_dirty | --- __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_file_write | --- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock jbd2_journal_put_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] fget_light | --- fget_light sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] ext4_clear_inode_state | --- ext4_clear_inode_state ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% swapper [kernel.kallsyms] [k] ahci_scr_read | --- ahci_scr_read sata_scr_read sata_async_notification ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] add_to_page_cache_lru | --- add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags _local_bh_enable irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] current_fs_time | --- current_fs_time | |--50.34%-- touch_atime | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --49.66%-- file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] need_resched | --- need_resched _cond_resched generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] mb_clear_bit | --- mb_clear_bit | |--57.54%-- ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --42.46%-- mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] __list_add | --- __list_add get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] task_dirty_inc | --- task_dirty_inc __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] vfs_read | --- vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock | |--55.69%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.31%-- jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] paravirt_read_tsc | --- paravirt_read_tsc read_tsc timekeeping_get_ns ktime_get tick_nohz_stop_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] irq_entries_start | --- irq_entries_start cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic | --- iov_iter_copy_from_user_atomic generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] clear_buffer_new | --- clear_buffer_new | |--57.80%-- __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --42.20%-- block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave | --- _raw_read_lock_irqsave dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] __prop_inc_single | --- __prop_inc_single | |--52.30%-- account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.70%-- task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] dm_merge_bvec | --- dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] menu_select | --- menu_select | |--59.21%-- cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --40.79%-- cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] scsi_decide_disposition | --- scsi_decide_disposition scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] bit_spin_trylock.constprop.23 | --- bit_spin_trylock.constprop.23 get_partial_node __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] exit_idle | --- exit_idle do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io | --- mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% swapper [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock_irq schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] get_cwq | --- get_cwq __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_lock inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ____pagevec_lru_add_fn | --- ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--57.19%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --42.81%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] alloc_buffer_head | --- alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] generic_make_request | --- generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] rcu_irq_enter | --- rcu_irq_enter irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] leave_mm | --- leave_mm intel_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] __lru_cache_add | --- __lru_cache_add grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar libc-2.13.90.so [.] __GI___libc_write | --- __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] task_rq_lock | --- task_rq_lock try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] __elv_add_request | --- __elv_add_request blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] may_link | --- may_link selinux_inode_unlink security_inode_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] __find_get_block_slow | --- __find_get_block_slow unmap_underlying_metadata __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% flush-253:2 [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] delayacct_blkio_start | --- delayacct_blkio_start io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] current_umask | --- current_umask ext4_init_acl ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% tar [kernel.kallsyms] [k] linear_merge | --- linear_merge dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip _raw_spin_lock ext4_discard_preallocations ext4_da_update_reserve_space ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __blk_run_queue | --- __blk_run_queue blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] __page_cache_alloc | --- __page_cache_alloc ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore account_system_vtime irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] ext4_new_inode | --- ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% tar [kernel.kallsyms] [k] __slab_alloc | --- __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount | --- ext4_ext_calc_metadata_amount ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] task_of | --- task_of dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] tick_program_event | --- tick_program_event smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% perf [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit unlock_page generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.01% perf [kernel.kallsyms] [k] generic_write_end | --- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.01% kworker/0:2 [kernel.kallsyms] [k] native_read_msr_safe | --- native_read_msr_safe acpi_cpufreq_init do_drv_write acpi_cpufreq_target __cpufreq_driver_target do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_journal_start_sb | --- ext4_journal_start_sb ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_get_inode_loc | --- ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27 | --- __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] hrtick_update | --- hrtick_update enqueue_task activate_task try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page | --- mem_cgroup_get_reclaim_stat_from_page ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] bio_endio | --- bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] kfree | --- kfree jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_free_blocks | --- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] ext4_test_inode_flag | --- ext4_test_inode_flag ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] scsi_run_queue | --- scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% kworker/0:2 [cpufreq_ondemand] [k] do_dbs_timer | --- do_dbs_timer worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] alloc_page_buffers | --- alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] linear_map | --- linear_map __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag.constprop.7 | --- test_ti_thread_flag.constprop.7 smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] delayacct_blkio_end | --- delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] inode_permission | --- inode_permission vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% kworker/0:2 [kernel.kallsyms] [k] kobject_get | --- kobject_get __cpufreq_driver_getavg do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __wake_up_common | --- __wake_up_common __wake_up jbd2_journal_stop __ext4_journal_stop ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_read_inode_bitmap | --- ext4_read_inode_bitmap ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] mnt_add_count | --- mnt_add_count mntget path_get nameidata_to_filp do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% tar [kernel.kallsyms] [k] schedule | --- schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] try_to_wake_up | --- try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] inode_doinit_with_dentry | --- inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate ext4_add_nondir ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] radix_tree_preload | --- radix_tree_preload add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __mod_zone_page_state | --- __mod_zone_page_state add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] unmap_single | --- unmap_single swiotlb_unmap_sg_attrs ata_sg_clean __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar libc-2.13.90.so [.] _int_malloc | --- _int_malloc 0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state | --- ext4_test_inode_state ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata | --- __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] queue_unplugged | --- queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] mutex_lock | --- mutex_lock ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] crc16 | --- crc16 ext4_group_desc_csum ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] lock_page_killable | --- lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] lock_page | --- lock_page find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr | --- balance_dirty_pages_ratelimited_nr generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] setup_object | --- setup_object new_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] sd_prep_fn | --- sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] system_call_fastpath | --- system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] unlock_page | --- unlock_page generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] blk_throtl_bio | --- blk_throtl_bio generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ihold | --- ihold do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] mb_find_buddy | --- mb_find_buddy mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access | --- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock unlock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] fput_light | --- fput_light system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] audit_inode.constprop.6 | --- audit_inode.constprop.6 system_call_fastpath __GI___fchown 0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state | --- ext4_test_inode_state ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] system_call | --- system_call __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] ktime_get | --- ktime_get tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] radix_tree_lookup_slot | --- radix_tree_lookup_slot generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] apic_timer_interrupt | --- apic_timer_interrupt generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] arch_local_irq_enable | --- arch_local_irq_enable _raw_spin_unlock_irq __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] file_read_actor | --- file_read_actor generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] dquot_active | --- dquot_active __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% sleep [kernel.kallsyms] [k] tlb_gather_mmu | --- tlb_gather_mmu mmput exit_mm do_exit do_group_exit __wake_up_parent system_call_fastpath 0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] bvec_free_bs | --- bvec_free_bs dm_bio_destructor bio_put clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] ktime_get_real | --- ktime_get_real cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] generic_write_sync | --- generic_write_sync generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] _raw_spin_lock_irq | --- _raw_spin_lock_irq add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] posix_acl_release | --- posix_acl_release ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] ext4_dirty_inode | --- ext4_dirty_inode generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] test_clear_page_writeback | --- test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] deadline_dispatch_requests | --- deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] deadline_move_request | --- deadline_move_request deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% kworker/0:2 [kernel.kallsyms] [k] schedule | --- schedule worker_thread kthread kernel_thread_helper 0.00% tar [kernel.kallsyms] [k] blk_start_plug | --- blk_start_plug ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] set_buffer_uptodate | --- set_buffer_uptodate __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] strncpy_from_user | --- strncpy_from_user getname user_path_parent do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc drive_stat_acct __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] update_curr | --- update_curr enqueue_task_fair enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id rcu_process_callbacks __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] page_cache_async_readahead | --- page_cache_async_readahead do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] bvec_alloc_bs | --- bvec_alloc_bs clone_bio __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] page_mapping | --- page_mapping test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free mempool_free_slab mempool_free dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] prop_norm_percpu | --- prop_norm_percpu __prop_inc_percpu_max test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] bit_waitqueue | --- bit_waitqueue wake_up_bit unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] apic_write | --- apic_write ack_APIC_irq ack_apic_edge handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] native_write_msr_safe | --- native_write_msr_safe 0.00% swapper [kernel.kallsyms] [k] req_bio_endio | --- req_bio_endio blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] __wake_up | --- __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] mem_cgroup_disabled | --- mem_cgroup_disabled add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __make_request | --- __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] tick_check_oneshot_broadcast | --- tick_check_oneshot_broadcast tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] raise_softirq | --- raise_softirq update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt radix_tree_preload_end add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ext4_setattr | --- ext4_setattr notify_change utimes_common do_utimes sys_utimensat system_call_fastpath futimens 0.00% swapper [kernel.kallsyms] [k] native_sched_clock | --- native_sched_clock sched_clock_cpu account_system_vtime irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] read_tsc | --- read_tsc timekeeping_get_ns ktime_get tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] cap_inode_need_killpriv | --- cap_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __cycles_2_ns | --- __cycles_2_ns native_sched_clock sched_clock blk_rq_init get_request get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] load_balance | --- load_balance schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] read_tsc | --- read_tsc ktime_get_ts delayacct_end __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] scsi_sg_alloc | --- scsi_sg_alloc __sg_alloc_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] cpuacct_charge | --- cpuacct_charge update_curr dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] elv_rqhash_add | --- elv_rqhash_add __elv_add_request blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22 | --- test_ti_thread_flag.constprop.22 alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] ata_find_dev.part.3 | --- ata_find_dev.part.3 __ata_scsi_find_dev ata_scsi_find_dev ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] fsnotify | --- fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% kworker/0:2 [kernel.kallsyms] [k] ns_to_timeval | --- ns_to_timeval get_cpu_iowait_time_us do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.00% tar [kernel.kallsyms] [k] expand | --- expand __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] preempt_schedule | --- preempt_schedule blk_finish_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] max_io_len | --- max_io_len __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] release_pages | --- release_pages pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] atomic_long_add | --- atomic_long_add zone_page_state_add __inc_zone_state __inc_zone_page_state account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] zone_statistics | --- zone_statistics __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] str2hashbuf_signed | --- str2hashbuf_signed ext4fs_dirhash dx_probe ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% swapper [kernel.kallsyms] [k] ata_scsi_qc_complete | --- ata_scsi_qc_complete __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] ext4_has_free_blocks | --- ext4_has_free_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] deadline_add_request | --- deadline_add_request blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] pagefault_enable | --- pagefault_enable generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] deadline_merge | --- deadline_merge elv_merge __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] rb_next | --- rb_next timerqueue_del __remove_hrtimer remove_hrtimer.part.4 hrtimer_try_to_cancel hrtimer_cancel tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge | --- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] sched_clock_tick | --- sched_clock_tick sched_clock_idle_wakeup_event tick_nohz_stop_idle tick_check_idle irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] put_bh | --- put_bh __brelse brelse ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] get_page | --- get_page __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] hashtab_search | --- hashtab_search string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] ata_qc_new_init | --- ata_qc_new_init __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] sd_done | --- sd_done scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% sleep [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip add_preempt_count get_page_from_freelist __alloc_pages_nodemask alloc_pages_current pte_alloc_one __pte_alloc handle_mm_fault do_page_fault page_fault _dl_vdso_vsym 0x7fff16d617f5 0x535345535f474458 0.00% sleep [kernel.kallsyms] [k] policy_zonelist | --- policy_zonelist pte_alloc_one __pte_alloc handle_mm_fault do_page_fault page_fault _dl_init_paths _dl_sysdep_start 0.00% tar [kernel.kallsyms] [k] find_lock_page | --- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] lock_buffer | --- lock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] _cond_resched | --- _cond_resched generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ext4_da_write_begin | --- ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] dquot_free_inode | --- dquot_free_inode ext4_free_inode ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] get_request_wait | --- get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] __local_bh_enable | --- __local_bh_enable __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] mutex_unlock | --- mutex_unlock generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] hrtimer_interrupt | --- hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] create_empty_buffers | --- create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] radix_tree_preload_end | --- radix_tree_preload_end add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] ext4_xattr_set_entry | --- ext4_xattr_set_entry ext4_xattr_ibody_set ext4_xattr_set_handle ext4_init_security ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% swapper [kernel.kallsyms] [k] SetPageUptodate | --- SetPageUptodate mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics | --- mem_cgroup_charge_statistics __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar libc-2.13.90.so [.] _int_free | --- _int_free 0.00% swapper [kernel.kallsyms] [k] schedule | --- schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] sysret_check | --- sysret_check __GI___libc_read 0.00% tar [kernel.kallsyms] [k] blk_peek_request | --- blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ata_scsi_rw_xlat | --- ata_scsi_rw_xlat ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] blk_flush_plug_list | --- blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] jbd2_journal_stop | --- jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __srcu_read_lock | --- __srcu_read_lock fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write # # (For a higher level overview, try: perf report --sort comm,dso) # ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 16:56 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 16:56 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote: > This is the output of perf record -g -a -f sleep 5 > > (hopefully the list won't choke) Um, this one actually shows kswapd James --- # Events: 6K cycles # # Overhead Command Shared Object Symbol # ........ ........... ................... ....................................... # 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab | --- shrink_slab | |--99.91%-- kswapd | kthread | kernel_thread_helper --0.09%-- [...] 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone | --- shrink_zone | |--99.46%-- kswapd | kthread | kernel_thread_helper | --0.54%-- kthread kernel_thread_helper 7.70% kswapd0 [kernel.kallsyms] [k] kswapd | --- kswapd kthread kernel_thread_helper 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe | --- zone_watermark_ok_safe | |--72.66%-- kswapd | kthread | kernel_thread_helper | |--20.88%-- sleeping_prematurely.part.12 | kswapd | kthread | kernel_thread_helper | --6.46%-- kthread kernel_thread_helper 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock | |--77.49%-- _raw_spin_lock | | | |--51.85%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | --48.15%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--19.47%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--1.73%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--0.88%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper --0.44%-- [...] 3.34% kswapd0 [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--48.58%-- _raw_spin_unlock | | | |--80.77%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --19.23%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--17.63%-- _raw_spin_unlock_irqrestore | | | |--52.96%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --47.04%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--13.83%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--6.67%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--5.54%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--2.76%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--2.24%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--1.68%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--0.55%-- irq_exit | smp_apic_timer_interrupt | apic_timer_interrupt | mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | --0.54%-- mem_cgroup_charge_statistics __mem_cgroup_uncharge_common mem_cgroup_uncharge_cache_page __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 2.96% kswapd0 [kernel.kallsyms] [k] __zone_watermark_ok | --- __zone_watermark_ok | |--85.77%-- zone_watermark_ok_safe | | | |--74.67%-- kswapd | | kthread | | kernel_thread_helper | | | --25.33%-- sleeping_prematurely.part.12 | kswapd | kthread | kernel_thread_helper | |--11.78%-- kswapd | kthread | kernel_thread_helper | --2.45%-- sleeping_prematurely.part.12 kswapd kthread kernel_thread_helper 2.67% kswapd0 [kernel.kallsyms] [k] global_dirty_limits | --- global_dirty_limits | |--97.28%-- throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | --2.72%-- shrink_zone kswapd kthread kernel_thread_helper 2.30% kswapd0 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--30.33%-- _raw_spin_lock | | | |--52.29%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --47.71%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--28.79%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--24.01%-- _raw_spin_lock_irqsave | | | |--63.28%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --36.72%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--6.31%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--4.82%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--3.28%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--1.64%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | --0.82%-- finish_wait kswapd kthread kernel_thread_helper 2.02% kswapd0 [kernel.kallsyms] [k] up_read | --- up_read | |--95.48%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --4.52%-- kswapd kthread kernel_thread_helper 2.01% kswapd0 [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip | |--37.59%-- sub_preempt_count | | | |--66.19%-- _raw_spin_unlock | | | | | |--55.77%-- mem_cgroup_soft_limit_reclaim | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --44.23%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--17.13%-- _raw_spin_unlock_irqrestore | | | | | |--57.38%-- finish_wait | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --42.62%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --16.68%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--32.16%-- add_preempt_count | | | |--71.32%-- _raw_spin_lock | | | | | |--56.02%-- mem_cgroup_soft_limit_reclaim | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --43.98%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--17.25%-- _raw_spin_lock_irqsave | | | | | |--83.09%-- finish_wait | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --16.91%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --11.43%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--8.20%-- _raw_spin_unlock | | | |--66.09%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --33.91%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--7.42%-- _raw_spin_lock | | | |--62.14%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | --37.86%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--5.55%-- _raw_spin_lock_irqsave | | | |--66.67%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --33.33%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--4.53%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--3.62%-- _raw_spin_unlock_irqrestore | | | |--51.20%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --48.80%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --0.94%-- _raw_spin_unlock_irq shrink_zone kswapd kthread kernel_thread_helper 1.88% kswapd0 [kernel.kallsyms] [k] zone_nr_lru_pages | --- zone_nr_lru_pages | |--85.21%-- shrink_zone | kswapd | kthread | kernel_thread_helper | --14.79%-- kswapd kthread kernel_thread_helper 1.79% kswapd0 [kernel.kallsyms] [k] down_read_trylock | --- down_read_trylock | |--92.80%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --7.20%-- kswapd kthread kernel_thread_helper 1.69% kswapd0 [kernel.kallsyms] [k] mutex_trylock | --- mutex_trylock i915_gem_inactive_shrink shrink_slab kswapd kthread kernel_thread_helper 1.66% kswapd0 [kernel.kallsyms] [k] sleeping_prematurely.part.12 | --- sleeping_prematurely.part.12 | |--93.41%-- kswapd | kthread | kernel_thread_helper | --6.59%-- kthread kernel_thread_helper 1.54% kswapd0 [kernel.kallsyms] [k] find_next_bit | --- find_next_bit | |--84.45%-- cpumask_next | zone_watermark_ok_safe | kswapd | kthread | kernel_thread_helper | --15.55%-- zone_watermark_ok_safe kswapd kthread kernel_thread_helper 1.51% kswapd0 [kernel.kallsyms] [k] throttle_vm_writeout | --- throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 1.48% kswapd0 [kernel.kallsyms] [k] mb_cache_shrink_fn | --- mb_cache_shrink_fn | |--96.31%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --3.69%-- kswapd kthread kernel_thread_helper 1.31% kswapd0 [kernel.kallsyms] [k] mem_cgroup_soft_limit_reclaim | --- mem_cgroup_soft_limit_reclaim | |--97.19%-- kswapd | kthread | kernel_thread_helper | --2.81%-- kthread kernel_thread_helper 1.26% kswapd0 [kernel.kallsyms] [k] mutex_unlock | --- mutex_unlock | |--98.57%-- i915_gem_inactive_shrink | shrink_slab | kswapd | kthread | kernel_thread_helper | --1.43%-- shrink_slab kswapd kthread kernel_thread_helper 1.19% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--47.70%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--47.65%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --4.65%-- kswapd kthread kernel_thread_helper 1.14% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock | |--59.71%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--38.63%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | --1.66%-- kswapd kthread kernel_thread_helper 0.96% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_largest_soft_limit_node | --- __mem_cgroup_largest_soft_limit_node | |--59.47%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | --40.53%-- kswapd kthread kernel_thread_helper 0.88% kswapd0 [kernel.kallsyms] [k] in_lock_functions | --- in_lock_functions | |--43.80%-- get_parent_ip | | | |--52.47%-- add_preempt_count | | | | | |--72.99%-- _raw_spin_lock | | | | | | | |--62.71%-- mem_cgroup_soft_limit_reclaim | | | | kswapd | | | | kthread | | | | kernel_thread_helper | | | | | | | --37.29%-- mb_cache_shrink_fn | | | shrink_slab | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | |--18.30%-- _raw_spin_lock_irq | | | shrink_zone | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --8.71%-- _raw_spin_lock_irqsave | | prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --47.53%-- sub_preempt_count | | | |--49.81%-- _raw_spin_unlock | | mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--30.32%-- _raw_spin_unlock_irqrestore | | finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --19.86%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | |--29.12%-- add_preempt_count | | | |--49.50%-- _raw_spin_lock | | | | | |--56.31%-- mem_cgroup_soft_limit_reclaim | | | kswapd | | | kthread | | | kernel_thread_helper | | | | | --43.69%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | |--29.19%-- _raw_spin_lock_irq | | shrink_zone | | kswapd | | kthread | | kernel_thread_helper | | | --21.32%-- _raw_spin_lock_irqsave | | | |--65.66%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --34.34%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --27.07%-- sub_preempt_count | |--62.26%-- _raw_spin_unlock | | | |--50.27%-- mb_cache_shrink_fn | | shrink_slab | | kswapd | | kthread | | kernel_thread_helper | | | --49.73%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--22.85%-- _raw_spin_unlock_irqrestore | | | |--67.34%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --32.66%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --14.89%-- _raw_spin_unlock_irq shrink_zone kswapd kthread kernel_thread_helper 0.81% kswapd0 [kernel.kallsyms] [k] cpumask_next | --- cpumask_next | |--79.81%-- zone_watermark_ok_safe | kswapd | kthread | kernel_thread_helper | --20.19%-- kswapd kthread kernel_thread_helper 0.79% kswapd0 [kernel.kallsyms] [k] zone_reclaimable_pages | --- zone_reclaimable_pages | |--93.08%-- kswapd | kthread | kernel_thread_helper | --6.92%-- kthread kernel_thread_helper 0.77% kswapd0 [i915] [k] i915_gem_inactive_shrink | --- i915_gem_inactive_shrink | |--97.55%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --2.45%-- kswapd kthread kernel_thread_helper 0.76% kswapd0 [kernel.kallsyms] [k] prepare_to_wait | --- prepare_to_wait | |--97.54%-- kswapd | kthread | kernel_thread_helper | --2.46%-- kthread kernel_thread_helper 0.74% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore | |--79.88%-- _raw_spin_unlock_irqrestore | | | |--62.49%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --37.51%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--15.16%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --4.96%-- finish_wait kswapd kthread kernel_thread_helper 0.65% kswapd0 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 | |--20.12%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--20.09%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | |--14.28%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--14.16%-- _raw_spin_unlock | | | |--80.68%-- mem_cgroup_soft_limit_reclaim | | kswapd | | kthread | | kernel_thread_helper | | | --19.32%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | kernel_thread_helper | |--11.40%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--8.63%-- finish_wait | kswapd | kthread | kernel_thread_helper | |--8.58%-- _raw_spin_unlock_irqrestore | | | |--66.13%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --33.87%-- finish_wait | kswapd | kthread | kernel_thread_helper | --2.74%-- _raw_spin_unlock_irq shrink_zone kswapd kthread kernel_thread_helper 0.63% tar [kernel.kallsyms] [k] copy_user_generic_string | --- copy_user_generic_string | |--56.11%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --43.89%-- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.59% kswapd0 [kernel.kallsyms] [k] test_tsk_thread_flag | --- test_tsk_thread_flag | |--50.16%-- kswapd | kthread | kernel_thread_helper | |--46.73%-- try_to_freeze | kswapd | kthread | kernel_thread_helper | --3.11%-- kthread kernel_thread_helper 0.59% kswapd0 [kernel.kallsyms] [k] shrink_icache_memory | --- shrink_icache_memory | |--81.19%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --18.81%-- kswapd kthread kernel_thread_helper 0.52% kswapd0 [kernel.kallsyms] [k] shrink_dqcache_memory | --- shrink_dqcache_memory | |--89.35%-- shrink_slab | kswapd | kthread | kernel_thread_helper | --10.65%-- kswapd kthread kernel_thread_helper 0.50% kswapd0 [kernel.kallsyms] [k] zone_clear_flag | --- zone_clear_flag | |--77.78%-- kswapd | kthread | kernel_thread_helper | --22.22%-- kthread kernel_thread_helper 0.45% kswapd0 [kernel.kallsyms] [k] kthread_should_stop | --- kthread_should_stop | |--54.28%-- kswapd | kthread | kernel_thread_helper | --45.72%-- kthread kernel_thread_helper 0.44% kswapd0 [kernel.kallsyms] [k] global_page_state | --- global_page_state | |--33.60%-- throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | |--29.47%-- determine_dirtyable_memory | global_dirty_limits | throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | |--20.52%-- shrink_zone | kswapd | kthread | kernel_thread_helper | --16.41%-- global_dirty_limits throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 0.44% kswapd0 [kernel.kallsyms] [k] shrink_dcache_memory | --- shrink_dcache_memory shrink_slab kswapd kthread kernel_thread_helper 0.41% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore | --- _raw_spin_unlock_irqrestore | |--59.01%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --40.99%-- finish_wait kswapd kthread kernel_thread_helper 0.39% kswapd0 [kernel.kallsyms] [k] global_reclaimable_pages | --- global_reclaimable_pages determine_dirtyable_memory global_dirty_limits throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 0.37% kswapd0 [kernel.kallsyms] [k] finish_wait | --- finish_wait | |--95.03%-- kswapd | kthread | kernel_thread_helper | --4.97%-- kthread kernel_thread_helper 0.33% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock | |--60.70%-- mem_cgroup_soft_limit_reclaim | kswapd | kthread | kernel_thread_helper | --39.30%-- mb_cache_shrink_fn shrink_slab kswapd kthread kernel_thread_helper 0.31% kswapd0 [kernel.kallsyms] [k] arch_local_irq_enable | --- arch_local_irq_enable | |--76.94%-- _raw_spin_unlock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | --23.06%-- shrink_zone kswapd kthread kernel_thread_helper 0.28% kswapd0 [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save | |--93.18%-- _raw_spin_lock_irqsave | | | |--64.21%-- prepare_to_wait | | kswapd | | kthread | | kernel_thread_helper | | | --35.79%-- finish_wait | kswapd | kthread | kernel_thread_helper | --6.82%-- finish_wait kswapd kthread kernel_thread_helper 0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irq | --- _raw_spin_unlock_irq | |--92.00%-- shrink_zone | kswapd | kthread | kernel_thread_helper | --8.00%-- __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.22% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable | |--58.81%-- arch_local_irq_save | _raw_spin_lock_irqsave | | | |--70.97%-- finish_wait | | kswapd | | kthread | | kernel_thread_helper | | | --29.03%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | |--33.06%-- _raw_spin_lock_irq | shrink_zone | kswapd | kthread | kernel_thread_helper | --8.13%-- shrink_zone kswapd kthread kernel_thread_helper 0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irq | --- _raw_spin_lock_irq | |--58.62%-- shrink_zone | kswapd | kthread | kernel_thread_helper | |--33.19%-- kswapd | kthread | kernel_thread_helper | --8.19%-- __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.22% swapper [kernel.kallsyms] [k] poll_idle | --- poll_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.20% kswapd0 [kernel.kallsyms] [k] __list_add | --- __list_add | |--90.97%-- prepare_to_wait | kswapd | kthread | kernel_thread_helper | --9.03%-- free_hot_cold_page __pagevec_free free_page_list shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.15% tar [kernel.kallsyms] [k] zero_user_segments | --- zero_user_segments __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.15% kswapd0 [kernel.kallsyms] [k] rb_last | --- rb_last mem_cgroup_soft_limit_reclaim kswapd kthread kernel_thread_helper 0.14% swapper [kernel.kallsyms] [k] intel_idle | --- intel_idle cpuidle_idle_call cpu_idle | |--65.05%-- rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --34.95%-- start_secondary 0.13% kswapd0 [kernel.kallsyms] [k] get_reclaim_stat | --- get_reclaim_stat | |--57.55%-- kswapd | kthread | kernel_thread_helper | --42.45%-- shrink_zone kswapd kthread kernel_thread_helper 0.12% perf [kernel.kallsyms] [k] selinux_file_alloc_security | --- selinux_file_alloc_security security_file_alloc get_empty_filp path_openat do_filp_open do_sys_open sys_open system_call_fastpath __GI___libc_open 0x429beb 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.12% tar [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--18.90%-- _raw_spin_lock | | | |--63.01%-- ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | |--20.46%-- get_page_from_freelist | | __alloc_pages_nodemask | | alloc_pages_current | | __page_cache_alloc | | grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --16.53%-- ext4_ext_map_blocks | ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--13.68%-- __lru_cache_add | add_to_page_cache_lru | | | |--51.23%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --48.77%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--13.32%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--12.87%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--9.79%-- _raw_spin_lock_irq | | | |--62.37%-- add_to_page_cache_locked | | add_to_page_cache_lru | | grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --37.63%-- __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--7.52%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.32%-- lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.69%-- __percpu_counter_add | __add_bdi_stat | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.54%-- bit_spin_lock | lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.41%-- create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.32%-- _raw_read_lock_irqsave | dm_get_live_table | __split_and_process_bio | dm_request | generic_make_request | submit_bio | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.29%-- _raw_read_lock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --2.33%-- dput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.11% kswapd0 [kernel.kallsyms] [k] try_to_freeze | --- try_to_freeze | |--83.76%-- kswapd | kthread | kernel_thread_helper | --16.24%-- kthread kernel_thread_helper 0.10% swapper [kernel.kallsyms] [k] ahci_interrupt | --- ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.09% kswapd0 [kernel.kallsyms] [k] determine_dirtyable_memory | --- determine_dirtyable_memory | |--80.46%-- global_dirty_limits | throttle_vm_writeout | shrink_zone | kswapd | kthread | kernel_thread_helper | --19.54%-- throttle_vm_writeout shrink_zone kswapd kthread kernel_thread_helper 0.08% tar [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--27.37%-- _raw_spin_unlock | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.90%-- blk_dequeue_request | blk_start_request | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--9.76%-- bit_spin_unlock | unlock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.85%-- _raw_spin_unlock_irq | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--6.81%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.86%-- _raw_spin_unlock_irqrestore | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.51%-- _raw_read_unlock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--5.37%-- jbd_unlock_bh_state | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_free_inode | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | |--4.68%-- deactivate_slab | __slab_alloc | kmem_cache_alloc | alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--4.15%-- __srcu_read_unlock | fsnotify | fsnotify_modify | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--3.72%-- radix_tree_preload_end | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--3.62%-- bit_spin_unlock | jbd2_journal_add_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --3.41%-- pagefault_enable iov_iter_copy_from_user_atomic generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.08% swapper [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--52.45%-- _raw_spin_lock | get_next_timer_interrupt | tick_nohz_stop_sched_tick | irq_exit | smp_call_function_single_interrupt | call_function_single_interrupt | cpuidle_idle_call | cpu_idle | start_secondary | |--26.09%-- bit_spin_lock.constprop.22 | __slab_free | kmem_cache_free | ext4_free_io_end | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --21.46%-- __percpu_counter_add __prop_inc_percpu_max test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.08% tar [kernel.kallsyms] [k] kmem_cache_alloc | --- kmem_cache_alloc | |--74.26%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--12.98%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.88%-- mempool_alloc_slab | mempool_alloc | scsi_sg_alloc | __sg_alloc_table | scsi_alloc_sgtable | scsi_init_sgtable | scsi_init_io | scsi_setup_fs_cmnd | sd_prep_fn | blk_peek_request | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --5.87%-- selinux_inode_alloc_security security_inode_alloc inode_init_always alloc_inode new_inode ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.07% kswapd0 [kernel.kallsyms] [k] __free_one_page | --- __free_one_page free_pcppages_bulk free_hot_cold_page __pagevec_free free_page_list shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.07% swapper [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count | |--56.43%-- __percpu_counter_add | __add_bdi_stat | test_clear_page_writeback | end_page_writeback | put_io_page | ext4_end_bio | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--17.99%-- _raw_spin_unlock_irqrestore | __hrtimer_start_range_ns | hrtimer_start_range_ns | hrtimer_start_expires.constprop.1 | tick_nohz_restart_sched_tick | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--15.61%-- _raw_spin_unlock | | | |--55.21%-- handle_irq_event | | handle_edge_irq | | handle_irq | | do_IRQ | | common_interrupt | | cpuidle_idle_call | | cpu_idle | | rest_init | | start_kernel | | x86_64_start_reservations | | x86_64_start_kernel | | | --44.79%-- scheduler_tick | update_process_times | tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--5.91%-- try_to_wake_up | default_wake_function | autoremove_wake_function | wake_bit_function | __wake_up_common | __wake_up | __wake_up_bit | unlock_page | mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --4.06%-- default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.06% tar [kernel.kallsyms] [k] mark_page_accessed | --- mark_page_accessed | |--87.24%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--6.42%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --6.35%-- __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] get_page_from_freelist | --- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--54.79%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --45.21%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar [kernel.kallsyms] [k] __ext4_get_inode_loc | --- __ext4_get_inode_loc ext4_get_inode_loc | |--55.97%-- ext4_reserve_inode_write | ext4_mark_inode_dirty | | | |--76.52%-- ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --23.48%-- ext4_ext_dirty | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --44.03%-- ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.06% tar tar [.] 0x2aa04 | --- 0x40753f | --- 0x4055c7 | --- 0x42aa04 | --- 0x4166d8 0x417081 | --- 0x40fdfb | --- 0x405f26 | --- 0x4332d6 | --- 0x40fe1a | --- 0x408ab1 | --- 0x40fdb8 | --- 0x408a87 | --- 0x4100c5 | --- 0x407564 0.06% perf [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4293b8 0x429c0a 0x418709 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.05% tar [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock | |--79.42%-- _raw_spin_lock_irq | | | |--57.06%-- __set_page_dirty | | mark_buffer_dirty | | __block_commit_write | | block_write_end | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --42.94%-- add_to_page_cache_locked | add_to_page_cache_lru | | | |--52.51%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --47.49%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--15.99%-- _raw_spin_lock | | | |--60.63%-- ext4_ext_map_blocks | | ext4_map_blocks | | ext4_da_get_block_prep | | __block_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --39.37%-- inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --4.59%-- inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28 | --- __mem_cgroup_commit_charge.constprop.28 | |--91.58%-- __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | | | |--51.67%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --48.33%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --8.42%-- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.05% tar [kernel.kallsyms] [k] radix_tree_lookup_element | --- radix_tree_lookup_element | |--69.74%-- radix_tree_lookup_slot | find_get_page | | | |--58.70%-- generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --41.30%-- find_lock_page | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.45%-- radix_tree_lookup | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --12.81%-- find_get_page generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] __alloc_pages_nodemask | --- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--71.39%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --28.61%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed alloc_pages_current __page_cache_alloc | |--70.79%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.21%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% swapper [kernel.kallsyms] [k] load_balance | --- load_balance rebalance_domains run_rebalance_domains __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle start_secondary 0.04% flush-253:2 [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit | |--89.40%-- unlock_page | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --10.60%-- wake_up_bit unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% tar [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip | |--29.28%-- sub_preempt_count | | | |--59.65%-- pagefault_enable | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --40.35%-- bit_spin_unlock | jbd2_journal_put_journal_head | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--19.16%-- avc_has_perm_noaudit | avc_has_perm | inode_has_perm | selinux_inode_permission | security_inode_exec_permission | exec_permission | link_path_walk | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | |--19.01%-- _raw_spin_lock | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--17.20%-- bit_spin_lock | lock_page_cgroup | __mem_cgroup_commit_charge.constprop.28 | __mem_cgroup_commit_charge_lrucare | mem_cgroup_cache_charge | add_to_page_cache_locked | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --15.36%-- add_preempt_count file_read_actor generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.04% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty | --- ext4_mark_iloc_dirty ext4_mark_inode_dirty | |--77.33%-- ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--13.35%-- ext4_new_inode | ext4_create | vfs_create | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --9.32%-- ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.04% kswapd0 [kernel.kallsyms] [k] native_write_msr_safe | --- native_write_msr_safe paravirt_write_msr intel_pmu_disable_all x86_pmu_disable perf_pmu_disable perf_event_task_tick scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--50.07%-- zone_watermark_ok_safe | sleeping_prematurely.part.12 | kswapd | kthread | kernel_thread_helper | --49.93%-- kswapd kthread kernel_thread_helper 0.04% swapper [kernel.kallsyms] [k] ext4_end_bio | --- ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.04% kswapd0 [kernel.kallsyms] [k] __list_del_entry | --- __list_del_entry finish_wait kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] mcount | --- mcount _raw_spin_lock_irq shrink_zone kswapd kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] __percpu_counter_add | --- __percpu_counter_add | |--87.00%-- ext4_claim_free_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --13.00%-- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% tar [kernel.kallsyms] [k] arch_read_unlock | --- arch_read_unlock | |--57.64%-- _raw_read_unlock_irqrestore | dm_get_live_table | dm_merge_bvec | __bio_add_page.part.2 | bio_add_page | do_mpage_readpage | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--31.54%-- _raw_read_unlock | start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --10.82%-- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% kswapd0 [kernel.kallsyms] [k] mem_cgroup_del_lru_list | --- mem_cgroup_del_lru_list mem_cgroup_del_lru isolate_lru_pages shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] test_and_set_bit | --- test_and_set_bit shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% tar [kernel.kallsyms] [k] __might_sleep | --- __might_sleep | |--22.45%-- kmem_cache_alloc | | | |--59.27%-- getname_flags | | getname | | do_sys_open | | sys_openat | | system_call_fastpath | | __GI___openat | | | --40.73%-- jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--22.30%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--54.37%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --45.63%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--15.00%-- lock_buffer | do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--11.33%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--11.31%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--10.36%-- unmap_underlying_metadata | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --7.25%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.04% kswapd0 [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--50.69%-- __pagevec_free | free_page_list | shrink_page_list | shrink_inactive_list | shrink_zone | kswapd | kthread | kernel_thread_helper | --49.31%-- free_hot_cold_page __pagevec_free free_page_list shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_uncharge_common | --- __mem_cgroup_uncharge_common mem_cgroup_uncharge_cache_page __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.04% kswapd0 [kernel.kallsyms] [k] tag_get | --- tag_get | |--50.59%-- radix_tree_delete | __delete_from_page_cache | __remove_mapping | shrink_page_list | shrink_inactive_list | shrink_zone | kswapd | kthread | kernel_thread_helper | --49.41%-- __delete_from_page_cache __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.03% flush-253:2 [kernel.kallsyms] [k] add_preempt_count | --- add_preempt_count | |--51.77%-- _raw_read_lock_irqsave | dm_get_live_table | dm_merge_bvec | __bio_add_page.part.2 | bio_add_page | ext4_bio_write_page | mpage_da_submit_io | mpage_da_map_and_submit | ext4_da_writepages | do_writepages | writeback_single_inode | writeback_sb_inodes | writeback_inodes_wb | wb_writeback | wb_do_writeback | bdi_writeback_thread | kthread | kernel_thread_helper | --48.23%-- blk_throtl_bio generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] __inc_zone_state | --- __inc_zone_state | |--49.98%-- __inc_zone_page_state | add_to_page_cache_locked | add_to_page_cache_lru | | | |--79.48%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --20.52%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--32.04%-- zone_statistics | get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --17.98%-- account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% tar [kernel.kallsyms] [k] __list_del_entry | --- __list_del_entry | |--56.86%-- list_del | __rmqueue | get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--78.10%-- __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --21.90%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.14%-- __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% flush-253:2 [kernel.kallsyms] [k] dm_table_find_target | --- dm_table_find_target dm_merge_bvec __bio_add_page.part.2 bio_add_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] jbd_lock_bh_state | --- jbd_lock_bh_state | |--61.84%-- do_get_write_access | jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | | | |--72.93%-- generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --27.07%-- ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --38.16%-- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% flush-253:2 [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% flush-253:2 [kernel.kallsyms] [k] radix_tree_lookup_element | --- radix_tree_lookup_element radix_tree_lookup_slot find_get_page __find_get_block_slow unmap_underlying_metadata mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] fcheck_files | --- fcheck_files fget_light | |--71.87%-- sys_read | system_call_fastpath | __GI___libc_read | --28.13%-- sys_write system_call_fastpath __GI___libc_write 0.03% flush-253:2 [kernel.kallsyms] [k] atomic_inc | --- atomic_inc ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% flush-253:2 [kernel.kallsyms] [k] lock_page | --- lock_page ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero | |--70.70%-- release_pages | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | | | |--77.62%-- mpage_readpages | | ext4_readpages | | __do_page_cache_readahead | | ra_submit | | ondemand_readahead | | page_cache_async_readahead | | generic_file_aio_read | | do_sync_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | | --22.38%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.30%-- put_page mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] zone_watermark_ok | --- zone_watermark_ok get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc | |--84.29%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --15.71%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] start_this_handle | --- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--86.30%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --13.70%-- ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages | --- ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.03% tar [kernel.kallsyms] [k] put_mems_allowed | --- put_mems_allowed | |--74.79%-- __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--65.17%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --34.83%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --25.21%-- alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% tar [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id | |--57.28%-- load_balance | schedule | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--25.57%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --17.15%-- get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.03% swapper [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.03% tar [kernel.kallsyms] [k] avc_has_perm_noaudit | --- avc_has_perm_noaudit avc_has_perm | |--82.82%-- inode_has_perm | | | |--71.72%-- dentry_has_perm | | selinux_inode_getattr | | security_inode_getattr | | vfs_getattr | | vfs_fstat | | sys_newfstat | | system_call_fastpath | | __GI___fxstat64 | | | --28.28%-- selinux_inode_permission | security_inode_exec_permission | exec_permission | link_path_walk | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --17.18%-- may_create selinux_inode_create security_inode_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.03% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] do_mpage_readpage | --- do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.03% tar [kernel.kallsyms] [k] list_del | --- list_del | |--55.44%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | | | |--67.14%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --32.86%-- __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--28.53%-- ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --16.02%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] __rmqueue | --- __rmqueue | |--77.01%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --22.99%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge_lrucare | --- __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--83.13%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --16.87%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] update_page_reclaim_stat | --- update_page_reclaim_stat ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--75.26%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --24.74%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% tar [kernel.kallsyms] [k] radix_tree_insert | --- radix_tree_insert add_to_page_cache_locked add_to_page_cache_lru | |--81.25%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --18.75%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] add_to_page_cache_locked | --- add_to_page_cache_locked add_to_page_cache_lru | |--62.06%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --37.94%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue | |--58.60%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.40%-- unlock_page generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] up_read | --- up_read | |--52.38%-- ext4_map_blocks | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.62%-- ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% swapper [kernel.kallsyms] [k] elv_completed_request | --- elv_completed_request blk_finish_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save | |--84.09%-- task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --15.91%-- account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] ext4_map_blocks | --- ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mcount | --- mcount _raw_spin_unlock_irqrestore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] clear_buffer_dirty | --- clear_buffer_dirty ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list | --- mem_cgroup_add_lru_list add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% swapper [kernel.kallsyms] [k] blk_complete_request | --- blk_complete_request scsi_done ata_scsi_qc_complete __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% swapper [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] touch_atime | --- touch_atime | |--90.35%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --9.65%-- do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% perf ld-2.13.90.so [.] _dl_fixup | --- _dl_fixup 0.02% tar [kernel.kallsyms] [k] __block_write_begin | --- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% kswapd0 [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt i915_gem_inactive_shrink shrink_slab kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] apic_timer_interrupt | --- apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kworker/2:1 [kernel.kallsyms] [k] ns_to_timeval | --- ns_to_timeval get_cpu_iowait_time_us do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] task_rq_lock | --- task_rq_lock try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt zone_watermark_ok_safe kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] hrtimer_run_queues | --- hrtimer_run_queues run_local_timers update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] next_zones_zonelist | --- next_zones_zonelist | |--71.41%-- get_page_from_freelist | __alloc_pages_nodemask | alloc_pages_current | __page_cache_alloc | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --28.59%-- __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable run_local_timers update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] __rcu_pending | --- __rcu_pending rcu_check_callbacks update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] write_cache_pages_da | --- write_cache_pages_da ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] intel_pmu_disable_all | --- intel_pmu_disable_all x86_pmu_disable perf_pmu_disable perf_event_task_tick scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt sleeping_prematurely.part.12 kswapd kthread kernel_thread_helper 0.02% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags arch_local_irq_save irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore call_rcu d_free dentry_kill shrink_dentry_list __shrink_dcache_sb shrink_dcache_memory shrink_slab kswapd kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] SetPageLRU | --- SetPageLRU | |--66.95%-- ____pagevec_lru_add_fn | pagevec_lru_move_fn | ____pagevec_lru_add | __lru_cache_add | add_to_page_cache_lru | | | |--56.88%-- grab_cache_page_write_begin | | ext4_da_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --43.12%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --33.05%-- pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% kswapd0 [kernel.kallsyms] [k] run_timer_softirq | --- run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt i915_gem_inactive_shrink shrink_slab kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page | --- __isolate_lru_page isolate_lru_pages shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] radix_tree_delete | --- radix_tree_delete __delete_from_page_cache __remove_mapping shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear | --- radix_tree_tag_clear test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_map_blocks | --- ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags __might_sleep | |--51.15%-- __getblk | __ext4_get_inode_loc | ext4_get_inode_loc | | | |--50.51%-- ext4_reserve_inode_write | | ext4_mark_inode_dirty | | ext4_ext_truncate | | ext4_truncate | | ext4_evict_inode | | evict | | iput | | do_unlinkat | | sys_unlinkat | | system_call_fastpath | | unlinkat | | | --49.49%-- ext4_xattr_get | ext4_xattr_security_get | generic_getxattr | cap_inode_need_killpriv | security_inode_need_killpriv | file_remove_suid | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.85%-- kmem_cache_alloc jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% kswapd0 [kernel.kallsyms] [k] mem_cgroup_uncharge_cache_page | --- mem_cgroup_uncharge_cache_page shrink_page_list shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] update_curr | --- update_curr task_tick_fair scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] timerqueue_del | --- timerqueue_del __remove_hrtimer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_unlock_irqrestore prepare_to_wait kswapd kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] ack_APIC_irq | --- ack_APIC_irq smp_apic_timer_interrupt apic_timer_interrupt kswapd kthread kernel_thread_helper 0.02% kworker/2:1 [kernel.kallsyms] [k] too_many_workers | --- too_many_workers worker_enter_idle worker_thread kthread kernel_thread_helper 0.02% kswapd0 [kernel.kallsyms] [k] update_isolated_counts | --- update_isolated_counts shrink_inactive_list shrink_zone kswapd kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 _raw_spin_unlock_irqrestore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] jbd2_journal_get_write_access | --- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_ext_dirty ext4_ext_insert_extent ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] __lookup | --- __lookup radix_tree_gang_lookup_slot find_get_pages pagevec_lookup mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_pages_tag pagevec_lookup_tag ext4_num_dirty_pages ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore _raw_spin_unlock_irqrestore test_set_page_writeback ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_ext_map_blocks | --- ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] jbd_lock_bh_state | --- jbd_lock_bh_state jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mb_mark_diskspace_used ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count | --- sub_preempt_count jbd_unlock_bh_state jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mb_mark_diskspace_used ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] alloc_pages_current | --- alloc_pages_current __page_cache_alloc | |--72.21%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --27.79%-- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page | --- ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% flush-253:2 [kernel.kallsyms] [k] ext4_mb_complex_scan_group | --- ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% top [kernel.kallsyms] [k] __d_lookup | --- __d_lookup d_lookup proc_fill_cache proc_pid_readdir proc_root_readdir vfs_readdir sys_getdents system_call_fastpath __getdents64 0.02% flush-253:2 [kernel.kallsyms] [k] kmem_cache_alloc | --- kmem_cache_alloc ext4_init_io_end ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] page_cache_get_speculative | --- page_cache_get_speculative find_get_page | |--51.46%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.54%-- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_extent.constprop.12 | --- mb_find_extent.constprop.12 ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] generic_file_buffered_write | --- generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% top libc-2.13.90.so [.] __GI_vfprintf | --- __GI_vfprintf ___vsprintf_chk 0.02% tar [kernel.kallsyms] [k] ext4_da_get_block_prep | --- ext4_da_get_block_prep | |--75.27%-- __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --24.73%-- ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% top [kernel.kallsyms] [k] getname_flags | --- getname_flags getname do_sys_open sys_open system_call_fastpath __GI___libc_open 0.02% swapper [kernel.kallsyms] [k] __count_vm_events | --- __count_vm_events __free_pages __free_slab discard_slab __slab_free kmem_cache_free ext4_free_io_end ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% swapper [kernel.kallsyms] [k] tick_check_idle | --- tick_check_idle irq_enter | |--73.47%-- smp_call_function_single_interrupt | call_function_single_interrupt | cpuidle_idle_call | cpu_idle | start_secondary | --26.53%-- smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% flush-253:2 [kernel.kallsyms] [k] find_next_zero_bit | --- find_next_zero_bit mb_find_next_zero_bit ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% tar [kernel.kallsyms] [k] arch_read_lock | --- arch_read_lock _raw_read_lock start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_buddy | --- mb_find_buddy mb_mark_used ext4_mb_use_best_found ext4_mb_check_limits ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.02% swapper [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock __prop_inc_percpu_max test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] mark_buffer_dirty | --- mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] new_slab | --- new_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% swapper [kernel.kallsyms] [k] __do_softirq | --- __do_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% swapper [kernel.kallsyms] [k] select_task_rq_fair | --- select_task_rq_fair select_task_rq try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.02% tar [kernel.kallsyms] [k] do_get_write_access | --- do_get_write_access | |--70.88%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | | | |--54.37%-- ext4_new_inode | | ext4_create | | vfs_create | | do_last | | path_openat | | do_filp_open | | do_sys_open | | sys_openat | | system_call_fastpath | | __GI___openat | | | --45.63%-- ext4_reserve_inode_write | ext4_orphan_add | ext4_unlink | vfs_unlink | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --29.12%-- __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] in_lock_functions | --- in_lock_functions | |--58.27%-- get_parent_ip | add_preempt_count | | | |--56.37%-- jbd_lock_bh_state | | jbd2_journal_dirty_metadata | | __ext4_handle_dirty_metadata | | ext4_mark_iloc_dirty | | ext4_mark_inode_dirty | | ext4_dirty_inode | | __mark_inode_dirty | | generic_write_end | | ext4_da_write_end | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | ext4_file_write | | do_sync_write | | vfs_write | | sys_write | | system_call_fastpath | | __GI___libc_write | | | --43.63%-- radix_tree_preload | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --41.73%-- sub_preempt_count bit_spin_unlock jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata | --- jbd2_journal_dirty_metadata __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] ext4_ext_map_blocks | --- ext4_ext_map_blocks ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% tar [kernel.kallsyms] [k] block_write_end | --- block_write_end | |--67.22%-- ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --32.78%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.02% flush-253:2 [kernel.kallsyms] [k] mb_test_bit | --- mb_test_bit mb_find_order_for_block mb_find_extent.constprop.12 ext4_mb_complex_scan_group ext4_mb_regular_allocator ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] attach_page_buffers | --- attach_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __phys_addr | --- __phys_addr virt_to_head_page kmem_cache_free jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] do_sync_read | --- do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs | --- swiotlb_map_sg_attrs ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] jbd2__journal_start | --- jbd2__journal_start jbd2_journal_start ext4_journal_start_sb | |--73.97%-- ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --26.03%-- ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] iov_iter_advance | --- iov_iter_advance __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] generic_getxattr | --- generic_getxattr security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% jbd2/dm-2-8 [kernel.kallsyms] [k] elv_rq_merge_ok | --- elv_rq_merge_ok elv_try_merge __make_request generic_make_request submit_bio submit_bh jbd2_journal_commit_transaction kjournald2 kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] clear_bit_unlock | --- clear_bit_unlock unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] page_waitqueue | --- page_waitqueue | |--52.26%-- unlock_page | mpage_end_io | bio_endio | dec_pending | clone_endio | bio_endio | req_bio_endio | blk_update_request | blk_update_bidi_request | blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --47.74%-- mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] ext4_xattr_find_entry | --- ext4_xattr_find_entry ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock lock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--76.11%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --23.89%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] writeback_single_inode | --- writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] find_get_page | --- find_get_page | |--70.62%-- __find_get_block_slow | __find_get_block | ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --29.38%-- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] __ext4_get_inode_loc | --- __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] fsnotify_modify | --- fsnotify_modify | |--73.71%-- vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --26.29%-- sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] blk_rq_map_sg | --- blk_rq_map_sg scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue blk_run_queue scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] get_page | --- get_page add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--44.23%-- blk_end_bidi_request | blk_end_request | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | |--32.84%-- blk_run_queue | scsi_run_queue | scsi_next_command | scsi_io_completion | scsi_finish_command | scsi_softirq_done | blk_done_softirq | __do_softirq | call_softirq | do_softirq | irq_exit | do_IRQ | common_interrupt | cpuidle_idle_call | cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --22.93%-- pm_qos_request menu_select cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] __bio_clone | --- __bio_clone clone_bio __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] sg_next | --- sg_next | |--36.30%-- ata_qc_issue | __ata_scsi_queuecmd | ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | |--32.06%-- swiotlb_map_sg_attrs | ata_qc_issue | __ata_scsi_queuecmd | ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --31.64%-- blk_rq_map_sg scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] __rcu_read_unlock | --- __rcu_read_unlock | |--70.72%-- __prop_inc_single | task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.28%-- avc_has_perm_noaudit task_has_capability selinux_capable security_capable ns_capable capable setattr_copy ext4_setattr notify_change sys_fchmod system_call_fastpath __GI___fchmod 0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock | |--36.11%-- smp_apic_timer_interrupt | apic_timer_interrupt | __find_get_block_slow | unmap_underlying_metadata | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | |--34.75%-- scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --29.14%-- __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] generic_file_aio_read | --- generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] map_bh | --- map_bh ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] memcmp | --- memcmp | |--67.73%-- ext4_dx_find_entry | ext4_find_entry | ext4_lookup | d_alloc_and_lookup | __lookup_hash.part.3 | lookup_hash | do_last | path_openat | do_filp_open | do_sys_open | sys_openat | system_call_fastpath | __GI___openat | --32.27%-- search_dirblock ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] clone_endio | --- clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] __slab_free | --- __slab_free kmem_cache_free put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] nr_iowait_cpu | --- nr_iowait_cpu tick_nohz_stop_idle tick_check_idle irq_enter smp_call_function_single_interrupt call_function_single_interrupt cpuidle_idle_call cpu_idle start_secondary 0.01% swapper [kernel.kallsyms] [k] timekeeping_get_ns | --- timekeeping_get_ns getnstimeofday ktime_get_real intel_idle cpuidle_idle_call cpu_idle start_secondary 0.01% flush-253:2 [kernel.kallsyms] [k] crc16 | --- crc16 ext4_group_desc_csum ext4_mb_mark_diskspace_used ext4_mb_new_blocks ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] ata_scsi_queuecmd | --- ata_scsi_queuecmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] jbd2_journal_put_journal_head | --- jbd2_journal_put_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--60.89%-- generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --39.11%-- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] ext4_da_write_end | --- ext4_da_write_end | |--68.79%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --31.21%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] acct_update_integrals | --- acct_update_integrals __account_system_time irqtime_account_process_tick account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | |--51.45%-- get_parent_ip | add_preempt_count | __lru_cache_add | add_to_page_cache_lru | mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --48.55%-- find_get_page __find_get_block_slow __find_get_block ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] __inc_zone_page_state | --- __inc_zone_page_state | |--61.16%-- account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --38.84%-- add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] account_idle_time | --- account_idle_time account_idle_ticks tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] recalc_bh_state | --- recalc_bh_state | |--62.12%-- alloc_buffer_head | alloc_page_buffers | create_empty_buffers | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --37.88%-- alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] put_page_testzero | --- put_page_testzero put_page put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] generic_segment_checks | --- generic_segment_checks | |--56.32%-- generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --43.68%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] arch_local_irq_save | --- arch_local_irq_save __free_pages __free_slab discard_slab __slab_free kmem_cache_free put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] rw_verify_area | --- rw_verify_area | |--65.20%-- vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --34.80%-- vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] file_update_time | --- file_update_time | |--70.97%-- __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --29.03%-- generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% flush-253:2 [kernel.kallsyms] [k] alloc_tio | --- alloc_tio dm_request generic_make_request submit_bio ext4_io_submit mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __mark_inode_dirty | --- __mark_inode_dirty __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore | --- _raw_spin_unlock_irqrestore __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke | --- jbd2_journal_cancel_revoke do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] lookup_page_cgroup | --- lookup_page_cgroup __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--53.32%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --46.68%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq | --- _raw_spin_unlock_irq add_to_page_cache_locked add_to_page_cache_lru | |--69.09%-- mpage_readpages | ext4_readpages | __do_page_cache_readahead | ra_submit | ondemand_readahead | page_cache_async_readahead | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --30.91%-- grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __do_page_cache_readahead | --- __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] ext4_acl_chmod | --- ext4_acl_chmod notify_change sys_fchmod system_call_fastpath __GI___fchmod 0.01% flush-253:2 [kernel.kallsyms] [k] start_this_handle | --- start_this_handle jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] _raw_read_lock | --- _raw_read_lock | |--56.63%-- start_this_handle | jbd2__journal_start | jbd2_journal_start | ext4_journal_start_sb | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.37%-- jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] mix_pool_bytes_extract | --- mix_pool_bytes_extract add_timer_randomness add_disk_randomness blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] system_call_after_swapgs | --- system_call_after_swapgs __GI___libc_read 0.01% tar [kernel.kallsyms] [k] ext4_claim_free_blocks | --- ext4_claim_free_blocks | |--53.08%-- __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --46.92%-- ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] sys_write | --- sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] mcount | --- mcount | |--52.17%-- radix_tree_preload_end | add_to_page_cache_locked | add_to_page_cache_lru | grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.83%-- _raw_spin_unlock ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] pagevec_lru_move_fn | --- pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head | --- jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __block_commit_write | --- __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] virt_to_head_page | --- virt_to_head_page | |--57.98%-- kmem_cache_free | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --42.02%-- jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] mb_test_bit | --- mb_test_bit | |--53.79%-- ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --46.21%-- mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc | |--57.00%-- ext4_mark_iloc_dirty | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.00%-- ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ahci_qc_prep | --- ahci_qc_prep | |--56.69%-- __ata_scsi_queuecmd | ata_scsi_queuecmd | scsi_dispatch_cmd | scsi_request_fn | __blk_run_queue | queue_unplugged | blk_flush_plug_list | io_schedule | sleep_on_page_killable | __wait_on_bit_lock | __lock_page_killable | lock_page_killable | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --43.31%-- ata_qc_issue __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] generic_write_end | --- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] iov_iter_fault_in_readable | --- iov_iter_fault_in_readable | |--51.45%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --48.55%-- __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] inode_reserved_space | --- inode_reserved_space | |--54.74%-- inode_add_rsv_space | __dquot_alloc_space | ext4_da_get_block_prep | __block_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --45.26%-- __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] update_curr | --- update_curr dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] atomic_sub | --- atomic_sub | |--56.22%-- __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --43.78%-- jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_get_inode_flags | --- ext4_get_inode_flags ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] prop_norm_single | --- prop_norm_single | |--55.85%-- task_dirty_inc | account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.15%-- __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] jbd_unlock_bh_state | --- jbd_unlock_bh_state | |--50.48%-- jbd2_journal_dirty_metadata | __ext4_handle_dirty_metadata | ext4_mark_iloc_dirty | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --49.52%-- do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] arch_local_irq_disable | --- arch_local_irq_disable arch_local_irq_save _raw_spin_lock_irqsave | |--65.53%-- __wake_up | jbd2_journal_stop | __ext4_journal_stop | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --34.47%-- pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __generic_file_aio_write | --- __generic_file_aio_write | |--53.33%-- generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --46.67%-- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] test_and_set_bit | --- test_and_set_bit mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% flush-253:2 [kernel.kallsyms] [k] _raw_read_lock_irqsave | --- _raw_read_lock_irqsave dm_merge_bvec __bio_add_page.part.2 bio_add_page ext4_bio_write_page mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __find_get_block | --- __find_get_block __getblk __ext4_get_inode_loc ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty | |--64.27%-- file_update_time | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --35.73%-- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __split_and_process_bio | --- __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] memcg_check_events | --- memcg_check_events __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru | |--52.26%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.74%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] xattr_resolve_name | --- xattr_resolve_name generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% kworker/0:2 [kernel.kallsyms] [k] bit_cursor | --- bit_cursor fb_flashcursor process_one_work worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __set_page_dirty | --- __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_file_write | --- ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock jbd2_journal_put_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] fget_light | --- fget_light sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] ext4_clear_inode_state | --- ext4_clear_inode_state ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% swapper [kernel.kallsyms] [k] ahci_scr_read | --- ahci_scr_read sata_scr_read sata_async_notification ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] add_to_page_cache_lru | --- add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] arch_local_save_flags | --- arch_local_save_flags _local_bh_enable irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] current_fs_time | --- current_fs_time | |--50.34%-- touch_atime | generic_file_aio_read | do_sync_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read | --49.66%-- file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] need_resched | --- need_resched _cond_resched generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] mb_clear_bit | --- mb_clear_bit | |--57.54%-- ext4_free_blocks | ext4_ext_truncate | ext4_truncate | ext4_evict_inode | evict | iput | do_unlinkat | sys_unlinkat | system_call_fastpath | unlinkat | --42.46%-- mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] __list_add | --- __list_add get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] task_dirty_inc | --- task_dirty_inc __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] vfs_read | --- vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] bit_spin_lock | --- bit_spin_lock | |--55.69%-- jbd2_journal_get_write_access | __ext4_journal_get_write_access | ext4_reserve_inode_write | ext4_mark_inode_dirty | ext4_dirty_inode | __mark_inode_dirty | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --44.31%-- jbd2_journal_add_journal_head jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] paravirt_read_tsc | --- paravirt_read_tsc read_tsc timekeeping_get_ns ktime_get tick_nohz_stop_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] irq_entries_start | --- irq_entries_start cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic | --- iov_iter_copy_from_user_atomic generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] clear_buffer_new | --- clear_buffer_new | |--57.80%-- __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --42.20%-- block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave | --- _raw_read_lock_irqsave dm_get_live_table dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] __prop_inc_single | --- __prop_inc_single | |--52.30%-- account_page_dirtied | __set_page_dirty | mark_buffer_dirty | __block_commit_write | block_write_end | generic_write_end | ext4_da_write_end | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --47.70%-- task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] dm_merge_bvec | --- dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] menu_select | --- menu_select | |--59.21%-- cpu_idle | rest_init | start_kernel | x86_64_start_reservations | x86_64_start_kernel | --40.79%-- cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] scsi_decide_disposition | --- scsi_decide_disposition scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] bit_spin_trylock.constprop.23 | --- bit_spin_trylock.constprop.23 get_partial_node __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] exit_idle | --- exit_idle do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io | --- mpage_da_submit_io mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% swapper [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock_irq schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] get_cwq | --- get_cwq __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt _raw_spin_lock inode_add_rsv_space __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ____pagevec_lru_add_fn | --- ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru | |--57.19%-- grab_cache_page_write_begin | ext4_da_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | ext4_file_write | do_sync_write | vfs_write | sys_write | system_call_fastpath | __GI___libc_write | --42.81%-- mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] alloc_buffer_head | --- alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] generic_make_request | --- generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] rcu_irq_enter | --- rcu_irq_enter irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% swapper [kernel.kallsyms] [k] leave_mm | --- leave_mm intel_idle cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] __lru_cache_add | --- __lru_cache_add grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar libc-2.13.90.so [.] __GI___libc_write | --- __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] task_rq_lock | --- task_rq_lock try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] __elv_add_request | --- __elv_add_request blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] may_link | --- may_link selinux_inode_unlink security_inode_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] __find_get_block_slow | --- __find_get_block_slow unmap_underlying_metadata __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] radix_tree_tag_set | --- radix_tree_tag_set __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% flush-253:2 [kernel.kallsyms] [k] do_raw_spin_lock | --- do_raw_spin_lock _raw_spin_lock writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] delayacct_blkio_start | --- delayacct_blkio_start io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] current_umask | --- current_umask ext4_init_acl ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% tar [kernel.kallsyms] [k] linear_merge | --- linear_merge dm_merge_bvec __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% flush-253:2 [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip _raw_spin_lock ext4_discard_preallocations ext4_da_update_reserve_space ext4_ext_map_blocks ext4_map_blocks mpage_da_map_and_submit ext4_da_writepages do_writepages writeback_single_inode writeback_sb_inodes writeback_inodes_wb wb_writeback wb_do_writeback bdi_writeback_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __blk_run_queue | --- __blk_run_queue blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] __page_cache_alloc | --- __page_cache_alloc ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore account_system_vtime irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] ext4_new_inode | --- ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% tar [kernel.kallsyms] [k] __slab_alloc | --- __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount | --- ext4_ext_calc_metadata_amount ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] task_of | --- task_of dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] tick_program_event | --- tick_program_event smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% perf [kernel.kallsyms] [k] __wake_up_bit | --- __wake_up_bit unlock_page generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.01% perf [kernel.kallsyms] [k] generic_write_end | --- generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __write_nocancel 0x4191c6 0x40f7a9 0x40ef8c __libc_start_main 0.01% kworker/0:2 [kernel.kallsyms] [k] native_read_msr_safe | --- native_read_msr_safe acpi_cpufreq_init do_drv_write acpi_cpufreq_target __cpufreq_driver_target do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_journal_start_sb | --- ext4_journal_start_sb ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_get_inode_loc | --- ext4_get_inode_loc ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27 | --- __mem_cgroup_try_charge.constprop.27 mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] hrtick_update | --- hrtick_update enqueue_task activate_task try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page | --- mem_cgroup_get_reclaim_stat_from_page ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] bio_endio | --- bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] kfree | --- kfree jbd2__journal_start jbd2_journal_start ext4_journal_start_sb ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_free_blocks | --- ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] ext4_test_inode_flag | --- ext4_test_inode_flag ext4_map_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% swapper [kernel.kallsyms] [k] scsi_run_queue | --- scsi_run_queue scsi_next_command scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% kworker/0:2 [cpufreq_ondemand] [k] do_dbs_timer | --- do_dbs_timer worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] alloc_page_buffers | --- alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] linear_map | --- linear_map __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag.constprop.7 | --- test_ti_thread_flag.constprop.7 smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.01% tar [kernel.kallsyms] [k] delayacct_blkio_end | --- delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.01% tar [kernel.kallsyms] [k] inode_permission | --- inode_permission vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% kworker/0:2 [kernel.kallsyms] [k] kobject_get | --- kobject_get __cpufreq_driver_getavg do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.01% tar [kernel.kallsyms] [k] __wake_up_common | --- __wake_up_common __wake_up jbd2_journal_stop __ext4_journal_stop ext4_dirty_inode __mark_inode_dirty file_update_time __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.01% tar [kernel.kallsyms] [k] ext4_read_inode_bitmap | --- ext4_read_inode_bitmap ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.01% tar [kernel.kallsyms] [k] mnt_add_count | --- mnt_add_count mntget path_get nameidata_to_filp do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.01% tar [kernel.kallsyms] [k] schedule | --- schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] try_to_wake_up | --- try_to_wake_up wake_up_process wake_up_worker insert_work __queue_work delayed_work_timer_fn run_timer_softirq __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] inode_doinit_with_dentry | --- inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate ext4_add_nondir ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] radix_tree_preload | --- radix_tree_preload add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __mod_zone_page_state | --- __mod_zone_page_state add_page_to_lru_list ____pagevec_lru_add_fn pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] unmap_single | --- unmap_single swiotlb_unmap_sg_attrs ata_sg_clean __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar libc-2.13.90.so [.] _int_malloc | --- _int_malloc 0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state | --- ext4_test_inode_state ext4_xattr_get ext4_xattr_security_get generic_getxattr cap_inode_need_killpriv security_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6 | --- test_ti_thread_flag.constprop.6 __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata | --- __ext4_handle_dirty_metadata ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] queue_unplugged | --- queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] mutex_lock | --- mutex_lock ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] crc16 | --- crc16 ext4_group_desc_csum ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] lock_page_killable | --- lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] lock_page | --- lock_page find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr | --- balance_dirty_pages_ratelimited_nr generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] setup_object | --- setup_object new_slab __slab_alloc kmem_cache_alloc alloc_buffer_head alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] sd_prep_fn | --- sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] system_call_fastpath | --- system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] unlock_page | --- unlock_page generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] blk_throtl_bio | --- blk_throtl_bio generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ihold | --- ihold do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] mb_find_buddy | --- mb_find_buddy mb_free_blocks ext4_free_blocks ext4_ext_truncate ext4_truncate ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access | --- jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] _raw_spin_unlock | --- _raw_spin_unlock scheduler_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] bit_spin_unlock | --- bit_spin_unlock unlock_page_cgroup __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] fput_light | --- fput_light system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] audit_inode.constprop.6 | --- audit_inode.constprop.6 system_call_fastpath __GI___fchown 0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state | --- ext4_test_inode_state ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] system_call | --- system_call __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] ktime_get | --- ktime_get tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] radix_tree_lookup_slot | --- radix_tree_lookup_slot generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] apic_timer_interrupt | --- apic_timer_interrupt generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] __rcu_read_lock | --- __rcu_read_lock __prop_inc_single task_dirty_inc account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] arch_local_irq_enable | --- arch_local_irq_enable _raw_spin_unlock_irq __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] file_read_actor | --- file_read_actor generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] dquot_active | --- dquot_active __dquot_alloc_space ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% sleep [kernel.kallsyms] [k] tlb_gather_mmu | --- tlb_gather_mmu mmput exit_mm do_exit do_group_exit __wake_up_parent system_call_fastpath 0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag | --- test_ti_thread_flag __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] bvec_free_bs | --- bvec_free_bs dm_bio_destructor bio_put clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] ktime_get_real | --- ktime_get_real cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] generic_write_sync | --- generic_write_sync generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] _raw_spin_lock_irq | --- _raw_spin_lock_irq add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] posix_acl_release | --- posix_acl_release ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] ext4_dirty_inode | --- ext4_dirty_inode generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] test_clear_page_writeback | --- test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] deadline_dispatch_requests | --- deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] deadline_move_request | --- deadline_move_request deadline_dispatch_requests blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% kworker/0:2 [kernel.kallsyms] [k] schedule | --- schedule worker_thread kthread kernel_thread_helper 0.00% tar [kernel.kallsyms] [k] blk_start_plug | --- blk_start_plug ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] set_buffer_uptodate | --- set_buffer_uptodate __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] strncpy_from_user | --- strncpy_from_user getname user_path_parent do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc drive_stat_acct __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] update_curr | --- update_curr enqueue_task_fair enqueue_task activate_task try_to_wake_up default_wake_function autoremove_wake_function wake_bit_function __wake_up_common __wake_up __wake_up_bit unlock_page mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] debug_smp_processor_id | --- debug_smp_processor_id rcu_process_callbacks __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] page_cache_async_readahead | --- page_cache_async_readahead do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] bvec_alloc_bs | --- bvec_alloc_bs clone_bio __split_and_process_bio dm_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] page_mapping | --- page_mapping test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] kmem_cache_free | --- kmem_cache_free mempool_free_slab mempool_free dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] prop_norm_percpu | --- prop_norm_percpu __prop_inc_percpu_max test_clear_page_writeback end_page_writeback put_io_page ext4_end_bio bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] sg_init_table | --- sg_init_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] atomic_inc | --- atomic_inc create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] bit_waitqueue | --- bit_waitqueue wake_up_bit unlock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] apic_write | --- apic_write ack_APIC_irq ack_apic_edge handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] native_write_msr_safe | --- native_write_msr_safe 0.00% swapper [kernel.kallsyms] [k] req_bio_endio | --- req_bio_endio blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] __wake_up | --- __wake_up jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] mem_cgroup_disabled | --- mem_cgroup_disabled add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __make_request | --- __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] tick_check_oneshot_broadcast | --- tick_check_oneshot_broadcast tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] raise_softirq | --- raise_softirq update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt radix_tree_preload_end add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ext4_setattr | --- ext4_setattr notify_change utimes_common do_utimes sys_utimensat system_call_fastpath futimens 0.00% swapper [kernel.kallsyms] [k] native_sched_clock | --- native_sched_clock sched_clock_cpu account_system_vtime irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% swapper [kernel.kallsyms] [k] read_tsc | --- read_tsc timekeeping_get_ns ktime_get tick_check_idle irq_enter smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] cap_inode_need_killpriv | --- cap_inode_need_killpriv file_remove_suid __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __cycles_2_ns | --- __cycles_2_ns native_sched_clock sched_clock blk_rq_init get_request get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] load_balance | --- load_balance schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] read_tsc | --- read_tsc ktime_get_ts delayacct_end __delayacct_blkio_end delayacct_blkio_end io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] scsi_sg_alloc | --- scsi_sg_alloc __sg_alloc_table scsi_alloc_sgtable scsi_init_sgtable scsi_init_io scsi_setup_fs_cmnd sd_prep_fn blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] cpuacct_charge | --- cpuacct_charge update_curr dequeue_task_fair dequeue_task deactivate_task schedule io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] elv_rqhash_add | --- elv_rqhash_add __elv_add_request blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22 | --- test_ti_thread_flag.constprop.22 alloc_page_buffers create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] ata_find_dev.part.3 | --- ata_find_dev.part.3 __ata_scsi_find_dev ata_scsi_find_dev ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] fsnotify | --- fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] arch_local_irq_restore | --- arch_local_irq_restore account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% kworker/0:2 [kernel.kallsyms] [k] ns_to_timeval | --- ns_to_timeval get_cpu_iowait_time_us do_dbs_timer process_one_work worker_thread kthread kernel_thread_helper 0.00% tar [kernel.kallsyms] [k] expand | --- expand __rmqueue get_page_from_freelist __alloc_pages_nodemask alloc_pages_current __page_cache_alloc __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] preempt_schedule | --- preempt_schedule blk_finish_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] max_io_len | --- max_io_len __bio_add_page.part.2 bio_add_page do_mpage_readpage mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] release_pages | --- release_pages pagevec_lru_move_fn ____pagevec_lru_add __lru_cache_add add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] irqtime_account_process_tick | --- irqtime_account_process_tick update_process_times tick_sched_timer __run_hrtimer hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] atomic_long_add | --- atomic_long_add zone_page_state_add __inc_zone_state __inc_zone_page_state account_page_dirtied __set_page_dirty mark_buffer_dirty __block_commit_write block_write_end generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] zone_statistics | --- zone_statistics __alloc_pages_nodemask alloc_pages_current __page_cache_alloc grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] str2hashbuf_signed | --- str2hashbuf_signed ext4fs_dirhash dx_probe ext4_dx_find_entry ext4_find_entry ext4_unlink vfs_unlink do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% swapper [kernel.kallsyms] [k] ata_scsi_qc_complete | --- ata_scsi_qc_complete __ata_qc_complete ata_qc_complete ata_qc_complete_multiple ahci_interrupt handle_irq_event_percpu handle_irq_event handle_edge_irq handle_irq do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] ext4_has_free_blocks | --- ext4_has_free_blocks ext4_da_get_block_prep __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] deadline_add_request | --- deadline_add_request blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] pagefault_enable | --- pagefault_enable generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] deadline_merge | --- deadline_merge elv_merge __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] rb_next | --- rb_next timerqueue_del __remove_hrtimer remove_hrtimer.part.4 hrtimer_try_to_cancel hrtimer_cancel tick_nohz_restart_sched_tick cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge | --- mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] sched_clock_tick | --- sched_clock_tick sched_clock_idle_wakeup_event tick_nohz_stop_idle tick_check_idle irq_enter do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] put_bh | --- put_bh __brelse brelse ext4_mark_iloc_dirty ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] get_page | --- get_page __lru_cache_add add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] hashtab_search | --- hashtab_search string_to_context_struct security_context_to_sid_core security_context_to_sid_default inode_doinit_with_dentry selinux_d_instantiate security_d_instantiate d_instantiate d_splice_alias ext4_lookup d_alloc_and_lookup __lookup_hash.part.3 lookup_hash do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% tar [kernel.kallsyms] [k] ata_qc_new_init | --- ata_qc_new_init __ata_scsi_queuecmd ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] sd_done | --- sd_done scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% sleep [kernel.kallsyms] [k] get_parent_ip | --- get_parent_ip add_preempt_count get_page_from_freelist __alloc_pages_nodemask alloc_pages_current pte_alloc_one __pte_alloc handle_mm_fault do_page_fault page_fault _dl_vdso_vsym 0x7fff16d617f5 0x535345535f474458 0.00% sleep [kernel.kallsyms] [k] policy_zonelist | --- policy_zonelist pte_alloc_one __pte_alloc handle_mm_fault do_page_fault page_fault _dl_init_paths _dl_sysdep_start 0.00% tar [kernel.kallsyms] [k] find_lock_page | --- find_lock_page grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] lock_buffer | --- lock_buffer do_get_write_access jbd2_journal_get_write_access __ext4_journal_get_write_access ext4_reserve_inode_write ext4_mark_inode_dirty ext4_dirty_inode __mark_inode_dirty generic_write_end ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] _cond_resched | --- _cond_resched generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ext4_da_write_begin | --- ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] dquot_free_inode | --- dquot_free_inode ext4_free_inode ext4_evict_inode evict iput do_unlinkat sys_unlinkat system_call_fastpath unlinkat 0.00% tar [kernel.kallsyms] [k] get_request_wait | --- get_request_wait __make_request generic_make_request submit_bio mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% swapper [kernel.kallsyms] [k] __local_bh_enable | --- __local_bh_enable __do_softirq call_softirq do_softirq irq_exit smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] mutex_unlock | --- mutex_unlock generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% swapper [kernel.kallsyms] [k] hrtimer_interrupt | --- hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] create_empty_buffers | --- create_empty_buffers __block_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] radix_tree_preload_end | --- radix_tree_preload_end add_to_page_cache_locked add_to_page_cache_lru grab_cache_page_write_begin ext4_da_write_begin generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] ext4_xattr_set_entry | --- ext4_xattr_set_entry ext4_xattr_ibody_set ext4_xattr_set_handle ext4_init_security ext4_new_inode ext4_create vfs_create do_last path_openat do_filp_open do_sys_open sys_openat system_call_fastpath __GI___openat 0.00% swapper [kernel.kallsyms] [k] SetPageUptodate | --- SetPageUptodate mpage_end_io bio_endio dec_pending clone_endio bio_endio req_bio_endio blk_update_request blk_update_bidi_request blk_end_bidi_request blk_end_request scsi_io_completion scsi_finish_command scsi_softirq_done blk_done_softirq __do_softirq call_softirq do_softirq irq_exit do_IRQ common_interrupt cpuidle_idle_call cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics | --- mem_cgroup_charge_statistics __mem_cgroup_commit_charge.constprop.28 __mem_cgroup_commit_charge_lrucare mem_cgroup_cache_charge add_to_page_cache_locked add_to_page_cache_lru mpage_readpages ext4_readpages __do_page_cache_readahead ra_submit ondemand_readahead page_cache_async_readahead generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar libc-2.13.90.so [.] _int_free | --- _int_free 0.00% swapper [kernel.kallsyms] [k] schedule | --- schedule cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.00% tar [kernel.kallsyms] [k] sysret_check | --- sysret_check __GI___libc_read 0.00% tar [kernel.kallsyms] [k] blk_peek_request | --- blk_peek_request scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] ata_scsi_rw_xlat | --- ata_scsi_rw_xlat ata_scsi_queuecmd scsi_dispatch_cmd scsi_request_fn __blk_run_queue queue_unplugged blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] blk_flush_plug_list | --- blk_flush_plug_list io_schedule sleep_on_page_killable __wait_on_bit_lock __lock_page_killable lock_page_killable generic_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath __GI___libc_read 0.00% tar [kernel.kallsyms] [k] jbd2_journal_stop | --- jbd2_journal_stop __ext4_journal_stop ext4_da_write_end generic_file_buffered_write __generic_file_aio_write generic_file_aio_write ext4_file_write do_sync_write vfs_write sys_write system_call_fastpath __GI___libc_write 0.00% tar [kernel.kallsyms] [k] __srcu_read_lock | --- __srcu_read_lock fsnotify fsnotify_modify vfs_write sys_write system_call_fastpath __GI___libc_write # # (For a higher level overview, try: perf report --sort comm,dso) # -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 16:56 ` James Bottomley @ 2011-04-28 17:18 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 17:18 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote: > > This is the output of perf record -g -a -f sleep 5 > > > > (hopefully the list won't choke) > > Um, this one actually shows kswapd > > James > > --- > > # Events: 6K cycles > # > # Overhead Command Shared Object Symbol > # ........ ........... ................... ....................................... > # > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > | > --- shrink_slab > | > |--99.91%-- kswapd > | kthread > | kernel_thread_helper > --0.09%-- [...] > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use compaction instead of lumpy reclaim" is related unless we are seeing two problems that happen to manifest in a similar manner. However, there were a number of changes made to dcache in particular for 2.6.38. Specifically thinks like dentry_kill use trylock and is happy to loop around if it fails to acquire anything. See things like this for example; static void try_prune_one_dentry(struct dentry *dentry) __releases(dentry->d_lock) { struct dentry *parent; parent = dentry_kill(dentry, 0); /* * If dentry_kill returns NULL, we have nothing more to do. * if it returns the same dentry, trylocks failed. In either * case, just loop again. If this in combination with many inodes being locked for whatever reason (writeback locking them maybe?) is causing the shrinker to return after zero progress, it could in turn cause kswapd to enter into a loop for longish periods of time in shrink_slab here; while (total_scan >= SHRINK_BATCH) { long this_scan = SHRINK_BATCH; int shrink_ret; int nr_before; nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask); shrink_ret = (*shrinker->shrink)(shrinker, this_scan, gfp_mask); if (shrink_ret == -1) break; if (shrink_ret < nr_before) ret += nr_before - shrink_ret; count_vm_events(SLABS_SCANNED, this_scan); total_scan -= this_scan; cond_resched(); } That would explain this trace. > 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone > | > --- shrink_zone > | > |--99.46%-- kswapd > | kthread > | kernel_thread_helper > | > --0.54%-- kthread > kernel_thread_helper > > 7.70% kswapd0 [kernel.kallsyms] [k] kswapd > | > --- kswapd > kthread > kernel_thread_helper > > 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe > | > --- zone_watermark_ok_safe > | > |--72.66%-- kswapd > | kthread > | kernel_thread_helper > | > |--20.88%-- sleeping_prematurely.part.12 > | kswapd > | kthread > | kernel_thread_helper > | > --6.46%-- kthread > kernel_thread_helper > We are also spending an astonishing amount of time in sleeping_prematurely leading me to believe we are failing to balance the zones and are continually under the min watermark for one of the zones. We are never going to sleep because of this check; if (total_scanned && (priority < DEF_PRIORITY - 2)) { if (has_under_min_watermark_zone) count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT); else congestion_wait(BLK_RW_ASYNC, HZ/10); } However, I think this is a secondary effect to the failure of shrinkers to do their work. If slabs were being shrunk, one would expect us to be getting over the min watermark. > 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock > | > --- do_raw_spin_lock > | > |--77.49%-- _raw_spin_lock > | | > | |--51.85%-- mb_cache_shrink_fn > | | shrink_slab > | | kswapd > | | kthread > | | kernel_thread_helper > | | > | --48.15%-- mem_cgroup_soft_limit_reclaim > | kswapd > | kthread > | kernel_thread_helper > | Way hey, cgroups are also in the mix. How jolly. Is systemd a common element of the machines hitting this bug by any chance? The remaining traces seem to be follow-on damage related to the three issues of "shrinkers are bust in some manner" causing "we are not getting over the min watermark" and as a side-show "we are spending lots of time doing something unspecified but unhelpful in cgroups". > <SNIP> -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 17:18 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 17:18 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote: > > This is the output of perf record -g -a -f sleep 5 > > > > (hopefully the list won't choke) > > Um, this one actually shows kswapd > > James > > --- > > # Events: 6K cycles > # > # Overhead Command Shared Object Symbol > # ........ ........... ................... ....................................... > # > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > | > --- shrink_slab > | > |--99.91%-- kswapd > | kthread > | kernel_thread_helper > --0.09%-- [...] > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use compaction instead of lumpy reclaim" is related unless we are seeing two problems that happen to manifest in a similar manner. However, there were a number of changes made to dcache in particular for 2.6.38. Specifically thinks like dentry_kill use trylock and is happy to loop around if it fails to acquire anything. See things like this for example; static void try_prune_one_dentry(struct dentry *dentry) __releases(dentry->d_lock) { struct dentry *parent; parent = dentry_kill(dentry, 0); /* * If dentry_kill returns NULL, we have nothing more to do. * if it returns the same dentry, trylocks failed. In either * case, just loop again. If this in combination with many inodes being locked for whatever reason (writeback locking them maybe?) is causing the shrinker to return after zero progress, it could in turn cause kswapd to enter into a loop for longish periods of time in shrink_slab here; while (total_scan >= SHRINK_BATCH) { long this_scan = SHRINK_BATCH; int shrink_ret; int nr_before; nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask); shrink_ret = (*shrinker->shrink)(shrinker, this_scan, gfp_mask); if (shrink_ret == -1) break; if (shrink_ret < nr_before) ret += nr_before - shrink_ret; count_vm_events(SLABS_SCANNED, this_scan); total_scan -= this_scan; cond_resched(); } That would explain this trace. > 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone > | > --- shrink_zone > | > |--99.46%-- kswapd > | kthread > | kernel_thread_helper > | > --0.54%-- kthread > kernel_thread_helper > > 7.70% kswapd0 [kernel.kallsyms] [k] kswapd > | > --- kswapd > kthread > kernel_thread_helper > > 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe > | > --- zone_watermark_ok_safe > | > |--72.66%-- kswapd > | kthread > | kernel_thread_helper > | > |--20.88%-- sleeping_prematurely.part.12 > | kswapd > | kthread > | kernel_thread_helper > | > --6.46%-- kthread > kernel_thread_helper > We are also spending an astonishing amount of time in sleeping_prematurely leading me to believe we are failing to balance the zones and are continually under the min watermark for one of the zones. We are never going to sleep because of this check; if (total_scanned && (priority < DEF_PRIORITY - 2)) { if (has_under_min_watermark_zone) count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT); else congestion_wait(BLK_RW_ASYNC, HZ/10); } However, I think this is a secondary effect to the failure of shrinkers to do their work. If slabs were being shrunk, one would expect us to be getting over the min watermark. > 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock > | > --- do_raw_spin_lock > | > |--77.49%-- _raw_spin_lock > | | > | |--51.85%-- mb_cache_shrink_fn > | | shrink_slab > | | kswapd > | | kthread > | | kernel_thread_helper > | | > | --48.15%-- mem_cgroup_soft_limit_reclaim > | kswapd > | kthread > | kernel_thread_helper > | Way hey, cgroups are also in the mix. How jolly. Is systemd a common element of the machines hitting this bug by any chance? The remaining traces seem to be follow-on damage related to the three issues of "shrinkers are bust in some manner" causing "we are not getting over the min watermark" and as a side-show "we are spending lots of time doing something unspecified but unhelpful in cgroups". > <SNIP> -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 17:18 ` Mel Gorman @ 2011-04-28 18:30 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 18:30 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > > # Events: 6K cycles > > # > > # Overhead Command Shared Object Symbol > > # ........ ........... ................... ....................................... > > # > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > > | > > --- shrink_slab > > | > > |--99.91%-- kswapd > > | kthread > > | kernel_thread_helper > > --0.09%-- [...] > > > > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use > compaction instead of lumpy reclaim" is related unless we are seeing > two problems that happen to manifest in a similar manner. > > However, there were a number of changes made to dcache in particular > for 2.6.38. Specifically thinks like dentry_kill use trylock and is > happy to loop around if it fails to acquire anything. See things like > this for example; OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well, networking is hosed for no reason I can see (probably systemd / cgroups problems). However, it runs enough for me to say that the tar proceeds to completion in a non-PREEMPT kernel. (I tried several times for good measure). That makes this definitely a regression of some sort, but it doesn't definitively identify the dcache code ... it could be an ext4 bug that got introduced in 2.6.38 either. > static void try_prune_one_dentry(struct dentry *dentry) > __releases(dentry->d_lock) > { > struct dentry *parent; > > parent = dentry_kill(dentry, 0); > /* > * If dentry_kill returns NULL, we have nothing more to do. > * if it returns the same dentry, trylocks failed. In either > * case, just loop again. > > > If this in combination with many inodes being locked for whatever > reason (writeback locking them maybe?) is causing the shrinker to > return after zero progress, it could in turn cause kswapd to enter > into a loop for longish periods of time in shrink_slab here; > > while (total_scan >= SHRINK_BATCH) { > long this_scan = SHRINK_BATCH; > int shrink_ret; > int nr_before; > > nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask); > shrink_ret = (*shrinker->shrink)(shrinker, this_scan, > gfp_mask); > if (shrink_ret == -1) > break; > if (shrink_ret < nr_before) > ret += nr_before - shrink_ret; > count_vm_events(SLABS_SCANNED, this_scan); > total_scan -= this_scan; > > cond_resched(); > } > > That would explain this trace. > > > 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone > > | > > --- shrink_zone > > | > > |--99.46%-- kswapd > > | kthread > > | kernel_thread_helper > > | > > --0.54%-- kthread > > kernel_thread_helper > > > > 7.70% kswapd0 [kernel.kallsyms] [k] kswapd > > | > > --- kswapd > > kthread > > kernel_thread_helper > > > > 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe > > | > > --- zone_watermark_ok_safe > > | > > |--72.66%-- kswapd > > | kthread > > | kernel_thread_helper > > | > > |--20.88%-- sleeping_prematurely.part.12 > > | kswapd > > | kthread > > | kernel_thread_helper > > | > > --6.46%-- kthread > > kernel_thread_helper > > > > We are also spending an astonishing amount of time in > sleeping_prematurely leading me to believe we are failing to balance the > zones and are continually under the min watermark for one of the zones. > We are never going to sleep because of this check; > > if (total_scanned && (priority < DEF_PRIORITY - 2)) { > if (has_under_min_watermark_zone) > count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT); > else > congestion_wait(BLK_RW_ASYNC, HZ/10); > } > > However, I think this is a secondary effect to the failure of shrinkers > to do their work. If slabs were being shrunk, one would expect us to > be getting over the min watermark. > > > 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock > > | > > --- do_raw_spin_lock > > | > > |--77.49%-- _raw_spin_lock > > | | > > | |--51.85%-- mb_cache_shrink_fn > > | | shrink_slab > > | | kswapd > > | | kthread > > | | kernel_thread_helper > > | | > > | --48.15%-- mem_cgroup_soft_limit_reclaim > > | kswapd > > | kthread > > | kernel_thread_helper > > | > > Way hey, cgroups are also in the mix. How jolly. > > Is systemd a common element of the machines hitting this bug by any > chance? Well, yes, the bug report is against FC15, which needs cgroups for systemd. > The remaining traces seem to be follow-on damage related to the three > issues of "shrinkers are bust in some manner" causing "we are not > getting over the min watermark" and as a side-show "we are spending lots > of time doing something unspecified but unhelpful in cgroups". Heh, well find a way for me to verify this: I can't turn off cgroups because systemd then won't work and the machine won't boot ... James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 18:30 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 18:30 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > > # Events: 6K cycles > > # > > # Overhead Command Shared Object Symbol > > # ........ ........... ................... ....................................... > > # > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > > | > > --- shrink_slab > > | > > |--99.91%-- kswapd > > | kthread > > | kernel_thread_helper > > --0.09%-- [...] > > > > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use > compaction instead of lumpy reclaim" is related unless we are seeing > two problems that happen to manifest in a similar manner. > > However, there were a number of changes made to dcache in particular > for 2.6.38. Specifically thinks like dentry_kill use trylock and is > happy to loop around if it fails to acquire anything. See things like > this for example; OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well, networking is hosed for no reason I can see (probably systemd / cgroups problems). However, it runs enough for me to say that the tar proceeds to completion in a non-PREEMPT kernel. (I tried several times for good measure). That makes this definitely a regression of some sort, but it doesn't definitively identify the dcache code ... it could be an ext4 bug that got introduced in 2.6.38 either. > static void try_prune_one_dentry(struct dentry *dentry) > __releases(dentry->d_lock) > { > struct dentry *parent; > > parent = dentry_kill(dentry, 0); > /* > * If dentry_kill returns NULL, we have nothing more to do. > * if it returns the same dentry, trylocks failed. In either > * case, just loop again. > > > If this in combination with many inodes being locked for whatever > reason (writeback locking them maybe?) is causing the shrinker to > return after zero progress, it could in turn cause kswapd to enter > into a loop for longish periods of time in shrink_slab here; > > while (total_scan >= SHRINK_BATCH) { > long this_scan = SHRINK_BATCH; > int shrink_ret; > int nr_before; > > nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask); > shrink_ret = (*shrinker->shrink)(shrinker, this_scan, > gfp_mask); > if (shrink_ret == -1) > break; > if (shrink_ret < nr_before) > ret += nr_before - shrink_ret; > count_vm_events(SLABS_SCANNED, this_scan); > total_scan -= this_scan; > > cond_resched(); > } > > That would explain this trace. > > > 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone > > | > > --- shrink_zone > > | > > |--99.46%-- kswapd > > | kthread > > | kernel_thread_helper > > | > > --0.54%-- kthread > > kernel_thread_helper > > > > 7.70% kswapd0 [kernel.kallsyms] [k] kswapd > > | > > --- kswapd > > kthread > > kernel_thread_helper > > > > 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe > > | > > --- zone_watermark_ok_safe > > | > > |--72.66%-- kswapd > > | kthread > > | kernel_thread_helper > > | > > |--20.88%-- sleeping_prematurely.part.12 > > | kswapd > > | kthread > > | kernel_thread_helper > > | > > --6.46%-- kthread > > kernel_thread_helper > > > > We are also spending an astonishing amount of time in > sleeping_prematurely leading me to believe we are failing to balance the > zones and are continually under the min watermark for one of the zones. > We are never going to sleep because of this check; > > if (total_scanned && (priority < DEF_PRIORITY - 2)) { > if (has_under_min_watermark_zone) > count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT); > else > congestion_wait(BLK_RW_ASYNC, HZ/10); > } > > However, I think this is a secondary effect to the failure of shrinkers > to do their work. If slabs were being shrunk, one would expect us to > be getting over the min watermark. > > > 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock > > | > > --- do_raw_spin_lock > > | > > |--77.49%-- _raw_spin_lock > > | | > > | |--51.85%-- mb_cache_shrink_fn > > | | shrink_slab > > | | kswapd > > | | kthread > > | | kernel_thread_helper > > | | > > | --48.15%-- mem_cgroup_soft_limit_reclaim > > | kswapd > > | kthread > > | kernel_thread_helper > > | > > Way hey, cgroups are also in the mix. How jolly. > > Is systemd a common element of the machines hitting this bug by any > chance? Well, yes, the bug report is against FC15, which needs cgroups for systemd. > The remaining traces seem to be follow-on damage related to the three > issues of "shrinkers are bust in some manner" causing "we are not > getting over the min watermark" and as a side-show "we are spending lots > of time doing something unspecified but unhelpful in cgroups". Heh, well find a way for me to verify this: I can't turn off cgroups because systemd then won't work and the machine won't boot ... James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 18:30 ` James Bottomley @ 2011-04-28 19:21 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 19:21 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote: > > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > > > # Events: 6K cycles > > > # > > > # Overhead Command Shared Object Symbol > > > # ........ ........... ................... ....................................... > > > # > > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > > > | > > > --- shrink_slab > > > | > > > |--99.91%-- kswapd > > > | kthread > > > | kernel_thread_helper > > > --0.09%-- [...] > > > > > > > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use > > compaction instead of lumpy reclaim" is related unless we are seeing > > two problems that happen to manifest in a similar manner. > > > > However, there were a number of changes made to dcache in particular > > for 2.6.38. Specifically thinks like dentry_kill use trylock and is > > happy to loop around if it fails to acquire anything. See things like > > this for example; > > OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well, > networking is hosed for no reason I can see (probably systemd / cgroups > problems). > > However, it runs enough for me to say that the tar proceeds to > completion in a non-PREEMPT kernel. (I tried several times for good > measure). That makes this definitely a regression of some sort, but it > doesn't definitively identify the dcache code ... it could be an ext4 > bug that got introduced in 2.6.38 either. > True, it could be any shrinker and dcache is just a guess. > > <SNIP> > > > > Way hey, cgroups are also in the mix. How jolly. > > > > Is systemd a common element of the machines hitting this bug by any > > chance? > > Well, yes, the bug report is against FC15, which needs cgroups for > systemd. > Ok although we do not have direct evidence that it's the problem yet. A broken shrinker could just mean we are also trying to aggressively reclaim in cgroups. > > The remaining traces seem to be follow-on damage related to the three > > issues of "shrinkers are bust in some manner" causing "we are not > > getting over the min watermark" and as a side-show "we are spending lots > > of time doing something unspecified but unhelpful in cgroups". > > Heh, well find a way for me to verify this: I can't turn off cgroups > because systemd then won't work and the machine won't boot ... > Same testcase, same kernel but a distro that is not using systemd to verify if cgroups are the problem. Not ideal I know. When I'm back online Tuesday, I'll try reproducing this on a !Fedora distribution. In the meantime, the following untested hatchet job might spit out which shrinker we are getting stuck in. It is also breaking out of the shrink_slab loop so it'd even be interesting to see if the bug is mitigated in any way. diff --git a/mm/vmscan.c b/mm/vmscan.c index c74a501..ed99104 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -225,6 +225,7 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask, { struct shrinker *shrinker; unsigned long ret = 0; + unsigned long shrink_expired = jiffies + HZ; if (scanned == 0) scanned = SWAP_CLUSTER_MAX; @@ -270,6 +271,14 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask, gfp_mask); if (shrink_ret == -1) break; + if (time_after(jiffies, shrink_expired)) { + printk(KERN_WARNING "Slab shrinker %p gone mental" + " comm=%s nr=%ld\n", + shrinker->shrink, + current->comm, + shrinker->nr); + break; + } if (shrink_ret < nr_before) ret += nr_before - shrink_ret; count_vm_events(SLABS_SCANNED, this_scan); ^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 19:21 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 19:21 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote: > > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > > > # Events: 6K cycles > > > # > > > # Overhead Command Shared Object Symbol > > > # ........ ........... ................... ....................................... > > > # > > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > > > | > > > --- shrink_slab > > > | > > > |--99.91%-- kswapd > > > | kthread > > > | kernel_thread_helper > > > --0.09%-- [...] > > > > > > > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use > > compaction instead of lumpy reclaim" is related unless we are seeing > > two problems that happen to manifest in a similar manner. > > > > However, there were a number of changes made to dcache in particular > > for 2.6.38. Specifically thinks like dentry_kill use trylock and is > > happy to loop around if it fails to acquire anything. See things like > > this for example; > > OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well, > networking is hosed for no reason I can see (probably systemd / cgroups > problems). > > However, it runs enough for me to say that the tar proceeds to > completion in a non-PREEMPT kernel. (I tried several times for good > measure). That makes this definitely a regression of some sort, but it > doesn't definitively identify the dcache code ... it could be an ext4 > bug that got introduced in 2.6.38 either. > True, it could be any shrinker and dcache is just a guess. > > <SNIP> > > > > Way hey, cgroups are also in the mix. How jolly. > > > > Is systemd a common element of the machines hitting this bug by any > > chance? > > Well, yes, the bug report is against FC15, which needs cgroups for > systemd. > Ok although we do not have direct evidence that it's the problem yet. A broken shrinker could just mean we are also trying to aggressively reclaim in cgroups. > > The remaining traces seem to be follow-on damage related to the three > > issues of "shrinkers are bust in some manner" causing "we are not > > getting over the min watermark" and as a side-show "we are spending lots > > of time doing something unspecified but unhelpful in cgroups". > > Heh, well find a way for me to verify this: I can't turn off cgroups > because systemd then won't work and the machine won't boot ... > Same testcase, same kernel but a distro that is not using systemd to verify if cgroups are the problem. Not ideal I know. When I'm back online Tuesday, I'll try reproducing this on a !Fedora distribution. In the meantime, the following untested hatchet job might spit out which shrinker we are getting stuck in. It is also breaking out of the shrink_slab loop so it'd even be interesting to see if the bug is mitigated in any way. diff --git a/mm/vmscan.c b/mm/vmscan.c index c74a501..ed99104 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -225,6 +225,7 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask, { struct shrinker *shrinker; unsigned long ret = 0; + unsigned long shrink_expired = jiffies + HZ; if (scanned == 0) scanned = SWAP_CLUSTER_MAX; @@ -270,6 +271,14 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask, gfp_mask); if (shrink_ret == -1) break; + if (time_after(jiffies, shrink_expired)) { + printk(KERN_WARNING "Slab shrinker %p gone mental" + " comm=%s nr=%ld\n", + shrinker->shrink, + current->comm, + shrinker->nr); + break; + } if (shrink_ret < nr_before) ret += nr_before - shrink_ret; count_vm_events(SLABS_SCANNED, this_scan); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 19:21 ` Mel Gorman @ 2011-04-28 19:59 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 19:59 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > > > Way hey, cgroups are also in the mix. How jolly. > > > > > > Is systemd a common element of the machines hitting this bug by any > > > chance? > > > > Well, yes, the bug report is against FC15, which needs cgroups for > > systemd. > > > > Ok although we do not have direct evidence that it's the problem yet. A > broken shrinker could just mean we are also trying to aggressively > reclaim in cgroups. > > > > The remaining traces seem to be follow-on damage related to the three > > > issues of "shrinkers are bust in some manner" causing "we are not > > > getting over the min watermark" and as a side-show "we are spending lots > > > of time doing something unspecified but unhelpful in cgroups". > > > > Heh, well find a way for me to verify this: I can't turn off cgroups > > because systemd then won't work and the machine won't boot ... > > > > Same testcase, same kernel but a distro that is not using systemd to > verify if cgroups are the problem. Not ideal I know. When I'm back > online Tuesday, I'll try reproducing this on a !Fedora distribution. In > the meantime, the following untested hatchet job might spit out > which shrinker we are getting stuck in. It is also breaking out of > the shrink_slab loop so it'd even be interesting to see if the bug > is mitigated in any way. Actually, talking to Chris, I think I can get the system up using init=/bin/bash without systemd, so I can try the no cgroup config. > diff --git a/mm/vmscan.c b/mm/vmscan.c > index c74a501..ed99104 100644 In the mean time, this patch produces: (that's nothing ... apparently the trace doesn't activate when kswapd goes mad). James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 19:59 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 19:59 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > > > Way hey, cgroups are also in the mix. How jolly. > > > > > > Is systemd a common element of the machines hitting this bug by any > > > chance? > > > > Well, yes, the bug report is against FC15, which needs cgroups for > > systemd. > > > > Ok although we do not have direct evidence that it's the problem yet. A > broken shrinker could just mean we are also trying to aggressively > reclaim in cgroups. > > > > The remaining traces seem to be follow-on damage related to the three > > > issues of "shrinkers are bust in some manner" causing "we are not > > > getting over the min watermark" and as a side-show "we are spending lots > > > of time doing something unspecified but unhelpful in cgroups". > > > > Heh, well find a way for me to verify this: I can't turn off cgroups > > because systemd then won't work and the machine won't boot ... > > > > Same testcase, same kernel but a distro that is not using systemd to > verify if cgroups are the problem. Not ideal I know. When I'm back > online Tuesday, I'll try reproducing this on a !Fedora distribution. In > the meantime, the following untested hatchet job might spit out > which shrinker we are getting stuck in. It is also breaking out of > the shrink_slab loop so it'd even be interesting to see if the bug > is mitigated in any way. Actually, talking to Chris, I think I can get the system up using init=/bin/bash without systemd, so I can try the no cgroup config. > diff --git a/mm/vmscan.c b/mm/vmscan.c > index c74a501..ed99104 100644 In the mean time, this patch produces: (that's nothing ... apparently the trace doesn't activate when kswapd goes mad). James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 19:59 ` James Bottomley @ 2011-04-28 20:27 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 20:27 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote: > > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > > > > Way hey, cgroups are also in the mix. How jolly. > > > > > > > > Is systemd a common element of the machines hitting this bug by any > > > > chance? > > > > > > Well, yes, the bug report is against FC15, which needs cgroups for > > > systemd. > > > > > > > Ok although we do not have direct evidence that it's the problem yet. A > > broken shrinker could just mean we are also trying to aggressively > > reclaim in cgroups. > > > > > > The remaining traces seem to be follow-on damage related to the three > > > > issues of "shrinkers are bust in some manner" causing "we are not > > > > getting over the min watermark" and as a side-show "we are spending lots > > > > of time doing something unspecified but unhelpful in cgroups". > > > > > > Heh, well find a way for me to verify this: I can't turn off cgroups > > > because systemd then won't work and the machine won't boot ... > > > > > > > Same testcase, same kernel but a distro that is not using systemd to > > verify if cgroups are the problem. Not ideal I know. When I'm back > > online Tuesday, I'll try reproducing this on a !Fedora distribution. In > > the meantime, the following untested hatchet job might spit out > > which shrinker we are getting stuck in. It is also breaking out of > > the shrink_slab loop so it'd even be interesting to see if the bug > > is mitigated in any way. > > Actually, talking to Chris, I think I can get the system up using > init=/bin/bash without systemd, so I can try the no cgroup config. > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index c74a501..ed99104 100644 > > In the mean time, this patch produces: > > (that's nothing ... apparently the trace doesn't activate when kswapd > goes mad). > Or is looping there for shorter than we expect. HZ/10? -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 20:27 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-04-28 20:27 UTC (permalink / raw) To: James Bottomley Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote: > > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > > > > Way hey, cgroups are also in the mix. How jolly. > > > > > > > > Is systemd a common element of the machines hitting this bug by any > > > > chance? > > > > > > Well, yes, the bug report is against FC15, which needs cgroups for > > > systemd. > > > > > > > Ok although we do not have direct evidence that it's the problem yet. A > > broken shrinker could just mean we are also trying to aggressively > > reclaim in cgroups. > > > > > > The remaining traces seem to be follow-on damage related to the three > > > > issues of "shrinkers are bust in some manner" causing "we are not > > > > getting over the min watermark" and as a side-show "we are spending lots > > > > of time doing something unspecified but unhelpful in cgroups". > > > > > > Heh, well find a way for me to verify this: I can't turn off cgroups > > > because systemd then won't work and the machine won't boot ... > > > > > > > Same testcase, same kernel but a distro that is not using systemd to > > verify if cgroups are the problem. Not ideal I know. When I'm back > > online Tuesday, I'll try reproducing this on a !Fedora distribution. In > > the meantime, the following untested hatchet job might spit out > > which shrinker we are getting stuck in. It is also breaking out of > > the shrink_slab loop so it'd even be interesting to see if the bug > > is mitigated in any way. > > Actually, talking to Chris, I think I can get the system up using > init=/bin/bash without systemd, so I can try the no cgroup config. > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index c74a501..ed99104 100644 > > In the mean time, this patch produces: > > (that's nothing ... apparently the trace doesn't activate when kswapd > goes mad). > Or is looping there for shorter than we expect. HZ/10? -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 20:27 ` Mel Gorman @ 2011-04-29 15:02 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 15:02 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 21:27 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote: > > > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > > > > > Way hey, cgroups are also in the mix. How jolly. > > > > > > > > > > Is systemd a common element of the machines hitting this bug by any > > > > > chance? > > > > > > > > Well, yes, the bug report is against FC15, which needs cgroups for > > > > systemd. > > > > > > > > > > Ok although we do not have direct evidence that it's the problem yet. A > > > broken shrinker could just mean we are also trying to aggressively > > > reclaim in cgroups. > > > > > > > > The remaining traces seem to be follow-on damage related to the three > > > > > issues of "shrinkers are bust in some manner" causing "we are not > > > > > getting over the min watermark" and as a side-show "we are spending lots > > > > > of time doing something unspecified but unhelpful in cgroups". > > > > > > > > Heh, well find a way for me to verify this: I can't turn off cgroups > > > > because systemd then won't work and the machine won't boot ... > > > > > > > > > > Same testcase, same kernel but a distro that is not using systemd to > > > verify if cgroups are the problem. Not ideal I know. When I'm back > > > online Tuesday, I'll try reproducing this on a !Fedora distribution. In > > > the meantime, the following untested hatchet job might spit out > > > which shrinker we are getting stuck in. It is also breaking out of > > > the shrink_slab loop so it'd even be interesting to see if the bug > > > is mitigated in any way. > > > > Actually, talking to Chris, I think I can get the system up using > > init=/bin/bash without systemd, so I can try the no cgroup config. > > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > > index c74a501..ed99104 100644 > > > > In the mean time, this patch produces: > > > > (that's nothing ... apparently the trace doesn't activate when kswapd > > goes mad). > > > > Or is looping there for shorter than we expect. HZ/10? Still doesn't print anything, even with HZ/10. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-29 15:02 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 15:02 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 21:27 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote: > > > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote: > > > > > Way hey, cgroups are also in the mix. How jolly. > > > > > > > > > > Is systemd a common element of the machines hitting this bug by any > > > > > chance? > > > > > > > > Well, yes, the bug report is against FC15, which needs cgroups for > > > > systemd. > > > > > > > > > > Ok although we do not have direct evidence that it's the problem yet. A > > > broken shrinker could just mean we are also trying to aggressively > > > reclaim in cgroups. > > > > > > > > The remaining traces seem to be follow-on damage related to the three > > > > > issues of "shrinkers are bust in some manner" causing "we are not > > > > > getting over the min watermark" and as a side-show "we are spending lots > > > > > of time doing something unspecified but unhelpful in cgroups". > > > > > > > > Heh, well find a way for me to verify this: I can't turn off cgroups > > > > because systemd then won't work and the machine won't boot ... > > > > > > > > > > Same testcase, same kernel but a distro that is not using systemd to > > > verify if cgroups are the problem. Not ideal I know. When I'm back > > > online Tuesday, I'll try reproducing this on a !Fedora distribution. In > > > the meantime, the following untested hatchet job might spit out > > > which shrinker we are getting stuck in. It is also breaking out of > > > the shrink_slab loop so it'd even be interesting to see if the bug > > > is mitigated in any way. > > > > Actually, talking to Chris, I think I can get the system up using > > init=/bin/bash without systemd, so I can try the no cgroup config. > > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > > index c74a501..ed99104 100644 > > > > In the mean time, this patch produces: > > > > (that's nothing ... apparently the trace doesn't activate when kswapd > > goes mad). > > > > Or is looping there for shorter than we expect. HZ/10? Still doesn't print anything, even with HZ/10. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 19:59 ` James Bottomley @ 2011-04-28 21:12 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 21:12 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > Actually, talking to Chris, I think I can get the system up using > init=/bin/bash without systemd, so I can try the no cgroup config. OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back runs of untar without locking or getting kswapd pegged, so I'm pretty certain this is cgroups related. The next steps are to turn cgroups back on but try disabling the memory and IO controllers. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 21:12 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 21:12 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > Actually, talking to Chris, I think I can get the system up using > init=/bin/bash without systemd, so I can try the no cgroup config. OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back runs of untar without locking or getting kswapd pegged, so I'm pretty certain this is cgroups related. The next steps are to turn cgroups back on but try disabling the memory and IO controllers. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 21:12 ` James Bottomley @ 2011-04-28 22:43 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 22:43 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > > Actually, talking to Chris, I think I can get the system up using > > init=/bin/bash without systemd, so I can try the no cgroup config. > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back > runs of untar without locking or getting kswapd pegged, so I'm pretty > certain this is cgroups related. The next steps are to turn cgroups > back on but try disabling the memory and IO controllers. I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR. The results are curious: the tar does complete (I've done three back to back). However, I did get one soft lockup in kswapd (below). But the system recovers instead of halting I/O and hanging like it did previously. The soft lockup is in shrink_slab, so perhaps it's a combination of slab shrinker and cgroup memory controller issues? James --- [ 670.823843] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 670.825472] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 670.830864] CPU 2 [ 670.830881] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 670.838385] [ 670.840289] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #3 LENOVO 4170CTO/4170CTO [ 670.842193] RIP: 0010:[<ffffffff810e07cb>] [<ffffffff810e07cb>] shrink_slab+0x86/0x166 [ 670.844063] RSP: 0018:ffff88006eea5da0 EFLAGS: 00000206 [ 670.845881] RAX: 0000000000000000 RBX: ffff88006eea5de0 RCX: 0000000000000002 [ 670.847652] RDX: 0000000000000000 RSI: ffff88006eea5d60 RDI: ffff88006eea5d60 [ 670.849394] RBP: ffff88006eea5de0 R08: 000000000000000c R09: 0000000000000000 [ 670.851091] R10: 0000000000000001 R11: 000000000000005f R12: ffffffff8147b50e [ 670.852733] R13: ffff8801005e6e00 R14: 0000000000000010 R15: 0000000000017fb6 [ 670.854351] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 670.855968] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 670.857555] CR2: 00000037d90ae040 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 670.859138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 670.860720] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 670.862320] Process kswapd0 (pid: 46, threadinfo ffff88006eea4000, task ffff88006eeb0000) [ 670.863932] Stack: [ 670.865477] 0000000000000001 0000000000000080 ffff880000000002 ffff8801005e6e00 [ 670.867023] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c [ 670.868558] ffff88006eea5ee0 ffffffff810e308c 0000000000000003 ffff88006eeb0000 [ 670.870120] Call Trace: [ 670.871652] [<ffffffff810e308c>] kswapd+0x4f0/0x774 [ 670.873218] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5 [ 670.874786] [<ffffffff8106ce57>] kthread+0x84/0x8c [ 670.876327] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10 [ 670.877871] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148 [ 670.879403] [<ffffffff8147bc60>] ? gs_change+0x13/0x13 [ 670.880932] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7 [ 670.881086] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7 [ 670.884285] Call Trace: [ 670.885884] [<ffffffff810e308c>] kswapd+0x4f0/0x774 [ 670.887462] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5 [ 670.889031] [<ffffffff8106ce57>] kthread+0x84/0x8c [ 670.890578] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10 [ 670.892130] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148 [ 670.893653] [<ffffffff8147bc60>] ? gs_change+0x13/0x13 ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 22:43 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-28 22:43 UTC (permalink / raw) To: Mel Gorman Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > > Actually, talking to Chris, I think I can get the system up using > > init=/bin/bash without systemd, so I can try the no cgroup config. > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back > runs of untar without locking or getting kswapd pegged, so I'm pretty > certain this is cgroups related. The next steps are to turn cgroups > back on but try disabling the memory and IO controllers. I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR. The results are curious: the tar does complete (I've done three back to back). However, I did get one soft lockup in kswapd (below). But the system recovers instead of halting I/O and hanging like it did previously. The soft lockup is in shrink_slab, so perhaps it's a combination of slab shrinker and cgroup memory controller issues? James --- [ 670.823843] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 670.825472] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 670.830864] CPU 2 [ 670.830881] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 670.838385] [ 670.840289] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #3 LENOVO 4170CTO/4170CTO [ 670.842193] RIP: 0010:[<ffffffff810e07cb>] [<ffffffff810e07cb>] shrink_slab+0x86/0x166 [ 670.844063] RSP: 0018:ffff88006eea5da0 EFLAGS: 00000206 [ 670.845881] RAX: 0000000000000000 RBX: ffff88006eea5de0 RCX: 0000000000000002 [ 670.847652] RDX: 0000000000000000 RSI: ffff88006eea5d60 RDI: ffff88006eea5d60 [ 670.849394] RBP: ffff88006eea5de0 R08: 000000000000000c R09: 0000000000000000 [ 670.851091] R10: 0000000000000001 R11: 000000000000005f R12: ffffffff8147b50e [ 670.852733] R13: ffff8801005e6e00 R14: 0000000000000010 R15: 0000000000017fb6 [ 670.854351] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 670.855968] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 670.857555] CR2: 00000037d90ae040 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 670.859138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 670.860720] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 670.862320] Process kswapd0 (pid: 46, threadinfo ffff88006eea4000, task ffff88006eeb0000) [ 670.863932] Stack: [ 670.865477] 0000000000000001 0000000000000080 ffff880000000002 ffff8801005e6e00 [ 670.867023] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c [ 670.868558] ffff88006eea5ee0 ffffffff810e308c 0000000000000003 ffff88006eeb0000 [ 670.870120] Call Trace: [ 670.871652] [<ffffffff810e308c>] kswapd+0x4f0/0x774 [ 670.873218] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5 [ 670.874786] [<ffffffff8106ce57>] kthread+0x84/0x8c [ 670.876327] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10 [ 670.877871] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148 [ 670.879403] [<ffffffff8147bc60>] ? gs_change+0x13/0x13 [ 670.880932] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7 [ 670.881086] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7 [ 670.884285] Call Trace: [ 670.885884] [<ffffffff810e308c>] kswapd+0x4f0/0x774 [ 670.887462] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5 [ 670.889031] [<ffffffff8106ce57>] kthread+0x84/0x8c [ 670.890578] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10 [ 670.892130] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148 [ 670.893653] [<ffffffff8147bc60>] ? gs_change+0x13/0x13 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 22:43 ` James Bottomley @ 2011-05-03 9:13 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-03 9:13 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > > > Actually, talking to Chris, I think I can get the system up using > > > init=/bin/bash without systemd, so I can try the no cgroup config. > > > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back > > runs of untar without locking or getting kswapd pegged, so I'm pretty > > certain this is cgroups related. The next steps are to turn cgroups > > back on but try disabling the memory and IO controllers. > > I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR. > > The results are curious: the tar does complete (I've done three back to > back). However, I did get one soft lockup in kswapd (below). But the > system recovers instead of halting I/O and hanging like it did > previously. > > The soft lockup is in shrink_slab, so perhaps it's a combination of slab > shrinker and cgroup memory controller issues? > So, kswapd is still looping in reclaim and spending a lot of time in shrink_slab but it must not be the shrinker itself or that debug patch would have triggered. It's curious that cgroups are involved with systemd considering that one would expect those groups to be fairly small. I still don't have a new theory but will get hold of a Fedora 15 install CD and see can I reproduce it locally. One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two of the reporting machines could be NUMA and if that proc file reads as 1, I'd be interested in hearing the results of a test with it set to 0. Thanks. ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-03 9:13 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-03 9:13 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote: > On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > > > Actually, talking to Chris, I think I can get the system up using > > > init=/bin/bash without systemd, so I can try the no cgroup config. > > > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back > > runs of untar without locking or getting kswapd pegged, so I'm pretty > > certain this is cgroups related. The next steps are to turn cgroups > > back on but try disabling the memory and IO controllers. > > I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR. > > The results are curious: the tar does complete (I've done three back to > back). However, I did get one soft lockup in kswapd (below). But the > system recovers instead of halting I/O and hanging like it did > previously. > > The soft lockup is in shrink_slab, so perhaps it's a combination of slab > shrinker and cgroup memory controller issues? > So, kswapd is still looping in reclaim and spending a lot of time in shrink_slab but it must not be the shrinker itself or that debug patch would have triggered. It's curious that cgroups are involved with systemd considering that one would expect those groups to be fairly small. I still don't have a new theory but will get hold of a Fedora 15 install CD and see can I reproduce it locally. One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two of the reporting machines could be NUMA and if that proc file reads as 1, I'd be interested in hearing the results of a test with it set to 0. Thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-03 9:13 ` Mel Gorman @ 2011-05-03 14:13 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-03 14:13 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-03 at 10:13 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote: > > > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > > > > Actually, talking to Chris, I think I can get the system up using > > > > init=/bin/bash without systemd, so I can try the no cgroup config. > > > > > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back > > > runs of untar without locking or getting kswapd pegged, so I'm pretty > > > certain this is cgroups related. The next steps are to turn cgroups > > > back on but try disabling the memory and IO controllers. > > > > I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR. > > > > The results are curious: the tar does complete (I've done three back to > > back). However, I did get one soft lockup in kswapd (below). But the > > system recovers instead of halting I/O and hanging like it did > > previously. > > > > The soft lockup is in shrink_slab, so perhaps it's a combination of slab > > shrinker and cgroup memory controller issues? > > > > So, kswapd is still looping in reclaim and spending a lot of time in > shrink_slab but it must not be the shrinker itself or that debug patch > would have triggered. It's curious that cgroups are involved with > systemd considering that one would expect those groups to be fairly > small. I still don't have a new theory but will get hold of a Fedora 15 > install CD and see can I reproduce it locally. I've got a ftrace output of kswapd ... it's 500k compressed, so I'll send under separate cover. > One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two > of the reporting machines could be NUMA and if that proc file reads as > 1, I'd be interested in hearing the results of a test with it set to 0. > Thanks. It's zero, I'm afraid James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-03 14:13 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-03 14:13 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-03 at 10:13 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote: > > > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote: > > > > Actually, talking to Chris, I think I can get the system up using > > > > init=/bin/bash without systemd, so I can try the no cgroup config. > > > > > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back > > > runs of untar without locking or getting kswapd pegged, so I'm pretty > > > certain this is cgroups related. The next steps are to turn cgroups > > > back on but try disabling the memory and IO controllers. > > > > I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR. > > > > The results are curious: the tar does complete (I've done three back to > > back). However, I did get one soft lockup in kswapd (below). But the > > system recovers instead of halting I/O and hanging like it did > > previously. > > > > The soft lockup is in shrink_slab, so perhaps it's a combination of slab > > shrinker and cgroup memory controller issues? > > > > So, kswapd is still looping in reclaim and spending a lot of time in > shrink_slab but it must not be the shrinker itself or that debug patch > would have triggered. It's curious that cgroups are involved with > systemd considering that one would expect those groups to be fairly > small. I still don't have a new theory but will get hold of a Fedora 15 > install CD and see can I reproduce it locally. I've got a ftrace output of kswapd ... it's 500k compressed, so I'll send under separate cover. > One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two > of the reporting machines could be NUMA and if that proc file reads as > 1, I'd be interested in hearing the results of a test with it set to 0. > Thanks. It's zero, I'm afraid James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-03 14:13 ` James Bottomley (?) @ 2011-05-03 14:22 ` James Bottomley 2011-05-06 7:42 ` Mel Gorman -1 siblings, 1 reply; 138+ messages in thread From: James Bottomley @ 2011-05-03 14:22 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 238 bytes --] On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote: > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll > send under separate cover. Here it is ... it's under 2.6.38.4 vanilla, but the code is similar. James [-- Attachment #2: kswapd-ftrace-2.6.38.4.txt.gz --] [-- Type: application/x-gzip, Size: 545984 bytes --] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-03 14:22 ` James Bottomley @ 2011-05-06 7:42 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 7:42 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote: > On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote: > > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll > > send under separate cover. > > Here it is ... it's under 2.6.38.4 vanilla, but the code is similar. > I was quiet because I was off trying to reproduce this but not having much luck. It doesn't seem directly related to filesystems or cgroups. For example, here is what I see with ext4 without cgroups 2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%) unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%) copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%) create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%) delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%) expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%) (all results are in seconds) When running in cgroups, the results are similar - bit slower but not remarkably so. ext3 is slower but not enough to count as the bug. The trace you posted is very short but kswapd is not going to sleep in it. It's less than a seconds worth on different cpus so it's hard to draw any conclusion from it other than sleeping_prematurely() is often deciding that kswapd should not sleep. So lets consider what keeps it awake. 1. High-order allocations? You machine is using i915 and RPC, something neither of my test machine uses. i915 is potentially a source for high-order allocations. I'm attaching a perl script. Please run it as ./watch-highorder.pl --output /tmp/highorders.txt while you are running tar. When kswapd is running for about 30 seconds, interrupt it with ctrl+c twice in quick succession and post /tmp/highorders.txt 2. All unreclaimable is not being set or we are not balancing at all. Can you post the output of sysrq+m while the machine is struggling please? 3. Slab may not be shrinking for some reason. Can you run a shell script like this during the whole test and record its output please? #!/bin/bash while [ 1 ]; do echo time: `date +%s` cat /proc/vmstat sleep 2 done Similarly if this is a slab issue, it'd be nice to know who it is so #!/bin/bash while [ 1 ]; do echo time: `date +%s` cat /proc/slabinfo sleep $MONITOR_UPDATE_FREQUENCY done 4. Lets get a better look at what is going on in kswapd echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt Out of curiousity, what's the exact machine you are testing on because I want to see what sort of hardware combination triggers this problem? Is the tar local or is it copied over NFS? What is the configuration of the disk or partition you are copying to? Thanks -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-06 7:42 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 7:42 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote: > On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote: > > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll > > send under separate cover. > > Here it is ... it's under 2.6.38.4 vanilla, but the code is similar. > I was quiet because I was off trying to reproduce this but not having much luck. It doesn't seem directly related to filesystems or cgroups. For example, here is what I see with ext4 without cgroups 2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%) unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%) copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%) create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%) delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%) expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%) (all results are in seconds) When running in cgroups, the results are similar - bit slower but not remarkably so. ext3 is slower but not enough to count as the bug. The trace you posted is very short but kswapd is not going to sleep in it. It's less than a seconds worth on different cpus so it's hard to draw any conclusion from it other than sleeping_prematurely() is often deciding that kswapd should not sleep. So lets consider what keeps it awake. 1. High-order allocations? You machine is using i915 and RPC, something neither of my test machine uses. i915 is potentially a source for high-order allocations. I'm attaching a perl script. Please run it as ./watch-highorder.pl --output /tmp/highorders.txt while you are running tar. When kswapd is running for about 30 seconds, interrupt it with ctrl+c twice in quick succession and post /tmp/highorders.txt 2. All unreclaimable is not being set or we are not balancing at all. Can you post the output of sysrq+m while the machine is struggling please? 3. Slab may not be shrinking for some reason. Can you run a shell script like this during the whole test and record its output please? #!/bin/bash while [ 1 ]; do echo time: `date +%s` cat /proc/vmstat sleep 2 done Similarly if this is a slab issue, it'd be nice to know who it is so #!/bin/bash while [ 1 ]; do echo time: `date +%s` cat /proc/slabinfo sleep $MONITOR_UPDATE_FREQUENCY done 4. Lets get a better look at what is going on in kswapd echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt Out of curiousity, what's the exact machine you are testing on because I want to see what sort of hardware combination triggers this problem? Is the tar local or is it copied over NFS? What is the configuration of the disk or partition you are copying to? Thanks -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 7:42 ` Mel Gorman (?) @ 2011-05-06 8:07 ` Mel Gorman 2011-05-09 18:16 ` James Bottomley -1 siblings, 1 reply; 138+ messages in thread From: Mel Gorman @ 2011-05-06 8:07 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 554 bytes --] On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > 1. High-order allocations? You machine is using i915 and RPC, something > neither of my test machine uses. i915 is potentially a source for > high-order allocations. I'm attaching a perl script. Please run it as > ./watch-highorder.pl --output /tmp/highorders.txt > while you are running tar. When kswapd is running for about 30 > seconds, interrupt it with ctrl+c twice in quick succession and > post /tmp/highorders.txt > Attached this time :/ -- Mel Gorman SUSE Labs [-- Attachment #2: watch-highorder.pl --] [-- Type: application/x-perl, Size: 7303 bytes --] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 8:07 ` Mel Gorman @ 2011-05-09 18:16 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-09 18:16 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote: > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > 1. High-order allocations? You machine is using i915 and RPC, something > > neither of my test machine uses. i915 is potentially a source for > > high-order allocations. I'm attaching a perl script. Please run it as > > ./watch-highorder.pl --output /tmp/highorders.txt > > while you are running tar. When kswapd is running for about 30 > > seconds, interrupt it with ctrl+c twice in quick succession and > > post /tmp/highorders.txt > > > > Attached this time :/ Here's the output (loaded with tar, evolution and firefox). The top trace is different this time because your perl script perturbs the system quite a bit. This was with your slub allocation fix applied. James --- 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => sg_kmalloc+0x24/0x26 <ffffffff81230eb9> => __sg_alloc_table+0x63/0x11c <ffffffff81230dbf> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b> => ra_submit+0x21/0x25 <ffffffff810dda8d> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a> => page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd49> => generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d0> 177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => radix_tree_preload+0x31/0x81 <ffffffff81229399> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => do_wp_page+0x348/0x560 <ffffffff810ef711> => handle_pte_fault+0x73b/0x7a5 <ffffffff810f13b7> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => prepare_creds+0x26/0xae <ffffffff81074d4b> => sys_faccessat+0x37/0x162 <ffffffff8111d255> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199> => __alloc_skb+0x40/0x133 <ffffffff813b5df7> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a> 1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => alloc_inode+0x30/0x78 <ffffffff811317f8> => new_inode+0x1b/0x4b <ffffffff8113185b> 16 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b6a> => handle_mm_fault+0x113/0x1ce <ffffffff810f16f0> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 5 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a> 252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => radix_tree_preload+0x31/0x81 <ffffffff81229399> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => skb_clone+0x50/0x70 <ffffffff813b5d97> => packet_rcv+0x101/0x2b2 <ffffffff81441e80> 1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => alloc_inode+0x30/0x78 <ffffffff811317f8> => iget_locked+0x61/0xdd <ffffffff8113239a> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => sg_kmalloc+0x24/0x26 <ffffffff81230eb9> => __sg_alloc_table+0x63/0x11c <ffffffff81230dbf> 1 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => copy_process+0x66e/0x10c5 <ffffffff81053961> => do_fork+0x104/0x286 <ffffffff810544f7> 1 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => wait_for_helper+0x6b/0xa3 <ffffffff81068fca> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 133 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => khugepaged+0x543/0xf2d <ffffffff81113cc1> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 2 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81> 14 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => shmem_file_setup+0xb6/0x160 <ffffffff810e70ad> 38 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mmap_region+0x1b7/0x446 <ffffffff810f6555> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a7c> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => idr_pre_get+0x2d/0x6f <ffffffff812266c4> => drm_gem_handle_create+0x2f/0x82 [drm] <ffffffffa0024f75> 400 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 9 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199> => __alloc_skb+0x40/0x133 <ffffffff813b5df7> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> => unix_stream_recvmsg+0x40f/0x536 <ffffffff8143dbe0> => sock_aio_read.part.7+0x10d/0x121 <ffffffff813afa54> 2 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => __call_usermodehelper+0x43/0x76 <ffffffff81069132> => process_one_work+0x186/0x298 <ffffffff8106b3ca> 14 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> 781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => kmalloc_large_node+0x56/0x95 <ffffffff8146a55d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217> => __sock_sendmsg+0x69/0x76 <ffffffff813af778> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => __pollwait+0x5e/0xcc <ffffffff8112c9e7> => sock_poll_wait+0x18/0x1d <ffffffff8143cc50> => unix_poll+0x1c/0x9a <ffffffff8143cc71> => sock_poll+0x1a/0x1c <ffffffff813ad41a> => do_sys_poll+0x1fa/0x386 <ffffffff8112d75d> 48 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => __split_vma+0x6c/0x21b <ffffffff810f5099> => split_vma+0x20/0x22 <ffffffff810f59a6> 3 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => sk_prot_alloc+0x37/0x13a <ffffffff813b27d4> => sk_alloc+0x2c/0x8e <ffffffff813b2935> 24 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => proc_alloc_inode+0x20/0x91 <ffffffff8116829c> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 1 instances order=1 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => ext4_mb_new_blocks+0x118/0x3c2 <ffffffff811b73a7> => ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a5d> 5 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => sock_alloc_inode+0x1d/0xaa <ffffffff813ad605> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 3 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => dst_alloc+0x48/0xaa <ffffffff813c6652> => __ip_route_output_key+0x561/0x764 <ffffffff813ee4f3> 3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => __pollwait+0x5e/0xcc <ffffffff8112c9e7> => sock_poll_wait+0x18/0x1d <ffffffff8143cc50> => unix_poll+0x1c/0x9a <ffffffff8143cc71> => sock_poll+0x1a/0x1c <ffffffff813ad41a> => do_select+0x2fb/0x4f5 <ffffffff8112cf13> 4 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => __split_vma+0x6c/0x21b <ffffffff810f5099> => do_munmap+0x194/0x30b <ffffffff810f57ad> 229 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => dup_mm+0x1f8/0x486 <ffffffff81053039> => copy_process+0x917/0x10c5 <ffffffff81053c0a> 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => bprm_mm_init+0x70/0x1a0 <ffffffff811239aa> => do_execve+0xd6/0x277 <ffffffff81123dd4> 2 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => find_skb+0x3a/0x82 <ffffffff813d1996> 2 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => __split_vma+0x6c/0x21b <ffffffff810f5099> => do_munmap+0x15f/0x30b <ffffffff810f5778> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1> => do_filp_open+0xe7/0x60a <ffffffff81129bcf> 18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d61> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4> 1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd> => i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05> 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => taskstats_exit+0x5b/0x2e3 <ffffffff810b0810> => do_exit+0x22a/0x6ef <ffffffff8105841f> 17 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => proc_fill_cache+0x82/0x135 <ffffffff8116903b> 99 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => inet_twsk_alloc+0x31/0x11e <ffffffff813f8a16> => tcp_time_wait+0xb9/0x29a <ffffffff8140e24f> 26 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943> => ext4_da_write_begin+0x13f/0x20e <ffffffff81196376> => generic_file_buffered_write+0x109/0x23a <ffffffff810d5408> => __generic_file_aio_write+0x242/0x272 <ffffffff810d617d> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6205> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249> 6 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => kthreadd+0xe7/0x124 <ffffffff8106f61f> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 2 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199> => __alloc_skb+0x40/0x133 <ffffffff813b5df7> => find_skb+0x3a/0x82 <ffffffff813d1996> 131 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3047> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30cd> 6 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => d_alloc_pseudo+0x18/0x46 <ffffffff8112e6a7> 6 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d61> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4> 11 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => shmem_alloc_inode+0x1a/0x2f <ffffffff810e58a0> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 4 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249> 680 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 8 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => copy_process+0xc6/0x10c5 <ffffffff810533b9> => do_fork+0x104/0x286 <ffffffff810544f7> 4 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a01> => handle_mm_fault+0x169/0x1ce <ffffffff810f1746> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 193 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 3 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> 17 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b> => ra_submit+0x21/0x25 <ffffffff810dda8d> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddce4> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510> 9 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956> => i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3> 14 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1> => alloc_file+0x1e/0xbf <ffffffff8111f3b6> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => kmalloc_order_trace+0x2c/0x5e <ffffffff8110c718> => __kmalloc+0x37/0x10d <ffffffff8110df41> => kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c> => tracing_open_pipe+0x40/0x147 <ffffffff810b907a> => __dentry_open+0x161/0x283 <ffffffff8111c95c> High-order normal allocations: 145450 High-order atomic allocations: 927 ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-09 18:16 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-09 18:16 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote: > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > 1. High-order allocations? You machine is using i915 and RPC, something > > neither of my test machine uses. i915 is potentially a source for > > high-order allocations. I'm attaching a perl script. Please run it as > > ./watch-highorder.pl --output /tmp/highorders.txt > > while you are running tar. When kswapd is running for about 30 > > seconds, interrupt it with ctrl+c twice in quick succession and > > post /tmp/highorders.txt > > > > Attached this time :/ Here's the output (loaded with tar, evolution and firefox). The top trace is different this time because your perl script perturbs the system quite a bit. This was with your slub allocation fix applied. James --- 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => sg_kmalloc+0x24/0x26 <ffffffff81230eb9> => __sg_alloc_table+0x63/0x11c <ffffffff81230dbf> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b> => ra_submit+0x21/0x25 <ffffffff810dda8d> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a> => page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd49> => generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d0> 177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => radix_tree_preload+0x31/0x81 <ffffffff81229399> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => do_wp_page+0x348/0x560 <ffffffff810ef711> => handle_pte_fault+0x73b/0x7a5 <ffffffff810f13b7> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => prepare_creds+0x26/0xae <ffffffff81074d4b> => sys_faccessat+0x37/0x162 <ffffffff8111d255> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199> => __alloc_skb+0x40/0x133 <ffffffff813b5df7> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a> 1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => alloc_inode+0x30/0x78 <ffffffff811317f8> => new_inode+0x1b/0x4b <ffffffff8113185b> 16 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b6a> => handle_mm_fault+0x113/0x1ce <ffffffff810f16f0> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 5 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a> 252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => radix_tree_preload+0x31/0x81 <ffffffff81229399> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => skb_clone+0x50/0x70 <ffffffff813b5d97> => packet_rcv+0x101/0x2b2 <ffffffff81441e80> 1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => alloc_inode+0x30/0x78 <ffffffff811317f8> => iget_locked+0x61/0xdd <ffffffff8113239a> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => sg_kmalloc+0x24/0x26 <ffffffff81230eb9> => __sg_alloc_table+0x63/0x11c <ffffffff81230dbf> 1 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => copy_process+0x66e/0x10c5 <ffffffff81053961> => do_fork+0x104/0x286 <ffffffff810544f7> 1 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => wait_for_helper+0x6b/0xa3 <ffffffff81068fca> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 133 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => khugepaged+0x543/0xf2d <ffffffff81113cc1> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 2 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81> 14 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => shmem_file_setup+0xb6/0x160 <ffffffff810e70ad> 38 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mmap_region+0x1b7/0x446 <ffffffff810f6555> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a7c> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => idr_pre_get+0x2d/0x6f <ffffffff812266c4> => drm_gem_handle_create+0x2f/0x82 [drm] <ffffffffa0024f75> 400 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 9 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199> => __alloc_skb+0x40/0x133 <ffffffff813b5df7> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> => unix_stream_recvmsg+0x40f/0x536 <ffffffff8143dbe0> => sock_aio_read.part.7+0x10d/0x121 <ffffffff813afa54> 2 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => __call_usermodehelper+0x43/0x76 <ffffffff81069132> => process_one_work+0x186/0x298 <ffffffff8106b3ca> 14 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> 781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => kmalloc_large_node+0x56/0x95 <ffffffff8146a55d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217> => __sock_sendmsg+0x69/0x76 <ffffffff813af778> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => __pollwait+0x5e/0xcc <ffffffff8112c9e7> => sock_poll_wait+0x18/0x1d <ffffffff8143cc50> => unix_poll+0x1c/0x9a <ffffffff8143cc71> => sock_poll+0x1a/0x1c <ffffffff813ad41a> => do_sys_poll+0x1fa/0x386 <ffffffff8112d75d> 48 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => __split_vma+0x6c/0x21b <ffffffff810f5099> => split_vma+0x20/0x22 <ffffffff810f59a6> 3 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => sk_prot_alloc+0x37/0x13a <ffffffff813b27d4> => sk_alloc+0x2c/0x8e <ffffffff813b2935> 24 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => proc_alloc_inode+0x20/0x91 <ffffffff8116829c> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 1 instances order=1 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => ext4_mb_new_blocks+0x118/0x3c2 <ffffffff811b73a7> => ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a5d> 5 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => sock_alloc_inode+0x1d/0xaa <ffffffff813ad605> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 3 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => dst_alloc+0x48/0xaa <ffffffff813c6652> => __ip_route_output_key+0x561/0x764 <ffffffff813ee4f3> 3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => __pollwait+0x5e/0xcc <ffffffff8112c9e7> => sock_poll_wait+0x18/0x1d <ffffffff8143cc50> => unix_poll+0x1c/0x9a <ffffffff8143cc71> => sock_poll+0x1a/0x1c <ffffffff813ad41a> => do_select+0x2fb/0x4f5 <ffffffff8112cf13> 4 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => __split_vma+0x6c/0x21b <ffffffff810f5099> => do_munmap+0x194/0x30b <ffffffff810f57ad> 229 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => dup_mm+0x1f8/0x486 <ffffffff81053039> => copy_process+0x917/0x10c5 <ffffffff81053c0a> 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => bprm_mm_init+0x70/0x1a0 <ffffffff811239aa> => do_execve+0xd6/0x277 <ffffffff81123dd4> 2 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => find_skb+0x3a/0x82 <ffffffff813d1996> 2 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => __split_vma+0x6c/0x21b <ffffffff810f5099> => do_munmap+0x15f/0x30b <ffffffff810f5778> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1> => do_filp_open+0xe7/0x60a <ffffffff81129bcf> 18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d61> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4> 1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd> => i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05> 1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => taskstats_exit+0x5b/0x2e3 <ffffffff810b0810> => do_exit+0x22a/0x6ef <ffffffff8105841f> 17 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => proc_fill_cache+0x82/0x135 <ffffffff8116903b> 99 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => inet_twsk_alloc+0x31/0x11e <ffffffff813f8a16> => tcp_time_wait+0xb9/0x29a <ffffffff8140e24f> 26 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943> => ext4_da_write_begin+0x13f/0x20e <ffffffff81196376> => generic_file_buffered_write+0x109/0x23a <ffffffff810d5408> => __generic_file_aio_write+0x242/0x272 <ffffffff810d617d> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6205> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249> 6 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => kthreadd+0xe7/0x124 <ffffffff8106f61f> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 2 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199> => __alloc_skb+0x40/0x133 <ffffffff813b5df7> => find_skb+0x3a/0x82 <ffffffff813d1996> 131 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3047> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30cd> 6 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => d_alloc+0x26/0x18d <ffffffff8112e4c5> => d_alloc_pseudo+0x18/0x46 <ffffffff8112e6a7> 6 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d61> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4> 11 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => shmem_alloc_inode+0x1a/0x2f <ffffffff810e58a0> => alloc_inode+0x1d/0x78 <ffffffff811317e5> 4 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249> 680 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 8 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => copy_process+0xc6/0x10c5 <ffffffff810533b9> => do_fork+0x104/0x286 <ffffffff810544f7> 4 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_vma+0xf5/0xfa <ffffffff81107001> => do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a01> => handle_mm_fault+0x169/0x1ce <ffffffff810f1746> => do_page_fault+0x358/0x37a <ffffffff8147416e> => page_fault+0x25/0x30 <ffffffff81471415> 193 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 3 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> 17 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b> => ra_submit+0x21/0x25 <ffffffff810dda8d> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddce4> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510> 9 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => __kmalloc+0xb1/0x10d <ffffffff8110dfbb> => kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956> => i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3> 14 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> => new_slab+0x50/0x199 <ffffffff8110dc24> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1> => alloc_file+0x1e/0xbf <ffffffff8111f3b6> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => kmalloc_order_trace+0x2c/0x5e <ffffffff8110c718> => __kmalloc+0x37/0x10d <ffffffff8110df41> => kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c> => tracing_open_pipe+0x40/0x147 <ffffffff810b907a> => __dentry_open+0x161/0x283 <ffffffff8111c95c> High-order normal allocations: 145450 High-order atomic allocations: 927 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-09 18:16 ` James Bottomley @ 2011-05-10 10:21 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-10 10:21 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Mon, May 09, 2011 at 01:16:20PM -0500, James Bottomley wrote: > On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote: > > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > > 1. High-order allocations? You machine is using i915 and RPC, something > > > neither of my test machine uses. i915 is potentially a source for > > > high-order allocations. I'm attaching a perl script. Please run it as > > > ./watch-highorder.pl --output /tmp/highorders.txt > > > while you are running tar. When kswapd is running for about 30 > > > seconds, interrupt it with ctrl+c twice in quick succession and > > > post /tmp/highorders.txt > > > > > > > Attached this time :/ > > Here's the output (loaded with tar, evolution and firefox). The top > trace is different this time because your perl script perturbs the > system quite a bit. This was with your slub allocation fix applied. > I note that certain flags like __GFP_NO_KSWAPD are not recognised by tracing which might explain why they are missing from the script output. I regret the script perturbs the system quite a bit. It's possible it can be made better by filtering events but it's not high on the list of things to do. How does the output compare without the fix? I can't find a similar report in my inbox. Does the fix help the system when the perl script is not running? > 177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => radix_tree_preload+0x31/0x81 <ffffffff81229399> > => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> > Ouch. > 46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => prepare_creds+0x26/0xae <ffffffff81074d4b> > => sys_faccessat+0x37/0x162 <ffffffff8111d255> > Less ouch, but still. > 252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => radix_tree_preload+0x31/0x81 <ffffffff81229399> > => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> > Ouch again. > 593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498> > => alloc_inode+0x1d/0x78 <ffffffff811317e5> > Again, filesystem-related calls are hitting high-order paths quite a bit. > 781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => kmalloc_large_node+0x56/0x95 <ffffffff8146a55d> > => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08> > => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> > => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> > => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2> > => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217> > => __sock_sendmsg+0x69/0x76 <ffffffff813af778> > A number of network paths are also being hit although this is the worst. > 501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1> > => do_filp_open+0xe7/0x60a <ffffffff81129bcf> > More filesystem impairment. > 1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => d_alloc+0x26/0x18d <ffffffff8112e4c5> > => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e> > *cries* > 140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> > => mempool_alloc+0x68/0x116 <ffffffff810d70f6> Wonder which pool this is! It goes on. A number of filesystem and network paths are being hit with high-order allocs. i915 was a red herring, it's present but not in massive numbers. The filesystem, network and mempool allocations are likely to be kicking kswapd awake frequently and hurting overall system performance as a result. I really would like to hear if the fix makes a big difference or if we need to consider forcing SLUB high-order allocations bailing at the first sign of trouble (e.g. by masking out __GFP_WAIT in allocate_slab). Even with the fix applied, kswapd might be waking up less but processes will still be getting stalled in direct compaction and direct reclaim so it would still be jittery. > High-order normal allocations: 145450 > High-order atomic allocations: 927 > I bet a shiny penny that the high-order allocations for SLAB are lower than this -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 10:21 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-10 10:21 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Mon, May 09, 2011 at 01:16:20PM -0500, James Bottomley wrote: > On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote: > > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > > 1. High-order allocations? You machine is using i915 and RPC, something > > > neither of my test machine uses. i915 is potentially a source for > > > high-order allocations. I'm attaching a perl script. Please run it as > > > ./watch-highorder.pl --output /tmp/highorders.txt > > > while you are running tar. When kswapd is running for about 30 > > > seconds, interrupt it with ctrl+c twice in quick succession and > > > post /tmp/highorders.txt > > > > > > > Attached this time :/ > > Here's the output (loaded with tar, evolution and firefox). The top > trace is different this time because your perl script perturbs the > system quite a bit. This was with your slub allocation fix applied. > I note that certain flags like __GFP_NO_KSWAPD are not recognised by tracing which might explain why they are missing from the script output. I regret the script perturbs the system quite a bit. It's possible it can be made better by filtering events but it's not high on the list of things to do. How does the output compare without the fix? I can't find a similar report in my inbox. Does the fix help the system when the perl script is not running? > 177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => radix_tree_preload+0x31/0x81 <ffffffff81229399> > => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> > Ouch. > 46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => prepare_creds+0x26/0xae <ffffffff81074d4b> > => sys_faccessat+0x37/0x162 <ffffffff8111d255> > Less ouch, but still. > 252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => radix_tree_preload+0x31/0x81 <ffffffff81229399> > => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> > Ouch again. > 593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498> > => alloc_inode+0x1d/0x78 <ffffffff811317e5> > Again, filesystem-related calls are hitting high-order paths quite a bit. > 781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => kmalloc_large_node+0x56/0x95 <ffffffff8146a55d> > => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08> > => __alloc_skb+0x75/0x133 <ffffffff813b5e2c> > => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a> > => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2> > => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217> > => __sock_sendmsg+0x69/0x76 <ffffffff813af778> > A number of network paths are also being hit although this is the worst. > 501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1> > => do_filp_open+0xe7/0x60a <ffffffff81129bcf> > More filesystem impairment. > 1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => d_alloc+0x26/0x18d <ffffffff8112e4c5> > => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e> > *cries* > 140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| > => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105435> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da> > => new_slab+0x50/0x199 <ffffffff8110dc24> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab66> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c> > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> > => mempool_alloc+0x68/0x116 <ffffffff810d70f6> Wonder which pool this is! It goes on. A number of filesystem and network paths are being hit with high-order allocs. i915 was a red herring, it's present but not in massive numbers. The filesystem, network and mempool allocations are likely to be kicking kswapd awake frequently and hurting overall system performance as a result. I really would like to hear if the fix makes a big difference or if we need to consider forcing SLUB high-order allocations bailing at the first sign of trouble (e.g. by masking out __GFP_WAIT in allocate_slab). Even with the fix applied, kswapd might be waking up less but processes will still be getting stalled in direct compaction and direct reclaim so it would still be jittery. > High-order normal allocations: 145450 > High-order atomic allocations: 927 > I bet a shiny penny that the high-order allocations for SLAB are lower than this -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 10:21 ` Mel Gorman @ 2011-05-10 10:33 ` Pekka Enberg -1 siblings, 0 replies; 138+ messages in thread From: Pekka Enberg @ 2011-05-10 10:33 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, Christoph Lameter On Tue, May 10, 2011 at 1:21 PM, Mel Gorman <mgorman@novell.com> wrote: > It goes on. A number of filesystem and network paths are being hit > with high-order allocs. i915 was a red herring, it's present but not > in massive numbers. The filesystem, network and mempool allocations > are likely to be kicking kswapd awake frequently and hurting overall > system performance as a result. > > I really would like to hear if the fix makes a big difference or > if we need to consider forcing SLUB high-order allocations bailing > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > allocate_slab). Even with the fix applied, kswapd might be waking up > less but processes will still be getting stalled in direct compaction > and direct reclaim so it would still be jittery. Yes, sounds reasonable to me. Pekka ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 10:33 ` Pekka Enberg 0 siblings, 0 replies; 138+ messages in thread From: Pekka Enberg @ 2011-05-10 10:33 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, Christoph Lameter On Tue, May 10, 2011 at 1:21 PM, Mel Gorman <mgorman@novell.com> wrote: > It goes on. A number of filesystem and network paths are being hit > with high-order allocs. i915 was a red herring, it's present but not > in massive numbers. The filesystem, network and mempool allocations > are likely to be kicking kswapd awake frequently and hurting overall > system performance as a result. > > I really would like to hear if the fix makes a big difference or > if we need to consider forcing SLUB high-order allocations bailing > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > allocate_slab). Even with the fix applied, kswapd might be waking up > less but processes will still be getting stalled in direct compaction > and direct reclaim so it would still be jittery. Yes, sounds reasonable to me. Pekka -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 10:21 ` Mel Gorman @ 2011-05-10 14:01 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 14:01 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > I really would like to hear if the fix makes a big difference or > if we need to consider forcing SLUB high-order allocations bailing > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > allocate_slab). Even with the fix applied, kswapd might be waking up > less but processes will still be getting stalled in direct compaction > and direct reclaim so it would still be jittery. "the fix" being this https://lkml.org/lkml/2011/3/5/121 In addition to your GFP_KSWAPD one? OK, will retry with that. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 14:01 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 14:01 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > I really would like to hear if the fix makes a big difference or > if we need to consider forcing SLUB high-order allocations bailing > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > allocate_slab). Even with the fix applied, kswapd might be waking up > less but processes will still be getting stalled in direct compaction > and direct reclaim so it would still be jittery. "the fix" being this https://lkml.org/lkml/2011/3/5/121 In addition to your GFP_KSWAPD one? OK, will retry with that. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 14:01 ` James Bottomley @ 2011-05-10 14:35 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-10 14:35 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 1044 bytes --] On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > I really would like to hear if the fix makes a big difference or > > if we need to consider forcing SLUB high-order allocations bailing > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > allocate_slab). Even with the fix applied, kswapd might be waking up > > less but processes will still be getting stalled in direct compaction > > and direct reclaim so it would still be jittery. > > "the fix" being this > > https://lkml.org/lkml/2011/3/5/121 > Drop this for the moment. It was a long shot at best and there is little evidence the problem is in this area. I'm attaching two patches. The first is the NO_KSWAPD one to stop kswapd being woken up by SLUB using speculative high-orders. The second one is more drastic and prevents slub entering direct reclaim or compaction. It applies on top of patch 1. These are both untested and afraid are a bit rushed as well :( -- Mel Gorman SUSE Labs [-- Attachment #2: mm-slub-do-not-wake-kswapd-for-slub-high-orders.patch --] [-- Type: text/x-patch, Size: 1395 bytes --] >From b48dee7d13980d4d901e3035dc6096c28c42c2ed Mon Sep 17 00:00:00 2001 From: Mel Gorman <mgorman@suse.de> Date: Tue, 10 May 2011 15:13:30 +0100 Subject: [PATCH] mm: slub: Do not wake kswapd for SLUBs speculative high-order allocations To avoid locking and per-cpu overhead, SLUB optimisically uses high-order allocations and falls back to lower allocations if they fail. However, by simply trying to allocate, kswapd is woken up to start reclaiming at that order. On a desktop system, two users report that the system is getting locked up with kswapd using large amounts of CPU. Using SLAB instead of SLUB makes this problem go away. This patch prevents kswapd being woken up for high-order allocations. Not-signed-off-yet: Mel Gorman <mgorman@suse.de> --- mm/slub.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 9d2e5e4..98c358d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1170,7 +1170,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) * Let the initial higher-order allocation fail under memory pressure * so we fall-back to the minimum order allocation. */ - alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL; + alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL; page = alloc_slab_page(alloc_gfp, node, oo); if (unlikely(!page)) { [-- Attachment #3: mm-slub-do-not-take-expensive-steps-for-slub-high-orders.patch --] [-- Type: text/x-patch, Size: 2400 bytes --] >From 59220aa310c0ba60afee29eeea1e602f4a374c60 Mon Sep 17 00:00:00 2001 From: Mel Gorman <mgorman@suse.de> Date: Tue, 10 May 2011 15:30:20 +0100 Subject: [PATCH] mm: slub: Do not take expensive steps for SLUBs speculative high-order allocations To avoid locking and per-cpu overhead, SLUB optimisically uses high-order allocations and falls back to lower allocations if they fail. However, by simply trying to allocate, the caller can enter compaction or reclaim - both of which are likely to cost more than the benefit of using high-order pages in SLUB. On a desktop system, two users report that the system is getting locked up with kswapd using large amounts of CPU. Using SLAB instead of SLUB makes this problem go away. This patch prevents SLUB taking any expensive steps when trying to use high-order allocations. Instead, it is expected to fall back to smaller orders more aggressively. Not-signed-off-yet: Mel Gorman <mgorman@suse.de> --- mm/page_alloc.c | 3 ++- mm/slub.c | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9f8a97b..f160d93 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1972,6 +1972,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) { int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET; const gfp_t wait = gfp_mask & __GFP_WAIT; + const gfp_t wakes_kswapd = !(gfp_mask & __GFP_NO_KSWAPD); /* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */ BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH); @@ -1984,7 +1985,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) */ alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH); - if (!wait) { + if (!wait && wakes_kswapd) { /* * Not worth trying to allocate harder for * __GFP_NOMEMALLOC even if it can't schedule. diff --git a/mm/slub.c b/mm/slub.c index 98c358d..1071723 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1170,7 +1170,8 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) * Let the initial higher-order allocation fail under memory pressure * so we fall-back to the minimum order allocation. */ - alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL; + alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & + ~(__GFP_NOFAIL | __GFP_WAIT); page = alloc_slab_page(alloc_gfp, node, oo); if (unlikely(!page)) { ^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 14:35 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-10 14:35 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 1044 bytes --] On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > I really would like to hear if the fix makes a big difference or > > if we need to consider forcing SLUB high-order allocations bailing > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > allocate_slab). Even with the fix applied, kswapd might be waking up > > less but processes will still be getting stalled in direct compaction > > and direct reclaim so it would still be jittery. > > "the fix" being this > > https://lkml.org/lkml/2011/3/5/121 > Drop this for the moment. It was a long shot at best and there is little evidence the problem is in this area. I'm attaching two patches. The first is the NO_KSWAPD one to stop kswapd being woken up by SLUB using speculative high-orders. The second one is more drastic and prevents slub entering direct reclaim or compaction. It applies on top of patch 1. These are both untested and afraid are a bit rushed as well :( -- Mel Gorman SUSE Labs [-- Attachment #2: mm-slub-do-not-wake-kswapd-for-slub-high-orders.patch --] [-- Type: text/x-patch, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 14:35 ` Mel Gorman @ 2011-05-10 15:29 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 15:29 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > I really would like to hear if the fix makes a big difference or > > > if we need to consider forcing SLUB high-order allocations bailing > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > less but processes will still be getting stalled in direct compaction > > > and direct reclaim so it would still be jittery. > > > > "the fix" being this > > > > https://lkml.org/lkml/2011/3/5/121 > > > > Drop this for the moment. It was a long shot at best and there is little > evidence the problem is in this area. > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > kswapd being woken up by SLUB using speculative high-orders. The second > one is more drastic and prevents slub entering direct reclaim or > compaction. It applies on top of patch 1. These are both untested and > afraid are a bit rushed as well :( Preliminary results with both patches applied still show kswapd periodically going up to 99% but it doesn't stay there, it comes back down again (and, obviously, the system doesn't hang). This is sysrq-M from a couple of times when it went up there: [ 426.736958] SysRq : Show Memory [ 426.736974] Mem-Info: [ 426.736977] Node 0 DMA per-cpu: [ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0 [ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0 [ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0 [ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0 [ 426.736996] Node 0 DMA32 per-cpu: [ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169 [ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40 [ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166 [ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168 [ 426.737015] Node 0 Normal per-cpu: [ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0 [ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0 [ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0 [ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0 [ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0 [ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32 [ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0 [ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477 [ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0 [ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [ 426.737064] lowmem_reserve[]: 0 1856 1862 1862 [ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no [ 426.737092] lowmem_reserve[]: 0 0 5 5 [ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 426.737119] lowmem_reserve[]: 0 0 0 0 [ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB [ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB [ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB [ 426.737257] 99595 total pagecache pages [ 426.737260] 0 pages in swap cache [ 426.737263] Swap cache stats: add 0, delete 0, find 0/0 [ 426.737266] Free swap = 3768316kB [ 426.737268] Total swap = 3768316kB [ 426.744603] 525808 pages RAM [ 426.744612] 57618 pages reserved [ 426.744614] 141551 pages shared [ 426.744617] 186065 pages non-shared [ 472.301810] SysRq : Show Memory [ 472.301826] Mem-Info: [ 472.301829] Node 0 DMA per-cpu: [ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0 [ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0 [ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0 [ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0 [ 472.301848] Node 0 DMA32 per-cpu: [ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184 [ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46 [ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158 [ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163 [ 472.301866] Node 0 Normal per-cpu: [ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0 [ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0 [ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0 [ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0 [ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0 [ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32 [ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0 [ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647 [ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0 [ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [ 472.301915] lowmem_reserve[]: 0 1856 1862 1862 [ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no [ 472.301943] lowmem_reserve[]: 0 0 5 5 [ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 472.301968] lowmem_reserve[]: 0 0 0 0 [ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB [ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB [ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB [ 472.302090] 91312 total pagecache pages [ 472.302093] 0 pages in swap cache [ 472.302096] Swap cache stats: add 0, delete 0, find 0/0 [ 472.302099] Free swap = 3768316kB [ 472.302102] Total swap = 3768316kB [ 472.309521] 525808 pages RAM [ 472.309529] 57618 pages reserved [ 472.309548] 142496 pages shared [ 472.309551] 177182 pages non-shared I'll finish off this verification, and then re-run with the watch-highorder script running. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 15:29 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 15:29 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > I really would like to hear if the fix makes a big difference or > > > if we need to consider forcing SLUB high-order allocations bailing > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > less but processes will still be getting stalled in direct compaction > > > and direct reclaim so it would still be jittery. > > > > "the fix" being this > > > > https://lkml.org/lkml/2011/3/5/121 > > > > Drop this for the moment. It was a long shot at best and there is little > evidence the problem is in this area. > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > kswapd being woken up by SLUB using speculative high-orders. The second > one is more drastic and prevents slub entering direct reclaim or > compaction. It applies on top of patch 1. These are both untested and > afraid are a bit rushed as well :( Preliminary results with both patches applied still show kswapd periodically going up to 99% but it doesn't stay there, it comes back down again (and, obviously, the system doesn't hang). This is sysrq-M from a couple of times when it went up there: [ 426.736958] SysRq : Show Memory [ 426.736974] Mem-Info: [ 426.736977] Node 0 DMA per-cpu: [ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0 [ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0 [ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0 [ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0 [ 426.736996] Node 0 DMA32 per-cpu: [ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169 [ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40 [ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166 [ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168 [ 426.737015] Node 0 Normal per-cpu: [ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0 [ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0 [ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0 [ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0 [ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0 [ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32 [ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0 [ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477 [ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0 [ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [ 426.737064] lowmem_reserve[]: 0 1856 1862 1862 [ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no [ 426.737092] lowmem_reserve[]: 0 0 5 5 [ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 426.737119] lowmem_reserve[]: 0 0 0 0 [ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB [ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB [ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB [ 426.737257] 99595 total pagecache pages [ 426.737260] 0 pages in swap cache [ 426.737263] Swap cache stats: add 0, delete 0, find 0/0 [ 426.737266] Free swap = 3768316kB [ 426.737268] Total swap = 3768316kB [ 426.744603] 525808 pages RAM [ 426.744612] 57618 pages reserved [ 426.744614] 141551 pages shared [ 426.744617] 186065 pages non-shared [ 472.301810] SysRq : Show Memory [ 472.301826] Mem-Info: [ 472.301829] Node 0 DMA per-cpu: [ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0 [ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0 [ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0 [ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0 [ 472.301848] Node 0 DMA32 per-cpu: [ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184 [ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46 [ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158 [ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163 [ 472.301866] Node 0 Normal per-cpu: [ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0 [ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0 [ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0 [ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0 [ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0 [ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32 [ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0 [ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647 [ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0 [ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [ 472.301915] lowmem_reserve[]: 0 1856 1862 1862 [ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no [ 472.301943] lowmem_reserve[]: 0 0 5 5 [ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 472.301968] lowmem_reserve[]: 0 0 0 0 [ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB [ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB [ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB [ 472.302090] 91312 total pagecache pages [ 472.302093] 0 pages in swap cache [ 472.302096] Swap cache stats: add 0, delete 0, find 0/0 [ 472.302099] Free swap = 3768316kB [ 472.302102] Total swap = 3768316kB [ 472.309521] 525808 pages RAM [ 472.309529] 57618 pages reserved [ 472.309548] 142496 pages shared [ 472.309551] 177182 pages non-shared I'll finish off this verification, and then re-run with the watch-highorder script running. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 15:29 ` James Bottomley @ 2011-05-10 15:57 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 15:57 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > I really would like to hear if the fix makes a big difference or > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > less but processes will still be getting stalled in direct compaction > > > > and direct reclaim so it would still be jittery. > > > > > > "the fix" being this > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > evidence the problem is in this area. > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > kswapd being woken up by SLUB using speculative high-orders. The second > > one is more drastic and prevents slub entering direct reclaim or > > compaction. It applies on top of patch 1. These are both untested and > > afraid are a bit rushed as well :( > > Preliminary results with both patches applied still show kswapd > periodically going up to 99% but it doesn't stay there, it comes back > down again (and, obviously, the system doesn't hang). This is a second run with the watch highorders. At the end of the run, the system hung temporarily and now comes back with CPU3 spinning in all system time at kswapd shrink_slab James --- 97 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> 127 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d81> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14> 47 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => radix_tree_preload+0x31/0x81 <ffffffff812293b9> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_trace+0x7b/0xd8 <ffffffff8110e2ce> => inode_doinit_with_dentry+0x124/0x45c <ffffffff811e96db> => selinux_d_instantiate+0x1c/0x1e <ffffffff811e9a2f> 3 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdb> => kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956> => i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3> 4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mmap_region+0x1b7/0x446 <ffffffff810f6575> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a9c> 27 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b> => ra_submit+0x21/0x25 <ffffffff810ddaad> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7a> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd04> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510> 159 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 446 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f1> => do_filp_open+0xe7/0x60a <ffffffff81129bef> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => arch_dup_task_struct+0x42/0x8b <ffffffff81010f47> => copy_process+0xf9/0x10c5 <ffffffff810533ec> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sk_stream_alloc_skb+0x38/0xd3 <ffffffff813fc5a9> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943> => ext4_da_write_begin+0x13f/0x20e <ffffffff81196396> => generic_file_buffered_write+0x109/0x23a <ffffffff810d5408> => __generic_file_aio_write+0x242/0x272 <ffffffff810d617d> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6205> 25 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f1> => alloc_file+0x1e/0xbf <ffffffff8111f3d6> 115 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d81> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14> 56 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => d_alloc+0x26/0x18d <ffffffff8112e4e5> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d2e> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => alloc_buffer_head+0x21/0x49 <ffffffff81141405> => alloc_page_buffers+0x32/0xce <ffffffff81141533> 9 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_vma+0xf5/0xfa <ffffffff81107021> => khugepaged+0x543/0xf2d <ffffffff81113ce1> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 347 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4b8> => alloc_inode+0x1d/0x78 <ffffffff81131805> 56 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943> => ext4_da_write_begin+0x13f/0x20e <ffffffff81196396> => generic_file_buffered_write+0x109/0x23a <ffffffff810d5408> => __generic_file_aio_write+0x242/0x272 <ffffffff810d617d> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6205> 60 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => dup_mm+0x1f8/0x486 <ffffffff81053039> => copy_process+0x917/0x10c5 <ffffffff81053c0a> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => bprm_mm_init+0x70/0x1a0 <ffffffff811239ca> => do_execve+0xd6/0x277 <ffffffff81123df4> 6 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_vma+0xf5/0xfa <ffffffff81107021> => do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8a> => handle_mm_fault+0x113/0x1ce <ffffffff810f1710> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 6 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => do_brk+0x1cd/0x293 <ffffffff810f5fd8> => load_elf_binary+0xd2d/0x175f <ffffffff8115ccf2> 4 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => kthreadd+0xe7/0x124 <ffffffff8106f61f> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1b9> => __alloc_skb+0x40/0x133 <ffffffff813b5e17> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> 534 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => kmalloc_large_node+0x56/0x95 <ffffffff8146a57d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff28> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => kmalloc_order_trace+0x2c/0x5e <ffffffff8110c738> => __kmalloc+0x37/0x10d <ffffffff8110df61> => kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c> => tracing_open_pipe+0x40/0x147 <ffffffff810b907a> => __dentry_open+0x161/0x283 <ffffffff8111c97c> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b> => ra_submit+0x21/0x25 <ffffffff810ddaad> => filemap_fault+0x17b/0x36c <ffffffff810d6ada> => __do_fault+0x55/0x377 <ffffffff810ee887> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f43> 55 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 8 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c0> => alloc_inode+0x1d/0x78 <ffffffff81131805> 563 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 12 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => __split_vma+0x6c/0x21b <ffffffff810f50b9> => split_vma+0x20/0x22 <ffffffff810f59c6> 5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => __pollwait+0x5e/0xcc <ffffffff8112ca07> => sock_poll_wait+0x18/0x1d <ffffffff8143cc70> => unix_poll+0x1c/0x9a <ffffffff8143cc91> => sock_poll+0x1a/0x1c <ffffffff813ad43a> => do_select+0x2fb/0x4f5 <ffffffff8112cf33> 145216 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 48 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => proc_alloc_inode+0x20/0x91 <ffffffff811682bc> => alloc_inode+0x1d/0x78 <ffffffff81131805> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => copy_process+0xc6/0x10c5 <ffffffff810533b9> => do_fork+0x104/0x286 <ffffffff810544f7> 31 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => prepare_creds+0x26/0xae <ffffffff81074d4b> => sys_faccessat+0x37/0x162 <ffffffff8111d275> 1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a> High-order normal allocations: 147471 High-order atomic allocations: 720 ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 15:57 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 15:57 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > I really would like to hear if the fix makes a big difference or > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > less but processes will still be getting stalled in direct compaction > > > > and direct reclaim so it would still be jittery. > > > > > > "the fix" being this > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > evidence the problem is in this area. > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > kswapd being woken up by SLUB using speculative high-orders. The second > > one is more drastic and prevents slub entering direct reclaim or > > compaction. It applies on top of patch 1. These are both untested and > > afraid are a bit rushed as well :( > > Preliminary results with both patches applied still show kswapd > periodically going up to 99% but it doesn't stay there, it comes back > down again (and, obviously, the system doesn't hang). This is a second run with the watch highorders. At the end of the run, the system hung temporarily and now comes back with CPU3 spinning in all system time at kswapd shrink_slab James --- 97 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> 127 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d81> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14> 47 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => radix_tree_preload+0x31/0x81 <ffffffff812293b9> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_trace+0x7b/0xd8 <ffffffff8110e2ce> => inode_doinit_with_dentry+0x124/0x45c <ffffffff811e96db> => selinux_d_instantiate+0x1c/0x1e <ffffffff811e9a2f> 3 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdb> => kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956> => i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3> 4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mmap_region+0x1b7/0x446 <ffffffff810f6575> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a9c> 27 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b> => ra_submit+0x21/0x25 <ffffffff810ddaad> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7a> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd04> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510> 159 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 446 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f1> => do_filp_open+0xe7/0x60a <ffffffff81129bef> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => arch_dup_task_struct+0x42/0x8b <ffffffff81010f47> => copy_process+0xf9/0x10c5 <ffffffff810533ec> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sk_stream_alloc_skb+0x38/0xd3 <ffffffff813fc5a9> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943> => ext4_da_write_begin+0x13f/0x20e <ffffffff81196396> => generic_file_buffered_write+0x109/0x23a <ffffffff810d5408> => __generic_file_aio_write+0x242/0x272 <ffffffff810d617d> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6205> 25 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f1> => alloc_file+0x1e/0xbf <ffffffff8111f3d6> 115 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d81> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14> 56 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => d_alloc+0x26/0x18d <ffffffff8112e4e5> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d2e> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => alloc_buffer_head+0x21/0x49 <ffffffff81141405> => alloc_page_buffers+0x32/0xce <ffffffff81141533> 9 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_vma+0xf5/0xfa <ffffffff81107021> => khugepaged+0x543/0xf2d <ffffffff81113ce1> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 347 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4b8> => alloc_inode+0x1d/0x78 <ffffffff81131805> 56 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943> => ext4_da_write_begin+0x13f/0x20e <ffffffff81196396> => generic_file_buffered_write+0x109/0x23a <ffffffff810d5408> => __generic_file_aio_write+0x242/0x272 <ffffffff810d617d> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6205> 60 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => dup_mm+0x1f8/0x486 <ffffffff81053039> => copy_process+0x917/0x10c5 <ffffffff81053c0a> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => bprm_mm_init+0x70/0x1a0 <ffffffff811239ca> => do_execve+0xd6/0x277 <ffffffff81123df4> 6 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_vma+0xf5/0xfa <ffffffff81107021> => do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8a> => handle_mm_fault+0x113/0x1ce <ffffffff810f1710> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 6 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => do_brk+0x1cd/0x293 <ffffffff810f5fd8> => load_elf_binary+0xd2d/0x175f <ffffffff8115ccf2> 4 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => kthreadd+0xe7/0x124 <ffffffff8106f61f> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1b9> => __alloc_skb+0x40/0x133 <ffffffff813b5e17> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> 534 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => kmalloc_large_node+0x56/0x95 <ffffffff8146a57d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff28> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => kmalloc_order_trace+0x2c/0x5e <ffffffff8110c738> => __kmalloc+0x37/0x10d <ffffffff8110df61> => kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c> => tracing_open_pipe+0x40/0x147 <ffffffff810b907a> => __dentry_open+0x161/0x283 <ffffffff8111c97c> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __page_cache_alloc+0x77/0x7e <ffffffff810d5778> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b> => ra_submit+0x21/0x25 <ffffffff810ddaad> => filemap_fault+0x17b/0x36c <ffffffff810d6ada> => __do_fault+0x55/0x377 <ffffffff810ee887> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f43> 55 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 8 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c0> => alloc_inode+0x1d/0x78 <ffffffff81131805> 563 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 12 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => __split_vma+0x6c/0x21b <ffffffff810f50b9> => split_vma+0x20/0x22 <ffffffff810f59c6> 5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => __get_free_pages+0xe/0x4a <ffffffff810d89d2> => __pollwait+0x5e/0xcc <ffffffff8112ca07> => sock_poll_wait+0x18/0x1d <ffffffff8143cc70> => unix_poll+0x1c/0x9a <ffffffff8143cc91> => sock_poll+0x1a/0x1c <ffffffff813ad43a> => do_select+0x2fb/0x4f5 <ffffffff8112cf33> 145216 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81> => mempool_alloc+0x68/0x116 <ffffffff810d70f6> 48 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => proc_alloc_inode+0x20/0x91 <ffffffff811682bc> => alloc_inode+0x1d/0x78 <ffffffff81131805> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => copy_process+0xc6/0x10c5 <ffffffff810533b9> => do_fork+0x104/0x286 <ffffffff810544f7> 31 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c> => prepare_creds+0x26/0xae <ffffffff81074d4b> => sys_faccessat+0x37/0x162 <ffffffff8111d275> 1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da> => alloc_pages_current+0xbe/0xd8 <ffffffff81105455> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa> => new_slab+0x50/0x199 <ffffffff8110dc44> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a> High-order normal allocations: 147471 High-order atomic allocations: 720 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 15:57 ` James Bottomley @ 2011-05-10 17:05 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 17:05 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote: > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > > I really would like to hear if the fix makes a big difference or > > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > > less but processes will still be getting stalled in direct compaction > > > > > and direct reclaim so it would still be jittery. > > > > > > > > "the fix" being this > > > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > > evidence the problem is in this area. > > > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > > kswapd being woken up by SLUB using speculative high-orders. The second > > > one is more drastic and prevents slub entering direct reclaim or > > > compaction. It applies on top of patch 1. These are both untested and > > > afraid are a bit rushed as well :( > > > > Preliminary results with both patches applied still show kswapd > > periodically going up to 99% but it doesn't stay there, it comes back > > down again (and, obviously, the system doesn't hang). > > This is a second run with the watch highorders. > > At the end of the run, the system hung temporarily and now comes back > with CPU3 spinning in all system time at kswapd shrink_slab Here's a trace in the same situation with the ftrace stack entries bumped to 16 as requested on IRC. There was no hang for this one. James --- 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c4> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => shmem_get_inode+0x45/0x1f6 <ffffffff810e49b3> => shmem_file_setup+0xe4/0x160 <ffffffff810e70ff> => drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc> => i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd> => i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> 701 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90> => __sg_alloc_table+0x63/0x11c <ffffffff81230ddf> => scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc> => scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02> => scsi_init_io+0x32/0x13b <ffffffff812f9f74> => scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> 189 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => get_request+0x180/0x262 <ffffffff812124f6> => get_request_wait+0x35/0x193 <ffffffff81212f71> => __make_request+0x2b4/0x400 <ffffffff81213898> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => inet_twsk_alloc+0x31/0x11e <ffffffff813f8a36> => tcp_time_wait+0xb9/0x29a <ffffffff8140e26f> => tcp_fin+0x7b/0x10f <ffffffff8140191a> => tcp_data_queue+0x1e4/0x876 <ffffffff8140207f> => tcp_rcv_state_process+0x880/0x8cf <ffffffff81404f67> => tcp_v4_do_rcv+0x312/0x363 <ffffffff8140b698> => tcp_v4_rcv+0x434/0x69b <ffffffff8140d209> => ip_local_deliver_finish+0x13d/0x1ae <ffffffff813f104d> => NF_HOOK.constprop.3+0x51/0x58 <ffffffff813f0eb2> => ip_local_deliver+0x4f/0x54 <ffffffff813f1205> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> 9 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd> => __alloc_skb+0x40/0x133 <ffffffff813b5e17> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => sk_prot_alloc+0x37/0x13a <ffffffff813b27f4> => sk_alloc+0x2c/0x8e <ffffffff813b2955> => unix_create1+0x4e/0x14d <ffffffff8143cae7> => unix_stream_connect+0x98/0x431 <ffffffff8143ecc6> => sys_connect+0x98/0xd6 <ffffffff813b0437> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => create_read_pipe+0x2b/0x5b <ffffffff81125ffe> => do_pipe_flags+0x4e/0xf4 <ffffffff8112607c> => sys_pipe2+0x1c/0x5c <ffffffff8112613e> => sys_pipe+0x10/0x12 <ffffffff8112618e> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 6 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a> => e1000_alloc_rx_buffers+0x8b/0x14e [e1000e] <ffffffffa01bfb58> => e1000_clean_rx_irq+0x2a6/0x2df [e1000e] <ffffffffa01c05a9> => e1000_clean+0x73/0x228 [e1000e] <ffffffffa01c0a07> => net_rx_action+0xa9/0x1ac <ffffffff813bfea2> => __do_softirq+0xd2/0x19d <ffffffff8105abe0> => call_softirq+0x1c/0x30 <ffffffff8100aa9c> => do_softirq+0x46/0x81 <ffffffff8100c0c1> => irq_exit+0x49/0x8b <ffffffff8105ad64> 48 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f> => ra_submit+0x21/0x25 <ffffffff810ddab1> => filemap_fault+0x17b/0x36c <ffffffff810d6ade> => __do_fault+0x55/0x377 <ffffffff810ee88b> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 16 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => do_read_cache_page+0x63/0x127 <ffffffff810d59f4> => read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6> => i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad> => i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609> => i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d> => i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> 409 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6> => kjournald2+0xc9/0x20a <ffffffff811c9764> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => mpage_readpages+0xf0/0x104 <ffffffff81148e05> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => copy_vma+0xe3/0x1e9 <ffffffff810f5cd3> => move_vma+0x8a/0x1fb <ffffffff810f7edb> => do_mremap+0x40e/0x46a <ffffffff810f845a> => sys_mremap+0x54/0x76 <ffffffff810f850a> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => do_read_cache_page+0x63/0x127 <ffffffff810d59f4> => read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6> => i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad> => i915_gem_fault+0x85/0x18e [i915] <ffffffffa007d287> => __do_fault+0x55/0x377 <ffffffff810ee88b> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc> 1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd> => i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => copy_process+0xc6/0x10c5 <ffffffff810533b9> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8> => ext4_mb_load_buddy+0x134/0x30f <ffffffff811b4b1c> => ext4_mb_regular_allocator+0x13c/0x275 <ffffffff811b6637> => ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432> => ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a81> 102 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> 1 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => get_request+0x180/0x262 <ffffffff812124f6> => get_request_wait+0x35/0x193 <ffffffff81212f71> => __make_request+0x2b4/0x400 <ffffffff81213898> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => mpage_readpages+0xf0/0x104 <ffffffff81148e05> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> 3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __split_vma+0x6c/0x21b <ffffffff810f50bd> => split_vma+0x20/0x22 <ffffffff810f59ca> => mprotect_fixup+0x105/0x65b <ffffffff810f70ad> => sys_mprotect+0x197/0x209 <ffffffff810f779a> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 14 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __generic_unplug_device+0x34/0x38 <ffffffff81212f38> => generic_unplug_device+0x2d/0x3d <ffffffff812131f0> 7 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __split_vma+0x6c/0x21b <ffffffff810f50bd> => split_vma+0x20/0x22 <ffffffff810f59ca> => mprotect_fixup+0x125/0x65b <ffffffff810f70cd> => sys_mprotect+0x197/0x209 <ffffffff810f779a> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => find_or_create_page+0x5a/0x8a <ffffffff810d5b6d> => __getblk+0x137/0x23a <ffffffff81141b55> => ext4_getblk+0x8e/0x153 <ffffffff811957c0> => ext4_bread+0x16/0x66 <ffffffff8119589b> => ext4_mkdir+0x159/0x33e <ffffffff8119b786> => vfs_mkdir+0x5f/0x9b <ffffffff81128b01> => sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd> 4 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_sendmsg+0xa1/0xb6 <ffffffff813af90f> => sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8> => ext4_mb_init_group+0x100/0x281 <ffffffff8146b9f8> => ext4_mb_good_group+0x58/0xdb <ffffffff811b4965> => ext4_mb_regular_allocator+0x125/0x275 <ffffffff811b6620> => ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432> 39 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_reserve_inode_write+0x26/0x83 <ffffffff81193de1> 66 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90> => __sg_alloc_table+0x63/0x11c <ffffffff81230ddf> => scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc> => scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02> => scsi_init_io+0x32/0x13b <ffffffff812f9f74> => scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> 2 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => do_read_cache_page+0x48/0x127 <ffffffff810d59d9> => read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6> => i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad> => i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609> => i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d> => i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 148 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => khugepaged+0x543/0xf2d <ffffffff81113ce5> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 526 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb> => ext4_create+0xbc/0x13e <ffffffff8119b32e> => vfs_create+0x6c/0x8e <ffffffff81128c64> => do_last+0x166/0x2df <ffffffff81129993> => do_filp_open+0x2ac/0x60a <ffffffff81129db8> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_openat+0x11/0x13 <ffffffff8111da20> 17 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __generic_unplug_device+0x34/0x38 <ffffffff81212f38> => __make_request+0x3a3/0x400 <ffffffff81213987> 43 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_get+0x57/0x213 <ffffffff811baf2b> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mmap_region+0x1b7/0x446 <ffffffff810f6579> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0> => sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f> => sys_mmap+0x22/0x24 <ffffffff8100da68> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 763 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => do_filp_open+0xe7/0x60a <ffffffff81129bf3> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 59 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb> => ext4_mkdir+0x10b/0x33e <ffffffff8119b738> => vfs_mkdir+0x5f/0x9b <ffffffff81128b01> => sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 83 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dup_mm+0x1f8/0x486 <ffffffff81053039> => copy_process+0x917/0x10c5 <ffffffff81053c0a> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 4 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => mpage_alloc+0x2a/0x84 <ffffffff8114866b> => do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e> 33 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __generic_unplug_device+0x34/0x38 <ffffffff81212f38> => elv_insert+0x172/0x17b <ffffffff8120dae8> 1 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => do_wp_page+0x348/0x560 <ffffffff810ef735> => handle_pte_fault+0x73b/0x7a5 <ffffffff810f13db> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> => do_path_lookup+0x4d/0xf6 <ffffffff8112881b> => do_filp_open+0x129/0x60a <ffffffff81129c35> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_get+0x57/0x213 <ffffffff811baf2b> => ext4_xattr_security_get+0x2a/0x2c <ffffffff811bc92d> => generic_getxattr+0x57/0x60 <ffffffff81137e27> 1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => kcalloc+0x38/0x3a <ffffffff811f5ff2> => security_get_user_sids+0x1c2/0x369 <ffffffff811f8503> => sel_write_user+0xd6/0x19d <ffffffff811ef5b3> => selinux_transaction_write+0x5b/0x77 <ffffffff811edbfc> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 15 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => proc_alloc_inode+0x20/0x91 <ffffffff811682c0> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => proc_pid_make_inode+0x14/0xb7 <ffffffff811697c0> => proc_pident_instantiate+0x22/0xb1 <ffffffff8116c202> => proc_pident_lookup+0x84/0xa3 <ffffffff8116c3ce> => proc_tgid_base_lookup+0x1a/0x1c <ffffffff8116c43f> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> 213 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> => block_write_full_page+0x15/0x17 <ffffffff81143693> => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8> => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> => lookup_hash+0x19/0x1b <ffffffff81127ecc> => lookup_create+0x4e/0x94 <ffffffff81127f1c> => sys_mkdirat+0x43/0xe8 <ffffffff8112a389> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 53 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6> 31 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __blk_run_queue+0x4a/0x7a <ffffffff81212b10> => blk_run_queue+0x28/0x3c <ffffffff81212bf9> 1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 8 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> 2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => __pollwait+0x5e/0xcc <ffffffff8112ca0b> => sock_poll_wait+0x18/0x1d <ffffffff8143cc70> => unix_poll+0x1c/0x9a <ffffffff8143cc91> => sock_poll+0x1a/0x1c <ffffffff813ad43a> => do_sys_poll+0x1fa/0x386 <ffffffff8112d781> => sys_poll+0x51/0xbb <ffffffff8112d9b0> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd> => __alloc_skb+0x40/0x133 <ffffffff813b5e17> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_sendmsg+0xa1/0xb6 <ffffffff813af90f> => sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => create_write_pipe+0x151/0x1bf <ffffffff81125f35> => do_pipe_flags+0x32/0xf4 <ffffffff81126060> => sys_pipe2+0x1c/0x5c <ffffffff8112613e> => sys_pipe+0x10/0x12 <ffffffff8112618e> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dup_fd+0x33/0x25f <ffffffff81133573> => copy_process+0x5b8/0x10c5 <ffffffff810538ab> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1> => ext4_init_security+0x61/0x82 <ffffffff811bc9d3> => ext4_new_inode+0xb69/0xcf2 <ffffffff81190493> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => skb_clone+0x50/0x70 <ffffffff813b5db7> => dev_hard_start_xmit+0xd1/0x535 <ffffffff813c1660> => sch_direct_xmit+0x6b/0x147 <ffffffff813d7173> => dev_queue_xmit+0x2ac/0x461 <ffffffff813c1d70> => neigh_resolve_output+0x205/0x236 <ffffffff813ca2be> => ip_finish_output2+0x1cc/0x20f <ffffffff813f3979> => ip_finish_output+0x65/0x6a <ffffffff813f42a9> => ip_output+0x8c/0x91 <ffffffff813f49f2> => dst_output+0x14/0x16 <ffffffff813f34d7> => ip_local_out+0x20/0x24 <ffffffff813f435f> 3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => __pollwait+0x5e/0xcc <ffffffff8112ca0b> => sock_poll_wait+0x18/0x1d <ffffffff8143cc70> => unix_poll+0x1c/0x9a <ffffffff8143cc91> => sock_poll+0x1a/0x1c <ffffffff813ad43a> => do_select+0x2fb/0x4f5 <ffffffff8112cf37> => core_sys_select+0x163/0x202 <ffffffff8112d294> => sys_select+0x91/0xb9 <ffffffff8112d3c4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => mpage_alloc+0x2a/0x84 <ffffffff8114866b> => do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => filemap_fault+0x17b/0x36c <ffffffff810d6ade> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => do_filp_open+0xe7/0x60a <ffffffff81129bf3> => open_exec+0x26/0x82 <ffffffff8112265e> => load_elf_binary+0x1df/0x175f <ffffffff8115c1a8> => search_binary_handler+0xb6/0x22e <ffffffff81122ce3> => do_execve+0x17e/0x277 <ffffffff81123ea0> => sys_execve+0x43/0x5a <ffffffff81011553> => stub_execve+0x6c/0xc0 <ffffffff81009fdc> 210 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => grab_cache_page_write_begin+0x6d/0x9e <ffffffff810d5960> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> 6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => do_filp_open+0xe7/0x60a <ffffffff81129bf3> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_openat+0x11/0x13 <ffffffff8111da20> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 8 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8e> => handle_mm_fault+0x113/0x1ce <ffffffff810f1714> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> 5 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> 257 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> => block_write_full_page+0x15/0x17 <ffffffff81143693> => ext4_writepage+0x201/0x210 <ffffffff81193854> => shrink_page_list+0x3c7/0x6de <ffffffff810e22a1> 809 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => kmalloc_large_node+0x56/0x95 <ffffffff8146a57d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => iget_locked+0x61/0xdd <ffffffff811323be> => ext4_iget+0x2c/0x64c <ffffffff81192a1c> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> => do_path_lookup+0x4d/0xf6 <ffffffff8112881b> => do_filp_open+0x129/0x60a <ffffffff81129c35> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => shmem_file_setup+0x125/0x160 <ffffffff810e7140> => drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc> => i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd> => i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 99 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => mpage_alloc+0x2a/0x84 <ffffffff8114866b> => do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> 24 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f> => ra_submit+0x21/0x25 <ffffffff810ddab1> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514> => do_sync_read+0xbf/0xff <ffffffff8111de6e> => vfs_read+0xa9/0xf0 <ffffffff8111e4d5> => sys_read+0x4a/0x6e <ffffffff8111e566> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 21 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => prepare_creds+0x26/0xae <ffffffff81074d4b> => sys_faccessat+0x37/0x162 <ffffffff8111d279> => sys_access+0x18/0x1a <ffffffff8111d3bc> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => kmalloc_order_trace+0x2c/0x5e <ffffffff8110c73c> => __kmalloc+0x37/0x10d <ffffffff8110df65> => kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c> => tracing_open_pipe+0x40/0x147 <ffffffff810b907a> => __dentry_open+0x161/0x283 <ffffffff8111c980> => nameidata_to_filp+0x60/0x67 <ffffffff8111d8f4> => finish_open+0xa1/0x17f <ffffffff8112974f> => do_filp_open+0x186/0x60a <ffffffff81129c92> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> 2 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => find_or_create_page+0x5a/0x8a <ffffffff810d5b6d> => __getblk+0x137/0x23a <ffffffff81141b55> => __ext4_get_inode_loc+0x118/0x344 <ffffffff8119115f> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1> => ext4_init_security+0x61/0x82 <ffffffff811bc9d3> => ext4_new_inode+0xb69/0xcf2 <ffffffff81190493> 8 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1> 58 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> => lookup_hash+0x19/0x1b <ffffffff81127ecc> => do_last+0xcf/0x2df <ffffffff811298fc> => do_filp_open+0x2ac/0x60a <ffffffff81129db8> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_openat+0x11/0x13 <ffffffff8111da20> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => mpage_readpages+0x99/0x104 <ffffffff81148dae> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ondemand_readahead+0x15a/0x1d8 <ffffffff810ddc0f> => page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd6d> => generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d4> => do_sync_read+0xbf/0xff <ffffffff8111de6e> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => create_read_pipe+0x2b/0x5b <ffffffff81125ffe> => do_pipe_flags+0x4e/0xf4 <ffffffff8112607c> => sys_pipe2+0x1c/0x5c <ffffffff8112613e> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 9 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => kthreadd+0xe7/0x124 <ffffffff8106f61f> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dst_alloc+0x48/0xaa <ffffffff813c6672> => __ip_route_output_key+0x561/0x764 <ffffffff813ee513> => tcp_v4_connect+0x121/0x47d <ffffffff8140becf> => inet_stream_connect+0xa8/0x25f <ffffffff81418e24> => sys_connect+0x98/0xd6 <ffffffff813b0437> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => kmalloc_large_node+0x56/0x95 <ffffffff8146a57d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xb3/0xf3 <ffffffff8111e3ec> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __nf_conntrack_alloc+0xc9/0x1af <ffffffff813e4533> => nf_conntrack_in+0x2f6/0x77a <ffffffff813e490f> => ipv4_conntrack_local+0x46/0x4a <ffffffff8142864c> => nf_iterate+0x48/0x83 <ffffffff813e1cb8> => nf_hook_slow+0x6a/0xe9 <ffffffff813e1d5d> => nf_hook_thresh.constprop.8+0x42/0x44 <ffffffff813f42f0> => __ip_local_out+0x49/0x4d <ffffffff813f433b> => ip_local_out+0x13/0x24 <ffffffff813f4352> => ip_queue_xmit+0x2b3/0x30e <ffffffff813f4616> => tcp_transmit_skb+0x708/0x739 <ffffffff81407217> 1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dst_alloc+0x48/0xaa <ffffffff813c6672> => __ip_route_output_key+0x561/0x764 <ffffffff813ee513> => ip_route_output_flow+0x2c/0x1ea <ffffffff813ee742> => ip4_datagram_connect+0x18d/0x298 <ffffffff8140ede5> => inet_dgram_connect+0x65/0x6c <ffffffff81418b9d> => sys_connect+0x98/0xd6 <ffffffff813b0437> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a25> => handle_mm_fault+0x169/0x1ce <ffffffff810f176a> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => find_or_create_page+0x41/0x8a <ffffffff810d5b54> => __getblk+0x137/0x23a <ffffffff81141b55> => __breadahead+0x12/0x39 <ffffffff81141c6a> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> => do_path_lookup+0x4d/0xf6 <ffffffff8112881b> => user_path_at+0x57/0x94 <ffffffff811294e0> => sys_faccessat+0xa0/0x162 <ffffffff8111d2e2> => sys_access+0x18/0x1a <ffffffff8111d3bc> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => shmem_file_setup+0xb6/0x160 <ffffffff810e70d1> => drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc> => i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd> => i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 416 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> => block_write_full_page+0x15/0x17 <ffffffff81143693> => blkdev_writepage+0x18/0x1a <ffffffff811460b8> => __writepage+0x15/0x2e <ffffffff810dca43> 13 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => mpage_readpages+0x99/0x104 <ffffffff81148dae> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => filemap_fault+0x17b/0x36c <ffffffff810d6ade> => __do_fault+0x55/0x377 <ffffffff810ee88b> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47> 58 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> => lookup_hash+0x19/0x1b <ffffffff81127ecc> => do_last+0xcf/0x2df <ffffffff811298fc> => do_filp_open+0x2ac/0x60a <ffffffff81129db8> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 7727 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __split_vma+0x6c/0x21b <ffffffff810f50bd> => do_munmap+0x194/0x30b <ffffffff810f57d1> => mmap_region+0x9b/0x446 <ffffffff810f645d> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0> => sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f> => sys_mmap+0x22/0x24 <ffffffff8100da68> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => __call_usermodehelper+0x43/0x76 <ffffffff81069132> => process_one_work+0x186/0x298 <ffffffff8106b3ca> => worker_thread+0xda/0x15d <ffffffff8106bf29> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => mpage_readpages+0xf0/0x104 <ffffffff81148e05> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => iget_locked+0x61/0xdd <ffffffff811323be> => ext4_iget+0x2c/0x64c <ffffffff81192a1c> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x14b/0x3b8 <ffffffff8112837b> => do_filp_open+0x1eb/0x60a <ffffffff81129cf7> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> 1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => find_or_create_page+0x5a/0x8a <ffffffff810d5b6d> => ext4_mb_load_buddy+0x1f7/0x30f <ffffffff811b4bdf> => ext4_free_blocks+0x41d/0x6ba <ffffffff811b7a92> => ext4_ext_truncate+0x361/0x78a <ffffffff811b1012> => ext4_truncate+0xa1/0x4d9 <ffffffff8119598c> => ext4_evict_inode+0x13f/0x228 <ffffffff811978e9> => evict+0x21/0x89 <ffffffff81131b92> 149 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => mpage_readpages+0x99/0x104 <ffffffff81148dae> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514> High-order normal allocations: 152966 High-order atomic allocations: 875 ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 17:05 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 17:05 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote: > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > > I really would like to hear if the fix makes a big difference or > > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > > less but processes will still be getting stalled in direct compaction > > > > > and direct reclaim so it would still be jittery. > > > > > > > > "the fix" being this > > > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > > evidence the problem is in this area. > > > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > > kswapd being woken up by SLUB using speculative high-orders. The second > > > one is more drastic and prevents slub entering direct reclaim or > > > compaction. It applies on top of patch 1. These are both untested and > > > afraid are a bit rushed as well :( > > > > Preliminary results with both patches applied still show kswapd > > periodically going up to 99% but it doesn't stay there, it comes back > > down again (and, obviously, the system doesn't hang). > > This is a second run with the watch highorders. > > At the end of the run, the system hung temporarily and now comes back > with CPU3 spinning in all system time at kswapd shrink_slab Here's a trace in the same situation with the ftrace stack entries bumped to 16 as requested on IRC. There was no hang for this one. James --- 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c4> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => shmem_get_inode+0x45/0x1f6 <ffffffff810e49b3> => shmem_file_setup+0xe4/0x160 <ffffffff810e70ff> => drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc> => i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd> => i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> 701 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90> => __sg_alloc_table+0x63/0x11c <ffffffff81230ddf> => scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc> => scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02> => scsi_init_io+0x32/0x13b <ffffffff812f9f74> => scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> 189 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => get_request+0x180/0x262 <ffffffff812124f6> => get_request_wait+0x35/0x193 <ffffffff81212f71> => __make_request+0x2b4/0x400 <ffffffff81213898> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => inet_twsk_alloc+0x31/0x11e <ffffffff813f8a36> => tcp_time_wait+0xb9/0x29a <ffffffff8140e26f> => tcp_fin+0x7b/0x10f <ffffffff8140191a> => tcp_data_queue+0x1e4/0x876 <ffffffff8140207f> => tcp_rcv_state_process+0x880/0x8cf <ffffffff81404f67> => tcp_v4_do_rcv+0x312/0x363 <ffffffff8140b698> => tcp_v4_rcv+0x434/0x69b <ffffffff8140d209> => ip_local_deliver_finish+0x13d/0x1ae <ffffffff813f104d> => NF_HOOK.constprop.3+0x51/0x58 <ffffffff813f0eb2> => ip_local_deliver+0x4f/0x54 <ffffffff813f1205> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> 9 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd> => __alloc_skb+0x40/0x133 <ffffffff813b5e17> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => sk_prot_alloc+0x37/0x13a <ffffffff813b27f4> => sk_alloc+0x2c/0x8e <ffffffff813b2955> => unix_create1+0x4e/0x14d <ffffffff8143cae7> => unix_stream_connect+0x98/0x431 <ffffffff8143ecc6> => sys_connect+0x98/0xd6 <ffffffff813b0437> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => create_read_pipe+0x2b/0x5b <ffffffff81125ffe> => do_pipe_flags+0x4e/0xf4 <ffffffff8112607c> => sys_pipe2+0x1c/0x5c <ffffffff8112613e> => sys_pipe+0x10/0x12 <ffffffff8112618e> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 6 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a> => e1000_alloc_rx_buffers+0x8b/0x14e [e1000e] <ffffffffa01bfb58> => e1000_clean_rx_irq+0x2a6/0x2df [e1000e] <ffffffffa01c05a9> => e1000_clean+0x73/0x228 [e1000e] <ffffffffa01c0a07> => net_rx_action+0xa9/0x1ac <ffffffff813bfea2> => __do_softirq+0xd2/0x19d <ffffffff8105abe0> => call_softirq+0x1c/0x30 <ffffffff8100aa9c> => do_softirq+0x46/0x81 <ffffffff8100c0c1> => irq_exit+0x49/0x8b <ffffffff8105ad64> 48 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f> => ra_submit+0x21/0x25 <ffffffff810ddab1> => filemap_fault+0x17b/0x36c <ffffffff810d6ade> => __do_fault+0x55/0x377 <ffffffff810ee88b> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 16 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => do_read_cache_page+0x63/0x127 <ffffffff810d59f4> => read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6> => i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad> => i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609> => i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d> => i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> 409 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6> => kjournald2+0xc9/0x20a <ffffffff811c9764> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => mpage_readpages+0xf0/0x104 <ffffffff81148e05> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => copy_vma+0xe3/0x1e9 <ffffffff810f5cd3> => move_vma+0x8a/0x1fb <ffffffff810f7edb> => do_mremap+0x40e/0x46a <ffffffff810f845a> => sys_mremap+0x54/0x76 <ffffffff810f850a> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => do_read_cache_page+0x63/0x127 <ffffffff810d59f4> => read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6> => i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad> => i915_gem_fault+0x85/0x18e [i915] <ffffffffa007d287> => __do_fault+0x55/0x377 <ffffffff810ee88b> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc> 1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd> => i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => copy_process+0xc6/0x10c5 <ffffffff810533b9> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8> => ext4_mb_load_buddy+0x134/0x30f <ffffffff811b4b1c> => ext4_mb_regular_allocator+0x13c/0x275 <ffffffff811b6637> => ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432> => ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a81> 102 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> 1 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => get_request+0x180/0x262 <ffffffff812124f6> => get_request_wait+0x35/0x193 <ffffffff81212f71> => __make_request+0x2b4/0x400 <ffffffff81213898> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => mpage_readpages+0xf0/0x104 <ffffffff81148e05> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> 3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __split_vma+0x6c/0x21b <ffffffff810f50bd> => split_vma+0x20/0x22 <ffffffff810f59ca> => mprotect_fixup+0x105/0x65b <ffffffff810f70ad> => sys_mprotect+0x197/0x209 <ffffffff810f779a> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 14 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __generic_unplug_device+0x34/0x38 <ffffffff81212f38> => generic_unplug_device+0x2d/0x3d <ffffffff812131f0> 7 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __split_vma+0x6c/0x21b <ffffffff810f50bd> => split_vma+0x20/0x22 <ffffffff810f59ca> => mprotect_fixup+0x125/0x65b <ffffffff810f70cd> => sys_mprotect+0x197/0x209 <ffffffff810f779a> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => find_or_create_page+0x5a/0x8a <ffffffff810d5b6d> => __getblk+0x137/0x23a <ffffffff81141b55> => ext4_getblk+0x8e/0x153 <ffffffff811957c0> => ext4_bread+0x16/0x66 <ffffffff8119589b> => ext4_mkdir+0x159/0x33e <ffffffff8119b786> => vfs_mkdir+0x5f/0x9b <ffffffff81128b01> => sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd> 4 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_sendmsg+0xa1/0xb6 <ffffffff813af90f> => sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8> => ext4_mb_init_group+0x100/0x281 <ffffffff8146b9f8> => ext4_mb_good_group+0x58/0xdb <ffffffff811b4965> => ext4_mb_regular_allocator+0x125/0x275 <ffffffff811b6620> => ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432> 39 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_reserve_inode_write+0x26/0x83 <ffffffff81193de1> 66 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90> => __sg_alloc_table+0x63/0x11c <ffffffff81230ddf> => scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc> => scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02> => scsi_init_io+0x32/0x13b <ffffffff812f9f74> => scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> 2 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => do_read_cache_page+0x48/0x127 <ffffffff810d59d9> => read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6> => i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad> => i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609> => i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d> => i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 148 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => khugepaged+0x543/0xf2d <ffffffff81113ce5> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 526 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb> => ext4_create+0xbc/0x13e <ffffffff8119b32e> => vfs_create+0x6c/0x8e <ffffffff81128c64> => do_last+0x166/0x2df <ffffffff81129993> => do_filp_open+0x2ac/0x60a <ffffffff81129db8> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_openat+0x11/0x13 <ffffffff8111da20> 17 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __generic_unplug_device+0x34/0x38 <ffffffff81212f38> => __make_request+0x3a3/0x400 <ffffffff81213987> 43 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_get+0x57/0x213 <ffffffff811baf2b> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mmap_region+0x1b7/0x446 <ffffffff810f6579> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0> => sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f> => sys_mmap+0x22/0x24 <ffffffff8100da68> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 763 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => do_filp_open+0xe7/0x60a <ffffffff81129bf3> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 59 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb> => ext4_mkdir+0x10b/0x33e <ffffffff8119b738> => vfs_mkdir+0x5f/0x9b <ffffffff81128b01> => sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 83 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dup_mm+0x1f8/0x486 <ffffffff81053039> => copy_process+0x917/0x10c5 <ffffffff81053c0a> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 4 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => mpage_alloc+0x2a/0x84 <ffffffff8114866b> => do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e> 33 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __generic_unplug_device+0x34/0x38 <ffffffff81212f38> => elv_insert+0x172/0x17b <ffffffff8120dae8> 1 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => do_wp_page+0x348/0x560 <ffffffff810ef735> => handle_pte_fault+0x73b/0x7a5 <ffffffff810f13db> => handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> => do_path_lookup+0x4d/0xf6 <ffffffff8112881b> => do_filp_open+0x129/0x60a <ffffffff81129c35> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_get+0x57/0x213 <ffffffff811baf2b> => ext4_xattr_security_get+0x2a/0x2c <ffffffff811bc92d> => generic_getxattr+0x57/0x60 <ffffffff81137e27> 1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => kcalloc+0x38/0x3a <ffffffff811f5ff2> => security_get_user_sids+0x1c2/0x369 <ffffffff811f8503> => sel_write_user+0xd6/0x19d <ffffffff811ef5b3> => selinux_transaction_write+0x5b/0x77 <ffffffff811edbfc> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 15 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => proc_alloc_inode+0x20/0x91 <ffffffff811682c0> => alloc_inode+0x1d/0x78 <ffffffff81131809> => new_inode+0x1b/0x4b <ffffffff8113187f> => proc_pid_make_inode+0x14/0xb7 <ffffffff811697c0> => proc_pident_instantiate+0x22/0xb1 <ffffffff8116c202> => proc_pident_lookup+0x84/0xa3 <ffffffff8116c3ce> => proc_tgid_base_lookup+0x1a/0x1c <ffffffff8116c43f> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> 213 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> => block_write_full_page+0x15/0x17 <ffffffff81143693> => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8> => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> => lookup_hash+0x19/0x1b <ffffffff81127ecc> => lookup_create+0x4e/0x94 <ffffffff81127f1c> => sys_mkdirat+0x43/0xe8 <ffffffff8112a389> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 53 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6> 31 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063> => scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9> => __scsi_get_command+0x1b/0x96 <ffffffff812f3150> => scsi_get_command+0x3d/0xa5 <ffffffff812f3283> => scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165> => sd_prep_fn+0x27b/0xb6d <ffffffff813026e9> => blk_peek_request+0xe1/0x1b6 <ffffffff81213411> => scsi_request_fn+0x82/0x421 <ffffffff812f9a5f> => __blk_run_queue+0x4a/0x7a <ffffffff81212b10> => blk_run_queue+0x28/0x3c <ffffffff81212bf9> 1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 8 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> 2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => __pollwait+0x5e/0xcc <ffffffff8112ca0b> => sock_poll_wait+0x18/0x1d <ffffffff8143cc70> => unix_poll+0x1c/0x9a <ffffffff8143cc91> => sock_poll+0x1a/0x1c <ffffffff813ad43a> => do_sys_poll+0x1fa/0x386 <ffffffff8112d781> => sys_poll+0x51/0xbb <ffffffff8112d9b0> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd> => __alloc_skb+0x40/0x133 <ffffffff813b5e17> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_sendmsg+0xa1/0xb6 <ffffffff813af90f> => sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => create_write_pipe+0x151/0x1bf <ffffffff81125f35> => do_pipe_flags+0x32/0xf4 <ffffffff81126060> => sys_pipe2+0x1c/0x5c <ffffffff8112613e> => sys_pipe+0x10/0x12 <ffffffff8112618e> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dup_fd+0x33/0x25f <ffffffff81133573> => copy_process+0x5b8/0x10c5 <ffffffff810538ab> => do_fork+0x104/0x286 <ffffffff810544f7> => sys_clone+0x28/0x2a <ffffffff8101150e> => stub_clone+0x13/0x20 <ffffffff81009ea3> 2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1> => ext4_init_security+0x61/0x82 <ffffffff811bc9d3> => ext4_new_inode+0xb69/0xcf2 <ffffffff81190493> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => skb_clone+0x50/0x70 <ffffffff813b5db7> => dev_hard_start_xmit+0xd1/0x535 <ffffffff813c1660> => sch_direct_xmit+0x6b/0x147 <ffffffff813d7173> => dev_queue_xmit+0x2ac/0x461 <ffffffff813c1d70> => neigh_resolve_output+0x205/0x236 <ffffffff813ca2be> => ip_finish_output2+0x1cc/0x20f <ffffffff813f3979> => ip_finish_output+0x65/0x6a <ffffffff813f42a9> => ip_output+0x8c/0x91 <ffffffff813f49f2> => dst_output+0x14/0x16 <ffffffff813f34d7> => ip_local_out+0x20/0x24 <ffffffff813f435f> 3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => __pollwait+0x5e/0xcc <ffffffff8112ca0b> => sock_poll_wait+0x18/0x1d <ffffffff8143cc70> => unix_poll+0x1c/0x9a <ffffffff8143cc91> => sock_poll+0x1a/0x1c <ffffffff813ad43a> => do_select+0x2fb/0x4f5 <ffffffff8112cf37> => core_sys_select+0x163/0x202 <ffffffff8112d294> => sys_select+0x91/0xb9 <ffffffff8112d3c4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => mpage_alloc+0x2a/0x84 <ffffffff8114866b> => do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => filemap_fault+0x17b/0x36c <ffffffff810d6ade> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => do_filp_open+0xe7/0x60a <ffffffff81129bf3> => open_exec+0x26/0x82 <ffffffff8112265e> => load_elf_binary+0x1df/0x175f <ffffffff8115c1a8> => search_binary_handler+0xb6/0x22e <ffffffff81122ce3> => do_execve+0x17e/0x277 <ffffffff81123ea0> => sys_execve+0x43/0x5a <ffffffff81011553> => stub_execve+0x6c/0xc0 <ffffffff81009fdc> 210 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => grab_cache_page_write_begin+0x6d/0x9e <ffffffff810d5960> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> 6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => do_filp_open+0xe7/0x60a <ffffffff81129bf3> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_openat+0x11/0x13 <ffffffff8111da20> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 8 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8e> => handle_mm_fault+0x113/0x1ce <ffffffff810f1714> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> 5 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> 257 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> => block_write_full_page+0x15/0x17 <ffffffff81143693> => ext4_writepage+0x201/0x210 <ffffffff81193854> => shrink_page_list+0x3c7/0x6de <ffffffff810e22a1> 809 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => kmalloc_large_node+0x56/0x95 <ffffffff8146a57d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d> => do_readv_writev+0xa7/0x11e <ffffffff8111ea03> => vfs_writev+0x45/0x47 <ffffffff8111eabf> => sys_writev+0x4a/0x93 <ffffffff8111ebe2> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => iget_locked+0x61/0xdd <ffffffff811323be> => ext4_iget+0x2c/0x64c <ffffffff81192a1c> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> => do_path_lookup+0x4d/0xf6 <ffffffff8112881b> => do_filp_open+0x129/0x60a <ffffffff81129c35> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => shmem_file_setup+0x125/0x160 <ffffffff810e7140> => drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc> => i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd> => i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 99 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => mpage_alloc+0x2a/0x84 <ffffffff8114866b> => do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> 24 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f> => ra_submit+0x21/0x25 <ffffffff810ddab1> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514> => do_sync_read+0xbf/0xff <ffffffff8111de6e> => vfs_read+0xa9/0xf0 <ffffffff8111e4d5> => sys_read+0x4a/0x6e <ffffffff8111e566> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 21 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => prepare_creds+0x26/0xae <ffffffff81074d4b> => sys_faccessat+0x37/0x162 <ffffffff8111d279> => sys_access+0x18/0x1a <ffffffff8111d3bc> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => kmalloc_order_trace+0x2c/0x5e <ffffffff8110c73c> => __kmalloc+0x37/0x10d <ffffffff8110df65> => kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c> => tracing_open_pipe+0x40/0x147 <ffffffff810b907a> => __dentry_open+0x161/0x283 <ffffffff8111c980> => nameidata_to_filp+0x60/0x67 <ffffffff8111d8f4> => finish_open+0xa1/0x17f <ffffffff8112974f> => do_filp_open+0x186/0x60a <ffffffff81129c92> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> 2 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => find_or_create_page+0x5a/0x8a <ffffffff810d5b6d> => __getblk+0x137/0x23a <ffffffff81141b55> => __ext4_get_inode_loc+0x118/0x344 <ffffffff8119115f> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1> => ext4_init_security+0x61/0x82 <ffffffff811bc9d3> => ext4_new_inode+0xb69/0xcf2 <ffffffff81190493> 8 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => ll_rw_block+0x84/0xa3 <ffffffff811410f4> => __breadahead+0x2e/0x39 <ffffffff81141c86> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936> => ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1> 58 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> => lookup_hash+0x19/0x1b <ffffffff81127ecc> => do_last+0xcf/0x2df <ffffffff811298fc> => do_filp_open+0x2ac/0x60a <ffffffff81129db8> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_openat+0x11/0x13 <ffffffff8111da20> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => mpage_readpages+0x99/0x104 <ffffffff81148dae> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ondemand_readahead+0x15a/0x1d8 <ffffffff810ddc0f> => page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd6d> => generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d4> => do_sync_read+0xbf/0xff <ffffffff8111de6e> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => get_empty_filp+0x7a/0x141 <ffffffff8111f2f5> => alloc_file+0x1e/0xbf <ffffffff8111f3da> => create_read_pipe+0x2b/0x5b <ffffffff81125ffe> => do_pipe_flags+0x4e/0xf4 <ffffffff8112607c> => sys_pipe2+0x1c/0x5c <ffffffff8112613e> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 9 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => kthreadd+0xe7/0x124 <ffffffff8106f61f> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947> => ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a> => generic_file_buffered_write+0x109/0x23a <ffffffff810d540c> => __generic_file_aio_write+0x242/0x272 <ffffffff810d6181> => generic_file_aio_write+0x58/0xa6 <ffffffff810d6209> => ext4_file_write+0x1dc/0x234 <ffffffff8118e898> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xac/0xf3 <ffffffff8111e3e5> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => __kmalloc+0xb1/0x10d <ffffffff8110dfdf> => drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2> => i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249> => i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dst_alloc+0x48/0xaa <ffffffff813c6672> => __ip_route_output_key+0x561/0x764 <ffffffff813ee513> => tcp_v4_connect+0x121/0x47d <ffffffff8140becf> => inet_stream_connect+0xa8/0x25f <ffffffff81418e24> => sys_connect+0x98/0xd6 <ffffffff813b0437> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => kmalloc_large_node+0x56/0x95 <ffffffff8146a57d> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c> => __alloc_skb+0x75/0x133 <ffffffff813b5e4c> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237> => __sock_sendmsg+0x69/0x76 <ffffffff813af798> => sock_aio_write+0xb9/0xc9 <ffffffff813af85e> => do_sync_write+0xbf/0xff <ffffffff8111dd6f> => vfs_write+0xb3/0xf3 <ffffffff8111e3ec> => sys_write+0x4a/0x6e <ffffffff8111e5d4> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __nf_conntrack_alloc+0xc9/0x1af <ffffffff813e4533> => nf_conntrack_in+0x2f6/0x77a <ffffffff813e490f> => ipv4_conntrack_local+0x46/0x4a <ffffffff8142864c> => nf_iterate+0x48/0x83 <ffffffff813e1cb8> => nf_hook_slow+0x6a/0xe9 <ffffffff813e1d5d> => nf_hook_thresh.constprop.8+0x42/0x44 <ffffffff813f42f0> => __ip_local_out+0x49/0x4d <ffffffff813f433b> => ip_local_out+0x13/0x24 <ffffffff813f4352> => ip_queue_xmit+0x2b3/0x30e <ffffffff813f4616> => tcp_transmit_skb+0x708/0x739 <ffffffff81407217> 1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => dst_alloc+0x48/0xaa <ffffffff813c6672> => __ip_route_output_key+0x561/0x764 <ffffffff813ee513> => ip_route_output_flow+0x2c/0x1ea <ffffffff813ee742> => ip4_datagram_connect+0x18d/0x298 <ffffffff8140ede5> => inet_dgram_connect+0x65/0x6c <ffffffff81418b9d> => sys_connect+0x98/0xd6 <ffffffff813b0437> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_vma+0xf5/0xfa <ffffffff81107025> => do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a25> => handle_mm_fault+0x169/0x1ce <ffffffff810f176a> => do_page_fault+0x358/0x37a <ffffffff814741ae> => page_fault+0x25/0x30 <ffffffff81471455> 1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __page_cache_alloc+0x77/0x7e <ffffffff810d577c> => find_or_create_page+0x41/0x8a <ffffffff810d5b54> => __getblk+0x137/0x23a <ffffffff81141b55> => __breadahead+0x12/0x39 <ffffffff81141c6a> => __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b> => ext4_iget+0x58/0x64c <ffffffff81192a48> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x239/0x3b8 <ffffffff81128469> => do_path_lookup+0x4d/0xf6 <ffffffff8112881b> => user_path_at+0x57/0x94 <ffffffff811294e0> => sys_faccessat+0xa0/0x162 <ffffffff8111d2e2> => sys_access+0x18/0x1a <ffffffff8111d3bc> 2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => shmem_file_setup+0xb6/0x160 <ffffffff810e70d1> => drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc> => i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd> => i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc> => drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815> => do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070> => sys_ioctl+0x56/0x7b <ffffffff8112c107> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 416 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => bio_alloc+0x15/0x24 <ffffffff81144ef5> => submit_bh+0x6d/0x105 <ffffffff811409f6> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> => block_write_full_page+0x15/0x17 <ffffffff81143693> => blkdev_writepage+0x18/0x1a <ffffffff811460b8> => __writepage+0x15/0x2e <ffffffff810dca43> 13 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => mpage_readpages+0x99/0x104 <ffffffff81148dae> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => filemap_fault+0x17b/0x36c <ffffffff810d6ade> => __do_fault+0x55/0x377 <ffffffff810ee88b> => handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47> 58 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d> => mpage_readpages+0xbe/0x104 <ffffffff81148dd3> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => d_alloc+0x26/0x18d <ffffffff8112e4e9> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32> => __lookup_hash+0x7f/0x88 <ffffffff81127eaa> => lookup_hash+0x19/0x1b <ffffffff81127ecc> => do_last+0xcf/0x2df <ffffffff811298fc> => do_filp_open+0x2ac/0x60a <ffffffff81129db8> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> => sys_open+0x20/0x22 <ffffffff8111da0d> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 7727 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> => mempool_alloc+0x68/0x116 <ffffffff810d70fa> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => submit_bh+0xe6/0x105 <ffffffff81140a6f> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> 1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => __split_vma+0x6c/0x21b <ffffffff810f50bd> => do_munmap+0x194/0x30b <ffffffff810f57d1> => mmap_region+0x9b/0x446 <ffffffff810f645d> => do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0> => sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f> => sys_mmap+0x22/0x24 <ffffffff8100da68> => system_call_fastpath+0x16/0x1b <ffffffff81009b82> 1 instances order=1 normal gfp_flags=GFP_KERNEL => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => __get_free_pages+0xe/0x4a <ffffffff810d89d6> => copy_process+0xe1/0x10c5 <ffffffff810533d4> => do_fork+0x104/0x286 <ffffffff810544f7> => kernel_thread+0x75/0x77 <ffffffff81010bb6> => __call_usermodehelper+0x43/0x76 <ffffffff81069132> => process_one_work+0x186/0x298 <ffffffff8106b3ca> => worker_thread+0xda/0x15d <ffffffff8106bf29> => kthread+0x84/0x8c <ffffffff8106f2df> => kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4> 18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => bvec_alloc_bs+0xae/0xcc <ffffffff81144d85> => bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18> => clone_bio+0x41/0xdb <ffffffff81382d3d> => __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0> => dm_request+0x160/0x172 <ffffffff81383bdd> => generic_make_request+0x2ae/0x328 <ffffffff8121212f> => submit_bio+0xde/0xfd <ffffffff81212287> => mpage_readpages+0xf0/0x104 <ffffffff81148e05> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> 1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc> => alloc_inode+0x1d/0x78 <ffffffff81131809> => iget_locked+0x61/0xdd <ffffffff811323be> => ext4_iget+0x2c/0x64c <ffffffff81192a1c> => ext4_lookup+0x93/0xf3 <ffffffff8119a79b> => d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b> => do_lookup+0x23a/0x287 <ffffffff81127dde> => link_path_walk+0x14b/0x3b8 <ffffffff8112837b> => do_filp_open+0x1eb/0x60a <ffffffff81129cf7> => do_sys_open+0x60/0xf2 <ffffffff8111d95b> 1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => find_or_create_page+0x5a/0x8a <ffffffff810d5b6d> => ext4_mb_load_buddy+0x1f7/0x30f <ffffffff811b4bdf> => ext4_free_blocks+0x41d/0x6ba <ffffffff811b7a92> => ext4_ext_truncate+0x361/0x78a <ffffffff811b1012> => ext4_truncate+0xa1/0x4d9 <ffffffff8119598c> => ext4_evict_inode+0x13f/0x228 <ffffffff811978e9> => evict+0x21/0x89 <ffffffff81131b92> 149 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE| => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> => new_slab+0x50/0x199 <ffffffff8110dc48> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> => radix_tree_preload+0x31/0x81 <ffffffff812293bd> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9> => add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5> => mpage_readpages+0x99/0x104 <ffffffff81148dae> => ext4_readpages+0x1d/0x1f <ffffffff8119174d> => __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4> => ra_submit+0x21/0x25 <ffffffff810ddab1> => ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e> => page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08> => generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514> High-order normal allocations: 152966 High-order atomic allocations: 875 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 17:05 ` James Bottomley @ 2011-05-10 17:17 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-10 17:17 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote: > On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote: > > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > > > I really would like to hear if the fix makes a big difference or > > > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > > > less but processes will still be getting stalled in direct compaction > > > > > > and direct reclaim so it would still be jittery. > > > > > > > > > > "the fix" being this > > > > > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > > > evidence the problem is in this area. > > > > > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > > > kswapd being woken up by SLUB using speculative high-orders. The second > > > > one is more drastic and prevents slub entering direct reclaim or > > > > compaction. It applies on top of patch 1. These are both untested and > > > > afraid are a bit rushed as well :( > > > > > > Preliminary results with both patches applied still show kswapd > > > periodically going up to 99% but it doesn't stay there, it comes back > > > down again (and, obviously, the system doesn't hang). > > > > This is a second run with the watch highorders. > > > > At the end of the run, the system hung temporarily and now comes back > > with CPU3 spinning in all system time at kswapd shrink_slab > > Here's a trace in the same situation with the ftrace stack entries > bumped to 16 as requested on IRC. There was no hang for this one. > Ok, so the bulk of the high-order allocs are coming from > 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| > => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> > => new_slab+0x50/0x199 <ffffffff8110dc48> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> > => mempool_alloc+0x68/0x116 <ffffffff810d70fa> > => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> > => bio_alloc+0x15/0x24 <ffffffff81144ef5> > => submit_bh+0x6d/0x105 <ffffffff811409f6> > => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> > => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> > => block_write_full_page+0x15/0x17 <ffffffff81143693> > => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8> > => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88> > That at least is in line with the large untar and absolves i915 from being the main cause of trouble. The lack of the hang implies that SLUB doing high order allocations is stressing the system too much and needs to be more willing to fall back to order-0 although it does not adequately explain why it hung as opposed to just being incredible slow. I'm also still concerned with the reports of getting stuck in a heavy loop on the i915 shrinker so will try again reproducing this locally with a greater focus on something X related happening at the same time. One thing at a time though, SLUB needs to be less aggressive so I'll prepare a series in the morning, have another go at generating data and see what shakes out. Thanks. -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 17:17 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-10 17:17 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote: > On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote: > > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > > > I really would like to hear if the fix makes a big difference or > > > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > > > less but processes will still be getting stalled in direct compaction > > > > > > and direct reclaim so it would still be jittery. > > > > > > > > > > "the fix" being this > > > > > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > > > evidence the problem is in this area. > > > > > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > > > kswapd being woken up by SLUB using speculative high-orders. The second > > > > one is more drastic and prevents slub entering direct reclaim or > > > > compaction. It applies on top of patch 1. These are both untested and > > > > afraid are a bit rushed as well :( > > > > > > Preliminary results with both patches applied still show kswapd > > > periodically going up to 99% but it doesn't stay there, it comes back > > > down again (and, obviously, the system doesn't hang). > > > > This is a second run with the watch highorders. > > > > At the end of the run, the system hung temporarily and now comes back > > with CPU3 spinning in all system time at kswapd shrink_slab > > Here's a trace in the same situation with the ftrace stack entries > bumped to 16 as requested on IRC. There was no hang for this one. > Ok, so the bulk of the high-order allocs are coming from > 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| > => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> > => new_slab+0x50/0x199 <ffffffff8110dc48> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> > => mempool_alloc+0x68/0x116 <ffffffff810d70fa> > => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> > => bio_alloc+0x15/0x24 <ffffffff81144ef5> > => submit_bh+0x6d/0x105 <ffffffff811409f6> > => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> > => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> > => block_write_full_page+0x15/0x17 <ffffffff81143693> > => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8> > => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88> > That at least is in line with the large untar and absolves i915 from being the main cause of trouble. The lack of the hang implies that SLUB doing high order allocations is stressing the system too much and needs to be more willing to fall back to order-0 although it does not adequately explain why it hung as opposed to just being incredible slow. I'm also still concerned with the reports of getting stuck in a heavy loop on the i915 shrinker so will try again reproducing this locally with a greater focus on something X related happening at the same time. One thing at a time though, SLUB needs to be less aggressive so I'll prepare a series in the morning, have another go at generating data and see what shakes out. Thanks. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 17:17 ` Mel Gorman @ 2011-05-10 17:29 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 17:29 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 18:17 +0100, Mel Gorman wrote: > On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote: > > On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote: > > > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > > > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > > > > I really would like to hear if the fix makes a big difference or > > > > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > > > > less but processes will still be getting stalled in direct compaction > > > > > > > and direct reclaim so it would still be jittery. > > > > > > > > > > > > "the fix" being this > > > > > > > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > > > > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > > > > evidence the problem is in this area. > > > > > > > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > > > > kswapd being woken up by SLUB using speculative high-orders. The second > > > > > one is more drastic and prevents slub entering direct reclaim or > > > > > compaction. It applies on top of patch 1. These are both untested and > > > > > afraid are a bit rushed as well :( > > > > > > > > Preliminary results with both patches applied still show kswapd > > > > periodically going up to 99% but it doesn't stay there, it comes back > > > > down again (and, obviously, the system doesn't hang). > > > > > > This is a second run with the watch highorders. > > > > > > At the end of the run, the system hung temporarily and now comes back > > > with CPU3 spinning in all system time at kswapd shrink_slab > > > > Here's a trace in the same situation with the ftrace stack entries > > bumped to 16 as requested on IRC. There was no hang for this one. > > > > Ok, so the bulk of the high-order allocs are coming from > > > 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| > > => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> > > => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> > > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> > > => new_slab+0x50/0x199 <ffffffff8110dc48> > > => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> > > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> > > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> > > => mempool_alloc+0x68/0x116 <ffffffff810d70fa> > > => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> > > => bio_alloc+0x15/0x24 <ffffffff81144ef5> > > => submit_bh+0x6d/0x105 <ffffffff811409f6> > > => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> > > => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> > > => block_write_full_page+0x15/0x17 <ffffffff81143693> > > => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8> > > => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88> > > > > That at least is in line with the large untar and absolves i915 > from being the main cause of trouble. The lack of the hang implies > that SLUB doing high order allocations is stressing the system too > much and needs to be more willing to fall back to order-0 although > it does not adequately explain why it hung as opposed to just being > incredible slow. > > I'm also still concerned with the reports of getting stuck in a heavy > loop on the i915 shrinker so will try again reproducing this locally > with a greater focus on something X related happening at the same time. > > One thing at a time though, SLUB needs to be less aggressive so I'll > prepare a series in the morning, have another go at generating data > and see what shakes out. As a final data point, I tried slub_max_order=0. kswapd is much more quiescent (it calms down to nothing). It shows spikes of up to 70% and once up to 99% though, but it doesn't go up to 99% and stay there like it did previously. This is just with top, so it's a bit inaccurate. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 17:29 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-10 17:29 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Tue, 2011-05-10 at 18:17 +0100, Mel Gorman wrote: > On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote: > > On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote: > > > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote: > > > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > > > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > > > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > > > > > > > I really would like to hear if the fix makes a big difference or > > > > > > > if we need to consider forcing SLUB high-order allocations bailing > > > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > > > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up > > > > > > > less but processes will still be getting stalled in direct compaction > > > > > > > and direct reclaim so it would still be jittery. > > > > > > > > > > > > "the fix" being this > > > > > > > > > > > > https://lkml.org/lkml/2011/3/5/121 > > > > > > > > > > > > > > > > Drop this for the moment. It was a long shot at best and there is little > > > > > evidence the problem is in this area. > > > > > > > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop > > > > > kswapd being woken up by SLUB using speculative high-orders. The second > > > > > one is more drastic and prevents slub entering direct reclaim or > > > > > compaction. It applies on top of patch 1. These are both untested and > > > > > afraid are a bit rushed as well :( > > > > > > > > Preliminary results with both patches applied still show kswapd > > > > periodically going up to 99% but it doesn't stay there, it comes back > > > > down again (and, obviously, the system doesn't hang). > > > > > > This is a second run with the watch highorders. > > > > > > At the end of the run, the system hung temporarily and now comes back > > > with CPU3 spinning in all system time at kswapd shrink_slab > > > > Here's a trace in the same situation with the ftrace stack entries > > bumped to 16 as requested on IRC. There was no hang for this one. > > > > Ok, so the bulk of the high-order allocs are coming from > > > 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC| > > => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de> > > => alloc_pages_current+0xbe/0xd8 <ffffffff81105459> > > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe> > > => new_slab+0x50/0x199 <ffffffff8110dc48> > > => __slab_alloc+0x24a/0x328 <ffffffff8146ab86> > > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450> > > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85> > > => mempool_alloc+0x68/0x116 <ffffffff810d70fa> > > => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8> > > => bio_alloc+0x15/0x24 <ffffffff81144ef5> > > => submit_bh+0x6d/0x105 <ffffffff811409f6> > > => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac> > > => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671> > > => block_write_full_page+0x15/0x17 <ffffffff81143693> > > => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8> > > => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88> > > > > That at least is in line with the large untar and absolves i915 > from being the main cause of trouble. The lack of the hang implies > that SLUB doing high order allocations is stressing the system too > much and needs to be more willing to fall back to order-0 although > it does not adequately explain why it hung as opposed to just being > incredible slow. > > I'm also still concerned with the reports of getting stuck in a heavy > loop on the i915 shrinker so will try again reproducing this locally > with a greater focus on something X related happening at the same time. > > One thing at a time though, SLUB needs to be less aggressive so I'll > prepare a series in the morning, have another go at generating data > and see what shakes out. As a final data point, I tried slub_max_order=0. kswapd is much more quiescent (it calms down to nothing). It shows spikes of up to 70% and once up to 99% though, but it doesn't go up to 99% and stay there like it did previously. This is just with top, so it's a bit inaccurate. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 15:29 ` James Bottomley (?) (?) @ 2011-05-10 21:08 ` Raghavendra D Prabhu 2011-05-11 9:16 ` Mel Gorman -1 siblings, 1 reply; 138+ messages in thread From: Raghavendra D Prabhu @ 2011-05-10 21:08 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 22814 bytes --] * On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <James.Bottomley@HansenPartnership.com> wrote: >On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: >> On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: >> > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: >> > > I really would like to hear if the fix makes a big difference or >> > > if we need to consider forcing SLUB high-order allocations bailing >> > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in >> > > allocate_slab). Even with the fix applied, kswapd might be waking up >> > > less but processes will still be getting stalled in direct compaction >> > > and direct reclaim so it would still be jittery. >> > "the fix" being this >> > https://lkml.org/lkml/2011/3/5/121 >> Drop this for the moment. It was a long shot at best and there is little >> evidence the problem is in this area. >> I'm attaching two patches. The first is the NO_KSWAPD one to stop >> kswapd being woken up by SLUB using speculative high-orders. The second >> one is more drastic and prevents slub entering direct reclaim or >> compaction. It applies on top of patch 1. These are both untested and >> afraid are a bit rushed as well :( > >Preliminary results with both patches applied still show kswapd >periodically going up to 99% but it doesn't stay there, it comes back >down again (and, obviously, the system doesn't hang). > >This is sysrq-M from a couple of times when it went up there: > >[ 426.736958] SysRq : Show Memory >[ 426.736974] Mem-Info: >[ 426.736977] Node 0 DMA per-cpu: >[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0 >[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0 >[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0 >[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0 >[ 426.736996] Node 0 DMA32 per-cpu: >[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169 >[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40 >[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166 >[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168 >[ 426.737015] Node 0 Normal per-cpu: >[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0 >[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0 >[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0 >[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0 >[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0 >[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32 >[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0 >[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477 >[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0 >[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no >[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862 >[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no >[ 426.737092] lowmem_reserve[]: 0 0 5 5 >[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes >[ 426.737119] lowmem_reserve[]: 0 0 0 0 >[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB >[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB >[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB >[ 426.737257] 99595 total pagecache pages >[ 426.737260] 0 pages in swap cache >[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0 >[ 426.737266] Free swap = 3768316kB >[ 426.737268] Total swap = 3768316kB >[ 426.744603] 525808 pages RAM >[ 426.744612] 57618 pages reserved >[ 426.744614] 141551 pages shared >[ 426.744617] 186065 pages non-shared >[ 472.301810] SysRq : Show Memory >[ 472.301826] Mem-Info: >[ 472.301829] Node 0 DMA per-cpu: >[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0 >[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0 >[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0 >[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0 >[ 472.301848] Node 0 DMA32 per-cpu: >[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184 >[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46 >[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158 >[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163 >[ 472.301866] Node 0 Normal per-cpu: >[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0 >[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0 >[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0 >[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0 >[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0 >[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32 >[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0 >[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647 >[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0 >[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no >[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862 >[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no >[ 472.301943] lowmem_reserve[]: 0 0 5 5 >[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes >[ 472.301968] lowmem_reserve[]: 0 0 0 0 >[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB >[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB >[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB >[ 472.302090] 91312 total pagecache pages >[ 472.302093] 0 pages in swap cache >[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0 >[ 472.302099] Free swap = 3768316kB >[ 472.302102] Total swap = 3768316kB >[ 472.309521] 525808 pages RAM >[ 472.309529] 57618 pages reserved >[ 472.309548] 142496 pages shared >[ 472.309551] 177182 pages non-shared > >I'll finish off this verification, and then re-run with the watch-highorder script running. > >James Hi, I saw similar lockups while doing a heavy copy routine (basically copying an entire partition). I don't think this may be related to ext4, since the source and destination filesystems were btrfs and xfs. =============================================================================================================== 9 00:04:58 localhost kernel: [39824.230679] INFO: task kswapd0:528 blocked for more than 120 seconds. May 9 00:04:58 localhost kernel: [39824.230684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 9 00:04:58 localhost kernel: [39824.230690] kswapd0 D ffff8800bb90d740 0 528 2 0x00000000 May 9 00:04:58 localhost kernel: [39824.230699] ffff88013f6a3750 0000000000000046 0000000000000246 ffff88013b290040 May 9 00:04:58 localhost kernel: [39824.230708] ffff88013b290000 0000000000000001 ffff88013f6ea210 ffff88013f6a3fd8 May 9 00:04:58 localhost kernel: [39824.230716] ffff88013f6a3fd8 ffff88013f6a3fd8 0000000000000004 ffff88013f6ea470 May 9 00:04:58 localhost kernel: [39824.230724] Call Trace: May 9 00:04:58 localhost kernel: [39824.230738] [<ffffffff81548aa3>] io_schedule+0x73/0xc0 May 9 00:04:58 localhost kernel: [39824.230747] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0 May 9 00:04:58 localhost kernel: [39824.230757] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40 May 9 00:05:22 localhost kernel: [39824.230766] [<ffffffff81334cb6>] __make_request+0x76/0x560 May 9 00:05:22 localhost kernel: [39824.230774] [<ffffffff810e1518>] ? account_page_writeback+0x28/0x30 May 9 00:05:22 localhost kernel: [39824.230782] [<ffffffff81332bce>] generic_make_request+0x28e/0x560 May 9 00:05:22 localhost kernel: [39824.230790] [<ffffffff81332f25>] submit_bio+0x85/0x110 May 9 00:05:22 localhost kernel: [39824.230797] [<ffffffff811552b2>] ? bio_alloc_bioset+0xb2/0xf0 May 9 00:05:22 localhost kernel: [39824.230806] [<ffffffff8127da17>] xfs_submit_ioend_bio.isra.11+0x57/0x80 May 9 00:05:22 localhost kernel: [39824.230816] [<ffffffff8127dafb>] xfs_submit_ioend+0xbb/0x110 May 9 00:05:22 localhost kernel: [39824.230818] [<ffffffff8127ed47>] xfs_vm_writepage+0x3f7/0x540 May 9 00:05:22 localhost kernel: [39824.230821] [<ffffffff810e90b0>] shrink_page_list+0x650/0x8f0 May 9 00:05:22 localhost kernel: [39824.230823] [<ffffffff810e9758>] shrink_inactive_list+0x178/0x410 May 9 00:05:22 localhost kernel: [39824.230826] [<ffffffff8104e89a>] ? del_timer_sync+0x3a/0x60 May 9 00:05:22 localhost kernel: [39824.230828] [<ffffffff810e9e6e>] shrink_zone+0x47e/0x540 May 9 00:05:22 localhost kernel: [39824.230831] [<ffffffff810e7a25>] ? shrink_slab+0x145/0x180 *May 9 00:05:22 localhost kernel: [39824.230833] [<ffffffff810ea5bd>] kswapd+0x68d/0xa20 *May 9 00:05:22 localhost kernel: [39824.230835] [<ffffffff810e9f30>] ? kswapd+0x0/0xa20 May 9 00:05:22 localhost kernel: [39824.230837] [<ffffffff8105fffc>] kthread+0x8c/0xa0 May 9 00:05:22 localhost kernel: [39824.230841] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0 May 9 00:05:22 localhost kernel: [39824.230843] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10 May 9 00:05:22 localhost kernel: [39824.230846] [<ffffffff8105ff70>] ? kthread+0x0/0xa0 May 9 00:05:22 localhost kernel: [39824.230848] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10 May 9 00:05:22 localhost kernel: [39824.230853] INFO: task xfsbufd/sda9:3757 blocked for more than 120 seconds. May 9 00:05:22 localhost kernel: [39824.230854] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 9 00:05:22 localhost kernel: [39824.230856] xfsbufd/sda9 D ffff8800bb84d740 0 3757 2 0x00000000 May 9 00:05:22 localhost kernel: [39824.230858] ffff88013b12fb10 0000000000000046 0000000000000246 ffff88013f6736c0 May 9 00:05:22 localhost kernel: [39824.230861] ffff88013f673680 0000000000000001 ffff88013cec2210 ffff88013b12ffd8 May 9 00:05:22 localhost kernel: [39824.230863] ffff88013b12ffd8 ffff88013b12ffd8 0000000000000001 ffff88013cec2470 May 9 00:05:22 localhost kernel: [39824.230866] Call Trace: May 9 00:05:22 localhost kernel: [39824.230868] [<ffffffff81548aa3>] io_schedule+0x73/0xc0 May 9 00:05:22 localhost kernel: [39824.230870] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0 May 9 00:05:22 localhost kernel: [39824.230873] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40 May 9 00:05:22 localhost kernel: [39824.230875] [<ffffffff81334cb6>] __make_request+0x76/0x560 May 9 00:05:22 localhost kernel: [39824.230878] [<ffffffff81037434>] ? resched_best_mask.isra.60+0x114/0x130 May 9 00:05:22 localhost kernel: [39824.230880] [<ffffffff81332bce>] generic_make_request+0x28e/0x560 May 9 00:05:22 localhost kernel: [39824.230883] [<ffffffff8103640c>] ? update_cpu_clock+0x19c/0x3d0 May 9 00:05:22 localhost kernel: [39824.230885] [<ffffffff81332f25>] submit_bio+0x85/0x110 May 9 00:05:22 localhost kernel: [39824.230887] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0 May 9 00:05:22 localhost kernel: [39824.230889] [<ffffffff8128159c>] _xfs_buf_ioapply+0x19c/0x300 May 9 00:05:22 localhost kernel: [39824.230892] [<ffffffff812825cd>] ? xfs_bdstrat_cb+0x5d/0x100 May 9 00:05:22 localhost kernel: [39824.230894] [<ffffffff8128194e>] xfs_buf_iorequest+0x5e/0x140 May 9 00:05:22 localhost kernel: [39824.230896] [<ffffffff812825cd>] xfs_bdstrat_cb+0x5d/0x100 May 9 00:05:22 localhost kernel: [39824.230899] [<ffffffff81282776>] xfsbufd+0x106/0x180 May 9 00:05:22 localhost kernel: [39824.230901] [<ffffffff81282670>] ? xfsbufd+0x0/0x180 May 9 00:05:22 localhost kernel: [39824.230903] [<ffffffff8105fffc>] kthread+0x8c/0xa0 May 9 00:05:22 localhost kernel: [39824.230905] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0 May 9 00:05:22 localhost kernel: [39824.230907] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10 May 9 00:05:22 localhost kernel: [39824.230909] [<ffffffff8105ff70>] ? kthread+0x0/0xa0 May 9 00:05:22 localhost kernel: [39824.230911] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10 May 9 00:05:22 localhost kernel: [39824.230913] INFO: task xfsbufd/sda11:3769 blocked for more than 120 seconds. May 9 00:05:22 localhost kernel: [39824.230914] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 9 00:05:22 localhost kernel: [39824.230915] xfsbufd/sda11 D ffff8800bb84d740 0 3769 2 0x00000000 May 9 00:05:22 localhost kernel: [39824.230918] ffff88013b183b10 0000000000000046 0000000000000246 ffff88013f6736c0 May 9 00:05:22 localhost kernel: [39824.230920] ffff88013f673680 0000000000000001 ffff88013b299b40 ffff88013b183fd8 May 9 00:05:22 localhost kernel: [39824.230923] ffff88013b183fd8 ffff88013b183fd8 0000000000000001 ffff88013b299da0 May 9 00:05:22 localhost kernel: [39824.230925] Call Trace: May 9 00:05:22 localhost kernel: [39824.230928] [<ffffffff81548aa3>] io_schedule+0x73/0xc0 May 9 00:05:22 localhost kernel: [39824.230930] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0 May 9 00:05:22 localhost kernel: [39824.230932] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40 May 9 00:05:22 localhost kernel: [39824.230935] [<ffffffff81334cb6>] __make_request+0x76/0x560 May 9 00:05:22 localhost kernel: [39824.230938] [<ffffffff8135d7af>] ? __sg_alloc_table+0x7f/0x140 May 9 00:05:22 localhost kernel: [39824.230941] [<ffffffff81332bce>] generic_make_request+0x28e/0x560 May 9 00:05:22 localhost kernel: [39824.230943] [<ffffffff8103640c>] ? update_cpu_clock+0x19c/0x3d0 May 9 00:05:22 localhost kernel: [39824.230946] [<ffffffff81431a60>] ? ata_scsi_rw_xlat+0x0/0x210 May 9 00:05:22 localhost kernel: [39824.230948] [<ffffffff81332f25>] submit_bio+0x85/0x110 May 9 00:05:22 localhost kernel: [39824.230950] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0 May 9 00:05:22 localhost kernel: [39824.230953] [<ffffffff8128159c>] _xfs_buf_ioapply+0x19c/0x300 May 9 00:05:22 localhost kernel: [39824.230955] [<ffffffff812825cd>] ? xfs_bdstrat_cb+0x5d/0x100 May 9 00:05:22 localhost kernel: [39824.230957] [<ffffffff8128194e>] xfs_buf_iorequest+0x5e/0x140 May 9 00:05:22 localhost kernel: [39824.230959] [<ffffffff812825cd>] xfs_bdstrat_cb+0x5d/0x100 May 9 00:05:22 localhost kernel: [39824.230962] [<ffffffff81282776>] xfsbufd+0x106/0x180 May 9 00:05:22 localhost kernel: [39824.230964] [<ffffffff81282670>] ? xfsbufd+0x0/0x180 May 9 00:05:22 localhost kernel: [39824.230966] [<ffffffff8105fffc>] kthread+0x8c/0xa0 May 9 00:05:22 localhost kernel: [39824.230968] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0 May 9 00:05:22 localhost kernel: [39824.230970] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10 May 9 00:05:22 localhost kernel: [39824.230972] [<ffffffff8105ff70>] ? kthread+0x0/0xa0 May 9 00:05:22 localhost kernel: [39824.230974] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10 May 9 00:05:22 localhost kernel: [39824.230989] INFO: task logrotate:1749 blocked for more than 120 seconds. May 9 00:05:22 localhost kernel: [39824.230990] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 9 00:05:22 localhost kernel: [39824.230992] logrotate D ffff8800bb90d740 0 1749 1748 0x00000000 May 9 00:05:22 localhost kernel: [39824.230994] ffff88000c6f3788 0000000000000082 ffff88000c6f3738 ffff88013f475890 May 9 00:05:22 localhost kernel: [39824.230997] 0000000000000001 0000000000000001 0000000000000001 ffff88000c6f3fd8 May 9 00:05:22 localhost kernel: [39824.230999] ffff88000c6f3fd8 ffff88000c6f3fd8 0000000000000004 ffff88013f475af0 May 9 00:05:22 localhost kernel: [39824.231002] Call Trace: May 9 00:05:22 localhost kernel: [39824.231004] [<ffffffff81548aa3>] io_schedule+0x73/0xc0 May 9 00:05:22 localhost kernel: [39824.231006] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0 May 9 00:05:22 localhost kernel: [39824.231008] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40 May 9 00:05:22 localhost kernel: [39824.231011] [<ffffffff81334cb6>] __make_request+0x76/0x560 May 9 00:05:22 localhost kernel: [39824.231014] [<ffffffff81150af4>] ? __getblk+0x24/0x280 May 9 00:05:22 localhost kernel: [39824.231016] [<ffffffff81332bce>] generic_make_request+0x28e/0x560 May 9 00:05:22 localhost kernel: [39824.231020] [<ffffffff811ae4fc>] ? ext4_mark_iloc_dirty+0x39c/0x5a0 May 9 00:05:22 localhost kernel: [39824.231022] [<ffffffff81332f25>] submit_bio+0x85/0x110 May 9 00:05:22 localhost kernel: [39824.231024] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0 May 9 00:05:22 localhost kernel: [39824.231027] [<ffffffff8114f66b>] submit_bh+0xeb/0x120 May 9 00:05:22 localhost kernel: [39824.231029] [<ffffffff81151720>] __block_write_full_page+0x210/0x390 May 9 00:05:22 localhost kernel: [39824.231031] [<ffffffff810605de>] ? wake_up_bit+0x2e/0x40 May 9 00:05:22 localhost kernel: [39824.231034] [<ffffffff811518a0>] ? end_buffer_async_write+0x0/0x200 May 9 00:05:22 localhost kernel: [39824.231036] [<ffffffff811b00c0>] ? noalloc_get_block_write+0x0/0x30 May 9 00:05:22 localhost kernel: [39824.231038] [<ffffffff811518a0>] ? end_buffer_async_write+0x0/0x200 May 9 00:05:22 localhost kernel: [39824.231041] [<ffffffff81151f65>] block_write_full_page_endio+0xf5/0x140 May 9 00:05:22 localhost kernel: [39824.231043] [<ffffffff811b00c0>] ? noalloc_get_block_write+0x0/0x30 May 9 00:05:22 localhost kernel: [39824.231045] [<ffffffff81151fc5>] block_write_full_page+0x15/0x20 May 9 00:05:22 localhost kernel: [39824.231048] [<ffffffff811add81>] mpage_da_submit_io+0x451/0x510 May 9 00:05:22 localhost kernel: [39824.231050] [<ffffffff811ae4fc>] ? ext4_mark_iloc_dirty+0x39c/0x5a0 May 9 00:05:22 localhost kernel: [39824.231053] [<ffffffff811ae822>] ? ext4_mark_inode_dirty+0x82/0x230 May 9 00:05:22 localhost kernel: [39824.231055] [<ffffffff811b26a6>] mpage_da_map_and_submit+0x1d6/0x410 May 9 00:05:22 localhost kernel: [39824.231059] [<ffffffff811e8983>] ? jbd2_journal_start+0x13/0x20 May 9 00:05:22 localhost kernel: [39824.231061] [<ffffffff811b31b1>] ext4_da_writepages+0x3e1/0x750 May 9 00:05:22 localhost kernel: [39824.231064] [<ffffffff810e2e91>] do_writepages+0x21/0x40 May 9 00:05:22 localhost kernel: [39824.231066] [<ffffffff810da00b>] __filemap_fdatawrite_range+0x5b/0x60 May 9 00:05:22 localhost kernel: [39824.231068] [<ffffffff810da74c>] filemap_flush+0x1c/0x20 May 9 00:05:22 localhost kernel: [39824.231070] [<ffffffff811acafc>] ext4_alloc_da_blocks+0x4c/0xe0 May 9 00:05:22 localhost kernel: [39824.231073] [<ffffffff811a78e9>] ext4_release_file+0x79/0xc0 May 9 00:05:22 localhost kernel: [39824.231075] [<ffffffff81124d7a>] fput+0xda/0x200 May 9 00:05:22 localhost kernel: [39824.231077] [<ffffffff811214e5>] filp_close+0x55/0x80 May 9 00:05:22 localhost kernel: [39824.231079] [<ffffffff811215c6>] sys_close+0xb6/0x120 May 9 00:05:22 localhost kernel: [39824.231082] [<ffffffff81002eeb>] system_call_fastpath+0x16/0x1b =========================================================================================================================================================== The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this. The kernel is ======================================================== Linux Xye 2.6.39-rc6 #78 SMP PREEMPT Mon May 9 14:38:17 IST 2011 x86_64 Intel(R) Core(TM) i7 CPU Q 740 @ 1.73GHz GenuineIntel GNU/Linux ====================================================== Also, I am not using cgroups or systemd at the moment. Also, this is not a Fedora system. (Arch linux). This happened only once. After which it has been fine, though I have not done such high throughput copying after that. --------------------- Raghavendra D Prabhu [-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-10 21:08 ` Raghavendra D Prabhu @ 2011-05-11 9:16 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-11 9:16 UTC (permalink / raw) To: Raghavendra D Prabhu Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, May 11, 2011 at 02:38:09AM +0530, Raghavendra D Prabhu wrote: > * On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <James.Bottomley@HansenPartnership.com> wrote: > >On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > >>On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > >>> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > >>> > I really would like to hear if the fix makes a big difference or > >>> > if we need to consider forcing SLUB high-order allocations bailing > >>> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > >>> > allocate_slab). Even with the fix applied, kswapd might be waking up > >>> > less but processes will still be getting stalled in direct compaction > >>> > and direct reclaim so it would still be jittery. > > >>> "the fix" being this > > >>> https://lkml.org/lkml/2011/3/5/121 > > > >>Drop this for the moment. It was a long shot at best and there is little > >>evidence the problem is in this area. > > >>I'm attaching two patches. The first is the NO_KSWAPD one to stop > >>kswapd being woken up by SLUB using speculative high-orders. The second > >>one is more drastic and prevents slub entering direct reclaim or > >>compaction. It applies on top of patch 1. These are both untested and > >>afraid are a bit rushed as well :( > > > >Preliminary results with both patches applied still show kswapd > >periodically going up to 99% but it doesn't stay there, it comes back > >down again (and, obviously, the system doesn't hang). > > > >This is sysrq-M from a couple of times when it went up there: > > > >[ 426.736958] SysRq : Show Memory > >[ 426.736974] Mem-Info: > >[ 426.736977] Node 0 DMA per-cpu: > >[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 426.736996] Node 0 DMA32 per-cpu: > >[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169 > >[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40 > >[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166 > >[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168 > >[ 426.737015] Node 0 Normal per-cpu: > >[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0 > >[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32 > >[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0 > >[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477 > >[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0 > >[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > >[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862 > >[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no > >[ 426.737092] lowmem_reserve[]: 0 0 5 5 > >[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > >[ 426.737119] lowmem_reserve[]: 0 0 0 0 > >[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB > >[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB > >[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB > >[ 426.737257] 99595 total pagecache pages > >[ 426.737260] 0 pages in swap cache > >[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0 > >[ 426.737266] Free swap = 3768316kB > >[ 426.737268] Total swap = 3768316kB > >[ 426.744603] 525808 pages RAM > >[ 426.744612] 57618 pages reserved > >[ 426.744614] 141551 pages shared > >[ 426.744617] 186065 pages non-shared > >[ 472.301810] SysRq : Show Memory > >[ 472.301826] Mem-Info: > >[ 472.301829] Node 0 DMA per-cpu: > >[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 472.301848] Node 0 DMA32 per-cpu: > >[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184 > >[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46 > >[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158 > >[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163 > >[ 472.301866] Node 0 Normal per-cpu: > >[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0 > >[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32 > >[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0 > >[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647 > >[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0 > >[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > >[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862 > >[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no > >[ 472.301943] lowmem_reserve[]: 0 0 5 5 > >[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > >[ 472.301968] lowmem_reserve[]: 0 0 0 0 > >[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB > >[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB > >[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB > >[ 472.302090] 91312 total pagecache pages > >[ 472.302093] 0 pages in swap cache > >[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0 > >[ 472.302099] Free swap = 3768316kB > >[ 472.302102] Total swap = 3768316kB > >[ 472.309521] 525808 pages RAM > >[ 472.309529] 57618 pages reserved > >[ 472.309548] 142496 pages shared > >[ 472.309551] 177182 pages non-shared > > > >I'll finish off this verification, and then re-run with the watch-highorder script running. > > > >James > Hi, > > I saw similar lockups while doing a heavy copy routine (basically > copying an entire partition). I don't think this may be related to > ext4, since the source and destination filesystems were btrfs and > xfs. > If you have SLUB configured, can you try again with the same two patches I attached earlier in the thread please? > <SNIP> > > The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this. > Is 1G higher than what /proc/sys/vm/dirty_ratio should allow? -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-11 9:16 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-11 9:16 UTC (permalink / raw) To: Raghavendra D Prabhu Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Wed, May 11, 2011 at 02:38:09AM +0530, Raghavendra D Prabhu wrote: > * On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <James.Bottomley@HansenPartnership.com> wrote: > >On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote: > >>On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote: > >>> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote: > >>> > I really would like to hear if the fix makes a big difference or > >>> > if we need to consider forcing SLUB high-order allocations bailing > >>> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in > >>> > allocate_slab). Even with the fix applied, kswapd might be waking up > >>> > less but processes will still be getting stalled in direct compaction > >>> > and direct reclaim so it would still be jittery. > > >>> "the fix" being this > > >>> https://lkml.org/lkml/2011/3/5/121 > > > >>Drop this for the moment. It was a long shot at best and there is little > >>evidence the problem is in this area. > > >>I'm attaching two patches. The first is the NO_KSWAPD one to stop > >>kswapd being woken up by SLUB using speculative high-orders. The second > >>one is more drastic and prevents slub entering direct reclaim or > >>compaction. It applies on top of patch 1. These are both untested and > >>afraid are a bit rushed as well :( > > > >Preliminary results with both patches applied still show kswapd > >periodically going up to 99% but it doesn't stay there, it comes back > >down again (and, obviously, the system doesn't hang). > > > >This is sysrq-M from a couple of times when it went up there: > > > >[ 426.736958] SysRq : Show Memory > >[ 426.736974] Mem-Info: > >[ 426.736977] Node 0 DMA per-cpu: > >[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 426.736996] Node 0 DMA32 per-cpu: > >[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169 > >[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40 > >[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166 > >[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168 > >[ 426.737015] Node 0 Normal per-cpu: > >[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0 > >[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32 > >[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0 > >[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477 > >[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0 > >[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > >[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862 > >[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no > >[ 426.737092] lowmem_reserve[]: 0 0 5 5 > >[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > >[ 426.737119] lowmem_reserve[]: 0 0 0 0 > >[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB > >[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB > >[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB > >[ 426.737257] 99595 total pagecache pages > >[ 426.737260] 0 pages in swap cache > >[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0 > >[ 426.737266] Free swap = 3768316kB > >[ 426.737268] Total swap = 3768316kB > >[ 426.744603] 525808 pages RAM > >[ 426.744612] 57618 pages reserved > >[ 426.744614] 141551 pages shared > >[ 426.744617] 186065 pages non-shared > >[ 472.301810] SysRq : Show Memory > >[ 472.301826] Mem-Info: > >[ 472.301829] Node 0 DMA per-cpu: > >[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 472.301848] Node 0 DMA32 per-cpu: > >[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184 > >[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46 > >[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158 > >[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163 > >[ 472.301866] Node 0 Normal per-cpu: > >[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0 > >[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0 > >[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0 > >[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0 > >[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0 > >[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32 > >[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0 > >[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647 > >[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0 > >[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > >[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862 > >[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no > >[ 472.301943] lowmem_reserve[]: 0 0 5 5 > >[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > >[ 472.301968] lowmem_reserve[]: 0 0 0 0 > >[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB > >[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB > >[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB > >[ 472.302090] 91312 total pagecache pages > >[ 472.302093] 0 pages in swap cache > >[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0 > >[ 472.302099] Free swap = 3768316kB > >[ 472.302102] Total swap = 3768316kB > >[ 472.309521] 525808 pages RAM > >[ 472.309529] 57618 pages reserved > >[ 472.309548] 142496 pages shared > >[ 472.309551] 177182 pages non-shared > > > >I'll finish off this verification, and then re-run with the watch-highorder script running. > > > >James > Hi, > > I saw similar lockups while doing a heavy copy routine (basically > copying an entire partition). I don't think this may be related to > ext4, since the source and destination filesystems were btrfs and > xfs. > If you have SLUB configured, can you try again with the same two patches I attached earlier in the thread please? > <SNIP> > > The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this. > Is 1G higher than what /proc/sys/vm/dirty_ratio should allow? -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 7:42 ` Mel Gorman @ 2011-05-06 11:42 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 11:42 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote: > > On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote: > > > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll > > > send under separate cover. > > > > Here it is ... it's under 2.6.38.4 vanilla, but the code is similar. > > > > I was quiet because I was off trying to reproduce this but not having > much luck. It doesn't seem directly related to filesystems or > cgroups. For example, here is what I see with ext4 without cgroups > > 2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla > download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%) > unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%) > copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%) > create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%) > delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%) > expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%) > > (all results are in seconds) > > When running in cgroups, the results are similar - bit slower but > not remarkably so. ext3 is slower but not enough to count as the bug. > > The trace you posted is very short but kswapd is not going to sleep > in it. It's less than a seconds worth on different cpus so it's hard > to draw any conclusion from it other than sleeping_prematurely() > is often deciding that kswapd should not sleep. > > So lets consider what keeps it awake. > > 1. High-order allocations? You machine is using i915 and RPC, something > neither of my test machine uses. i915 is potentially a source for > high-order allocations. I'm attaching a perl script. Please run it as > ./watch-highorder.pl --output /tmp/highorders.txt > while you are running tar. When kswapd is running for about 30 > seconds, interrupt it with ctrl+c twice in quick succession and > post /tmp/highorders.txt > > 2. All unreclaimable is not being set or we are not balancing at all. > Can you post the output of sysrq+m while the machine is struggling > please? > > 3. Slab may not be shrinking for some reason. Can you run a shell > script like this during the whole test and record its output please? > > #!/bin/bash > while [ 1 ]; do > echo time: `date +%s` > cat /proc/vmstat > sleep 2 > done > > Similarly if this is a slab issue, it'd be nice to know who it is so > > #!/bin/bash > while [ 1 ]; do > echo time: `date +%s` > cat /proc/slabinfo > sleep $MONITOR_UPDATE_FREQUENCY > done > > 4. Lets get a better look at what is going on in kswapd > > echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable > cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt > Also, could you test the patch at https://lkml.org/lkml/2011/3/5/121 please? -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-06 11:42 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 11:42 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote: > > On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote: > > > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll > > > send under separate cover. > > > > Here it is ... it's under 2.6.38.4 vanilla, but the code is similar. > > > > I was quiet because I was off trying to reproduce this but not having > much luck. It doesn't seem directly related to filesystems or > cgroups. For example, here is what I see with ext4 without cgroups > > 2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla > download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%) > unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%) > copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%) > create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%) > delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%) > expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%) > > (all results are in seconds) > > When running in cgroups, the results are similar - bit slower but > not remarkably so. ext3 is slower but not enough to count as the bug. > > The trace you posted is very short but kswapd is not going to sleep > in it. It's less than a seconds worth on different cpus so it's hard > to draw any conclusion from it other than sleeping_prematurely() > is often deciding that kswapd should not sleep. > > So lets consider what keeps it awake. > > 1. High-order allocations? You machine is using i915 and RPC, something > neither of my test machine uses. i915 is potentially a source for > high-order allocations. I'm attaching a perl script. Please run it as > ./watch-highorder.pl --output /tmp/highorders.txt > while you are running tar. When kswapd is running for about 30 > seconds, interrupt it with ctrl+c twice in quick succession and > post /tmp/highorders.txt > > 2. All unreclaimable is not being set or we are not balancing at all. > Can you post the output of sysrq+m while the machine is struggling > please? > > 3. Slab may not be shrinking for some reason. Can you run a shell > script like this during the whole test and record its output please? > > #!/bin/bash > while [ 1 ]; do > echo time: `date +%s` > cat /proc/vmstat > sleep 2 > done > > Similarly if this is a slab issue, it'd be nice to know who it is so > > #!/bin/bash > while [ 1 ]; do > echo time: `date +%s` > cat /proc/slabinfo > sleep $MONITOR_UPDATE_FREQUENCY > done > > 4. Lets get a better look at what is going on in kswapd > > echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable > cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt > Also, could you test the patch at https://lkml.org/lkml/2011/3/5/121 please? -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 7:42 ` Mel Gorman @ 2011-05-06 15:44 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 15:44 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > 1. High-order allocations? You machine is using i915 and RPC, something > neither of my test machine uses. i915 is potentially a source for > high-order allocations. I'm attaching a perl script. Please run it as > ./watch-highorder.pl --output /tmp/highorders.txt > while you are running tar. When kswapd is running for about 30 > seconds, interrupt it with ctrl+c twice in quick succession and > post /tmp/highorders.txt > Colin send me this information for his test case at least and I see 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5> => new_slab+0x1f5/0x290 <ffffffff81153645> => __slab_alloc+0x262/0x390 <ffffffff81155192> => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5> => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705> => mempool_alloc+0x59/0x140 <ffffffff8110ea49> => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae> => bio_alloc+0x15/0x30 <ffffffff81197805> Colin and James: Did you happen to switch from SLAB to SLUB between 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I didn't see the problem. Am restarting the tests with SLUB. -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-06 15:44 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 15:44 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > 1. High-order allocations? You machine is using i915 and RPC, something > neither of my test machine uses. i915 is potentially a source for > high-order allocations. I'm attaching a perl script. Please run it as > ./watch-highorder.pl --output /tmp/highorders.txt > while you are running tar. When kswapd is running for about 30 > seconds, interrupt it with ctrl+c twice in quick succession and > post /tmp/highorders.txt > Colin send me this information for his test case at least and I see 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5> => new_slab+0x1f5/0x290 <ffffffff81153645> => __slab_alloc+0x262/0x390 <ffffffff81155192> => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5> => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705> => mempool_alloc+0x59/0x140 <ffffffff8110ea49> => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae> => bio_alloc+0x15/0x30 <ffffffff81197805> Colin and James: Did you happen to switch from SLAB to SLUB between 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I didn't see the problem. Am restarting the tests with SLUB. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 15:44 ` Mel Gorman @ 2011-05-06 19:14 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-06 19:14 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote: > Colin and James: Did you happen to switch from SLAB to SLUB between > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I > didn't see the problem. Am restarting the tests with SLUB. Aargh ... I'm an idiot. I should have thought of SLUB immediately ... it's been causing oopses since debian switched to it. So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and the problem goes away ... at least from three untar runs on a loaded box ... of course it could manifest a few ms after I send this email ... There are material differences, as well: SLAB isn't taking my system down to very low memory on the untar ... it's keeping about 0.5Gb listed as free. SLUB took that to under 100kb, so it could just be that SLAB isn't wandering as close to the cliff edge? James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-06 19:14 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-06 19:14 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote: > Colin and James: Did you happen to switch from SLAB to SLUB between > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I > didn't see the problem. Am restarting the tests with SLUB. Aargh ... I'm an idiot. I should have thought of SLUB immediately ... it's been causing oopses since debian switched to it. So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and the problem goes away ... at least from three untar runs on a loaded box ... of course it could manifest a few ms after I send this email ... There are material differences, as well: SLAB isn't taking my system down to very low memory on the untar ... it's keeping about 0.5Gb listed as free. SLUB took that to under 100kb, so it could just be that SLAB isn't wandering as close to the cliff edge? James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 19:14 ` James Bottomley @ 2011-05-06 19:37 ` Mel Gorman -1 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 19:37 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, May 06, 2011 at 02:14:37PM -0500, James Bottomley wrote: > On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote: > > Colin and James: Did you happen to switch from SLAB to SLUB between > > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I > > didn't see the problem. Am restarting the tests with SLUB. > > Aargh ... I'm an idiot. I should have thought of SLUB immediately ... > it's been causing oopses since debian switched to it. > > So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and > the problem goes away ... at least from three untar runs on a loaded > box ... of course it could manifest a few ms after I send this email ... > > There are material differences, as well: SLAB isn't taking my system > down to very low memory on the untar ... it's keeping about 0.5Gb listed > as free. SLUB took that to under 100kb, so it could just be that SLAB > isn't wandering as close to the cliff edge? > A comparison of watch-highorder.pl with SLAB and SLUB may be enlightening as well as testing SLUB altering allocate_slab() to read alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL; i.e. try adding the __GFP_NO_KSWAPD. My own tests are still in progress but I'm still not seeing the problem. I'm installing Fedora on another test machine at the moment to see if X and other applications have to be running to pressure high-order allocations properly. -- Mel Gorman SUSE Labs ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-06 19:37 ` Mel Gorman 0 siblings, 0 replies; 138+ messages in thread From: Mel Gorman @ 2011-05-06 19:37 UTC (permalink / raw) To: James Bottomley Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, May 06, 2011 at 02:14:37PM -0500, James Bottomley wrote: > On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote: > > Colin and James: Did you happen to switch from SLAB to SLUB between > > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I > > didn't see the problem. Am restarting the tests with SLUB. > > Aargh ... I'm an idiot. I should have thought of SLUB immediately ... > it's been causing oopses since debian switched to it. > > So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and > the problem goes away ... at least from three untar runs on a loaded > box ... of course it could manifest a few ms after I send this email ... > > There are material differences, as well: SLAB isn't taking my system > down to very low memory on the untar ... it's keeping about 0.5Gb listed > as free. SLUB took that to under 100kb, so it could just be that SLAB > isn't wandering as close to the cliff edge? > A comparison of watch-highorder.pl with SLAB and SLUB may be enlightening as well as testing SLUB altering allocate_slab() to read alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL; i.e. try adding the __GFP_NO_KSWAPD. My own tests are still in progress but I'm still not seeing the problem. I'm installing Fedora on another test machine at the moment to see if X and other applications have to be running to pressure high-order allocations properly. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 15:44 ` Mel Gorman (?) @ 2011-05-10 5:37 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-05-10 5:37 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Mel Gorman, Jan Kara, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote: > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > > > 1. High-order allocations? You machine is using i915 and RPC, something > > neither of my test machine uses. i915 is potentially a source for > > high-order allocations. I'm attaching a perl script. Please run it as > > ./watch-highorder.pl --output /tmp/highorders.txt > > while you are running tar. When kswapd is running for about 30 > > seconds, interrupt it with ctrl+c twice in quick succession and > > post /tmp/highorders.txt > > > > Colin send me this information for his test case at least and I see > > 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC > => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5> > => new_slab+0x1f5/0x290 <ffffffff81153645> > => __slab_alloc+0x262/0x390 <ffffffff81155192> > => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5> > => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705> > => mempool_alloc+0x59/0x140 <ffffffff8110ea49> > => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae> > => bio_alloc+0x15/0x30 <ffffffff81197805> > > Colin and James: Did you happen to switch from SLAB to SLUB between > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I > didn't see the problem. Am restarting the tests with SLUB. So I tested with SLAB instead of SLUB and I reliably ran my copy test for 4+ hours with several hundred iterations of the test. (Apologies for taking time to respond, but I was travelling). > ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 5:37 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-05-10 5:37 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Mel Gorman, Jan Kara, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote: > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > > > 1. High-order allocations? You machine is using i915 and RPC, something > > neither of my test machine uses. i915 is potentially a source for > > high-order allocations. I'm attaching a perl script. Please run it as > > ./watch-highorder.pl --output /tmp/highorders.txt > > while you are running tar. When kswapd is running for about 30 > > seconds, interrupt it with ctrl+c twice in quick succession and > > post /tmp/highorders.txt > > > > Colin send me this information for his test case at least and I see > > 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC > => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5> > => new_slab+0x1f5/0x290 <ffffffff81153645> > => __slab_alloc+0x262/0x390 <ffffffff81155192> > => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5> > => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705> > => mempool_alloc+0x59/0x140 <ffffffff8110ea49> > => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae> > => bio_alloc+0x15/0x30 <ffffffff81197805> > > Colin and James: Did you happen to switch from SLAB to SLUB between > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I > didn't see the problem. Am restarting the tests with SLUB. So I tested with SLAB instead of SLUB and I reliably ran my copy test for 4+ hours with several hundred iterations of the test. (Apologies for taking time to respond, but I was travelling). > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-10 5:37 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-05-10 5:37 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Mel Gorman, Jan Kara, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote: > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote: > > > > 1. High-order allocations? You machine is using i915 and RPC, something > > neither of my test machine uses. i915 is potentially a source for > > high-order allocations. I'm attaching a perl script. Please run it as > > ./watch-highorder.pl --output /tmp/highorders.txt > > while you are running tar. When kswapd is running for about 30 > > seconds, interrupt it with ctrl+c twice in quick succession and > > post /tmp/highorders.txt > > > > Colin send me this information for his test case at least and I see > > 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC > => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5> > => new_slab+0x1f5/0x290 <ffffffff81153645> > => __slab_alloc+0x262/0x390 <ffffffff81155192> > => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5> > => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705> > => mempool_alloc+0x59/0x140 <ffffffff8110ea49> > => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae> > => bio_alloc+0x15/0x30 <ffffffff81197805> > > Colin and James: Did you happen to switch from SLAB to SLUB between > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I > didn't see the problem. Am restarting the tests with SLUB. So I tested with SLAB instead of SLUB and I reliably ran my copy test for 4+ hours with several hundred iterations of the test. (Apologies for taking time to respond, but I was travelling). > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-05-06 7:42 ` Mel Gorman ` (3 preceding siblings ...) (?) @ 2011-05-06 15:58 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-06 15:58 UTC (permalink / raw) To: Mel Gorman Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4 [-- Attachment #1: Type: text/plain, Size: 972 bytes --] On Fri, 2011-05-06 at 08:42 +0100, Mel Gorman wrote: [...] Will try tests and post results > Out of curiousity, what's the exact machine you are testing on > because I want to see what sort of hardware combination triggers > this problem? Is the tar local or is it copied over NFS? What is the > configuration of the disk or partition you are copying to? It's a Lenovo T420s with 2GB of memory. It's core i5, so it has two cores each with two threads, so it emulates a 4 cpu system. It's running Fedora 15 Beta, so the 2.6.38.4 kernel (although I've tried the default 2.6.38.5 as well, since the fedora i915 updates are a bit unstable). I'm also booting 2.6.39-rc on it (with and without PREEMPT). The standard Fedora config is SLUB. Because it's FC15, it has all of the cgroups stuff enabled for systemd. Just in case, I've attached the FC15 config ... I modify it slightly for 2.6.39-rc but since the bug shows up in standard FC15, that should be enough. James [-- Attachment #2: config-2.6.38.4-20.fc15.x86_64 --] [-- Type: text/x-mpsub, Size: 114129 bytes --] # # Automatically generated make config: don't edit # Linux/x86_64 2.6.38.4-20.fc15.x86_64 Kernel Configuration # Thu Apr 28 23:31:51 2011 # CONFIG_64BIT=y # CONFIG_X86_32 is not set CONFIG_X86_64=y CONFIG_X86=y CONFIG_INSTRUCTION_DECODER=y CONFIG_OUTPUT_FORMAT="elf64-x86-64" CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig" CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_CLOCKSOURCE_WATCHDOG=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_MMU=y CONFIG_ZONE_DMA=y CONFIG_NEED_DMA_MAP_STATE=y CONFIG_NEED_SG_DMA_LENGTH=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_IOMAP=y CONFIG_GENERIC_BUG=y CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_GPIO=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y # CONFIG_RWSEM_GENERIC_SPINLOCK is not set CONFIG_RWSEM_XCHGADD_ALGORITHM=y CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_ARCH_HAS_CPU_RELAX=y CONFIG_ARCH_HAS_DEFAULT_IDLE=y CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y CONFIG_HAVE_SETUP_PER_CPU_AREA=y CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y CONFIG_ARCH_SUSPEND_POSSIBLE=y CONFIG_ZONE_DMA32=y CONFIG_ARCH_POPULATES_NODE_MAP=y CONFIG_AUDIT_ARCH=y CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_HAVE_INTEL_TXT=y CONFIG_X86_64_SMP=y CONFIG_X86_HT=y CONFIG_X86_TRAMPOLINE=y CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" # CONFIG_KTIME_SCALAR is not set CONFIG_ARCH_CPU_PROBE_RELEASE=y CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" CONFIG_CONSTRUCTORS=y CONFIG_HAVE_IRQ_WORK=y CONFIG_IRQ_WORK=y # # General setup # CONFIG_EXPERIMENTAL=y CONFIG_LOCK_KERNEL=y CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_CROSS_COMPILE="" CONFIG_LOCALVERSION="" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_HAVE_KERNEL_GZIP=y CONFIG_HAVE_KERNEL_BZIP2=y CONFIG_HAVE_KERNEL_LZMA=y CONFIG_HAVE_KERNEL_XZ=y CONFIG_HAVE_KERNEL_LZO=y CONFIG_KERNEL_GZIP=y # CONFIG_KERNEL_BZIP2 is not set # CONFIG_KERNEL_LZMA is not set # CONFIG_KERNEL_XZ is not set # CONFIG_KERNEL_LZO is not set CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y CONFIG_POSIX_MQUEUE=y CONFIG_POSIX_MQUEUE_SYSCTL=y CONFIG_BSD_PROCESS_ACCT=y # CONFIG_BSD_PROCESS_ACCT_V3 is not set CONFIG_TASKSTATS=y CONFIG_TASK_DELAY_ACCT=y CONFIG_TASK_XACCT=y CONFIG_TASK_IO_ACCOUNTING=y CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y CONFIG_AUDIT_WATCH=y CONFIG_AUDIT_TREE=y CONFIG_UTRACE=y CONFIG_HAVE_GENERIC_HARDIRQS=y # # IRQ subsystem # CONFIG_GENERIC_HARDIRQS=y # CONFIG_GENERIC_HARDIRQS_NO_DEPRECATED is not set CONFIG_HAVE_SPARSE_IRQ=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_GENERIC_PENDING_IRQ=y # CONFIG_AUTO_IRQ_AFFINITY is not set # CONFIG_IRQ_PER_CPU is not set # CONFIG_HARDIRQS_SW_RESEND is not set CONFIG_SPARSE_IRQ=y # # RCU Subsystem # CONFIG_TREE_RCU=y # CONFIG_PREEMPT_RCU is not set # CONFIG_RCU_TRACE is not set CONFIG_RCU_FANOUT=64 # CONFIG_RCU_FANOUT_EXACT is not set CONFIG_RCU_FAST_NO_HZ=y # CONFIG_TREE_RCU_TRACE is not set # CONFIG_IKCONFIG is not set CONFIG_LOG_BUF_SHIFT=17 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_CGROUPS=y # CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_NS=y CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_DEVICE=y CONFIG_CPUSETS=y CONFIG_PROC_PID_CPUSET=y CONFIG_CGROUP_CPUACCT=y CONFIG_RESOURCE_COUNTERS=y CONFIG_CGROUP_MEM_RES_CTLR=y CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y # CONFIG_CGROUP_MEM_RES_CTLR_SWAP_ENABLED is not set CONFIG_CGROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y CONFIG_RT_GROUP_SCHED=y CONFIG_BLK_CGROUP=y # CONFIG_DEBUG_BLK_CGROUP is not set CONFIG_NAMESPACES=y CONFIG_UTS_NS=y CONFIG_IPC_NS=y CONFIG_USER_NS=y CONFIG_PID_NS=y CONFIG_NET_NS=y CONFIG_SCHED_AUTOGROUP=y CONFIG_MM_OWNER=y # CONFIG_SYSFS_DEPRECATED is not set CONFIG_RELAY=y CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE="" CONFIG_RD_GZIP=y CONFIG_RD_BZIP2=y CONFIG_RD_LZMA=y CONFIG_RD_XZ=y CONFIG_RD_LZO=y CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y CONFIG_ANON_INODES=y # CONFIG_EXPERT is not set # CONFIG_EMBEDDED is not set CONFIG_UID16=y CONFIG_SYSCTL_SYSCALL=y CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y CONFIG_KALLSYMS_EXTRA_PASS=y CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_PCSPKR_PLATFORM=y CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_AIO=y CONFIG_HAVE_PERF_EVENTS=y # # Kernel Performance Events And Counters # CONFIG_PERF_EVENTS=y CONFIG_PERF_COUNTERS=y # CONFIG_DEBUG_PERF_USE_VMALLOC is not set CONFIG_VM_EVENT_COUNTERS=y CONFIG_PCI_QUIRKS=y CONFIG_SLUB_DEBUG=y # CONFIG_COMPAT_BRK is not set # CONFIG_SLAB is not set CONFIG_SLUB=y CONFIG_PROFILING=y CONFIG_TRACEPOINTS=y CONFIG_OPROFILE=m CONFIG_OPROFILE_EVENT_MULTIPLEX=y CONFIG_HAVE_OPROFILE=y CONFIG_KPROBES=y CONFIG_JUMP_LABEL=y CONFIG_OPTPROBES=y CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y CONFIG_KRETPROBES=y CONFIG_USER_RETURN_NOTIFIER=y CONFIG_HAVE_IOREMAP_PROT=y CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y CONFIG_HAVE_OPTPROBES=y CONFIG_HAVE_ARCH_TRACEHOOK=y CONFIG_HAVE_DMA_ATTRS=y CONFIG_USE_GENERIC_SMP_HELPERS=y CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y CONFIG_HAVE_DMA_API_DEBUG=y CONFIG_HAVE_HW_BREAKPOINT=y CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y CONFIG_HAVE_USER_RETURN_NOTIFIER=y CONFIG_HAVE_PERF_EVENTS_NMI=y CONFIG_HAVE_ARCH_JUMP_LABEL=y # # GCOV-based kernel profiling # # CONFIG_GCOV_KERNEL is not set # CONFIG_HAVE_GENERIC_DMA_COHERENT is not set CONFIG_SLABINFO=y CONFIG_RT_MUTEXES=y CONFIG_BASE_SMALL=0 CONFIG_MODULES=y # CONFIG_MODULE_FORCE_LOAD is not set CONFIG_MODULE_UNLOAD=y # CONFIG_MODULE_FORCE_UNLOAD is not set # CONFIG_MODVERSIONS is not set CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_STOP_MACHINE=y CONFIG_BLOCK=y CONFIG_BLK_DEV_BSG=y CONFIG_BLK_DEV_INTEGRITY=y CONFIG_BLK_DEV_THROTTLING=y CONFIG_BLOCK_COMPAT=y # # IO Schedulers # CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y CONFIG_CFQ_GROUP_IOSCHED=y # CONFIG_DEFAULT_DEADLINE is not set CONFIG_DEFAULT_CFQ=y # CONFIG_DEFAULT_NOOP is not set CONFIG_DEFAULT_IOSCHED="cfq" CONFIG_PREEMPT_NOTIFIERS=y CONFIG_PADATA=y # CONFIG_INLINE_SPIN_TRYLOCK is not set # CONFIG_INLINE_SPIN_TRYLOCK_BH is not set # CONFIG_INLINE_SPIN_LOCK is not set # CONFIG_INLINE_SPIN_LOCK_BH is not set # CONFIG_INLINE_SPIN_LOCK_IRQ is not set # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set CONFIG_INLINE_SPIN_UNLOCK=y # CONFIG_INLINE_SPIN_UNLOCK_BH is not set CONFIG_INLINE_SPIN_UNLOCK_IRQ=y # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_READ_TRYLOCK is not set # CONFIG_INLINE_READ_LOCK is not set # CONFIG_INLINE_READ_LOCK_BH is not set # CONFIG_INLINE_READ_LOCK_IRQ is not set # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set CONFIG_INLINE_READ_UNLOCK=y # CONFIG_INLINE_READ_UNLOCK_BH is not set CONFIG_INLINE_READ_UNLOCK_IRQ=y # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_WRITE_TRYLOCK is not set # CONFIG_INLINE_WRITE_LOCK is not set # CONFIG_INLINE_WRITE_LOCK_BH is not set # CONFIG_INLINE_WRITE_LOCK_IRQ is not set # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set CONFIG_INLINE_WRITE_UNLOCK=y # CONFIG_INLINE_WRITE_UNLOCK_BH is not set CONFIG_INLINE_WRITE_UNLOCK_IRQ=y # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set CONFIG_MUTEX_SPIN_ON_OWNER=y CONFIG_FREEZER=y # # Processor type and features # CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_SMP=y CONFIG_X86_X2APIC=y CONFIG_X86_MPPARSE=y CONFIG_X86_EXTENDED_PLATFORM=y # CONFIG_X86_VSMP is not set # CONFIG_X86_UV is not set CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_PARAVIRT_GUEST=y CONFIG_XEN=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PVHVM=y CONFIG_XEN_MAX_DOMAIN_MEMORY=128 CONFIG_XEN_SAVE_RESTORE=y CONFIG_XEN_DEBUG_FS=y CONFIG_KVM_CLOCK=y CONFIG_KVM_GUEST=y CONFIG_PARAVIRT=y # CONFIG_PARAVIRT_SPINLOCKS is not set CONFIG_PARAVIRT_CLOCK=y # CONFIG_PARAVIRT_DEBUG is not set CONFIG_NO_BOOTMEM=y # CONFIG_MEMTEST is not set # CONFIG_MK8 is not set # CONFIG_MPSC is not set # CONFIG_MCORE2 is not set # CONFIG_MATOM is not set CONFIG_GENERIC_CPU=y CONFIG_X86_CPU=y CONFIG_X86_INTERNODE_CACHE_SHIFT=7 CONFIG_X86_CMPXCHG=y CONFIG_CMPXCHG_LOCAL=y CONFIG_X86_L1_CACHE_SHIFT=6 CONFIG_X86_XADD=y CONFIG_X86_WP_WORKS_OK=y CONFIG_X86_TSC=y CONFIG_X86_CMPXCHG64=y CONFIG_X86_CMOV=y CONFIG_X86_MINIMUM_CPU_FAMILY=64 CONFIG_X86_DEBUGCTLMSR=y CONFIG_CPU_SUP_INTEL=y CONFIG_CPU_SUP_AMD=y CONFIG_CPU_SUP_CENTAUR=y CONFIG_HPET_TIMER=y CONFIG_HPET_EMULATE_RTC=y CONFIG_DMI=y CONFIG_GART_IOMMU=y CONFIG_CALGARY_IOMMU=y CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_STATS=y CONFIG_SWIOTLB=y CONFIG_IOMMU_HELPER=y CONFIG_IOMMU_API=y # CONFIG_MAXSMP is not set CONFIG_NR_CPUS=256 CONFIG_SCHED_SMT=y CONFIG_SCHED_MC=y CONFIG_IRQ_TIME_ACCOUNTING=y # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y # CONFIG_PREEMPT is not set CONFIG_X86_LOCAL_APIC=y CONFIG_X86_IO_APIC=y CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y CONFIG_X86_MCE=y CONFIG_X86_MCE_INTEL=y CONFIG_X86_MCE_AMD=y CONFIG_X86_MCE_THRESHOLD=y # CONFIG_X86_MCE_INJECT is not set CONFIG_X86_THERMAL_VECTOR=y CONFIG_I8K=m CONFIG_MICROCODE=m CONFIG_MICROCODE_INTEL=y CONFIG_MICROCODE_AMD=y CONFIG_MICROCODE_OLD_INTERFACE=y CONFIG_X86_MSR=y CONFIG_X86_CPUID=y CONFIG_ARCH_PHYS_ADDR_T_64BIT=y CONFIG_ARCH_DMA_ADDR_T_64BIT=y CONFIG_DIRECT_GBPAGES=y CONFIG_NUMA=y CONFIG_AMD_NUMA=y CONFIG_X86_64_ACPI_NUMA=y CONFIG_NODES_SPAN_OTHER_NODES=y # CONFIG_NUMA_EMU is not set CONFIG_NODES_SHIFT=9 CONFIG_ARCH_PROC_KCORE_TEXT=y CONFIG_ARCH_SPARSEMEM_DEFAULT=y CONFIG_ARCH_SPARSEMEM_ENABLE=y CONFIG_ARCH_SELECT_MEMORY_MODEL=y CONFIG_ARCH_MEMORY_PROBE=y CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000 CONFIG_SELECT_MEMORY_MODEL=y CONFIG_SPARSEMEM_MANUAL=y CONFIG_SPARSEMEM=y CONFIG_NEED_MULTIPLE_NODES=y CONFIG_HAVE_MEMORY_PRESENT=y CONFIG_SPARSEMEM_EXTREME=y CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y CONFIG_SPARSEMEM_VMEMMAP=y CONFIG_HAVE_MEMBLOCK=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTPLUG_SPARSE=y CONFIG_MEMORY_HOTREMOVE=y CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=4 CONFIG_COMPACTION=y CONFIG_MIGRATION=y CONFIG_PHYS_ADDR_T_64BIT=y CONFIG_ZONE_DMA_FLAG=1 CONFIG_BOUNCE=y CONFIG_VIRT_TO_BUS=y CONFIG_MMU_NOTIFIER=y CONFIG_KSM=y CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y CONFIG_MEMORY_FAILURE=y CONFIG_HWPOISON_INJECT=m CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y # CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set # CONFIG_X86_CHECK_BIOS_CORRUPTION is not set CONFIG_X86_RESERVE_LOW=64 CONFIG_MTRR=y CONFIG_MTRR_SANITIZER=y CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1 CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1 CONFIG_X86_PAT=y CONFIG_ARCH_USES_PG_UNCACHED=y CONFIG_EFI=y CONFIG_SECCOMP=y CONFIG_CC_STACKPROTECTOR=y # CONFIG_HZ_100 is not set # CONFIG_HZ_250 is not set # CONFIG_HZ_300 is not set CONFIG_HZ_1000=y CONFIG_HZ=1000 CONFIG_SCHED_HRTICK=y CONFIG_KEXEC=y CONFIG_CRASH_DUMP=y CONFIG_KEXEC_JUMP=y CONFIG_PHYSICAL_START=0x1000000 CONFIG_RELOCATABLE=y CONFIG_PHYSICAL_ALIGN=0x1000000 CONFIG_HOTPLUG_CPU=y # CONFIG_COMPAT_VDSO is not set # CONFIG_CMDLINE_BOOL is not set CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y CONFIG_USE_PERCPU_NUMA_NODE_ID=y # # Power management and ACPI options # CONFIG_ARCH_HIBERNATION_HEADER=y CONFIG_PM=y CONFIG_PM_DEBUG=y CONFIG_PM_ADVANCED_DEBUG=y # CONFIG_PM_VERBOSE is not set CONFIG_CAN_PM_TRACE=y CONFIG_PM_TRACE=y CONFIG_PM_TRACE_RTC=y CONFIG_PM_SLEEP_SMP=y CONFIG_PM_SLEEP=y # CONFIG_PM_SLEEP_ADVANCED_DEBUG is not set CONFIG_SUSPEND=y # CONFIG_PM_TEST_SUSPEND is not set CONFIG_SUSPEND_FREEZER=y CONFIG_HIBERNATION=y CONFIG_PM_STD_PARTITION="" CONFIG_PM_RUNTIME=y CONFIG_PM_OPS=y CONFIG_ACPI=y CONFIG_ACPI_SLEEP=y CONFIG_ACPI_PROCFS=y CONFIG_ACPI_PROCFS_POWER=y CONFIG_ACPI_POWER_METER=m CONFIG_ACPI_EC_DEBUGFS=m # CONFIG_ACPI_PROC_EVENT is not set CONFIG_ACPI_AC=y CONFIG_ACPI_BATTERY=y CONFIG_ACPI_BUTTON=y CONFIG_ACPI_VIDEO=m CONFIG_ACPI_FAN=y CONFIG_ACPI_DOCK=y CONFIG_ACPI_PROCESSOR=y CONFIG_ACPI_IPMI=m CONFIG_ACPI_HOTPLUG_CPU=y CONFIG_ACPI_PROCESSOR_AGGREGATOR=m CONFIG_ACPI_THERMAL=y CONFIG_ACPI_NUMA=y # CONFIG_ACPI_CUSTOM_DSDT is not set CONFIG_ACPI_BLACKLIST_YEAR=0 # CONFIG_ACPI_DEBUG is not set CONFIG_ACPI_PCI_SLOT=y CONFIG_X86_PM_TIMER=y CONFIG_ACPI_CONTAINER=y CONFIG_ACPI_HOTPLUG_MEMORY=m CONFIG_ACPI_SBS=m CONFIG_ACPI_HED=m CONFIG_ACPI_APEI=y CONFIG_ACPI_APEI_GHES=m # CONFIG_ACPI_APEI_EINJ is not set # CONFIG_ACPI_APEI_ERST_DEBUG is not set CONFIG_SFI=y # # CPU Frequency scaling # CONFIG_CPU_FREQ=y CONFIG_CPU_FREQ_TABLE=m CONFIG_CPU_FREQ_DEBUG=y CONFIG_CPU_FREQ_STAT=m CONFIG_CPU_FREQ_STAT_DETAILS=y # CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set CONFIG_CPU_FREQ_GOV_PERFORMANCE=y CONFIG_CPU_FREQ_GOV_POWERSAVE=m CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=m CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m # # CPUFreq processor drivers # CONFIG_X86_PCC_CPUFREQ=m CONFIG_X86_ACPI_CPUFREQ=m CONFIG_X86_POWERNOW_K8=m # CONFIG_X86_SPEEDSTEP_CENTRINO is not set CONFIG_X86_P4_CLOCKMOD=m # # shared options # CONFIG_X86_SPEEDSTEP_LIB=m CONFIG_CPU_IDLE=y CONFIG_CPU_IDLE_GOV_LADDER=y CONFIG_CPU_IDLE_GOV_MENU=y CONFIG_INTEL_IDLE=y # # Memory power savings # CONFIG_I7300_IDLE_IOAT_CHANNEL=y CONFIG_I7300_IDLE=m # # Bus options (PCI etc.) # CONFIG_PCI=y CONFIG_PCI_DIRECT=y CONFIG_PCI_MMCONFIG=y CONFIG_PCI_XEN=y CONFIG_PCI_DOMAINS=y # CONFIG_PCI_CNB20LE_QUIRK is not set CONFIG_DMAR=y CONFIG_DMAR_DEFAULT_ON=y CONFIG_DMAR_FLOPPY_WA=y CONFIG_INTR_REMAP=y CONFIG_PCIEPORTBUS=y CONFIG_HOTPLUG_PCI_PCIE=y CONFIG_PCIEAER=y CONFIG_PCIE_ECRC=y CONFIG_PCIEAER_INJECT=m CONFIG_PCIEASPM=y # CONFIG_PCIEASPM_DEBUG is not set CONFIG_PCIE_PME=y CONFIG_ARCH_SUPPORTS_MSI=y CONFIG_PCI_MSI=y CONFIG_PCI_MSI_DEFAULT_ON=y # CONFIG_PCI_DEBUG is not set CONFIG_PCI_STUB=y CONFIG_XEN_PCIDEV_FRONTEND=m CONFIG_HT_IRQ=y CONFIG_PCI_IOV=y CONFIG_PCI_IOAPIC=y CONFIG_ISA_DMA_API=y CONFIG_AMD_NB=y CONFIG_PCCARD=y CONFIG_PCMCIA=y CONFIG_PCMCIA_LOAD_CIS=y CONFIG_CARDBUS=y # # PC-card bridges # CONFIG_YENTA=m CONFIG_YENTA_O2=y CONFIG_YENTA_RICOH=y CONFIG_YENTA_TI=y CONFIG_YENTA_ENE_TUNE=y CONFIG_YENTA_TOSHIBA=y CONFIG_PD6729=m CONFIG_I82092=m CONFIG_PCCARD_NONSTATIC=y CONFIG_HOTPLUG_PCI=y CONFIG_HOTPLUG_PCI_FAKE=m CONFIG_HOTPLUG_PCI_ACPI=y CONFIG_HOTPLUG_PCI_ACPI_IBM=m # CONFIG_HOTPLUG_PCI_CPCI is not set CONFIG_HOTPLUG_PCI_SHPC=m # # Executable file formats / Emulations # CONFIG_BINFMT_ELF=y CONFIG_COMPAT_BINFMT_ELF=y CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y # CONFIG_HAVE_AOUT is not set CONFIG_BINFMT_MISC=y CONFIG_IA32_EMULATION=y # CONFIG_IA32_AOUT is not set CONFIG_COMPAT=y CONFIG_COMPAT_FOR_U64_ALIGNMENT=y CONFIG_SYSVIPC_COMPAT=y CONFIG_HAVE_TEXT_POKE_SMP=y CONFIG_NET=y CONFIG_COMPAT_NETLINK_MESSAGES=y # # Networking options # CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_XFRM=y CONFIG_XFRM_USER=y CONFIG_XFRM_SUB_POLICY=y CONFIG_XFRM_MIGRATE=y CONFIG_XFRM_STATISTICS=y CONFIG_XFRM_IPCOMP=m CONFIG_NET_KEY=m CONFIG_NET_KEY_MIGRATE=y CONFIG_INET=y CONFIG_IP_MULTICAST=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_ASK_IP_FIB_HASH=y # CONFIG_IP_FIB_TRIE is not set CONFIG_IP_FIB_HASH=y CONFIG_IP_MULTIPLE_TABLES=y CONFIG_IP_ROUTE_MULTIPATH=y CONFIG_IP_ROUTE_VERBOSE=y # CONFIG_IP_PNP is not set CONFIG_NET_IPIP=m CONFIG_NET_IPGRE_DEMUX=m CONFIG_NET_IPGRE=m CONFIG_NET_IPGRE_BROADCAST=y CONFIG_IP_MROUTE=y CONFIG_IP_MROUTE_MULTIPLE_TABLES=y CONFIG_IP_PIMSM_V1=y CONFIG_IP_PIMSM_V2=y CONFIG_ARPD=y CONFIG_SYN_COOKIES=y CONFIG_INET_AH=m CONFIG_INET_ESP=m CONFIG_INET_IPCOMP=m CONFIG_INET_XFRM_TUNNEL=m CONFIG_INET_TUNNEL=m CONFIG_INET_XFRM_MODE_TRANSPORT=m CONFIG_INET_XFRM_MODE_TUNNEL=m CONFIG_INET_XFRM_MODE_BEET=m CONFIG_INET_LRO=y CONFIG_INET_DIAG=m CONFIG_INET_TCP_DIAG=m CONFIG_TCP_CONG_ADVANCED=y CONFIG_TCP_CONG_BIC=m CONFIG_TCP_CONG_CUBIC=y CONFIG_TCP_CONG_WESTWOOD=m CONFIG_TCP_CONG_HTCP=m CONFIG_TCP_CONG_HSTCP=m CONFIG_TCP_CONG_HYBLA=m CONFIG_TCP_CONG_VEGAS=m CONFIG_TCP_CONG_SCALABLE=m CONFIG_TCP_CONG_LP=m CONFIG_TCP_CONG_VENO=m CONFIG_TCP_CONG_YEAH=m CONFIG_TCP_CONG_ILLINOIS=m CONFIG_DEFAULT_CUBIC=y # CONFIG_DEFAULT_RENO is not set CONFIG_DEFAULT_TCP_CONG="cubic" CONFIG_TCP_MD5SIG=y CONFIG_IPV6=m CONFIG_IPV6_PRIVACY=y CONFIG_IPV6_ROUTER_PREF=y CONFIG_IPV6_ROUTE_INFO=y CONFIG_IPV6_OPTIMISTIC_DAD=y CONFIG_INET6_AH=m CONFIG_INET6_ESP=m CONFIG_INET6_IPCOMP=m CONFIG_IPV6_MIP6=m CONFIG_INET6_XFRM_TUNNEL=m CONFIG_INET6_TUNNEL=m CONFIG_INET6_XFRM_MODE_TRANSPORT=m CONFIG_INET6_XFRM_MODE_TUNNEL=m CONFIG_INET6_XFRM_MODE_BEET=m CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m CONFIG_IPV6_SIT=m CONFIG_IPV6_SIT_6RD=y CONFIG_IPV6_NDISC_NODETYPE=y CONFIG_IPV6_TUNNEL=m CONFIG_IPV6_MULTIPLE_TABLES=y CONFIG_IPV6_SUBTREES=y CONFIG_IPV6_MROUTE=y CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y CONFIG_IPV6_PIMSM_V2=y CONFIG_NETLABEL=y CONFIG_NETWORK_SECMARK=y # CONFIG_NETWORK_PHY_TIMESTAMPING is not set CONFIG_NETFILTER=y # CONFIG_NETFILTER_DEBUG is not set CONFIG_NETFILTER_ADVANCED=y CONFIG_BRIDGE_NETFILTER=y # # Core Netfilter Configuration # CONFIG_NETFILTER_NETLINK=m CONFIG_NETFILTER_NETLINK_QUEUE=m CONFIG_NETFILTER_NETLINK_LOG=m CONFIG_NF_CONNTRACK=y CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_CONNTRACK_SECMARK=y CONFIG_NF_CONNTRACK_ZONES=y CONFIG_NF_CONNTRACK_EVENTS=y CONFIG_NF_CT_PROTO_DCCP=m CONFIG_NF_CT_PROTO_GRE=m CONFIG_NF_CT_PROTO_SCTP=m CONFIG_NF_CT_PROTO_UDPLITE=m CONFIG_NF_CONNTRACK_AMANDA=m CONFIG_NF_CONNTRACK_FTP=m CONFIG_NF_CONNTRACK_H323=m CONFIG_NF_CONNTRACK_IRC=m CONFIG_NF_CONNTRACK_NETBIOS_NS=m CONFIG_NF_CONNTRACK_PPTP=m CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_CT_NETLINK=m CONFIG_NETFILTER_TPROXY=m CONFIG_NETFILTER_XTABLES=y # # Xtables combined modules # CONFIG_NETFILTER_XT_MARK=m CONFIG_NETFILTER_XT_CONNMARK=m # # Xtables targets # CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m CONFIG_NETFILTER_XT_TARGET_CONNMARK=m CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m CONFIG_NETFILTER_XT_TARGET_CT=m CONFIG_NETFILTER_XT_TARGET_DSCP=m CONFIG_NETFILTER_XT_TARGET_HL=m CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m CONFIG_NETFILTER_XT_TARGET_LED=m CONFIG_NETFILTER_XT_TARGET_MARK=m CONFIG_NETFILTER_XT_TARGET_NFLOG=m CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m CONFIG_NETFILTER_XT_TARGET_NOTRACK=m CONFIG_NETFILTER_XT_TARGET_RATEEST=m CONFIG_NETFILTER_XT_TARGET_TEE=m CONFIG_NETFILTER_XT_TARGET_TPROXY=m CONFIG_NETFILTER_XT_TARGET_TRACE=m CONFIG_NETFILTER_XT_TARGET_SECMARK=m CONFIG_NETFILTER_XT_TARGET_TCPMSS=m CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m # # Xtables matches # CONFIG_NETFILTER_XT_MATCH_CLUSTER=m CONFIG_NETFILTER_XT_MATCH_COMMENT=m CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m CONFIG_NETFILTER_XT_MATCH_CONNMARK=m CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y CONFIG_NETFILTER_XT_MATCH_CPU=m CONFIG_NETFILTER_XT_MATCH_DCCP=m CONFIG_NETFILTER_XT_MATCH_DSCP=m CONFIG_NETFILTER_XT_MATCH_ESP=m CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m CONFIG_NETFILTER_XT_MATCH_HELPER=m CONFIG_NETFILTER_XT_MATCH_HL=m CONFIG_NETFILTER_XT_MATCH_IPRANGE=m CONFIG_NETFILTER_XT_MATCH_IPVS=m CONFIG_NETFILTER_XT_MATCH_LENGTH=m CONFIG_NETFILTER_XT_MATCH_LIMIT=m CONFIG_NETFILTER_XT_MATCH_MAC=m CONFIG_NETFILTER_XT_MATCH_MARK=m CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m CONFIG_NETFILTER_XT_MATCH_OSF=m CONFIG_NETFILTER_XT_MATCH_OWNER=m CONFIG_NETFILTER_XT_MATCH_POLICY=m CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m CONFIG_NETFILTER_XT_MATCH_QUOTA=m CONFIG_NETFILTER_XT_MATCH_RATEEST=m CONFIG_NETFILTER_XT_MATCH_REALM=m CONFIG_NETFILTER_XT_MATCH_RECENT=m CONFIG_NETFILTER_XT_MATCH_SCTP=m CONFIG_NETFILTER_XT_MATCH_SOCKET=m CONFIG_NETFILTER_XT_MATCH_STATE=y CONFIG_NETFILTER_XT_MATCH_STATISTIC=m CONFIG_NETFILTER_XT_MATCH_STRING=m CONFIG_NETFILTER_XT_MATCH_TCPMSS=m CONFIG_NETFILTER_XT_MATCH_TIME=m CONFIG_NETFILTER_XT_MATCH_U32=m CONFIG_IP_VS=m # CONFIG_IP_VS_IPV6 is not set # CONFIG_IP_VS_DEBUG is not set CONFIG_IP_VS_TAB_BITS=12 # # IPVS transport protocol load balancing support # CONFIG_IP_VS_PROTO_TCP=y CONFIG_IP_VS_PROTO_UDP=y CONFIG_IP_VS_PROTO_AH_ESP=y CONFIG_IP_VS_PROTO_ESP=y CONFIG_IP_VS_PROTO_AH=y CONFIG_IP_VS_PROTO_SCTP=y # # IPVS scheduler # CONFIG_IP_VS_RR=m CONFIG_IP_VS_WRR=m CONFIG_IP_VS_LC=m CONFIG_IP_VS_WLC=m CONFIG_IP_VS_LBLC=m CONFIG_IP_VS_LBLCR=m CONFIG_IP_VS_DH=m CONFIG_IP_VS_SH=m CONFIG_IP_VS_SED=m CONFIG_IP_VS_NQ=m # # IPVS application helper # CONFIG_IP_VS_FTP=m CONFIG_IP_VS_NFCT=y CONFIG_IP_VS_PE_SIP=m # # IP: Netfilter Configuration # CONFIG_NF_DEFRAG_IPV4=y CONFIG_NF_CONNTRACK_IPV4=y # CONFIG_NF_CONNTRACK_PROC_COMPAT is not set CONFIG_IP_NF_QUEUE=m CONFIG_IP_NF_IPTABLES=y CONFIG_IP_NF_MATCH_ADDRTYPE=m CONFIG_IP_NF_MATCH_AH=m CONFIG_IP_NF_MATCH_ECN=m CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=y CONFIG_IP_NF_TARGET_REJECT=y CONFIG_IP_NF_TARGET_LOG=m CONFIG_IP_NF_TARGET_ULOG=m CONFIG_NF_NAT=m CONFIG_NF_NAT_NEEDED=y CONFIG_IP_NF_TARGET_MASQUERADE=m CONFIG_IP_NF_TARGET_NETMAP=m CONFIG_IP_NF_TARGET_REDIRECT=m CONFIG_NF_NAT_SNMP_BASIC=m CONFIG_NF_NAT_PROTO_DCCP=m CONFIG_NF_NAT_PROTO_GRE=m CONFIG_NF_NAT_PROTO_UDPLITE=m CONFIG_NF_NAT_PROTO_SCTP=m CONFIG_NF_NAT_FTP=m CONFIG_NF_NAT_IRC=m CONFIG_NF_NAT_TFTP=m CONFIG_NF_NAT_AMANDA=m CONFIG_NF_NAT_PPTP=m CONFIG_NF_NAT_H323=m CONFIG_NF_NAT_SIP=m CONFIG_IP_NF_MANGLE=m CONFIG_IP_NF_TARGET_CLUSTERIP=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m CONFIG_IP_NF_SECURITY=m CONFIG_IP_NF_ARPTABLES=m CONFIG_IP_NF_ARPFILTER=m CONFIG_IP_NF_ARP_MANGLE=m # # IPv6: Netfilter Configuration # CONFIG_NF_DEFRAG_IPV6=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_IP6_NF_QUEUE=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m CONFIG_IP6_NF_MATCH_FRAG=m CONFIG_IP6_NF_MATCH_OPTS=m CONFIG_IP6_NF_MATCH_HL=m CONFIG_IP6_NF_MATCH_IPV6HEADER=m CONFIG_IP6_NF_MATCH_MH=m CONFIG_IP6_NF_MATCH_RT=m CONFIG_IP6_NF_TARGET_HL=m CONFIG_IP6_NF_TARGET_LOG=m CONFIG_IP6_NF_FILTER=m CONFIG_IP6_NF_TARGET_REJECT=m CONFIG_IP6_NF_MANGLE=m CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_SECURITY=m # # DECnet: Netfilter Configuration # # CONFIG_DECNET_NF_GRABULATOR is not set CONFIG_BRIDGE_NF_EBTABLES=m CONFIG_BRIDGE_EBT_BROUTE=m CONFIG_BRIDGE_EBT_T_FILTER=m CONFIG_BRIDGE_EBT_T_NAT=m CONFIG_BRIDGE_EBT_802_3=m CONFIG_BRIDGE_EBT_AMONG=m CONFIG_BRIDGE_EBT_ARP=m CONFIG_BRIDGE_EBT_IP=m CONFIG_BRIDGE_EBT_IP6=m CONFIG_BRIDGE_EBT_LIMIT=m CONFIG_BRIDGE_EBT_MARK=m CONFIG_BRIDGE_EBT_PKTTYPE=m CONFIG_BRIDGE_EBT_STP=m CONFIG_BRIDGE_EBT_VLAN=m CONFIG_BRIDGE_EBT_ARPREPLY=m CONFIG_BRIDGE_EBT_DNAT=m CONFIG_BRIDGE_EBT_MARK_T=m CONFIG_BRIDGE_EBT_REDIRECT=m CONFIG_BRIDGE_EBT_SNAT=m CONFIG_BRIDGE_EBT_LOG=m CONFIG_BRIDGE_EBT_ULOG=m CONFIG_BRIDGE_EBT_NFLOG=m CONFIG_IP_DCCP=m CONFIG_INET_DCCP_DIAG=m # # DCCP CCIDs Configuration (EXPERIMENTAL) # # CONFIG_IP_DCCP_CCID2_DEBUG is not set CONFIG_IP_DCCP_CCID3=y # CONFIG_IP_DCCP_CCID3_DEBUG is not set CONFIG_IP_DCCP_TFRC_LIB=y # # DCCP Kernel Hacking # # CONFIG_IP_DCCP_DEBUG is not set CONFIG_NET_DCCPPROBE=m CONFIG_IP_SCTP=m CONFIG_NET_SCTPPROBE=m # CONFIG_SCTP_DBG_MSG is not set # CONFIG_SCTP_DBG_OBJCNT is not set # CONFIG_SCTP_HMAC_NONE is not set CONFIG_SCTP_HMAC_SHA1=y # CONFIG_SCTP_HMAC_MD5 is not set CONFIG_RDS=m CONFIG_RDS_RDMA=m CONFIG_RDS_TCP=m # CONFIG_RDS_DEBUG is not set # CONFIG_TIPC is not set CONFIG_ATM=m CONFIG_ATM_CLIP=m # CONFIG_ATM_CLIP_NO_ICMP is not set CONFIG_ATM_LANE=m # CONFIG_ATM_MPOA is not set CONFIG_ATM_BR2684=m # CONFIG_ATM_BR2684_IPFILTER is not set CONFIG_L2TP=m CONFIG_L2TP_DEBUGFS=m CONFIG_L2TP_V3=y CONFIG_L2TP_IP=m CONFIG_L2TP_ETH=m CONFIG_STP=m CONFIG_GARP=m CONFIG_BRIDGE=m CONFIG_BRIDGE_IGMP_SNOOPING=y CONFIG_NET_DSA=y CONFIG_NET_DSA_TAG_DSA=y CONFIG_NET_DSA_TAG_EDSA=y CONFIG_NET_DSA_TAG_TRAILER=y CONFIG_NET_DSA_MV88E6XXX=y CONFIG_NET_DSA_MV88E6060=y CONFIG_NET_DSA_MV88E6XXX_NEED_PPU=y CONFIG_NET_DSA_MV88E6131=y CONFIG_NET_DSA_MV88E6123_61_65=y CONFIG_VLAN_8021Q=m CONFIG_VLAN_8021Q_GVRP=y CONFIG_DECNET=m CONFIG_DECNET_ROUTER=y CONFIG_LLC=m # CONFIG_LLC2 is not set CONFIG_IPX=m # CONFIG_IPX_INTERN is not set CONFIG_ATALK=m CONFIG_DEV_APPLETALK=m CONFIG_IPDDP=m CONFIG_IPDDP_ENCAP=y CONFIG_IPDDP_DECAP=y # CONFIG_X25 is not set # CONFIG_LAPB is not set # CONFIG_ECONET is not set CONFIG_WAN_ROUTER=m CONFIG_PHONET=m # CONFIG_PHONET_PIPECTRLR is not set CONFIG_IEEE802154=m CONFIG_NET_SCHED=y # # Queueing/Scheduling # CONFIG_NET_SCH_CBQ=m CONFIG_NET_SCH_HTB=m CONFIG_NET_SCH_HFSC=m CONFIG_NET_SCH_ATM=m CONFIG_NET_SCH_PRIO=m CONFIG_NET_SCH_MULTIQ=m CONFIG_NET_SCH_RED=m CONFIG_NET_SCH_SFQ=m CONFIG_NET_SCH_TEQL=m CONFIG_NET_SCH_TBF=m CONFIG_NET_SCH_GRED=m CONFIG_NET_SCH_DSMARK=m CONFIG_NET_SCH_NETEM=m CONFIG_NET_SCH_DRR=m CONFIG_NET_SCH_INGRESS=m # # Classification # CONFIG_NET_CLS=y CONFIG_NET_CLS_BASIC=m CONFIG_NET_CLS_TCINDEX=m CONFIG_NET_CLS_ROUTE4=m CONFIG_NET_CLS_ROUTE=y CONFIG_NET_CLS_FW=m CONFIG_NET_CLS_U32=m CONFIG_CLS_U32_PERF=y CONFIG_CLS_U32_MARK=y CONFIG_NET_CLS_RSVP=m CONFIG_NET_CLS_RSVP6=m CONFIG_NET_CLS_FLOW=m CONFIG_NET_CLS_CGROUP=y CONFIG_NET_EMATCH=y CONFIG_NET_EMATCH_STACK=32 CONFIG_NET_EMATCH_CMP=m CONFIG_NET_EMATCH_NBYTE=m CONFIG_NET_EMATCH_U32=m CONFIG_NET_EMATCH_META=m CONFIG_NET_EMATCH_TEXT=m CONFIG_NET_CLS_ACT=y CONFIG_NET_ACT_POLICE=m CONFIG_NET_ACT_GACT=m CONFIG_GACT_PROB=y CONFIG_NET_ACT_MIRRED=m CONFIG_NET_ACT_IPT=m CONFIG_NET_ACT_NAT=m CONFIG_NET_ACT_PEDIT=m CONFIG_NET_ACT_SIMP=m CONFIG_NET_ACT_SKBEDIT=m CONFIG_NET_ACT_CSUM=m CONFIG_NET_CLS_IND=y CONFIG_NET_SCH_FIFO=y CONFIG_DCB=y CONFIG_DNS_RESOLVER=y # CONFIG_BATMAN_ADV is not set CONFIG_RPS=y CONFIG_XPS=y # # Network testing # CONFIG_NET_PKTGEN=m # CONFIG_NET_TCPPROBE is not set CONFIG_NET_DROP_MONITOR=y CONFIG_HAMRADIO=y # # Packet Radio protocols # CONFIG_AX25=m CONFIG_AX25_DAMA_SLAVE=y CONFIG_NETROM=m CONFIG_ROSE=m # # AX.25 network device drivers # CONFIG_MKISS=m CONFIG_6PACK=m CONFIG_BPQETHER=m CONFIG_BAYCOM_SER_FDX=m CONFIG_BAYCOM_SER_HDX=m CONFIG_BAYCOM_PAR=m CONFIG_YAM=m CONFIG_CAN=m CONFIG_CAN_RAW=m CONFIG_CAN_BCM=m # # CAN Device Drivers # CONFIG_CAN_VCAN=m CONFIG_CAN_SLCAN=m CONFIG_CAN_DEV=m CONFIG_CAN_CALC_BITTIMING=y # CONFIG_PCH_CAN is not set CONFIG_CAN_SJA1000=m CONFIG_CAN_SJA1000_PLATFORM=m CONFIG_CAN_EMS_PCI=m CONFIG_CAN_KVASER_PCI=m CONFIG_CAN_PLX_PCI=m # # CAN USB interfaces # CONFIG_CAN_EMS_USB=m CONFIG_CAN_ESD_USB2=m CONFIG_CAN_SOFTING=m CONFIG_CAN_SOFTING_CS=m # CONFIG_CAN_DEBUG_DEVICES is not set CONFIG_IRDA=m # # IrDA protocols # CONFIG_IRLAN=m CONFIG_IRNET=m CONFIG_IRCOMM=m # CONFIG_IRDA_ULTRA is not set # # IrDA options # CONFIG_IRDA_CACHE_LAST_LSAP=y CONFIG_IRDA_FAST_RR=y # CONFIG_IRDA_DEBUG is not set # # Infrared-port device drivers # # # SIR device drivers # CONFIG_IRTTY_SIR=m # # Dongle support # CONFIG_DONGLE=y CONFIG_ESI_DONGLE=m CONFIG_ACTISYS_DONGLE=m CONFIG_TEKRAM_DONGLE=m CONFIG_TOIM3232_DONGLE=m CONFIG_LITELINK_DONGLE=m CONFIG_MA600_DONGLE=m CONFIG_GIRBIL_DONGLE=m CONFIG_MCP2120_DONGLE=m CONFIG_OLD_BELKIN_DONGLE=m CONFIG_ACT200L_DONGLE=m CONFIG_KINGSUN_DONGLE=m CONFIG_KSDAZZLE_DONGLE=m CONFIG_KS959_DONGLE=m # # FIR device drivers # CONFIG_USB_IRDA=m CONFIG_SIGMATEL_FIR=m CONFIG_NSC_FIR=m CONFIG_WINBOND_FIR=m CONFIG_SMC_IRCC_FIR=m CONFIG_ALI_FIR=m CONFIG_VLSI_FIR=m CONFIG_VIA_FIR=m CONFIG_MCS_FIR=m CONFIG_BT=m CONFIG_BT_L2CAP=m CONFIG_BT_SCO=m CONFIG_BT_RFCOMM=m CONFIG_BT_RFCOMM_TTY=y CONFIG_BT_BNEP=m CONFIG_BT_BNEP_MC_FILTER=y CONFIG_BT_BNEP_PROTO_FILTER=y CONFIG_BT_CMTP=m CONFIG_BT_HIDP=m # # Bluetooth device drivers # CONFIG_BT_HCIBTUSB=m CONFIG_BT_HCIBTSDIO=m CONFIG_BT_HCIUART=m CONFIG_BT_HCIUART_H4=y CONFIG_BT_HCIUART_BCSP=y CONFIG_BT_HCIUART_ATH3K=y CONFIG_BT_HCIUART_LL=y CONFIG_BT_HCIBCM203X=m CONFIG_BT_HCIBPA10X=m CONFIG_BT_HCIBFUSB=m CONFIG_BT_HCIDTL1=m CONFIG_BT_HCIBT3C=m CONFIG_BT_HCIBLUECARD=m CONFIG_BT_HCIBTUART=m CONFIG_BT_HCIVHCI=m CONFIG_BT_MRVL=m CONFIG_BT_MRVL_SDIO=m CONFIG_BT_ATH3K=m # CONFIG_AF_RXRPC is not set CONFIG_FIB_RULES=y CONFIG_WIRELESS=y CONFIG_WIRELESS_EXT=y CONFIG_WEXT_CORE=y CONFIG_WEXT_PROC=y CONFIG_WEXT_SPY=y CONFIG_WEXT_PRIV=y CONFIG_CFG80211=m # CONFIG_NL80211_TESTMODE is not set # CONFIG_CFG80211_DEVELOPER_WARNINGS is not set # CONFIG_CFG80211_REG_DEBUG is not set CONFIG_CFG80211_DEFAULT_PS=y CONFIG_CFG80211_DEBUGFS=y # CONFIG_CFG80211_INTERNAL_REGDB is not set CONFIG_CFG80211_WEXT=y CONFIG_WIRELESS_EXT_SYSFS=y CONFIG_LIB80211=m CONFIG_LIB80211_CRYPT_WEP=m CONFIG_LIB80211_CRYPT_CCMP=m CONFIG_LIB80211_CRYPT_TKIP=m # CONFIG_LIB80211_DEBUG is not set CONFIG_MAC80211=m CONFIG_MAC80211_HAS_RC=y CONFIG_MAC80211_RC_MINSTREL=y CONFIG_MAC80211_RC_MINSTREL_HT=y CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y CONFIG_MAC80211_RC_DEFAULT="minstrel_ht" CONFIG_MAC80211_MESH=y CONFIG_MAC80211_LEDS=y CONFIG_MAC80211_DEBUGFS=y # CONFIG_MAC80211_DEBUG_MENU is not set CONFIG_WIMAX=m CONFIG_WIMAX_DEBUG_LEVEL=8 CONFIG_RFKILL=m CONFIG_RFKILL_LEDS=y CONFIG_RFKILL_INPUT=y CONFIG_NET_9P=m CONFIG_NET_9P_VIRTIO=m CONFIG_NET_9P_RDMA=m # CONFIG_NET_9P_DEBUG is not set # CONFIG_CAIF is not set CONFIG_CEPH_LIB=m # CONFIG_CEPH_LIB_PRETTYDEBUG is not set # # Device Drivers # # # Generic Driver Options # CONFIG_UEVENT_HELPER_PATH="" CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_STANDALONE=y CONFIG_PREVENT_FIRMWARE_BUILD=y CONFIG_FW_LOADER=y # CONFIG_FIRMWARE_IN_KERNEL is not set CONFIG_EXTRA_FIRMWARE="" # CONFIG_DEBUG_DRIVER is not set CONFIG_DEBUG_DEVRES=y CONFIG_SYS_HYPERVISOR=y CONFIG_CONNECTOR=y CONFIG_PROC_EVENTS=y CONFIG_MTD=m # CONFIG_MTD_DEBUG is not set # CONFIG_MTD_TESTS is not set CONFIG_MTD_CONCAT=m CONFIG_MTD_PARTITIONS=y CONFIG_MTD_REDBOOT_PARTS=m CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1 # CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set # CONFIG_MTD_REDBOOT_PARTS_READONLY is not set CONFIG_MTD_AR7_PARTS=m # # User Modules And Translation Layers # CONFIG_MTD_CHAR=m CONFIG_MTD_BLKDEVS=m CONFIG_MTD_BLOCK=m CONFIG_MTD_BLOCK_RO=m CONFIG_FTL=m CONFIG_NFTL=m CONFIG_NFTL_RW=y CONFIG_INFTL=m CONFIG_RFD_FTL=m CONFIG_SSFDC=m # CONFIG_SM_FTL is not set CONFIG_MTD_OOPS=m # # RAM/ROM/Flash chip drivers # CONFIG_MTD_CFI=m CONFIG_MTD_JEDECPROBE=m CONFIG_MTD_GEN_PROBE=m # CONFIG_MTD_CFI_ADV_OPTIONS is not set CONFIG_MTD_MAP_BANK_WIDTH_1=y CONFIG_MTD_MAP_BANK_WIDTH_2=y CONFIG_MTD_MAP_BANK_WIDTH_4=y # CONFIG_MTD_MAP_BANK_WIDTH_8 is not set # CONFIG_MTD_MAP_BANK_WIDTH_16 is not set # CONFIG_MTD_MAP_BANK_WIDTH_32 is not set CONFIG_MTD_CFI_I1=y CONFIG_MTD_CFI_I2=y # CONFIG_MTD_CFI_I4 is not set # CONFIG_MTD_CFI_I8 is not set CONFIG_MTD_CFI_INTELEXT=m CONFIG_MTD_CFI_AMDSTD=m CONFIG_MTD_CFI_STAA=m CONFIG_MTD_CFI_UTIL=m CONFIG_MTD_RAM=m CONFIG_MTD_ROM=m CONFIG_MTD_ABSENT=m # # Mapping drivers for chip access # CONFIG_MTD_COMPLEX_MAPPINGS=y # CONFIG_MTD_PHYSMAP is not set CONFIG_MTD_SC520CDP=m CONFIG_MTD_NETSC520=m CONFIG_MTD_TS5500=m # CONFIG_MTD_SBC_GXX is not set # CONFIG_MTD_AMD76XROM is not set # CONFIG_MTD_ICHXROM is not set CONFIG_MTD_ESB2ROM=m CONFIG_MTD_CK804XROM=m CONFIG_MTD_SCB2_FLASH=m # CONFIG_MTD_NETtel is not set # CONFIG_MTD_L440GX is not set CONFIG_MTD_PCI=m # CONFIG_MTD_PCMCIA is not set # CONFIG_MTD_GPIO_ADDR is not set # CONFIG_MTD_INTEL_VR_NOR is not set # CONFIG_MTD_PLATRAM is not set # # Self-contained MTD device drivers # CONFIG_MTD_PMC551=m # CONFIG_MTD_PMC551_BUGFIX is not set # CONFIG_MTD_PMC551_DEBUG is not set # CONFIG_MTD_SLRAM is not set # CONFIG_MTD_PHRAM is not set CONFIG_MTD_MTDRAM=m CONFIG_MTDRAM_TOTAL_SIZE=4096 CONFIG_MTDRAM_ERASE_SIZE=128 CONFIG_MTD_BLOCK2MTD=m # # Disk-On-Chip Device Drivers # # CONFIG_MTD_DOC2000 is not set # CONFIG_MTD_DOC2001 is not set # CONFIG_MTD_DOC2001PLUS is not set CONFIG_MTD_NAND_ECC=m CONFIG_MTD_NAND_ECC_SMC=y CONFIG_MTD_NAND=m # CONFIG_MTD_NAND_VERIFY_WRITE is not set CONFIG_MTD_SM_COMMON=m # CONFIG_MTD_NAND_MUSEUM_IDS is not set # CONFIG_MTD_NAND_DENALI is not set CONFIG_MTD_NAND_IDS=m CONFIG_MTD_NAND_RICOH=m CONFIG_MTD_NAND_DISKONCHIP=m # CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0 # CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set # CONFIG_MTD_NAND_CAFE is not set CONFIG_MTD_NAND_NANDSIM=m # CONFIG_MTD_NAND_PLATFORM is not set CONFIG_MTD_ALAUDA=m # CONFIG_MTD_ONENAND is not set # # LPDDR flash memory drivers # CONFIG_MTD_LPDDR=m CONFIG_MTD_QINFO_PROBE=m CONFIG_MTD_UBI=m CONFIG_MTD_UBI_WL_THRESHOLD=4096 CONFIG_MTD_UBI_BEB_RESERVE=1 # CONFIG_MTD_UBI_GLUEBI is not set # # UBI debugging options # # CONFIG_MTD_UBI_DEBUG is not set CONFIG_PARPORT=m CONFIG_PARPORT_PC=m CONFIG_PARPORT_SERIAL=m # CONFIG_PARPORT_PC_FIFO is not set # CONFIG_PARPORT_PC_SUPERIO is not set CONFIG_PARPORT_PC_PCMCIA=m # CONFIG_PARPORT_GSC is not set # CONFIG_PARPORT_AX88796 is not set CONFIG_PARPORT_1284=y CONFIG_PARPORT_NOT_PC=y CONFIG_PNP=y # CONFIG_PNP_DEBUG_MESSAGES is not set # # Protocols # CONFIG_PNPACPI=y CONFIG_BLK_DEV=y CONFIG_BLK_DEV_FD=m # CONFIG_PARIDE is not set CONFIG_BLK_CPQ_DA=m CONFIG_BLK_CPQ_CISS_DA=m CONFIG_CISS_SCSI_TAPE=y CONFIG_BLK_DEV_DAC960=m CONFIG_BLK_DEV_UMEM=m # CONFIG_BLK_DEV_COW_COMMON is not set CONFIG_BLK_DEV_LOOP=y CONFIG_BLK_DEV_CRYPTOLOOP=m CONFIG_BLK_DEV_DRBD=m # CONFIG_DRBD_FAULT_INJECTION is not set CONFIG_BLK_DEV_NBD=m CONFIG_BLK_DEV_OSD=m CONFIG_BLK_DEV_SX8=m # CONFIG_BLK_DEV_UB is not set CONFIG_BLK_DEV_RAM=y CONFIG_BLK_DEV_RAM_COUNT=16 CONFIG_BLK_DEV_RAM_SIZE=16384 # CONFIG_BLK_DEV_XIP is not set CONFIG_CDROM_PKTCDVD=m CONFIG_CDROM_PKTCDVD_BUFFERS=8 # CONFIG_CDROM_PKTCDVD_WCACHE is not set CONFIG_ATA_OVER_ETH=m CONFIG_XEN_BLKDEV_FRONTEND=m CONFIG_VIRTIO_BLK=m # CONFIG_BLK_DEV_HD is not set CONFIG_BLK_DEV_RBD=m CONFIG_MISC_DEVICES=y # CONFIG_AD525X_DPOT is not set # CONFIG_IBM_ASM is not set # CONFIG_PHANTOM is not set CONFIG_SGI_IOC4=m CONFIG_TIFM_CORE=m CONFIG_TIFM_7XX1=m CONFIG_ICS932S401=m CONFIG_ENCLOSURE_SERVICES=m CONFIG_CS5535_MFGPT=m CONFIG_CS5535_MFGPT_DEFAULT_IRQ=7 CONFIG_CS5535_CLOCK_EVENT_SRC=m CONFIG_HP_ILO=m CONFIG_APDS9802ALS=m CONFIG_ISL29003=m CONFIG_ISL29020=m CONFIG_SENSORS_TSL2550=m # CONFIG_SENSORS_BH1780 is not set CONFIG_SENSORS_BH1770=m CONFIG_SENSORS_APDS990X=m # CONFIG_HMC6352 is not set # CONFIG_DS1682 is not set CONFIG_VMWARE_BALLOON=m # CONFIG_BMP085 is not set CONFIG_PCH_PHUB=m # CONFIG_C2PORT is not set # # EEPROM support # CONFIG_EEPROM_AT24=m CONFIG_EEPROM_LEGACY=m CONFIG_EEPROM_MAX6875=m CONFIG_EEPROM_93CX6=m CONFIG_CB710_CORE=m # CONFIG_CB710_DEBUG is not set CONFIG_CB710_DEBUG_ASSUMPTIONS=y CONFIG_IWMC3200TOP=m # CONFIG_IWMC3200TOP_DEBUG is not set CONFIG_IWMC3200TOP_DEBUGFS=y # # Texas Instruments shared transport line discipline # # CONFIG_TI_ST is not set CONFIG_HAVE_IDE=y # CONFIG_IDE is not set # # SCSI device support # CONFIG_SCSI_MOD=y CONFIG_RAID_ATTRS=m CONFIG_SCSI=y CONFIG_SCSI_DMA=y CONFIG_SCSI_TGT=m CONFIG_SCSI_NETLINK=y CONFIG_SCSI_PROC_FS=y # # SCSI support type (disk, tape, CD-ROM) # CONFIG_BLK_DEV_SD=y CONFIG_CHR_DEV_ST=m CONFIG_CHR_DEV_OSST=m CONFIG_BLK_DEV_SR=y CONFIG_BLK_DEV_SR_VENDOR=y CONFIG_CHR_DEV_SG=y CONFIG_CHR_DEV_SCH=m CONFIG_SCSI_ENCLOSURE=m CONFIG_SCSI_MULTI_LUN=y CONFIG_SCSI_CONSTANTS=y CONFIG_SCSI_LOGGING=y CONFIG_SCSI_SCAN_ASYNC=y CONFIG_SCSI_WAIT_SCAN=m # # SCSI Transports # CONFIG_SCSI_SPI_ATTRS=m CONFIG_SCSI_FC_ATTRS=m CONFIG_SCSI_FC_TGT_ATTRS=y CONFIG_SCSI_ISCSI_ATTRS=m CONFIG_SCSI_SAS_ATTRS=m CONFIG_SCSI_SAS_LIBSAS=m CONFIG_SCSI_SAS_ATA=y CONFIG_SCSI_SAS_HOST_SMP=y # CONFIG_SCSI_SAS_LIBSAS_DEBUG is not set CONFIG_SCSI_SRP_ATTRS=m CONFIG_SCSI_SRP_TGT_ATTRS=y CONFIG_SCSI_LOWLEVEL=y CONFIG_ISCSI_TCP=m CONFIG_ISCSI_BOOT_SYSFS=m CONFIG_SCSI_CXGB3_ISCSI=m CONFIG_SCSI_CXGB4_ISCSI=m CONFIG_SCSI_BNX2_ISCSI=m CONFIG_BE2ISCSI=m CONFIG_BLK_DEV_3W_XXXX_RAID=m CONFIG_SCSI_HPSA=m CONFIG_SCSI_3W_9XXX=m CONFIG_SCSI_3W_SAS=m CONFIG_SCSI_ACARD=m CONFIG_SCSI_AACRAID=m CONFIG_SCSI_AIC7XXX=m CONFIG_AIC7XXX_CMDS_PER_DEVICE=4 CONFIG_AIC7XXX_RESET_DELAY_MS=15000 # CONFIG_AIC7XXX_DEBUG_ENABLE is not set CONFIG_AIC7XXX_DEBUG_MASK=0 # CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set CONFIG_SCSI_AIC7XXX_OLD=m CONFIG_SCSI_AIC79XX=m CONFIG_AIC79XX_CMDS_PER_DEVICE=4 CONFIG_AIC79XX_RESET_DELAY_MS=15000 # CONFIG_AIC79XX_DEBUG_ENABLE is not set CONFIG_AIC79XX_DEBUG_MASK=0 # CONFIG_AIC79XX_REG_PRETTY_PRINT is not set CONFIG_SCSI_AIC94XX=m # CONFIG_AIC94XX_DEBUG is not set CONFIG_SCSI_MVSAS=m # CONFIG_SCSI_MVSAS_DEBUG is not set # CONFIG_SCSI_DPT_I2O is not set CONFIG_SCSI_ADVANSYS=m CONFIG_SCSI_ARCMSR=m CONFIG_SCSI_ARCMSR_AER=y CONFIG_MEGARAID_NEWGEN=y CONFIG_MEGARAID_MM=m CONFIG_MEGARAID_MAILBOX=m CONFIG_MEGARAID_LEGACY=m CONFIG_MEGARAID_SAS=m CONFIG_SCSI_MPT2SAS=m CONFIG_SCSI_MPT2SAS_MAX_SGE=128 CONFIG_SCSI_MPT2SAS_LOGGING=y CONFIG_SCSI_HPTIOP=m CONFIG_SCSI_BUSLOGIC=m CONFIG_VMWARE_PVSCSI=m CONFIG_LIBFC=m CONFIG_LIBFCOE=m CONFIG_FCOE=m CONFIG_FCOE_FNIC=m # CONFIG_SCSI_DMX3191D is not set # CONFIG_SCSI_EATA is not set # CONFIG_SCSI_FUTURE_DOMAIN is not set CONFIG_SCSI_GDTH=m CONFIG_SCSI_IPS=m CONFIG_SCSI_INITIO=m CONFIG_SCSI_INIA100=m CONFIG_SCSI_PPA=m CONFIG_SCSI_IMM=m # CONFIG_SCSI_IZIP_EPP16 is not set # CONFIG_SCSI_IZIP_SLOW_CTR is not set CONFIG_SCSI_STEX=m CONFIG_SCSI_SYM53C8XX_2=m CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1 CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16 CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64 CONFIG_SCSI_SYM53C8XX_MMIO=y # CONFIG_SCSI_IPR is not set CONFIG_SCSI_QLOGIC_1280=m CONFIG_SCSI_QLA_FC=m CONFIG_SCSI_QLA_ISCSI=m CONFIG_SCSI_LPFC=m # CONFIG_SCSI_LPFC_DEBUG_FS is not set CONFIG_SCSI_DC395x=m CONFIG_SCSI_DC390T=m CONFIG_SCSI_DEBUG=m CONFIG_SCSI_PMCRAID=m CONFIG_SCSI_PM8001=m CONFIG_SCSI_SRP=m CONFIG_SCSI_BFA_FC=m CONFIG_SCSI_LOWLEVEL_PCMCIA=y CONFIG_PCMCIA_AHA152X=m # CONFIG_PCMCIA_FDOMAIN is not set CONFIG_PCMCIA_QLOGIC=m CONFIG_PCMCIA_SYM53C500=m CONFIG_SCSI_DH=y CONFIG_SCSI_DH_RDAC=m CONFIG_SCSI_DH_HP_SW=m CONFIG_SCSI_DH_EMC=m CONFIG_SCSI_DH_ALUA=m CONFIG_SCSI_OSD_INITIATOR=m CONFIG_SCSI_OSD_ULD=m CONFIG_SCSI_OSD_DPRINT_SENSE=1 # CONFIG_SCSI_OSD_DEBUG is not set CONFIG_ATA=y # CONFIG_ATA_NONSTANDARD is not set CONFIG_ATA_VERBOSE_ERROR=y CONFIG_ATA_ACPI=y CONFIG_SATA_PMP=y # # Controllers with non-SFF native interface # CONFIG_SATA_AHCI=y CONFIG_SATA_AHCI_PLATFORM=m CONFIG_SATA_INIC162X=m CONFIG_SATA_ACARD_AHCI=m CONFIG_SATA_SIL24=m CONFIG_ATA_SFF=y # # SFF controllers with custom DMA interface # CONFIG_PDC_ADMA=m CONFIG_SATA_QSTOR=m CONFIG_SATA_SX4=m CONFIG_ATA_BMDMA=y # # SATA SFF controllers with BMDMA # CONFIG_ATA_PIIX=y CONFIG_SATA_MV=m CONFIG_SATA_NV=m CONFIG_SATA_PROMISE=m CONFIG_SATA_SIL=m CONFIG_SATA_SIS=m CONFIG_SATA_SVW=m CONFIG_SATA_ULI=m CONFIG_SATA_VIA=m CONFIG_SATA_VITESSE=m # # PATA SFF controllers with BMDMA # CONFIG_PATA_ALI=m CONFIG_PATA_AMD=m CONFIG_PATA_ARTOP=m CONFIG_PATA_ATIIXP=m CONFIG_PATA_ATP867X=m CONFIG_PATA_CMD64X=m CONFIG_PATA_CS5520=m CONFIG_PATA_CS5530=m CONFIG_PATA_CS5536=m CONFIG_PATA_CYPRESS=m CONFIG_PATA_EFAR=m CONFIG_PATA_HPT366=m CONFIG_PATA_HPT37X=m CONFIG_PATA_HPT3X2N=m CONFIG_PATA_HPT3X3=m # CONFIG_PATA_HPT3X3_DMA is not set CONFIG_PATA_IT8213=m CONFIG_PATA_IT821X=m CONFIG_PATA_JMICRON=m CONFIG_PATA_MARVELL=m CONFIG_PATA_NETCELL=m CONFIG_PATA_NINJA32=m CONFIG_PATA_NS87415=m CONFIG_PATA_OLDPIIX=m CONFIG_PATA_OPTIDMA=m CONFIG_PATA_PDC2027X=m CONFIG_PATA_PDC_OLD=m # CONFIG_PATA_RADISYS is not set CONFIG_PATA_RDC=m # CONFIG_PATA_SC1200 is not set CONFIG_PATA_SCH=m CONFIG_PATA_SERVERWORKS=m CONFIG_PATA_SIL680=m CONFIG_PATA_SIS=m CONFIG_PATA_TOSHIBA=m CONFIG_PATA_TRIFLEX=m CONFIG_PATA_VIA=m CONFIG_PATA_WINBOND=m # # PIO-only SFF controllers # CONFIG_PATA_CMD640_PCI=m CONFIG_PATA_MPIIX=m CONFIG_PATA_NS87410=m CONFIG_PATA_OPTI=m CONFIG_PATA_PCMCIA=m # CONFIG_PATA_RZ1000 is not set # # Generic fallback / legacy drivers # CONFIG_PATA_ACPI=m CONFIG_ATA_GENERIC=m # CONFIG_PATA_LEGACY is not set CONFIG_MD=y CONFIG_BLK_DEV_MD=y CONFIG_MD_AUTODETECT=y CONFIG_MD_LINEAR=m CONFIG_MD_RAID0=m CONFIG_MD_RAID1=m CONFIG_MD_RAID10=m CONFIG_MD_RAID456=m # CONFIG_MULTICORE_RAID456 is not set CONFIG_MD_MULTIPATH=m CONFIG_MD_FAULTY=m CONFIG_BLK_DEV_DM=y CONFIG_DM_DEBUG=y CONFIG_DM_CRYPT=m CONFIG_DM_SNAPSHOT=y CONFIG_DM_MIRROR=y CONFIG_DM_RAID=m CONFIG_DM_LOG_USERSPACE=m CONFIG_DM_ZERO=y CONFIG_DM_MULTIPATH=m CONFIG_DM_MULTIPATH_QL=m CONFIG_DM_MULTIPATH_ST=m # CONFIG_DM_DELAY is not set CONFIG_DM_UEVENT=y CONFIG_TARGET_CORE=m CONFIG_TCM_IBLOCK=m CONFIG_TCM_FILEIO=m CONFIG_TCM_PSCSI=m CONFIG_FUSION=y CONFIG_FUSION_SPI=m CONFIG_FUSION_FC=m CONFIG_FUSION_SAS=m CONFIG_FUSION_MAX_SGE=40 CONFIG_FUSION_CTL=m CONFIG_FUSION_LAN=m CONFIG_FUSION_LOGGING=y # # IEEE 1394 (FireWire) support # CONFIG_FIREWIRE=m CONFIG_FIREWIRE_OHCI=m CONFIG_FIREWIRE_OHCI_DEBUG=y CONFIG_FIREWIRE_SBP2=m CONFIG_FIREWIRE_NET=m CONFIG_FIREWIRE_NOSY=m # CONFIG_I2O is not set CONFIG_MACINTOSH_DRIVERS=y CONFIG_MAC_EMUMOUSEBTN=y CONFIG_NETDEVICES=y CONFIG_IFB=m CONFIG_DUMMY=m CONFIG_BONDING=m CONFIG_MACVLAN=m CONFIG_MACVTAP=m CONFIG_EQUALIZER=m CONFIG_TUN=m CONFIG_VETH=m CONFIG_NET_SB1000=m # CONFIG_ARCNET is not set CONFIG_MII=m CONFIG_PHYLIB=y # # MII PHY device drivers # CONFIG_MARVELL_PHY=m CONFIG_DAVICOM_PHY=m CONFIG_QSEMI_PHY=m CONFIG_LXT_PHY=m CONFIG_CICADA_PHY=m CONFIG_VITESSE_PHY=m CONFIG_SMSC_PHY=m CONFIG_BROADCOM_PHY=m CONFIG_BCM63XX_PHY=m CONFIG_ICPLUS_PHY=m CONFIG_REALTEK_PHY=m CONFIG_NATIONAL_PHY=m CONFIG_STE10XP=m CONFIG_LSI_ET1011C_PHY=m CONFIG_MICREL_PHY=m CONFIG_FIXED_PHY=y CONFIG_MDIO_BITBANG=m # CONFIG_MDIO_GPIO is not set CONFIG_NET_ETHERNET=y CONFIG_HAPPYMEAL=m CONFIG_SUNGEM=m CONFIG_CASSINI=m CONFIG_NET_VENDOR_3COM=y CONFIG_VORTEX=m CONFIG_TYPHOON=m CONFIG_ETHOC=m CONFIG_DNET=m CONFIG_NET_TULIP=y CONFIG_DE2104X=m CONFIG_DE2104X_DSL=0 CONFIG_TULIP=m # CONFIG_TULIP_MWI is not set CONFIG_TULIP_MMIO=y # CONFIG_TULIP_NAPI is not set CONFIG_DE4X5=m CONFIG_WINBOND_840=m CONFIG_DM9102=m CONFIG_ULI526X=m CONFIG_PCMCIA_XIRCOM=m # CONFIG_HP100 is not set # CONFIG_IBM_NEW_EMAC_ZMII is not set # CONFIG_IBM_NEW_EMAC_RGMII is not set # CONFIG_IBM_NEW_EMAC_TAH is not set # CONFIG_IBM_NEW_EMAC_EMAC4 is not set # CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set # CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set # CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set CONFIG_NET_PCI=y CONFIG_PCNET32=m CONFIG_AMD8111_ETH=m CONFIG_ADAPTEC_STARFIRE=m CONFIG_KSZ884X_PCI=m CONFIG_B44=m CONFIG_B44_PCI_AUTOSELECT=y CONFIG_B44_PCICORE_AUTOSELECT=y CONFIG_B44_PCI=y CONFIG_FORCEDETH=m CONFIG_E100=m CONFIG_FEALNX=m CONFIG_NATSEMI=m CONFIG_NE2K_PCI=m CONFIG_8139CP=m CONFIG_8139TOO=m # CONFIG_8139TOO_PIO is not set # CONFIG_8139TOO_TUNE_TWISTER is not set CONFIG_8139TOO_8129=y # CONFIG_8139_OLD_RX_RESET is not set CONFIG_R6040=m CONFIG_SIS900=m CONFIG_EPIC100=m CONFIG_SMSC9420=m CONFIG_SUNDANCE=m # CONFIG_SUNDANCE_MMIO is not set CONFIG_TLAN=m # CONFIG_KS8842 is not set # CONFIG_KS8851_MLL is not set CONFIG_VIA_RHINE=m CONFIG_VIA_RHINE_MMIO=y CONFIG_SC92031=m CONFIG_NET_POCKET=y CONFIG_ATP=m CONFIG_DE600=m CONFIG_DE620=m CONFIG_ATL2=m CONFIG_NETDEV_1000=y CONFIG_ACENIC=m # CONFIG_ACENIC_OMIT_TIGON_I is not set CONFIG_DL2K=m CONFIG_E1000=m CONFIG_E1000E=m CONFIG_IP1000=m CONFIG_IGB=m CONFIG_IGB_DCA=y CONFIG_IGBVF=m CONFIG_NS83820=m CONFIG_HAMACHI=m CONFIG_YELLOWFIN=m CONFIG_R8169=m CONFIG_R8169_VLAN=y CONFIG_SIS190=m CONFIG_SKGE=m # CONFIG_SKGE_DEBUG is not set CONFIG_SKY2=m # CONFIG_SKY2_DEBUG is not set CONFIG_VIA_VELOCITY=m CONFIG_TIGON3=m CONFIG_BNX2=m CONFIG_CNIC=m CONFIG_QLA3XXX=m CONFIG_ATL1=m CONFIG_ATL1E=m CONFIG_ATL1C=m CONFIG_JME=m CONFIG_STMMAC_ETH=m # CONFIG_STMMAC_DA is not set # CONFIG_STMMAC_DUAL_MAC is not set CONFIG_PCH_GBE=m CONFIG_NETDEV_10000=y CONFIG_MDIO=m CONFIG_CHELSIO_T1=m CONFIG_CHELSIO_T1_1G=y CONFIG_CHELSIO_T3_DEPENDS=y CONFIG_CHELSIO_T3=m CONFIG_CHELSIO_T4_DEPENDS=y CONFIG_CHELSIO_T4=m CONFIG_CHELSIO_T4VF_DEPENDS=y CONFIG_CHELSIO_T4VF=m CONFIG_ENIC=m CONFIG_IXGBE=m CONFIG_IXGBE_DCA=y CONFIG_IXGBE_DCB=y CONFIG_IXGBEVF=m CONFIG_IXGB=m CONFIG_S2IO=m CONFIG_VXGE=m # CONFIG_VXGE_DEBUG_TRACE_ALL is not set CONFIG_MYRI10GE=m CONFIG_MYRI10GE_DCA=y CONFIG_NETXEN_NIC=m CONFIG_NIU=m CONFIG_MLX4_EN=m CONFIG_MLX4_CORE=m CONFIG_MLX4_DEBUG=y CONFIG_TEHUTI=m CONFIG_BNX2X=m CONFIG_QLCNIC=m CONFIG_QLGE=m CONFIG_BNA=m CONFIG_SFC=m CONFIG_SFC_MTD=y CONFIG_BE2NET=m # CONFIG_TR is not set CONFIG_WLAN=y # CONFIG_PCMCIA_RAYCS is not set CONFIG_LIBERTAS_THINFIRM=m # CONFIG_LIBERTAS_THINFIRM_DEBUG is not set CONFIG_LIBERTAS_THINFIRM_USB=m CONFIG_AIRO=m CONFIG_ATMEL=m CONFIG_PCI_ATMEL=m CONFIG_PCMCIA_ATMEL=m CONFIG_AT76C50X_USB=m CONFIG_AIRO_CS=m CONFIG_PCMCIA_WL3501=m # CONFIG_PRISM54 is not set CONFIG_USB_ZD1201=m CONFIG_USB_NET_RNDIS_WLAN=m CONFIG_RTL8180=m CONFIG_RTL8187=m CONFIG_RTL8187_LEDS=y CONFIG_ADM8211=m CONFIG_MAC80211_HWSIM=m CONFIG_MWL8K=m CONFIG_ATH_COMMON=m # CONFIG_ATH_DEBUG is not set CONFIG_ATH5K=m CONFIG_ATH5K_DEBUG=y CONFIG_ATH5K_PCI=y CONFIG_ATH9K_HW=m CONFIG_ATH9K_COMMON=m CONFIG_ATH9K=m CONFIG_ATH9K_DEBUGFS=y CONFIG_ATH9K_RATE_CONTROL=y CONFIG_ATH9K_HTC=m # CONFIG_ATH9K_HTC_DEBUGFS is not set CONFIG_AR9170_USB=m CONFIG_AR9170_LEDS=y CONFIG_CARL9170=m CONFIG_CARL9170_LEDS=y # CONFIG_CARL9170_DEBUGFS is not set CONFIG_CARL9170_WPC=y CONFIG_B43=m CONFIG_B43_PCI_AUTOSELECT=y CONFIG_B43_PCICORE_AUTOSELECT=y CONFIG_B43_PCMCIA=y CONFIG_B43_SDIO=y CONFIG_B43_PIO=y CONFIG_B43_PHY_N=y CONFIG_B43_PHY_LP=y CONFIG_B43_LEDS=y CONFIG_B43_HWRNG=y # CONFIG_B43_DEBUG is not set CONFIG_B43LEGACY=m CONFIG_B43LEGACY_PCI_AUTOSELECT=y CONFIG_B43LEGACY_PCICORE_AUTOSELECT=y CONFIG_B43LEGACY_LEDS=y CONFIG_B43LEGACY_HWRNG=y # CONFIG_B43LEGACY_DEBUG is not set CONFIG_B43LEGACY_DMA=y CONFIG_B43LEGACY_PIO=y CONFIG_B43LEGACY_DMA_AND_PIO_MODE=y # CONFIG_B43LEGACY_DMA_MODE is not set # CONFIG_B43LEGACY_PIO_MODE is not set CONFIG_HOSTAP=m CONFIG_HOSTAP_FIRMWARE=y CONFIG_HOSTAP_FIRMWARE_NVRAM=y CONFIG_HOSTAP_PLX=m CONFIG_HOSTAP_PCI=m CONFIG_HOSTAP_CS=m CONFIG_IPW2100=m CONFIG_IPW2100_MONITOR=y # CONFIG_IPW2100_DEBUG is not set CONFIG_IPW2200=m CONFIG_IPW2200_MONITOR=y CONFIG_IPW2200_RADIOTAP=y CONFIG_IPW2200_PROMISCUOUS=y CONFIG_IPW2200_QOS=y # CONFIG_IPW2200_DEBUG is not set CONFIG_LIBIPW=m # CONFIG_LIBIPW_DEBUG is not set CONFIG_IWLWIFI=m # # Debugging Options # CONFIG_IWLWIFI_DEBUG=y CONFIG_IWLWIFI_DEBUGFS=y # CONFIG_IWLWIFI_DEBUG_EXPERIMENTAL_UCODE is not set # CONFIG_IWLWIFI_DEVICE_TRACING is not set CONFIG_IWLAGN=m CONFIG_IWL4965=y CONFIG_IWL5000=y CONFIG_IWL3945=m CONFIG_IWM=m # CONFIG_IWM_DEBUG is not set # CONFIG_IWM_TRACING is not set CONFIG_LIBERTAS=m CONFIG_LIBERTAS_USB=m CONFIG_LIBERTAS_CS=m CONFIG_LIBERTAS_SDIO=m # CONFIG_LIBERTAS_DEBUG is not set CONFIG_LIBERTAS_MESH=y CONFIG_HERMES=m # CONFIG_HERMES_PRISM is not set CONFIG_HERMES_CACHE_FW_ON_INIT=y CONFIG_PLX_HERMES=m CONFIG_TMD_HERMES=m CONFIG_NORTEL_HERMES=m CONFIG_PCMCIA_HERMES=m CONFIG_PCMCIA_SPECTRUM=m CONFIG_ORINOCO_USB=m CONFIG_P54_COMMON=m CONFIG_P54_USB=m CONFIG_P54_PCI=m CONFIG_P54_LEDS=y CONFIG_RT2X00=m CONFIG_RT2400PCI=m CONFIG_RT2500PCI=m CONFIG_RT61PCI=m CONFIG_RT2800PCI=m CONFIG_RT2800PCI_RT33XX=y CONFIG_RT2800PCI_RT35XX=y CONFIG_RT2500USB=m CONFIG_RT73USB=m CONFIG_RT2800USB=m CONFIG_RT2800USB_RT33XX=y CONFIG_RT2800USB_RT35XX=y CONFIG_RT2800USB_UNKNOWN=y CONFIG_RT2800_LIB=m CONFIG_RT2X00_LIB_PCI=m CONFIG_RT2X00_LIB_USB=m CONFIG_RT2X00_LIB=m CONFIG_RT2X00_LIB_HT=y CONFIG_RT2X00_LIB_FIRMWARE=y CONFIG_RT2X00_LIB_CRYPTO=y CONFIG_RT2X00_LIB_LEDS=y CONFIG_RT2X00_LIB_DEBUGFS=y # CONFIG_RT2X00_DEBUG is not set CONFIG_RTL8192CE=m CONFIG_RTLWIFI=m CONFIG_WL1251=m CONFIG_WL1251_SDIO=m CONFIG_WL12XX_MENU=m CONFIG_WL12XX=m # CONFIG_WL12XX_HT is not set CONFIG_WL12XX_SDIO=m # CONFIG_WL12XX_SDIO_TEST is not set CONFIG_WL12XX_PLATFORM_DATA=y CONFIG_ZD1211RW=m # CONFIG_ZD1211RW_DEBUG is not set # # WiMAX Wireless Broadband devices # CONFIG_WIMAX_I2400M=m CONFIG_WIMAX_I2400M_USB=m CONFIG_WIMAX_I2400M_SDIO=m CONFIG_WIMAX_IWMC3200_SDIO=y CONFIG_WIMAX_I2400M_DEBUG_LEVEL=8 # # USB Network Adapters # CONFIG_USB_CATC=m CONFIG_USB_KAWETH=m CONFIG_USB_PEGASUS=m CONFIG_USB_RTL8150=m CONFIG_USB_USBNET=m CONFIG_USB_NET_AX8817X=m CONFIG_USB_NET_CDCETHER=m CONFIG_USB_NET_CDC_EEM=m CONFIG_USB_NET_CDC_NCM=m CONFIG_USB_NET_DM9601=m CONFIG_USB_NET_SMSC75XX=m CONFIG_USB_NET_SMSC95XX=m CONFIG_USB_NET_GL620A=m CONFIG_USB_NET_NET1080=m CONFIG_USB_NET_PLUSB=m CONFIG_USB_NET_MCS7830=m CONFIG_USB_NET_RNDIS_HOST=m CONFIG_USB_NET_CDC_SUBSET=m CONFIG_USB_ALI_M5632=y CONFIG_USB_AN2720=y CONFIG_USB_BELKIN=y CONFIG_USB_ARMLINUX=y CONFIG_USB_EPSON2888=y CONFIG_USB_KC2190=y CONFIG_USB_NET_ZAURUS=m CONFIG_USB_NET_CX82310_ETH=m CONFIG_USB_HSO=m CONFIG_USB_NET_INT51X1=m CONFIG_USB_CDC_PHONET=m CONFIG_USB_IPHETH=m CONFIG_USB_SIERRA_NET=m CONFIG_NET_PCMCIA=y CONFIG_PCMCIA_3C589=m CONFIG_PCMCIA_3C574=m CONFIG_PCMCIA_FMVJ18X=m CONFIG_PCMCIA_PCNET=m CONFIG_PCMCIA_NMCLAN=m CONFIG_PCMCIA_SMC91C92=m CONFIG_PCMCIA_XIRC2PS=m CONFIG_PCMCIA_AXNET=m # CONFIG_WAN is not set CONFIG_ATM_DRIVERS=y # CONFIG_ATM_DUMMY is not set CONFIG_ATM_TCP=m # CONFIG_ATM_LANAI is not set CONFIG_ATM_ENI=m # CONFIG_ATM_ENI_DEBUG is not set # CONFIG_ATM_ENI_TUNE_BURST is not set CONFIG_ATM_FIRESTREAM=m # CONFIG_ATM_ZATM is not set CONFIG_ATM_NICSTAR=m # CONFIG_ATM_NICSTAR_USE_SUNI is not set # CONFIG_ATM_NICSTAR_USE_IDT77105 is not set # CONFIG_ATM_IDT77252 is not set # CONFIG_ATM_AMBASSADOR is not set # CONFIG_ATM_HORIZON is not set # CONFIG_ATM_IA is not set # CONFIG_ATM_FORE200E is not set CONFIG_ATM_HE=m # CONFIG_ATM_HE_USE_SUNI is not set CONFIG_ATM_SOLOS=m CONFIG_IEEE802154_DRIVERS=m CONFIG_IEEE802154_FAKEHARD=m # # CAIF transport drivers # CONFIG_XEN_NETDEV_FRONTEND=m CONFIG_FDDI=y # CONFIG_DEFXX is not set CONFIG_SKFP=m # CONFIG_HIPPI is not set CONFIG_PLIP=m CONFIG_PPP=m CONFIG_PPP_MULTILINK=y CONFIG_PPP_FILTER=y CONFIG_PPP_ASYNC=m CONFIG_PPP_SYNC_TTY=m CONFIG_PPP_DEFLATE=m # CONFIG_PPP_BSDCOMP is not set CONFIG_PPP_MPPE=m CONFIG_PPPOE=m CONFIG_PPTP=m CONFIG_PPPOATM=m CONFIG_PPPOL2TP=m CONFIG_SLIP=m CONFIG_SLIP_COMPRESSED=y CONFIG_SLHC=m CONFIG_SLIP_SMART=y # CONFIG_SLIP_MODE_SLIP6 is not set CONFIG_NET_FC=y CONFIG_NETCONSOLE=m CONFIG_NETCONSOLE_DYNAMIC=y CONFIG_NETPOLL=y CONFIG_NETPOLL_TRAP=y CONFIG_NET_POLL_CONTROLLER=y CONFIG_VIRTIO_NET=m CONFIG_VMXNET3=m CONFIG_ISDN=y CONFIG_ISDN_I4L=m CONFIG_ISDN_PPP=y CONFIG_ISDN_PPP_VJ=y CONFIG_ISDN_MPP=y CONFIG_IPPP_FILTER=y # CONFIG_ISDN_PPP_BSDCOMP is not set CONFIG_ISDN_AUDIO=y CONFIG_ISDN_TTY_FAX=y # # ISDN feature submodules # CONFIG_ISDN_DIVERSION=m # # ISDN4Linux hardware drivers # # # Passive cards # CONFIG_ISDN_DRV_HISAX=m # # D-channel protocol features # CONFIG_HISAX_EURO=y CONFIG_DE_AOC=y CONFIG_HISAX_NO_SENDCOMPLETE=y CONFIG_HISAX_NO_LLC=y CONFIG_HISAX_NO_KEYPAD=y CONFIG_HISAX_1TR6=y CONFIG_HISAX_NI1=y CONFIG_HISAX_MAX_CARDS=8 # # HiSax supported cards # CONFIG_HISAX_16_3=y CONFIG_HISAX_TELESPCI=y CONFIG_HISAX_S0BOX=y CONFIG_HISAX_FRITZPCI=y CONFIG_HISAX_AVM_A1_PCMCIA=y CONFIG_HISAX_ELSA=y CONFIG_HISAX_DIEHLDIVA=y CONFIG_HISAX_SEDLBAUER=y CONFIG_HISAX_NETJET=y CONFIG_HISAX_NETJET_U=y CONFIG_HISAX_NICCY=y CONFIG_HISAX_BKM_A4T=y CONFIG_HISAX_SCT_QUADRO=y CONFIG_HISAX_GAZEL=y CONFIG_HISAX_HFC_PCI=y CONFIG_HISAX_W6692=y CONFIG_HISAX_HFC_SX=y CONFIG_HISAX_ENTERNOW_PCI=y # CONFIG_HISAX_DEBUG is not set # # HiSax PCMCIA card service modules # CONFIG_HISAX_SEDLBAUER_CS=m CONFIG_HISAX_ELSA_CS=m CONFIG_HISAX_AVM_A1_CS=m CONFIG_HISAX_TELES_CS=m # # HiSax sub driver modules # CONFIG_HISAX_ST5481=m # CONFIG_HISAX_HFCUSB is not set CONFIG_HISAX_HFC4S8S=m CONFIG_HISAX_FRITZ_PCIPNP=m # # Active cards # CONFIG_ISDN_CAPI=m CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y # CONFIG_CAPI_TRACE is not set CONFIG_ISDN_CAPI_MIDDLEWARE=y CONFIG_ISDN_CAPI_CAPI20=m CONFIG_ISDN_CAPI_CAPIFS_BOOL=y CONFIG_ISDN_CAPI_CAPIFS=m CONFIG_ISDN_CAPI_CAPIDRV=m # # CAPI hardware drivers # CONFIG_CAPI_AVM=y CONFIG_ISDN_DRV_AVMB1_B1PCI=m CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y CONFIG_ISDN_DRV_AVMB1_B1PCMCIA=m CONFIG_ISDN_DRV_AVMB1_AVM_CS=m CONFIG_ISDN_DRV_AVMB1_T1PCI=m CONFIG_ISDN_DRV_AVMB1_C4=m CONFIG_CAPI_EICON=y CONFIG_ISDN_DIVAS=m CONFIG_ISDN_DIVAS_BRIPCI=y CONFIG_ISDN_DIVAS_PRIPCI=y CONFIG_ISDN_DIVAS_DIVACAPI=m CONFIG_ISDN_DIVAS_USERIDI=m CONFIG_ISDN_DIVAS_MAINT=m CONFIG_ISDN_DRV_GIGASET=m CONFIG_GIGASET_CAPI=y # CONFIG_GIGASET_I4L is not set # CONFIG_GIGASET_DUMMYLL is not set CONFIG_GIGASET_BASE=m CONFIG_GIGASET_M105=m CONFIG_GIGASET_M101=m # CONFIG_GIGASET_DEBUG is not set CONFIG_HYSDN=m CONFIG_HYSDN_CAPI=y CONFIG_MISDN=m CONFIG_MISDN_DSP=m CONFIG_MISDN_L1OIP=m # # mISDN hardware drivers # CONFIG_MISDN_HFCPCI=m CONFIG_MISDN_HFCMULTI=m CONFIG_MISDN_HFCUSB=m CONFIG_MISDN_AVMFRITZ=m CONFIG_MISDN_SPEEDFAX=m CONFIG_MISDN_INFINEON=m CONFIG_MISDN_W6692=m CONFIG_MISDN_NETJET=m CONFIG_MISDN_IPAC=m CONFIG_MISDN_ISAR=m CONFIG_ISDN_HDLC=m # CONFIG_PHONE is not set # # Input device support # CONFIG_INPUT=y CONFIG_INPUT_FF_MEMLESS=y CONFIG_INPUT_POLLDEV=m CONFIG_INPUT_SPARSEKMAP=m # # Userland interfaces # CONFIG_INPUT_MOUSEDEV=y # CONFIG_INPUT_MOUSEDEV_PSAUX is not set CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024 CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768 CONFIG_INPUT_JOYDEV=m CONFIG_INPUT_EVDEV=y # CONFIG_INPUT_EVBUG is not set CONFIG_XEN_KBDDEV_FRONTEND=y # # Input Device Drivers # CONFIG_INPUT_KEYBOARD=y CONFIG_KEYBOARD_ADP5588=m CONFIG_KEYBOARD_ATKBD=y CONFIG_KEYBOARD_QT2160=m # CONFIG_KEYBOARD_LKKBD is not set # CONFIG_KEYBOARD_GPIO is not set # CONFIG_KEYBOARD_GPIO_POLLED is not set # CONFIG_KEYBOARD_TCA6416 is not set # CONFIG_KEYBOARD_MATRIX is not set # CONFIG_KEYBOARD_LM8323 is not set CONFIG_KEYBOARD_MAX7359=m # CONFIG_KEYBOARD_MCS is not set # CONFIG_KEYBOARD_NEWTON is not set CONFIG_KEYBOARD_OPENCORES=m # CONFIG_KEYBOARD_STOWAWAY is not set # CONFIG_KEYBOARD_SUNKBD is not set # CONFIG_KEYBOARD_XTKBD is not set CONFIG_INPUT_MOUSE=y CONFIG_MOUSE_PS2=y CONFIG_MOUSE_PS2_ALPS=y CONFIG_MOUSE_PS2_LOGIPS2PP=y CONFIG_MOUSE_PS2_SYNAPTICS=y CONFIG_MOUSE_PS2_LIFEBOOK=y CONFIG_MOUSE_PS2_TRACKPOINT=y CONFIG_MOUSE_PS2_ELANTECH=y CONFIG_MOUSE_PS2_SENTELIC=y # CONFIG_MOUSE_PS2_TOUCHKIT is not set CONFIG_MOUSE_SERIAL=m CONFIG_MOUSE_APPLETOUCH=m CONFIG_MOUSE_BCM5974=m CONFIG_MOUSE_VSXXXAA=m # CONFIG_MOUSE_GPIO is not set CONFIG_MOUSE_SYNAPTICS_I2C=m CONFIG_INPUT_JOYSTICK=y CONFIG_JOYSTICK_ANALOG=m CONFIG_JOYSTICK_A3D=m CONFIG_JOYSTICK_ADI=m CONFIG_JOYSTICK_COBRA=m CONFIG_JOYSTICK_GF2K=m CONFIG_JOYSTICK_GRIP=m CONFIG_JOYSTICK_GRIP_MP=m CONFIG_JOYSTICK_GUILLEMOT=m CONFIG_JOYSTICK_INTERACT=m CONFIG_JOYSTICK_SIDEWINDER=m CONFIG_JOYSTICK_TMDC=m CONFIG_JOYSTICK_IFORCE=m CONFIG_JOYSTICK_IFORCE_USB=y CONFIG_JOYSTICK_IFORCE_232=y CONFIG_JOYSTICK_WARRIOR=m CONFIG_JOYSTICK_MAGELLAN=m CONFIG_JOYSTICK_SPACEORB=m CONFIG_JOYSTICK_SPACEBALL=m CONFIG_JOYSTICK_STINGER=m CONFIG_JOYSTICK_TWIDJOY=m CONFIG_JOYSTICK_ZHENHUA=m CONFIG_JOYSTICK_DB9=m CONFIG_JOYSTICK_GAMECON=m CONFIG_JOYSTICK_TURBOGRAFX=m # CONFIG_JOYSTICK_AS5011 is not set CONFIG_JOYSTICK_JOYDUMP=m CONFIG_JOYSTICK_XPAD=m CONFIG_JOYSTICK_XPAD_FF=y CONFIG_JOYSTICK_XPAD_LEDS=y CONFIG_JOYSTICK_WALKERA0701=m CONFIG_INPUT_TABLET=y CONFIG_TABLET_USB_ACECAD=m CONFIG_TABLET_USB_AIPTEK=m CONFIG_TABLET_USB_GTCO=m CONFIG_TABLET_USB_HANWANG=m CONFIG_TABLET_USB_KBTAB=m CONFIG_TABLET_USB_WACOM=m CONFIG_INPUT_TOUCHSCREEN=y # CONFIG_TOUCHSCREEN_AD7879 is not set # CONFIG_TOUCHSCREEN_BU21013 is not set # CONFIG_TOUCHSCREEN_CY8CTMG110 is not set CONFIG_TOUCHSCREEN_DYNAPRO=m # CONFIG_TOUCHSCREEN_HAMPSHIRE is not set CONFIG_TOUCHSCREEN_EETI=m CONFIG_TOUCHSCREEN_FUJITSU=m CONFIG_TOUCHSCREEN_GUNZE=m CONFIG_TOUCHSCREEN_ELO=m CONFIG_TOUCHSCREEN_WACOM_W8001=m CONFIG_TOUCHSCREEN_MCS5000=m CONFIG_TOUCHSCREEN_MTOUCH=m CONFIG_TOUCHSCREEN_INEXIO=m CONFIG_TOUCHSCREEN_MK712=m CONFIG_TOUCHSCREEN_PENMOUNT=m # CONFIG_TOUCHSCREEN_QT602240 is not set CONFIG_TOUCHSCREEN_TOUCHRIGHT=m CONFIG_TOUCHSCREEN_TOUCHWIN=m # CONFIG_TOUCHSCREEN_WM97XX is not set CONFIG_TOUCHSCREEN_USB_COMPOSITE=m CONFIG_TOUCHSCREEN_USB_EGALAX=y CONFIG_TOUCHSCREEN_USB_PANJIT=y CONFIG_TOUCHSCREEN_USB_3M=y CONFIG_TOUCHSCREEN_USB_ITM=y CONFIG_TOUCHSCREEN_USB_ETURBO=y CONFIG_TOUCHSCREEN_USB_GUNZE=y CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y CONFIG_TOUCHSCREEN_USB_IRTOUCH=y CONFIG_TOUCHSCREEN_USB_IDEALTEK=y CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y CONFIG_TOUCHSCREEN_USB_GOTOP=y CONFIG_TOUCHSCREEN_USB_JASTEC=y CONFIG_TOUCHSCREEN_USB_E2I=y CONFIG_TOUCHSCREEN_USB_ZYTRONIC=y CONFIG_TOUCHSCREEN_USB_ETT_TC45USB=y CONFIG_TOUCHSCREEN_USB_NEXIO=y CONFIG_TOUCHSCREEN_TOUCHIT213=m CONFIG_TOUCHSCREEN_TSC2007=m CONFIG_TOUCHSCREEN_ST1232=m # CONFIG_TOUCHSCREEN_TPS6507X is not set CONFIG_INPUT_MISC=y # CONFIG_INPUT_AD714X is not set CONFIG_INPUT_PCSPKR=m CONFIG_INPUT_APANEL=m CONFIG_INPUT_ATLAS_BTNS=m CONFIG_INPUT_ATI_REMOTE=m CONFIG_INPUT_ATI_REMOTE2=m CONFIG_INPUT_KEYSPAN_REMOTE=m CONFIG_INPUT_APPLEIR=m CONFIG_INPUT_POWERMATE=m CONFIG_INPUT_YEALINK=m CONFIG_INPUT_CM109=m CONFIG_INPUT_UINPUT=m # CONFIG_INPUT_PCF8574 is not set CONFIG_INPUT_GPIO_ROTARY_ENCODER=m # CONFIG_INPUT_ADXL34X is not set CONFIG_INPUT_CMA3000=m CONFIG_INPUT_CMA3000_I2C=m # # Hardware I/O ports # CONFIG_SERIO=y CONFIG_SERIO_I8042=y CONFIG_SERIO_SERPORT=y # CONFIG_SERIO_CT82C710 is not set # CONFIG_SERIO_PARKBD is not set # CONFIG_SERIO_PCIPS2 is not set CONFIG_SERIO_LIBPS2=y CONFIG_SERIO_RAW=m CONFIG_SERIO_ALTERA_PS2=m # CONFIG_SERIO_PS2MULT is not set CONFIG_GAMEPORT=m CONFIG_GAMEPORT_NS558=m CONFIG_GAMEPORT_L4=m CONFIG_GAMEPORT_EMU10K1=m CONFIG_GAMEPORT_FM801=m # # Character devices # CONFIG_VT=y CONFIG_CONSOLE_TRANSLATIONS=y CONFIG_VT_CONSOLE=y CONFIG_HW_CONSOLE=y CONFIG_VT_HW_CONSOLE_BINDING=y # CONFIG_DEVKMEM is not set CONFIG_SERIAL_NONSTANDARD=y # CONFIG_COMPUTONE is not set CONFIG_ROCKETPORT=m CONFIG_CYCLADES=m # CONFIG_CYZ_INTR is not set # CONFIG_DIGIEPCA is not set # CONFIG_MOXA_INTELLIO is not set # CONFIG_MOXA_SMARTIO is not set # CONFIG_ISI is not set CONFIG_SYNCLINK=m CONFIG_SYNCLINKMP=m CONFIG_SYNCLINK_GT=m CONFIG_N_HDLC=m CONFIG_N_GSM=m # CONFIG_RISCOM8 is not set # CONFIG_SPECIALIX is not set # CONFIG_STALDRV is not set CONFIG_NOZOMI=m # # Serial drivers # CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_FIX_EARLYCON_MEM=y CONFIG_SERIAL_8250_PCI=y CONFIG_SERIAL_8250_PNP=y CONFIG_SERIAL_8250_CS=m CONFIG_SERIAL_8250_NR_UARTS=32 CONFIG_SERIAL_8250_RUNTIME_UARTS=4 CONFIG_SERIAL_8250_EXTENDED=y CONFIG_SERIAL_8250_MANY_PORTS=y CONFIG_SERIAL_8250_SHARE_IRQ=y # CONFIG_SERIAL_8250_DETECT_IRQ is not set CONFIG_SERIAL_8250_RSA=y # # Non-8250 serial port support # # CONFIG_SERIAL_MFD_HSU is not set CONFIG_SERIAL_CORE=y CONFIG_SERIAL_CORE_CONSOLE=y CONFIG_CONSOLE_POLL=y CONFIG_SERIAL_JSM=m # CONFIG_SERIAL_TIMBERDALE is not set # CONFIG_SERIAL_ALTERA_JTAGUART is not set # CONFIG_SERIAL_ALTERA_UART is not set # CONFIG_SERIAL_PCH_UART is not set CONFIG_UNIX98_PTYS=y CONFIG_DEVPTS_MULTIPLE_INSTANCES=y # CONFIG_LEGACY_PTYS is not set CONFIG_CRASH=m CONFIG_PRINTER=m CONFIG_LP_CONSOLE=y CONFIG_PPDEV=m CONFIG_HVC_DRIVER=y CONFIG_HVC_IRQ=y CONFIG_HVC_XEN=y CONFIG_VIRTIO_CONSOLE=y CONFIG_IPMI_HANDLER=m # CONFIG_IPMI_PANIC_EVENT is not set CONFIG_IPMI_DEVICE_INTERFACE=m CONFIG_IPMI_SI=m CONFIG_IPMI_WATCHDOG=m CONFIG_IPMI_POWEROFF=m CONFIG_HW_RANDOM=y CONFIG_HW_RANDOM_TIMERIOMEM=m CONFIG_HW_RANDOM_INTEL=m CONFIG_HW_RANDOM_AMD=m CONFIG_HW_RANDOM_VIA=m CONFIG_HW_RANDOM_VIRTIO=m CONFIG_NVRAM=y CONFIG_R3964=m # CONFIG_APPLICOM is not set # # PCMCIA character devices # # CONFIG_SYNCLINK_CS is not set CONFIG_CARDMAN_4000=m CONFIG_CARDMAN_4040=m CONFIG_IPWIRELESS=m CONFIG_MWAVE=m CONFIG_RAW_DRIVER=y CONFIG_MAX_RAW_DEVS=8192 CONFIG_HPET=y # CONFIG_HPET_MMAP is not set CONFIG_HANGCHECK_TIMER=m CONFIG_TCG_TPM=y CONFIG_TCG_TIS=y CONFIG_TCG_NSC=m CONFIG_TCG_ATMEL=m CONFIG_TCG_INFINEON=m CONFIG_TELCLOCK=m CONFIG_DEVPORT=y # CONFIG_RAMOOPS is not set CONFIG_I2C=m CONFIG_I2C_BOARDINFO=y CONFIG_I2C_COMPAT=y CONFIG_I2C_CHARDEV=m # CONFIG_I2C_MUX is not set CONFIG_I2C_HELPER_AUTO=y CONFIG_I2C_SMBUS=m CONFIG_I2C_ALGOBIT=m CONFIG_I2C_ALGOPCA=m # # I2C Hardware Bus support # # # PC SMBus host controller drivers # # CONFIG_I2C_ALI1535 is not set # CONFIG_I2C_ALI1563 is not set # CONFIG_I2C_ALI15X3 is not set CONFIG_I2C_AMD756=m CONFIG_I2C_AMD756_S4882=m CONFIG_I2C_AMD8111=m CONFIG_I2C_I801=m CONFIG_I2C_ISCH=m CONFIG_I2C_PIIX4=m CONFIG_I2C_NFORCE2=m CONFIG_I2C_NFORCE2_S4985=m # CONFIG_I2C_SIS5595 is not set # CONFIG_I2C_SIS630 is not set CONFIG_I2C_SIS96X=m CONFIG_I2C_VIA=m CONFIG_I2C_VIAPRO=m # # ACPI drivers # CONFIG_I2C_SCMI=m # # I2C system bus drivers (mostly embedded / system-on-chip) # # CONFIG_I2C_GPIO is not set # CONFIG_I2C_INTEL_MID is not set # CONFIG_I2C_OCORES is not set CONFIG_I2C_PCA_PLATFORM=m CONFIG_I2C_SIMTEC=m # CONFIG_I2C_XILINX is not set # CONFIG_I2C_EG20T is not set # # External I2C/SMBus adapter drivers # CONFIG_I2C_PARPORT=m CONFIG_I2C_PARPORT_LIGHT=m # CONFIG_I2C_TAOS_EVM is not set CONFIG_I2C_TINY_USB=m # # Other I2C/SMBus bus drivers # CONFIG_I2C_STUB=m # CONFIG_I2C_DEBUG_CORE is not set # CONFIG_I2C_DEBUG_ALGO is not set # CONFIG_I2C_DEBUG_BUS is not set # CONFIG_SPI is not set # # PPS support # CONFIG_PPS=m # CONFIG_PPS_DEBUG is not set # # PPS clients support # # CONFIG_PPS_CLIENT_KTIMER is not set CONFIG_PPS_CLIENT_LDISC=m CONFIG_PPS_CLIENT_PARPORT=m # # PPS generators support # CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y CONFIG_GPIOLIB=y # CONFIG_DEBUG_GPIO is not set CONFIG_GPIO_SYSFS=y # # Memory mapped GPIO expanders: # # CONFIG_GPIO_BASIC_MMIO is not set # CONFIG_GPIO_IT8761E is not set CONFIG_GPIO_SCH=m # CONFIG_GPIO_VX855 is not set # # I2C GPIO expanders: # # CONFIG_GPIO_MAX7300 is not set # CONFIG_GPIO_MAX732X is not set # CONFIG_GPIO_PCA953X is not set # CONFIG_GPIO_PCF857X is not set # CONFIG_GPIO_ADP5588 is not set # # PCI GPIO expanders: # # CONFIG_GPIO_CS5535 is not set CONFIG_GPIO_LANGWELL=y # CONFIG_GPIO_PCH is not set # CONFIG_GPIO_ML_IOH is not set # CONFIG_GPIO_RDC321X is not set # # SPI GPIO expanders: # # # AC97 GPIO expanders: # # # MODULbus GPIO expanders: # CONFIG_W1=m CONFIG_W1_CON=y # # 1-wire Bus Masters # # CONFIG_W1_MASTER_MATROX is not set CONFIG_W1_MASTER_DS2490=m CONFIG_W1_MASTER_DS2482=m # CONFIG_W1_MASTER_GPIO is not set # # 1-wire Slaves # CONFIG_W1_SLAVE_THERM=m CONFIG_W1_SLAVE_SMEM=m CONFIG_W1_SLAVE_DS2423=m CONFIG_W1_SLAVE_DS2431=m CONFIG_W1_SLAVE_DS2433=m CONFIG_W1_SLAVE_DS2433_CRC=y CONFIG_W1_SLAVE_DS2760=m CONFIG_W1_SLAVE_BQ27000=m CONFIG_POWER_SUPPLY=y # CONFIG_POWER_SUPPLY_DEBUG is not set # CONFIG_PDA_POWER is not set # CONFIG_TEST_POWER is not set # CONFIG_BATTERY_DS2760 is not set # CONFIG_BATTERY_DS2782 is not set # CONFIG_BATTERY_BQ20Z75 is not set # CONFIG_BATTERY_BQ27x00 is not set # CONFIG_BATTERY_MAX17040 is not set # CONFIG_BATTERY_MAX17042 is not set # CONFIG_CHARGER_ISP1704 is not set # CONFIG_CHARGER_GPIO is not set CONFIG_HWMON=y CONFIG_HWMON_VID=m # CONFIG_HWMON_DEBUG_CHIP is not set # # Native drivers # CONFIG_SENSORS_ABITUGURU=m CONFIG_SENSORS_ABITUGURU3=m CONFIG_SENSORS_AD7414=m CONFIG_SENSORS_AD7418=m CONFIG_SENSORS_ADM1021=m CONFIG_SENSORS_ADM1025=m CONFIG_SENSORS_ADM1026=m CONFIG_SENSORS_ADM1029=m CONFIG_SENSORS_ADM1031=m CONFIG_SENSORS_ADM9240=m CONFIG_SENSORS_ADT7411=m CONFIG_SENSORS_ADT7462=m CONFIG_SENSORS_ADT7470=m CONFIG_SENSORS_ADT7475=m CONFIG_SENSORS_ASC7621=m CONFIG_SENSORS_K8TEMP=m CONFIG_SENSORS_K10TEMP=m CONFIG_SENSORS_ASB100=m CONFIG_SENSORS_ATXP1=m CONFIG_SENSORS_DS620=m CONFIG_SENSORS_DS1621=m CONFIG_SENSORS_I5K_AMB=m CONFIG_SENSORS_F71805F=m CONFIG_SENSORS_F71882FG=m CONFIG_SENSORS_F75375S=m CONFIG_SENSORS_FSCHMD=m CONFIG_SENSORS_G760A=m CONFIG_SENSORS_GL518SM=m CONFIG_SENSORS_GL520SM=m # CONFIG_SENSORS_GPIO_FAN is not set CONFIG_SENSORS_CORETEMP=m CONFIG_SENSORS_PKGTEMP=m CONFIG_SENSORS_IBMAEM=m CONFIG_SENSORS_IBMPEX=m CONFIG_SENSORS_IT87=m # CONFIG_SENSORS_JC42 is not set CONFIG_SENSORS_LM63=m CONFIG_SENSORS_LM73=m CONFIG_SENSORS_LM75=m CONFIG_SENSORS_LM77=m CONFIG_SENSORS_LM78=m CONFIG_SENSORS_LM80=m CONFIG_SENSORS_LM83=m CONFIG_SENSORS_LM85=m CONFIG_SENSORS_LM87=m CONFIG_SENSORS_LM90=m CONFIG_SENSORS_LM92=m CONFIG_SENSORS_LM93=m CONFIG_SENSORS_LTC4215=m CONFIG_SENSORS_LTC4245=m CONFIG_SENSORS_LTC4261=m CONFIG_SENSORS_LM95241=m CONFIG_SENSORS_MAX1619=m CONFIG_SENSORS_MAX6650=m CONFIG_SENSORS_PC87360=m CONFIG_SENSORS_PC87427=m CONFIG_SENSORS_PCF8591=m CONFIG_SENSORS_SHT15=m CONFIG_SENSORS_SHT21=m CONFIG_SENSORS_SIS5595=m # CONFIG_SENSORS_SMM665 is not set CONFIG_SENSORS_DME1737=m CONFIG_SENSORS_EMC1403=m # CONFIG_SENSORS_EMC2103 is not set CONFIG_SENSORS_SMSC47M1=m CONFIG_SENSORS_SMSC47M192=m CONFIG_SENSORS_SMSC47B397=m CONFIG_SENSORS_ADS7828=m CONFIG_SENSORS_AMC6821=m CONFIG_SENSORS_THMC50=m CONFIG_SENSORS_TMP102=m CONFIG_SENSORS_TMP401=m CONFIG_SENSORS_TMP421=m CONFIG_SENSORS_VIA_CPUTEMP=m CONFIG_SENSORS_VIA686A=m CONFIG_SENSORS_VT1211=m CONFIG_SENSORS_VT8231=m CONFIG_SENSORS_W83781D=m CONFIG_SENSORS_W83791D=m CONFIG_SENSORS_W83792D=m CONFIG_SENSORS_W83793=m CONFIG_SENSORS_W83795=m # CONFIG_SENSORS_W83795_FANCTRL is not set CONFIG_SENSORS_W83L785TS=m CONFIG_SENSORS_W83L786NG=m CONFIG_SENSORS_W83627HF=m CONFIG_SENSORS_W83627EHF=m CONFIG_SENSORS_LIS3_I2C=m CONFIG_SENSORS_APPLESMC=m # # ACPI drivers # CONFIG_SENSORS_ATK0110=m CONFIG_SENSORS_LIS3LV02D=m CONFIG_THERMAL=y CONFIG_THERMAL_HWMON=y CONFIG_WATCHDOG=y # CONFIG_WATCHDOG_NOWAYOUT is not set # # Watchdog Device Drivers # CONFIG_SOFT_WATCHDOG=m # CONFIG_ACQUIRE_WDT is not set # CONFIG_ADVANTECH_WDT is not set CONFIG_ALIM1535_WDT=m CONFIG_ALIM7101_WDT=m CONFIG_F71808E_WDT=m CONFIG_SP5100_TCO=m CONFIG_GEODE_WDT=m # CONFIG_SC520_WDT is not set CONFIG_SBC_FITPC2_WATCHDOG=m # CONFIG_EUROTECH_WDT is not set CONFIG_IB700_WDT=m CONFIG_IBMASR=m # CONFIG_WAFER_WDT is not set CONFIG_I6300ESB_WDT=m CONFIG_ITCO_WDT=m CONFIG_ITCO_VENDOR_SUPPORT=y CONFIG_IT8712F_WDT=m CONFIG_IT87_WDT=m CONFIG_HP_WATCHDOG=m CONFIG_HPWDT_NMI_DECODING=y # CONFIG_SC1200_WDT is not set # CONFIG_PC87413_WDT is not set CONFIG_NV_TCO=m # CONFIG_60XX_WDT is not set # CONFIG_SBC8360_WDT is not set # CONFIG_CPU5_WDT is not set CONFIG_SMSC_SCH311X_WDT=m # CONFIG_SMSC37B787_WDT is not set CONFIG_W83627HF_WDT=m CONFIG_W83697HF_WDT=m CONFIG_W83697UG_WDT=m CONFIG_W83877F_WDT=m CONFIG_W83977F_WDT=m CONFIG_MACHZ_WDT=m # CONFIG_SBC_EPX_C3_WATCHDOG is not set # # PCI-based Watchdog Cards # CONFIG_PCIPCWATCHDOG=m CONFIG_WDTPCI=m # # USB-based Watchdog Cards # CONFIG_USBPCWATCHDOG=m CONFIG_SSB_POSSIBLE=y # # Sonics Silicon Backplane # CONFIG_SSB=m CONFIG_SSB_SPROM=y CONFIG_SSB_BLOCKIO=y CONFIG_SSB_PCIHOST_POSSIBLE=y CONFIG_SSB_PCIHOST=y CONFIG_SSB_B43_PCI_BRIDGE=y CONFIG_SSB_PCMCIAHOST_POSSIBLE=y CONFIG_SSB_PCMCIAHOST=y CONFIG_SSB_SDIOHOST_POSSIBLE=y CONFIG_SSB_SDIOHOST=y # CONFIG_SSB_DEBUG is not set CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y CONFIG_SSB_DRIVER_PCICORE=y CONFIG_MFD_SUPPORT=y CONFIG_MFD_CORE=m CONFIG_MFD_SM501=m CONFIG_MFD_SM501_GPIO=y # CONFIG_HTC_PASIC3 is not set # CONFIG_UCB1400_CORE is not set # CONFIG_TPS65010 is not set # CONFIG_TPS6507X is not set # CONFIG_MFD_TMIO is not set CONFIG_MFD_WM8400=m # CONFIG_MFD_PCF50633 is not set # CONFIG_ABX500_CORE is not set CONFIG_MFD_CS5535=m # CONFIG_MFD_TIMBERDALE is not set CONFIG_LPC_SCH=m # CONFIG_MFD_RDC321X is not set # CONFIG_MFD_JANZ_CMODIO is not set CONFIG_MFD_VX855=m CONFIG_MFD_WL1273_CORE=m # CONFIG_REGULATOR is not set CONFIG_MEDIA_SUPPORT=m # # Multimedia core support # CONFIG_VIDEO_DEV=m CONFIG_VIDEO_V4L2_COMMON=m CONFIG_DVB_CORE=m CONFIG_VIDEO_MEDIA=m # # Multimedia drivers # CONFIG_VIDEO_SAA7146=m CONFIG_VIDEO_SAA7146_VV=m CONFIG_RC_CORE=m CONFIG_LIRC=m CONFIG_RC_MAP=m CONFIG_IR_NEC_DECODER=m CONFIG_IR_RC5_DECODER=m CONFIG_IR_RC6_DECODER=m CONFIG_IR_JVC_DECODER=m CONFIG_IR_SONY_DECODER=m CONFIG_IR_RC5_SZ_DECODER=m CONFIG_IR_LIRC_CODEC=m CONFIG_IR_ENE=m CONFIG_IR_IMON=m CONFIG_IR_MCEUSB=m CONFIG_IR_ITE_CIR=m CONFIG_IR_NUVOTON=m CONFIG_IR_STREAMZAP=m CONFIG_IR_WINBOND_CIR=m CONFIG_RC_LOOPBACK=m CONFIG_MEDIA_ATTACH=y CONFIG_MEDIA_TUNER=m CONFIG_MEDIA_TUNER_CUSTOMISE=y # # Customize TV tuners # CONFIG_MEDIA_TUNER_SIMPLE=m CONFIG_MEDIA_TUNER_TDA8290=m CONFIG_MEDIA_TUNER_TDA827X=m CONFIG_MEDIA_TUNER_TDA18271=m CONFIG_MEDIA_TUNER_TDA9887=m CONFIG_MEDIA_TUNER_TEA5761=m CONFIG_MEDIA_TUNER_TEA5767=m CONFIG_MEDIA_TUNER_MT20XX=m CONFIG_MEDIA_TUNER_MT2060=m CONFIG_MEDIA_TUNER_MT2266=m CONFIG_MEDIA_TUNER_MT2131=m CONFIG_MEDIA_TUNER_QT1010=m CONFIG_MEDIA_TUNER_XC2028=m CONFIG_MEDIA_TUNER_XC5000=m CONFIG_MEDIA_TUNER_MXL5005S=m CONFIG_MEDIA_TUNER_MXL5007T=m CONFIG_MEDIA_TUNER_MC44S803=m CONFIG_MEDIA_TUNER_MAX2165=m CONFIG_MEDIA_TUNER_TDA18218=m CONFIG_VIDEO_V4L2=m CONFIG_VIDEOBUF_GEN=m CONFIG_VIDEOBUF_DMA_SG=m CONFIG_VIDEOBUF_VMALLOC=m CONFIG_VIDEOBUF_DVB=m CONFIG_VIDEO_BTCX=m CONFIG_VIDEO_TVEEPROM=m CONFIG_VIDEO_TUNER=m CONFIG_VIDEO_CAPTURE_DRIVERS=y # CONFIG_VIDEO_ADV_DEBUG is not set # CONFIG_VIDEO_FIXED_MINOR_RANGES is not set CONFIG_VIDEO_HELPER_CHIPS_AUTO=y CONFIG_VIDEO_IR_I2C=m # # Audio decoders # CONFIG_VIDEO_TVAUDIO=m CONFIG_VIDEO_TDA7432=m CONFIG_VIDEO_TDA9840=m CONFIG_VIDEO_TEA6415C=m CONFIG_VIDEO_TEA6420=m CONFIG_VIDEO_MSP3400=m CONFIG_VIDEO_CS5345=m CONFIG_VIDEO_CS53L32A=m CONFIG_VIDEO_M52790=m CONFIG_VIDEO_WM8775=m CONFIG_VIDEO_WM8739=m CONFIG_VIDEO_VP27SMPX=m # # RDS decoders # CONFIG_VIDEO_SAA6588=m # # Video decoders # CONFIG_VIDEO_BT819=m CONFIG_VIDEO_BT856=m CONFIG_VIDEO_BT866=m CONFIG_VIDEO_KS0127=m CONFIG_VIDEO_OV7670=m CONFIG_VIDEO_MT9V011=m CONFIG_VIDEO_SAA7110=m CONFIG_VIDEO_SAA711X=m CONFIG_VIDEO_SAA717X=m CONFIG_VIDEO_TVP5150=m CONFIG_VIDEO_VPX3220=m # # Video and audio decoders # CONFIG_VIDEO_CX25840=m # # MPEG video encoders # CONFIG_VIDEO_CX2341X=m # # Video encoders # CONFIG_VIDEO_SAA7127=m CONFIG_VIDEO_SAA7185=m CONFIG_VIDEO_ADV7170=m CONFIG_VIDEO_ADV7175=m # # Video improvement chips # CONFIG_VIDEO_UPD64031A=m CONFIG_VIDEO_UPD64083=m # CONFIG_VIDEO_VIVI is not set CONFIG_VIDEO_BT848=m CONFIG_VIDEO_BT848_DVB=y CONFIG_VIDEO_BWQCAM=m CONFIG_VIDEO_CQCAM=m CONFIG_VIDEO_W9966=m CONFIG_VIDEO_CPIA2=m CONFIG_VIDEO_ZORAN=m CONFIG_VIDEO_ZORAN_DC30=m CONFIG_VIDEO_ZORAN_ZR36060=m CONFIG_VIDEO_ZORAN_BUZ=m CONFIG_VIDEO_ZORAN_DC10=m CONFIG_VIDEO_ZORAN_LML33=m CONFIG_VIDEO_ZORAN_LML33R10=m CONFIG_VIDEO_ZORAN_AVS6EYES=m CONFIG_VIDEO_MEYE=m CONFIG_VIDEO_SAA7134=m CONFIG_VIDEO_SAA7134_ALSA=m CONFIG_VIDEO_SAA7134_RC=y CONFIG_VIDEO_SAA7134_DVB=m CONFIG_VIDEO_MXB=m CONFIG_VIDEO_HEXIUM_ORION=m CONFIG_VIDEO_HEXIUM_GEMINI=m # CONFIG_VIDEO_TIMBERDALE is not set CONFIG_VIDEO_CX88=m CONFIG_VIDEO_CX88_ALSA=m CONFIG_VIDEO_CX88_BLACKBIRD=m CONFIG_VIDEO_CX88_DVB=m CONFIG_VIDEO_CX88_MPEG=m CONFIG_VIDEO_CX88_VP3054=m CONFIG_VIDEO_CX23885=m CONFIG_VIDEO_AU0828=m CONFIG_VIDEO_IVTV=m CONFIG_VIDEO_FB_IVTV=m CONFIG_VIDEO_CX18=m CONFIG_VIDEO_CX18_ALSA=m CONFIG_VIDEO_SAA7164=m # CONFIG_VIDEO_CAFE_CCIC is not set CONFIG_VIDEO_SR030PC30=m CONFIG_VIDEO_VIA_CAMERA=m CONFIG_SOC_CAMERA=m CONFIG_SOC_CAMERA_IMX074=m CONFIG_SOC_CAMERA_MT9M001=m CONFIG_SOC_CAMERA_MT9M111=m CONFIG_SOC_CAMERA_MT9T031=m CONFIG_SOC_CAMERA_MT9T112=m CONFIG_SOC_CAMERA_MT9V022=m CONFIG_SOC_CAMERA_RJ54N1=m CONFIG_SOC_CAMERA_TW9910=m CONFIG_SOC_CAMERA_PLATFORM=m CONFIG_SOC_CAMERA_OV2640=m CONFIG_SOC_CAMERA_OV6650=m CONFIG_SOC_CAMERA_OV772X=m CONFIG_SOC_CAMERA_OV9640=m CONFIG_V4L_USB_DRIVERS=y CONFIG_USB_VIDEO_CLASS=m CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y CONFIG_USB_GSPCA=m CONFIG_USB_M5602=m CONFIG_USB_STV06XX=m CONFIG_USB_GL860=m CONFIG_USB_GSPCA_BENQ=m CONFIG_USB_GSPCA_CONEX=m CONFIG_USB_GSPCA_CPIA1=m CONFIG_USB_GSPCA_ETOMS=m CONFIG_USB_GSPCA_FINEPIX=m CONFIG_USB_GSPCA_JEILINJ=m CONFIG_USB_GSPCA_KONICA=m CONFIG_USB_GSPCA_MARS=m CONFIG_USB_GSPCA_MR97310A=m CONFIG_USB_GSPCA_OV519=m CONFIG_USB_GSPCA_OV534=m CONFIG_USB_GSPCA_OV534_9=m CONFIG_USB_GSPCA_PAC207=m CONFIG_USB_GSPCA_PAC7302=m CONFIG_USB_GSPCA_PAC7311=m CONFIG_USB_GSPCA_SN9C2028=m CONFIG_USB_GSPCA_SN9C20X=m CONFIG_USB_GSPCA_SONIXB=m CONFIG_USB_GSPCA_SONIXJ=m CONFIG_USB_GSPCA_SPCA500=m CONFIG_USB_GSPCA_SPCA501=m CONFIG_USB_GSPCA_SPCA505=m CONFIG_USB_GSPCA_SPCA506=m CONFIG_USB_GSPCA_SPCA508=m CONFIG_USB_GSPCA_SPCA561=m CONFIG_USB_GSPCA_SPCA1528=m CONFIG_USB_GSPCA_SQ905=m CONFIG_USB_GSPCA_SQ905C=m CONFIG_USB_GSPCA_SQ930X=m CONFIG_USB_GSPCA_STK014=m CONFIG_USB_GSPCA_STV0680=m CONFIG_USB_GSPCA_SUNPLUS=m CONFIG_USB_GSPCA_T613=m CONFIG_USB_GSPCA_TV8532=m CONFIG_USB_GSPCA_VC032X=m CONFIG_USB_GSPCA_XIRLINK_CIT=m CONFIG_USB_GSPCA_ZC3XX=m CONFIG_VIDEO_PVRUSB2=m CONFIG_VIDEO_PVRUSB2_SYSFS=y CONFIG_VIDEO_PVRUSB2_DVB=y # CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set CONFIG_VIDEO_HDPVR=m CONFIG_VIDEO_EM28XX=m CONFIG_VIDEO_EM28XX_ALSA=m CONFIG_VIDEO_EM28XX_DVB=m CONFIG_VIDEO_TLG2300=m CONFIG_VIDEO_CX231XX=m CONFIG_VIDEO_CX231XX_RC=y CONFIG_VIDEO_CX231XX_ALSA=m CONFIG_VIDEO_CX231XX_DVB=m CONFIG_VIDEO_USBVISION=m # CONFIG_USB_ET61X251 is not set # CONFIG_USB_SN9C102 is not set CONFIG_USB_PWC=m # CONFIG_USB_PWC_DEBUG is not set CONFIG_USB_PWC_INPUT_EVDEV=y CONFIG_USB_ZR364XX=m CONFIG_USB_STKWEBCAM=m CONFIG_USB_S2255=m CONFIG_V4L_MEM2MEM_DRIVERS=y # CONFIG_VIDEO_MEM2MEM_TESTDEV is not set CONFIG_RADIO_ADAPTERS=y CONFIG_RADIO_MAXIRADIO=m CONFIG_RADIO_MAESTRO=m CONFIG_I2C_SI4713=m CONFIG_RADIO_SI4713=m CONFIG_USB_DSBR=m CONFIG_RADIO_SI470X=y CONFIG_USB_SI470X=m CONFIG_I2C_SI470X=m CONFIG_USB_MR800=m # CONFIG_RADIO_TEA5764 is not set # CONFIG_RADIO_SAA7706H is not set # CONFIG_RADIO_TEF6862 is not set CONFIG_RADIO_WL1273=m CONFIG_DVB_MAX_ADAPTERS=8 CONFIG_DVB_DYNAMIC_MINORS=y CONFIG_DVB_CAPTURE_DRIVERS=y # # Supported SAA7146 based PCI Adapters # CONFIG_TTPCI_EEPROM=m CONFIG_DVB_AV7110=m CONFIG_DVB_AV7110_OSD=y CONFIG_DVB_BUDGET_CORE=m CONFIG_DVB_BUDGET=m CONFIG_DVB_BUDGET_CI=m CONFIG_DVB_BUDGET_AV=m CONFIG_DVB_BUDGET_PATCH=m # # Supported USB Adapters # CONFIG_DVB_USB=m # CONFIG_DVB_USB_DEBUG is not set CONFIG_DVB_USB_A800=m CONFIG_DVB_USB_DIBUSB_MB=m # CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set CONFIG_DVB_USB_DIBUSB_MC=m CONFIG_DVB_USB_DIB0700=m CONFIG_DVB_USB_UMT_010=m CONFIG_DVB_USB_CXUSB=m CONFIG_DVB_USB_M920X=m CONFIG_DVB_USB_GL861=m CONFIG_DVB_USB_AU6610=m CONFIG_DVB_USB_DIGITV=m CONFIG_DVB_USB_VP7045=m CONFIG_DVB_USB_VP702X=m CONFIG_DVB_USB_GP8PSK=m CONFIG_DVB_USB_NOVA_T_USB2=m CONFIG_DVB_USB_TTUSB2=m CONFIG_DVB_USB_DTT200U=m CONFIG_DVB_USB_OPERA1=m CONFIG_DVB_USB_AF9005=m CONFIG_DVB_USB_AF9005_REMOTE=m CONFIG_DVB_USB_DW2102=m CONFIG_DVB_USB_CINERGY_T2=m CONFIG_DVB_USB_ANYSEE=m CONFIG_DVB_USB_DTV5100=m CONFIG_DVB_USB_AF9015=m CONFIG_DVB_USB_CE6230=m CONFIG_DVB_USB_FRIIO=m CONFIG_DVB_USB_EC168=m CONFIG_DVB_USB_AZ6027=m CONFIG_DVB_USB_LME2510=m CONFIG_DVB_TTUSB_BUDGET=m CONFIG_DVB_TTUSB_DEC=m CONFIG_SMS_SIANO_MDTV=m # # Siano module components # CONFIG_SMS_USB_DRV=m CONFIG_SMS_SDIO_DRV=m # # Supported FlexCopII (B2C2) Adapters # CONFIG_DVB_B2C2_FLEXCOP=m CONFIG_DVB_B2C2_FLEXCOP_PCI=m CONFIG_DVB_B2C2_FLEXCOP_USB=m # CONFIG_DVB_B2C2_FLEXCOP_DEBUG is not set # # Supported BT878 Adapters # CONFIG_DVB_BT8XX=m # # Supported Pluto2 Adapters # CONFIG_DVB_PLUTO2=m # # Supported SDMC DM1105 Adapters # CONFIG_DVB_DM1105=m CONFIG_DVB_FIREDTV=m CONFIG_DVB_FIREDTV_FIREWIRE=y # CONFIG_DVB_FIREDTV_IEEE1394 is not set CONFIG_DVB_FIREDTV_INPUT=y # # Supported Earthsoft PT1 Adapters # CONFIG_DVB_PT1=m # # Supported Mantis Adapters # CONFIG_MANTIS_CORE=m CONFIG_DVB_MANTIS=m CONFIG_DVB_HOPPER=m # # Supported nGene Adapters # CONFIG_DVB_NGENE=m # # Supported DVB Frontends # CONFIG_DVB_FE_CUSTOMISE=y # # Customise DVB Frontends # # # Multistandard (satellite) frontends # CONFIG_DVB_STB0899=m CONFIG_DVB_STB6100=m CONFIG_DVB_STV090x=m CONFIG_DVB_STV6110x=m # # DVB-S (satellite) frontends # CONFIG_DVB_CX24110=m CONFIG_DVB_CX24123=m CONFIG_DVB_MT312=m CONFIG_DVB_ZL10036=m CONFIG_DVB_ZL10039=m CONFIG_DVB_S5H1420=m CONFIG_DVB_STV0288=m CONFIG_DVB_STB6000=m CONFIG_DVB_STV0299=m CONFIG_DVB_STV6110=m CONFIG_DVB_STV0900=m CONFIG_DVB_TDA8083=m CONFIG_DVB_TDA10086=m CONFIG_DVB_TDA8261=m CONFIG_DVB_VES1X93=m CONFIG_DVB_TUNER_ITD1000=m CONFIG_DVB_TUNER_CX24113=m CONFIG_DVB_TDA826X=m CONFIG_DVB_TUA6100=m CONFIG_DVB_CX24116=m CONFIG_DVB_SI21XX=m CONFIG_DVB_DS3000=m CONFIG_DVB_MB86A16=m # # DVB-T (terrestrial) frontends # CONFIG_DVB_SP8870=m CONFIG_DVB_SP887X=m CONFIG_DVB_CX22700=m CONFIG_DVB_CX22702=m CONFIG_DVB_S5H1432=m CONFIG_DVB_DRX397XD=m CONFIG_DVB_L64781=m CONFIG_DVB_TDA1004X=m CONFIG_DVB_NXT6000=m CONFIG_DVB_MT352=m CONFIG_DVB_ZL10353=m CONFIG_DVB_DIB3000MB=m CONFIG_DVB_DIB3000MC=m CONFIG_DVB_DIB7000M=m CONFIG_DVB_DIB7000P=m CONFIG_DVB_TDA10048=m CONFIG_DVB_AF9013=m CONFIG_DVB_EC100=m # # DVB-C (cable) frontends # CONFIG_DVB_VES1820=m CONFIG_DVB_TDA10021=m CONFIG_DVB_TDA10023=m CONFIG_DVB_STV0297=m # # ATSC (North American/Korean Terrestrial/Cable DTV) frontends # CONFIG_DVB_NXT200X=m CONFIG_DVB_OR51211=m CONFIG_DVB_OR51132=m CONFIG_DVB_BCM3510=m CONFIG_DVB_LGDT330X=m CONFIG_DVB_LGDT3305=m CONFIG_DVB_S5H1409=m CONFIG_DVB_AU8522=m CONFIG_DVB_S5H1411=m # # ISDB-T (terrestrial) frontends # CONFIG_DVB_S921=m CONFIG_DVB_DIB8000=m CONFIG_DVB_MB86A20S=m # # Digital terrestrial only tuners/PLL # CONFIG_DVB_PLL=m CONFIG_DVB_TUNER_DIB0070=m CONFIG_DVB_TUNER_DIB0090=m # # SEC control devices for DVB-S # CONFIG_DVB_LNBP21=m CONFIG_DVB_ISL6405=m CONFIG_DVB_ISL6421=m CONFIG_DVB_ISL6423=m CONFIG_DVB_LGS8GL5=m CONFIG_DVB_LGS8GXX=m CONFIG_DVB_ATBM8830=m CONFIG_DVB_TDA665x=m CONFIG_DVB_IX2505V=m # # Tools to develop new frontends # CONFIG_DVB_DUMMY_FE=m # # Graphics support # CONFIG_AGP=y CONFIG_AGP_AMD64=y CONFIG_AGP_INTEL=y CONFIG_AGP_SIS=y CONFIG_AGP_VIA=y CONFIG_VGA_ARB=y CONFIG_VGA_ARB_MAX_GPUS=16 CONFIG_VGA_SWITCHEROO=y CONFIG_DRM=m CONFIG_DRM_KMS_HELPER=m CONFIG_DRM_TTM=m CONFIG_DRM_TDFX=m CONFIG_DRM_R128=m CONFIG_DRM_RADEON=m CONFIG_DRM_RADEON_KMS=y CONFIG_DRM_I810=m # CONFIG_DRM_I830 is not set CONFIG_DRM_I915=m CONFIG_DRM_I915_KMS=y CONFIG_DRM_MGA=m CONFIG_DRM_SIS=m CONFIG_DRM_VIA=m CONFIG_DRM_SAVAGE=m CONFIG_STUB_POULSBO=m CONFIG_VGASTATE=m CONFIG_VIDEO_OUTPUT_CONTROL=m CONFIG_FB=y # CONFIG_FIRMWARE_EDID is not set CONFIG_FB_DDC=m CONFIG_FB_BOOT_VESA_SUPPORT=y CONFIG_FB_CFB_FILLRECT=y CONFIG_FB_CFB_COPYAREA=y CONFIG_FB_CFB_IMAGEBLIT=y # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set CONFIG_FB_SYS_FILLRECT=y CONFIG_FB_SYS_COPYAREA=y CONFIG_FB_SYS_IMAGEBLIT=y # CONFIG_FB_FOREIGN_ENDIAN is not set CONFIG_FB_SYS_FOPS=y # CONFIG_FB_WMT_GE_ROPS is not set CONFIG_FB_DEFERRED_IO=y CONFIG_FB_SVGALIB=m # CONFIG_FB_MACMODES is not set CONFIG_FB_BACKLIGHT=y CONFIG_FB_MODE_HELPERS=y CONFIG_FB_TILEBLITTING=y # # Frame buffer hardware drivers # CONFIG_FB_CIRRUS=m # CONFIG_FB_PM2 is not set # CONFIG_FB_CYBER2000 is not set # CONFIG_FB_ARC is not set # CONFIG_FB_ASILIANT is not set # CONFIG_FB_IMSTT is not set CONFIG_FB_VGA16=m # CONFIG_FB_UVESA is not set CONFIG_FB_VESA=y CONFIG_FB_EFI=y # CONFIG_FB_N411 is not set # CONFIG_FB_HGA is not set # CONFIG_FB_S1D13XXX is not set CONFIG_FB_NVIDIA=m CONFIG_FB_NVIDIA_I2C=y # CONFIG_FB_NVIDIA_DEBUG is not set CONFIG_FB_NVIDIA_BACKLIGHT=y CONFIG_FB_RIVA=m # CONFIG_FB_RIVA_I2C is not set # CONFIG_FB_RIVA_DEBUG is not set CONFIG_FB_RIVA_BACKLIGHT=y # CONFIG_FB_LE80578 is not set CONFIG_FB_MATROX=m CONFIG_FB_MATROX_MILLENIUM=y CONFIG_FB_MATROX_MYSTIQUE=y CONFIG_FB_MATROX_G=y CONFIG_FB_MATROX_I2C=m CONFIG_FB_MATROX_MAVEN=m CONFIG_FB_RADEON=m CONFIG_FB_RADEON_I2C=y CONFIG_FB_RADEON_BACKLIGHT=y # CONFIG_FB_RADEON_DEBUG is not set CONFIG_FB_ATY128=m CONFIG_FB_ATY128_BACKLIGHT=y CONFIG_FB_ATY=m CONFIG_FB_ATY_CT=y CONFIG_FB_ATY_GENERIC_LCD=y CONFIG_FB_ATY_GX=y CONFIG_FB_ATY_BACKLIGHT=y CONFIG_FB_S3=m CONFIG_FB_SAVAGE=m CONFIG_FB_SAVAGE_I2C=y CONFIG_FB_SAVAGE_ACCEL=y # CONFIG_FB_SIS is not set CONFIG_FB_VIA=m # CONFIG_FB_VIA_DIRECT_PROCFS is not set CONFIG_FB_NEOMAGIC=m CONFIG_FB_KYRO=m CONFIG_FB_3DFX=m CONFIG_FB_3DFX_ACCEL=y CONFIG_FB_3DFX_I2C=y CONFIG_FB_VOODOO1=m # CONFIG_FB_VT8623 is not set CONFIG_FB_TRIDENT=m # CONFIG_FB_ARK is not set # CONFIG_FB_PM3 is not set # CONFIG_FB_CARMINE is not set # CONFIG_FB_GEODE is not set # CONFIG_FB_TMIO is not set CONFIG_FB_SM501=m # CONFIG_FB_UDL is not set CONFIG_FB_VIRTUAL=m CONFIG_XEN_FBDEV_FRONTEND=y CONFIG_FB_METRONOME=m CONFIG_FB_MB862XX=m CONFIG_FB_MB862XX_PCI_GDC=y # CONFIG_FB_BROADSHEET is not set CONFIG_BACKLIGHT_LCD_SUPPORT=y CONFIG_LCD_CLASS_DEVICE=m CONFIG_LCD_PLATFORM=m CONFIG_BACKLIGHT_CLASS_DEVICE=y # CONFIG_BACKLIGHT_GENERIC is not set CONFIG_BACKLIGHT_PROGEAR=m CONFIG_BACKLIGHT_APPLE=m # CONFIG_BACKLIGHT_SAHARA is not set # CONFIG_BACKLIGHT_ADP8860 is not set # # Display device support # CONFIG_DISPLAY_SUPPORT=m # # Display hardware drivers # # # Console display driver support # CONFIG_VGA_CONSOLE=y CONFIG_VGACON_SOFT_SCROLLBACK=y CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64 CONFIG_DUMMY_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y # CONFIG_FONTS is not set CONFIG_FONT_8x8=y CONFIG_FONT_8x16=y CONFIG_LOGO=y # CONFIG_LOGO_LINUX_MONO is not set # CONFIG_LOGO_LINUX_VGA16 is not set CONFIG_LOGO_LINUX_CLUT224=y CONFIG_SOUND=m CONFIG_SOUND_OSS_CORE=y CONFIG_SOUND_OSS_CORE_PRECLAIM=y CONFIG_SND=m CONFIG_SND_TIMER=m CONFIG_SND_PCM=m CONFIG_SND_HWDEP=m CONFIG_SND_RAWMIDI=m CONFIG_SND_JACK=y CONFIG_SND_SEQUENCER=m CONFIG_SND_SEQ_DUMMY=m CONFIG_SND_OSSEMUL=y CONFIG_SND_MIXER_OSS=m CONFIG_SND_PCM_OSS=m CONFIG_SND_PCM_OSS_PLUGINS=y CONFIG_SND_SEQUENCER_OSS=y CONFIG_SND_HRTIMER=m CONFIG_SND_SEQ_HRTIMER_DEFAULT=y CONFIG_SND_DYNAMIC_MINORS=y # CONFIG_SND_SUPPORT_OLD_API is not set CONFIG_SND_VERBOSE_PROCFS=y CONFIG_SND_VERBOSE_PRINTK=y CONFIG_SND_DEBUG=y # CONFIG_SND_DEBUG_VERBOSE is not set CONFIG_SND_PCM_XRUN_DEBUG=y CONFIG_SND_VMASTER=y CONFIG_SND_DMA_SGBUF=y CONFIG_SND_RAWMIDI_SEQ=m CONFIG_SND_OPL3_LIB_SEQ=m # CONFIG_SND_OPL4_LIB_SEQ is not set # CONFIG_SND_SBAWE_SEQ is not set CONFIG_SND_EMU10K1_SEQ=m CONFIG_SND_MPU401_UART=m CONFIG_SND_OPL3_LIB=m CONFIG_SND_VX_LIB=m CONFIG_SND_AC97_CODEC=m CONFIG_SND_DRIVERS=y CONFIG_SND_PCSP=m CONFIG_SND_DUMMY=m CONFIG_SND_ALOOP=m CONFIG_SND_VIRMIDI=m CONFIG_SND_MTPAV=m CONFIG_SND_MTS64=m CONFIG_SND_SERIAL_U16550=m CONFIG_SND_MPU401=m CONFIG_SND_PORTMAN2X4=m CONFIG_SND_AC97_POWER_SAVE=y CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0 CONFIG_SND_SB_COMMON=m CONFIG_SND_SB16_DSP=m CONFIG_SND_PCI=y CONFIG_SND_AD1889=m CONFIG_SND_ALS300=m CONFIG_SND_ALS4000=m CONFIG_SND_ALI5451=m CONFIG_SND_ASIHPI=m CONFIG_SND_ATIIXP=m CONFIG_SND_ATIIXP_MODEM=m CONFIG_SND_AU8810=m CONFIG_SND_AU8820=m CONFIG_SND_AU8830=m # CONFIG_SND_AW2 is not set CONFIG_SND_AZT3328=m CONFIG_SND_BT87X=m # CONFIG_SND_BT87X_OVERCLOCK is not set CONFIG_SND_CA0106=m CONFIG_SND_CMIPCI=m CONFIG_SND_OXYGEN_LIB=m CONFIG_SND_OXYGEN=m CONFIG_SND_CS4281=m CONFIG_SND_CS46XX=m CONFIG_SND_CS46XX_NEW_DSP=y CONFIG_SND_CS5530=m CONFIG_SND_CS5535AUDIO=m CONFIG_SND_CTXFI=m CONFIG_SND_DARLA20=m CONFIG_SND_GINA20=m CONFIG_SND_LAYLA20=m CONFIG_SND_DARLA24=m CONFIG_SND_GINA24=m CONFIG_SND_LAYLA24=m CONFIG_SND_MONA=m CONFIG_SND_MIA=m CONFIG_SND_ECHO3G=m CONFIG_SND_INDIGO=m CONFIG_SND_INDIGOIO=m CONFIG_SND_INDIGODJ=m CONFIG_SND_INDIGOIOX=m CONFIG_SND_INDIGODJX=m CONFIG_SND_EMU10K1=m CONFIG_SND_EMU10K1X=m CONFIG_SND_ENS1370=m CONFIG_SND_ENS1371=m CONFIG_SND_ES1938=m CONFIG_SND_ES1968=m CONFIG_SND_ES1968_INPUT=y CONFIG_SND_FM801=m CONFIG_SND_FM801_TEA575X_BOOL=y CONFIG_SND_FM801_TEA575X=m CONFIG_SND_HDA_INTEL=m CONFIG_SND_HDA_HWDEP=y CONFIG_SND_HDA_RECONFIG=y CONFIG_SND_HDA_INPUT_BEEP=y CONFIG_SND_HDA_INPUT_BEEP_MODE=0 CONFIG_SND_HDA_INPUT_JACK=y CONFIG_SND_HDA_PATCH_LOADER=y CONFIG_SND_HDA_CODEC_REALTEK=y CONFIG_SND_HDA_CODEC_ANALOG=y CONFIG_SND_HDA_CODEC_SIGMATEL=y CONFIG_SND_HDA_CODEC_VIA=y CONFIG_SND_HDA_CODEC_HDMI=y CONFIG_SND_HDA_CODEC_CIRRUS=y CONFIG_SND_HDA_CODEC_CONEXANT=y CONFIG_SND_HDA_CODEC_CA0110=y CONFIG_SND_HDA_CODEC_CMEDIA=y CONFIG_SND_HDA_CODEC_SI3054=y CONFIG_SND_HDA_GENERIC=y CONFIG_SND_HDA_POWER_SAVE=y CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0 CONFIG_SND_HDSP=m CONFIG_SND_HDSPM=m CONFIG_SND_ICE1712=m CONFIG_SND_ICE1724=m CONFIG_SND_INTEL8X0=m CONFIG_SND_INTEL8X0M=m CONFIG_SND_KORG1212=m CONFIG_SND_LX6464ES=m CONFIG_SND_MAESTRO3=m CONFIG_SND_MAESTRO3_INPUT=y CONFIG_SND_MIXART=m CONFIG_SND_NM256=m CONFIG_SND_PCXHR=m CONFIG_SND_RIPTIDE=m CONFIG_SND_RME32=m CONFIG_SND_RME96=m CONFIG_SND_RME9652=m CONFIG_SND_SONICVIBES=m CONFIG_SND_TRIDENT=m CONFIG_SND_VIA82XX=m CONFIG_SND_VIA82XX_MODEM=m CONFIG_SND_VIRTUOSO=m CONFIG_SND_VX222=m CONFIG_SND_YMFPCI=m CONFIG_SND_USB=y CONFIG_SND_USB_AUDIO=m CONFIG_SND_USB_UA101=m CONFIG_SND_USB_USX2Y=m CONFIG_SND_USB_CAIAQ=m CONFIG_SND_USB_CAIAQ_INPUT=y CONFIG_SND_USB_US122L=m CONFIG_SND_PCMCIA=y CONFIG_SND_VXPOCKET=m CONFIG_SND_PDAUDIOCF=m # CONFIG_SND_SOC is not set # CONFIG_SOUND_PRIME is not set CONFIG_AC97_BUS=m CONFIG_HID_SUPPORT=y CONFIG_HID=y CONFIG_HIDRAW=y # # USB Input Devices # CONFIG_USB_HID=y CONFIG_HID_PID=y CONFIG_USB_HIDDEV=y # # Special HID drivers # CONFIG_HID_3M_PCT=y CONFIG_HID_A4TECH=y CONFIG_HID_ACRUX=m CONFIG_HID_ACRUX_FF=m CONFIG_HID_APPLE=y CONFIG_HID_BELKIN=y CONFIG_HID_CANDO=m CONFIG_HID_CHERRY=y CONFIG_HID_CHICONY=y CONFIG_HID_PRODIKEYS=m CONFIG_HID_CYPRESS=y CONFIG_HID_DRAGONRISE=m CONFIG_DRAGONRISE_FF=y CONFIG_HID_EMS_FF=m CONFIG_HID_EGALAX=m CONFIG_HID_ELECOM=m CONFIG_HID_EZKEY=y CONFIG_HID_KYE=y CONFIG_HID_UCLOGIC=m CONFIG_HID_WALTOP=m CONFIG_HID_GYRATION=m CONFIG_HID_TWINHAN=m CONFIG_HID_KENSINGTON=y CONFIG_HID_LOGITECH=y CONFIG_LOGITECH_FF=y CONFIG_LOGIRUMBLEPAD2_FF=y CONFIG_LOGIG940_FF=y CONFIG_LOGIWII_FF=y CONFIG_HID_MAGICMOUSE=m CONFIG_HID_MICROSOFT=y CONFIG_HID_MOSART=y CONFIG_HID_MONTEREY=y CONFIG_HID_MULTITOUCH=m CONFIG_HID_NTRIG=y CONFIG_HID_ORTEK=m CONFIG_HID_PANTHERLORD=m CONFIG_PANTHERLORD_FF=y CONFIG_HID_PETALYNX=m CONFIG_HID_PICOLCD=m CONFIG_HID_PICOLCD_FB=y CONFIG_HID_PICOLCD_BACKLIGHT=y CONFIG_HID_PICOLCD_LCD=y CONFIG_HID_PICOLCD_LEDS=y CONFIG_HID_QUANTA=y CONFIG_HID_ROCCAT=m CONFIG_HID_ROCCAT_KONE=m CONFIG_HID_ROCCAT_KONEPLUS=m CONFIG_HID_ROCCAT_PYRA=m CONFIG_HID_SAMSUNG=m CONFIG_HID_SONY=m CONFIG_HID_STANTUM=y CONFIG_HID_SUNPLUS=m CONFIG_HID_GREENASIA=m CONFIG_GREENASIA_FF=y CONFIG_HID_SMARTJOYPLUS=m CONFIG_SMARTJOYPLUS_FF=y CONFIG_HID_TIVO_SLIDE=m CONFIG_HID_TOPSEED=m CONFIG_HID_THRUSTMASTER=m CONFIG_THRUSTMASTER_FF=y CONFIG_HID_WACOM=m CONFIG_HID_WACOM_POWER_SUPPLY=y CONFIG_HID_ZEROPLUS=m CONFIG_ZEROPLUS_FF=y CONFIG_HID_ZYDACRON=m CONFIG_USB_SUPPORT=y CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB_ARCH_HAS_OHCI=y CONFIG_USB_ARCH_HAS_EHCI=y CONFIG_USB=y # CONFIG_USB_DEBUG is not set CONFIG_USB_ANNOUNCE_NEW_DEVICES=y # # Miscellaneous USB options # CONFIG_USB_DEVICEFS=y # CONFIG_USB_DEVICE_CLASS is not set # CONFIG_USB_DYNAMIC_MINORS is not set CONFIG_USB_SUSPEND=y # CONFIG_USB_OTG is not set CONFIG_USB_MON=y CONFIG_USB_WUSB=m CONFIG_USB_WUSB_CBAF=m # CONFIG_USB_WUSB_CBAF_DEBUG is not set # # USB Host Controller Drivers # # CONFIG_USB_C67X00_HCD is not set CONFIG_USB_XHCI_HCD=m # CONFIG_USB_XHCI_HCD_DEBUGGING is not set CONFIG_USB_EHCI_HCD=y CONFIG_USB_EHCI_ROOT_HUB_TT=y CONFIG_USB_EHCI_TT_NEWSCHED=y # CONFIG_USB_OXU210HP_HCD is not set # CONFIG_USB_ISP116X_HCD is not set # CONFIG_USB_ISP1760_HCD is not set CONFIG_USB_ISP1362_HCD=m CONFIG_USB_OHCI_HCD=y # CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set # CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set CONFIG_USB_OHCI_LITTLE_ENDIAN=y CONFIG_USB_UHCI_HCD=y CONFIG_USB_U132_HCD=m CONFIG_USB_SL811_HCD=m # CONFIG_USB_SL811_CS is not set # CONFIG_USB_R8A66597_HCD is not set CONFIG_USB_WHCI_HCD=m CONFIG_USB_HWA_HCD=m # # USB Device Class drivers # CONFIG_USB_ACM=m CONFIG_USB_PRINTER=m CONFIG_USB_WDM=m CONFIG_USB_TMC=m # # NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may # # # also be needed; see USB_STORAGE Help for more info # CONFIG_USB_STORAGE=m # CONFIG_USB_STORAGE_DEBUG is not set CONFIG_USB_STORAGE_DATAFAB=m CONFIG_USB_STORAGE_FREECOM=m CONFIG_USB_STORAGE_ISD200=m CONFIG_USB_STORAGE_USBAT=m CONFIG_USB_STORAGE_SDDR09=m CONFIG_USB_STORAGE_SDDR55=m CONFIG_USB_STORAGE_JUMPSHOT=m CONFIG_USB_STORAGE_ALAUDA=m CONFIG_USB_STORAGE_ONETOUCH=m CONFIG_USB_STORAGE_KARMA=m CONFIG_USB_STORAGE_CYPRESS_ATACB=m CONFIG_USB_UAS=m # CONFIG_USB_LIBUSUAL is not set # # USB Imaging devices # CONFIG_USB_MDC800=m CONFIG_USB_MICROTEK=m # # USB port drivers # CONFIG_USB_USS720=m CONFIG_USB_SERIAL=m CONFIG_USB_EZUSB=y CONFIG_USB_SERIAL_GENERIC=y CONFIG_USB_SERIAL_AIRCABLE=m CONFIG_USB_SERIAL_ARK3116=m CONFIG_USB_SERIAL_BELKIN=m CONFIG_USB_SERIAL_CH341=m CONFIG_USB_SERIAL_WHITEHEAT=m CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m CONFIG_USB_SERIAL_CP210X=m CONFIG_USB_SERIAL_CYPRESS_M8=m CONFIG_USB_SERIAL_EMPEG=m CONFIG_USB_SERIAL_FTDI_SIO=m CONFIG_USB_SERIAL_FUNSOFT=m CONFIG_USB_SERIAL_VISOR=m CONFIG_USB_SERIAL_IPAQ=m CONFIG_USB_SERIAL_IR=m CONFIG_USB_SERIAL_EDGEPORT=m CONFIG_USB_SERIAL_EDGEPORT_TI=m CONFIG_USB_SERIAL_GARMIN=m CONFIG_USB_SERIAL_IPW=m CONFIG_USB_SERIAL_IUU=m CONFIG_USB_SERIAL_KEYSPAN_PDA=m CONFIG_USB_SERIAL_KEYSPAN=m CONFIG_USB_SERIAL_KLSI=m CONFIG_USB_SERIAL_KOBIL_SCT=m CONFIG_USB_SERIAL_MCT_U232=m CONFIG_USB_SERIAL_MOS7720=m CONFIG_USB_SERIAL_MOS7715_PARPORT=y CONFIG_USB_SERIAL_MOS7840=m CONFIG_USB_SERIAL_MOTOROLA=m CONFIG_USB_SERIAL_NAVMAN=m CONFIG_USB_SERIAL_PL2303=m CONFIG_USB_SERIAL_OTI6858=m CONFIG_USB_SERIAL_QCAUX=m CONFIG_USB_SERIAL_QUALCOMM=m CONFIG_USB_SERIAL_SPCP8X5=m CONFIG_USB_SERIAL_HP4X=m CONFIG_USB_SERIAL_SAFE=m CONFIG_USB_SERIAL_SAFE_PADDED=y CONFIG_USB_SERIAL_SAMBA=m CONFIG_USB_SERIAL_SIEMENS_MPI=m CONFIG_USB_SERIAL_SIERRAWIRELESS=m CONFIG_USB_SERIAL_SYMBOL=m CONFIG_USB_SERIAL_TI=m CONFIG_USB_SERIAL_CYBERJACK=m CONFIG_USB_SERIAL_XIRCOM=m CONFIG_USB_SERIAL_WWAN=m CONFIG_USB_SERIAL_OPTION=m CONFIG_USB_SERIAL_OMNINET=m CONFIG_USB_SERIAL_OPTICON=m CONFIG_USB_SERIAL_VIVOPAY_SERIAL=m # CONFIG_USB_SERIAL_ZIO is not set CONFIG_USB_SERIAL_SSU100=m CONFIG_USB_SERIAL_DEBUG=m # # USB Miscellaneous drivers # CONFIG_USB_EMI62=m CONFIG_USB_EMI26=m CONFIG_USB_ADUTUX=m CONFIG_USB_SEVSEG=m # CONFIG_USB_RIO500 is not set CONFIG_USB_LEGOTOWER=m CONFIG_USB_LCD=m CONFIG_USB_LED=m # CONFIG_USB_CYPRESS_CY7C63 is not set # CONFIG_USB_CYTHERM is not set CONFIG_USB_IDMOUSE=m CONFIG_USB_FTDI_ELAN=m CONFIG_USB_APPLEDISPLAY=m CONFIG_USB_SISUSBVGA=m CONFIG_USB_SISUSBVGA_CON=y CONFIG_USB_LD=m CONFIG_USB_TRANCEVIBRATOR=m CONFIG_USB_IOWARRIOR=m # CONFIG_USB_TEST is not set CONFIG_USB_ISIGHTFW=m CONFIG_USB_YUREX=m CONFIG_USB_ATM=m CONFIG_USB_SPEEDTOUCH=m CONFIG_USB_CXACRU=m CONFIG_USB_UEAGLEATM=m CONFIG_USB_XUSBATM=m # CONFIG_USB_GADGET is not set # # OTG and related infrastructure # CONFIG_USB_OTG_UTILS=y # CONFIG_USB_GPIO_VBUS is not set CONFIG_NOP_USB_XCEIV=m CONFIG_UWB=m CONFIG_UWB_HWA=m CONFIG_UWB_WHCI=m CONFIG_UWB_I1480U=m CONFIG_MMC=m # CONFIG_MMC_DEBUG is not set # CONFIG_MMC_UNSAFE_RESUME is not set # CONFIG_MMC_CLKGATE is not set # # MMC/SD/SDIO Card Drivers # CONFIG_MMC_BLOCK=m CONFIG_MMC_BLOCK_MINORS=8 CONFIG_MMC_BLOCK_BOUNCE=y CONFIG_SDIO_UART=m # CONFIG_MMC_TEST is not set # # MMC/SD/SDIO Host Controller Drivers # CONFIG_MMC_SDHCI=m CONFIG_MMC_SDHCI_PCI=m CONFIG_MMC_RICOH_MMC=y CONFIG_MMC_SDHCI_PLTFM=m CONFIG_MMC_WBSD=m CONFIG_MMC_TIFM_SD=m CONFIG_MMC_SDRICOH_CS=m CONFIG_MMC_CB710=m CONFIG_MMC_VIA_SDMMC=m CONFIG_MMC_USHC=m CONFIG_MEMSTICK=m # CONFIG_MEMSTICK_DEBUG is not set # # MemoryStick drivers # # CONFIG_MEMSTICK_UNSAFE_RESUME is not set CONFIG_MSPRO_BLOCK=m # # MemoryStick Host Controller Drivers # CONFIG_MEMSTICK_TIFM_MS=m CONFIG_MEMSTICK_JMICRON_38X=m CONFIG_NEW_LEDS=y CONFIG_LEDS_CLASS=y # # LED drivers # CONFIG_LEDS_ALIX2=m # CONFIG_LEDS_PCA9532 is not set # CONFIG_LEDS_GPIO is not set CONFIG_LEDS_LP3944=m CONFIG_LEDS_LP5521=m CONFIG_LEDS_LP5523=m CONFIG_LEDS_CLEVO_MAIL=m # CONFIG_LEDS_PCA955X is not set # CONFIG_LEDS_BD2802 is not set CONFIG_LEDS_INTEL_SS4200=m CONFIG_LEDS_LT3593=m CONFIG_LEDS_DELL_NETBOOKS=m CONFIG_LEDS_TRIGGERS=y # # LED Triggers # CONFIG_LEDS_TRIGGER_TIMER=m CONFIG_LEDS_TRIGGER_HEARTBEAT=m CONFIG_LEDS_TRIGGER_BACKLIGHT=m CONFIG_LEDS_TRIGGER_GPIO=m CONFIG_LEDS_TRIGGER_DEFAULT_ON=m # # iptables trigger is under Netfilter config (LED target) # CONFIG_NFC_DEVICES=y CONFIG_PN544_NFC=m CONFIG_ACCESSIBILITY=y CONFIG_A11Y_BRAILLE_CONSOLE=y CONFIG_INFINIBAND=m CONFIG_INFINIBAND_USER_MAD=m CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_USER_MEM=y CONFIG_INFINIBAND_ADDR_TRANS=y CONFIG_INFINIBAND_MTHCA=m CONFIG_INFINIBAND_MTHCA_DEBUG=y CONFIG_INFINIBAND_IPATH=m CONFIG_INFINIBAND_QIB=m CONFIG_INFINIBAND_AMSO1100=m # CONFIG_INFINIBAND_AMSO1100_DEBUG is not set CONFIG_INFINIBAND_CXGB3=m # CONFIG_INFINIBAND_CXGB3_DEBUG is not set CONFIG_INFINIBAND_CXGB4=m CONFIG_MLX4_INFINIBAND=m CONFIG_INFINIBAND_NES=m # CONFIG_INFINIBAND_NES_DEBUG is not set CONFIG_INFINIBAND_IPOIB=m CONFIG_INFINIBAND_IPOIB_CM=y CONFIG_INFINIBAND_IPOIB_DEBUG=y CONFIG_INFINIBAND_IPOIB_DEBUG_DATA=y CONFIG_INFINIBAND_SRP=m CONFIG_INFINIBAND_ISER=m CONFIG_EDAC=y # # Reporting subsystems # # CONFIG_EDAC_DEBUG is not set CONFIG_EDAC_DECODE_MCE=m CONFIG_EDAC_MCE_INJ=m CONFIG_EDAC_MM_EDAC=m CONFIG_EDAC_MCE=y CONFIG_EDAC_AMD64=m # CONFIG_EDAC_AMD64_ERROR_INJECTION is not set CONFIG_EDAC_E752X=m CONFIG_EDAC_I82975X=m CONFIG_EDAC_I3000=m CONFIG_EDAC_I3200=m CONFIG_EDAC_X38=m CONFIG_EDAC_I5400=m CONFIG_EDAC_I7CORE=m CONFIG_EDAC_I5000=m CONFIG_EDAC_I5100=m CONFIG_EDAC_I7300=m CONFIG_RTC_LIB=y CONFIG_RTC_CLASS=y CONFIG_RTC_HCTOSYS=y CONFIG_RTC_HCTOSYS_DEVICE="rtc0" # CONFIG_RTC_DEBUG is not set # # RTC interfaces # CONFIG_RTC_INTF_SYSFS=y CONFIG_RTC_INTF_PROC=y CONFIG_RTC_INTF_DEV=y # CONFIG_RTC_INTF_DEV_UIE_EMUL is not set # CONFIG_RTC_DRV_TEST is not set # # I2C RTC drivers # CONFIG_RTC_DRV_DS1307=m CONFIG_RTC_DRV_DS1374=m CONFIG_RTC_DRV_DS1672=m CONFIG_RTC_DRV_DS3232=m CONFIG_RTC_DRV_MAX6900=m CONFIG_RTC_DRV_RS5C372=m CONFIG_RTC_DRV_ISL1208=m CONFIG_RTC_DRV_ISL12022=m CONFIG_RTC_DRV_X1205=m CONFIG_RTC_DRV_PCF8563=m CONFIG_RTC_DRV_PCF8583=m CONFIG_RTC_DRV_M41T80=m CONFIG_RTC_DRV_M41T80_WDT=y CONFIG_RTC_DRV_BQ32K=m # CONFIG_RTC_DRV_S35390A is not set CONFIG_RTC_DRV_FM3130=m CONFIG_RTC_DRV_RX8581=m CONFIG_RTC_DRV_RX8025=m # # SPI RTC drivers # # # Platform RTC drivers # CONFIG_RTC_DRV_CMOS=y CONFIG_RTC_DRV_DS1286=m CONFIG_RTC_DRV_DS1511=m CONFIG_RTC_DRV_DS1553=m CONFIG_RTC_DRV_DS1742=m CONFIG_RTC_DRV_STK17TA8=m # CONFIG_RTC_DRV_M48T86 is not set CONFIG_RTC_DRV_M48T35=m CONFIG_RTC_DRV_M48T59=m CONFIG_RTC_DRV_MSM6242=m CONFIG_RTC_DRV_BQ4802=m CONFIG_RTC_DRV_RP5C01=m CONFIG_RTC_DRV_V3020=m # # on-CPU RTC drivers # CONFIG_DMADEVICES=y # CONFIG_DMADEVICES_DEBUG is not set # # DMA Devices # # CONFIG_INTEL_MID_DMAC is not set CONFIG_INTEL_IOATDMA=m # CONFIG_TIMB_DMA is not set CONFIG_PCH_DMA=m CONFIG_DMA_ENGINE=y # # DMA Clients # CONFIG_NET_DMA=y CONFIG_ASYNC_TX_DMA=y # CONFIG_DMATEST is not set CONFIG_DCA=m CONFIG_AUXDISPLAY=y CONFIG_KS0108=m CONFIG_KS0108_PORT=0x378 CONFIG_KS0108_DELAY=2 CONFIG_CFAG12864B=m CONFIG_CFAG12864B_RATE=20 CONFIG_UIO=m CONFIG_UIO_CIF=m # CONFIG_UIO_PDRV is not set # CONFIG_UIO_PDRV_GENIRQ is not set CONFIG_UIO_AEC=m CONFIG_UIO_SERCOS3=m CONFIG_UIO_PCI_GENERIC=m # CONFIG_UIO_NETX is not set # # Xen driver support # CONFIG_XEN_BALLOON=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=m CONFIG_XEN_BACKEND=y CONFIG_XENFS=m CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_GNTDEV=m CONFIG_XEN_PLATFORM_PCI=m CONFIG_SWIOTLB_XEN=y CONFIG_STAGING=y # CONFIG_STAGING_EXCLUDE_BUILD is not set # CONFIG_ET131X is not set # CONFIG_SLICOSS is not set # CONFIG_VIDEO_GO7007 is not set # CONFIG_VIDEO_CX25821 is not set # CONFIG_VIDEO_TM6000 is not set CONFIG_USB_DABUSB=m CONFIG_USB_SE401=m CONFIG_VIDEO_USBVIDEO=m CONFIG_USB_VICAM=m # CONFIG_USB_IP_COMMON is not set # CONFIG_W35UND is not set # CONFIG_PRISM2_USB is not set # CONFIG_ECHO is not set # CONFIG_BRCM80211 is not set # CONFIG_RT2860 is not set # CONFIG_RT2870 is not set # CONFIG_COMEDI is not set # CONFIG_ASUS_OLED is not set # CONFIG_PANEL is not set # CONFIG_R8187SE is not set # CONFIG_RTL8192U is not set # CONFIG_RTL8192E is not set # CONFIG_R8712U is not set # CONFIG_TRANZPORT is not set # CONFIG_POHMELFS is not set # CONFIG_AUTOFS_FS is not set # CONFIG_IDE_PHISON is not set # CONFIG_LINE6_USB is not set CONFIG_DRM_VMWGFX=m CONFIG_DRM_NOUVEAU=m CONFIG_DRM_NOUVEAU_BACKLIGHT=y CONFIG_DRM_NOUVEAU_DEBUG=y # # I2C encoder or helper chips # CONFIG_DRM_I2C_CH7006=m CONFIG_DRM_I2C_SIL164=m # CONFIG_USB_SERIAL_QUATECH2 is not set # CONFIG_USB_SERIAL_QUATECH_USB2 is not set # CONFIG_VT6655 is not set # CONFIG_VT6656 is not set # CONFIG_HYPERV is not set # CONFIG_VME_BUS is not set # CONFIG_DX_SEP is not set # CONFIG_IIO is not set # CONFIG_ZRAM is not set # CONFIG_WLAGS49_H2 is not set # CONFIG_WLAGS49_H25 is not set # CONFIG_SAMSUNG_LAPTOP is not set # CONFIG_FB_SM7XX is not set # CONFIG_VIDEO_DT3155 is not set CONFIG_CRYSTALHD=m # # Texas Instruments shared transport line discipline # # CONFIG_ST_BT is not set # CONFIG_FB_XGI is not set CONFIG_LIRC_STAGING=y CONFIG_LIRC_BT829=m CONFIG_LIRC_IGORPLUGUSB=m CONFIG_LIRC_IMON=m CONFIG_LIRC_PARALLEL=m CONFIG_LIRC_SASEM=m CONFIG_LIRC_SERIAL=m CONFIG_LIRC_SERIAL_TRANSMITTER=y CONFIG_LIRC_SIR=m CONFIG_LIRC_TTUSBIR=m CONFIG_LIRC_ZILOG=m # CONFIG_SMB_FS is not set # CONFIG_EASYCAP is not set # CONFIG_SOLO6X10 is not set # CONFIG_ACPI_QUICKSTART is not set CONFIG_MACH_NO_WESTBRIDGE=y # CONFIG_ATH6K_LEGACY is not set # CONFIG_USB_ENESTORAGE is not set # CONFIG_BCM_WIMAX is not set # CONFIG_FT1000 is not set # # Speakup console speech # # CONFIG_SPEAKUP is not set # CONFIG_TOUCHSCREEN_CLEARPAD_TM1217 is not set # CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4 is not set CONFIG_X86_PLATFORM_DEVICES=y CONFIG_ACER_WMI=m CONFIG_ACERHDF=m CONFIG_ASUS_LAPTOP=m CONFIG_DELL_LAPTOP=m CONFIG_DELL_WMI=m CONFIG_FUJITSU_LAPTOP=m # CONFIG_FUJITSU_LAPTOP_DEBUG is not set CONFIG_HP_WMI=m CONFIG_MSI_LAPTOP=m CONFIG_PANASONIC_LAPTOP=m CONFIG_COMPAL_LAPTOP=m CONFIG_SONY_LAPTOP=m CONFIG_SONYPI_COMPAT=y CONFIG_IDEAPAD_LAPTOP=m CONFIG_THINKPAD_ACPI=m CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y # CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set # CONFIG_THINKPAD_ACPI_DEBUG is not set # CONFIG_THINKPAD_ACPI_UNSAFE_LEDS is not set CONFIG_THINKPAD_ACPI_VIDEO=y CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y CONFIG_SENSORS_HDAPS=m # CONFIG_INTEL_MENLOW is not set CONFIG_EEEPC_LAPTOP=m CONFIG_EEEPC_WMI=m CONFIG_ACPI_WMI=m CONFIG_MSI_WMI=m # CONFIG_ACPI_ASUS is not set CONFIG_TOPSTAR_LAPTOP=m CONFIG_ACPI_TOSHIBA=m CONFIG_TOSHIBA_BT_RFKILL=m CONFIG_ACPI_CMPC=m CONFIG_INTEL_IPS=m # CONFIG_IBM_RTL is not set # # Firmware Drivers # CONFIG_EDD=m # CONFIG_EDD_OFF is not set CONFIG_FIRMWARE_MEMMAP=y CONFIG_EFI_VARS=y CONFIG_DELL_RBU=m CONFIG_DCDBAS=m CONFIG_DMIID=y CONFIG_ISCSI_IBFT_FIND=y CONFIG_ISCSI_IBFT=m # # File systems # CONFIG_EXT2_FS=m CONFIG_EXT2_FS_XATTR=y CONFIG_EXT2_FS_POSIX_ACL=y CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT2_FS_XIP=y CONFIG_EXT3_FS=y CONFIG_EXT3_DEFAULTS_TO_ORDERED=y CONFIG_EXT3_FS_XATTR=y CONFIG_EXT3_FS_POSIX_ACL=y CONFIG_EXT3_FS_SECURITY=y CONFIG_EXT4_FS=y CONFIG_EXT4_FS_XATTR=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y # CONFIG_EXT4_DEBUG is not set CONFIG_FS_XIP=y CONFIG_JBD=y # CONFIG_JBD_DEBUG is not set CONFIG_JBD2=y # CONFIG_JBD2_DEBUG is not set CONFIG_FS_MBCACHE=y CONFIG_REISERFS_FS=m # CONFIG_REISERFS_CHECK is not set CONFIG_REISERFS_PROC_INFO=y CONFIG_REISERFS_FS_XATTR=y CONFIG_REISERFS_FS_POSIX_ACL=y CONFIG_REISERFS_FS_SECURITY=y CONFIG_JFS_FS=m CONFIG_JFS_POSIX_ACL=y CONFIG_JFS_SECURITY=y # CONFIG_JFS_DEBUG is not set # CONFIG_JFS_STATISTICS is not set CONFIG_XFS_FS=m CONFIG_XFS_QUOTA=y CONFIG_XFS_POSIX_ACL=y # CONFIG_XFS_RT is not set # CONFIG_XFS_DEBUG is not set CONFIG_GFS2_FS=m CONFIG_GFS2_FS_LOCKING_DLM=y CONFIG_OCFS2_FS=m CONFIG_OCFS2_FS_O2CB=m CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m # CONFIG_OCFS2_FS_STATS is not set # CONFIG_OCFS2_DEBUG_MASKLOG is not set # CONFIG_OCFS2_DEBUG_FS is not set CONFIG_BTRFS_FS=m CONFIG_BTRFS_FS_POSIX_ACL=y CONFIG_NILFS2_FS=m CONFIG_FS_POSIX_ACL=y CONFIG_EXPORTFS=m CONFIG_FILE_LOCKING=y CONFIG_FSNOTIFY=y CONFIG_DNOTIFY=y CONFIG_INOTIFY_USER=y CONFIG_FANOTIFY=y CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y CONFIG_QUOTA=y CONFIG_QUOTA_NETLINK_INTERFACE=y # CONFIG_PRINT_QUOTA_WARNING is not set # CONFIG_QUOTA_DEBUG is not set CONFIG_QUOTA_TREE=y # CONFIG_QFMT_V1 is not set CONFIG_QFMT_V2=y CONFIG_QUOTACTL=y CONFIG_QUOTACTL_COMPAT=y CONFIG_AUTOFS4_FS=y CONFIG_FUSE_FS=m CONFIG_CUSE=m CONFIG_GENERIC_ACL=y # # Caches # CONFIG_FSCACHE=m CONFIG_FSCACHE_STATS=y # CONFIG_FSCACHE_HISTOGRAM is not set # CONFIG_FSCACHE_DEBUG is not set CONFIG_FSCACHE_OBJECT_LIST=y CONFIG_CACHEFILES=m # CONFIG_CACHEFILES_DEBUG is not set # CONFIG_CACHEFILES_HISTOGRAM is not set # # CD-ROM/DVD Filesystems # CONFIG_ISO9660_FS=y CONFIG_JOLIET=y CONFIG_ZISOFS=y CONFIG_UDF_FS=m CONFIG_UDF_NLS=y # # DOS/FAT/NT Filesystems # CONFIG_FAT_FS=m CONFIG_MSDOS_FS=m CONFIG_VFAT_FS=m CONFIG_FAT_DEFAULT_CODEPAGE=437 CONFIG_FAT_DEFAULT_IOCHARSET="ascii" # CONFIG_NTFS_FS is not set # # Pseudo filesystems # CONFIG_PROC_FS=y CONFIG_PROC_KCORE=y CONFIG_PROC_VMCORE=y CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y CONFIG_SYSFS=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_HUGETLBFS=y CONFIG_HUGETLB_PAGE=y CONFIG_CONFIGFS_FS=m CONFIG_MISC_FILESYSTEMS=y # CONFIG_ADFS_FS is not set CONFIG_AFFS_FS=m CONFIG_ECRYPT_FS=m CONFIG_HFS_FS=m CONFIG_HFSPLUS_FS=m CONFIG_BEFS_FS=m # CONFIG_BEFS_DEBUG is not set CONFIG_BFS_FS=m CONFIG_EFS_FS=m CONFIG_JFFS2_FS=m CONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y # CONFIG_JFFS2_FS_WBUF_VERIFY is not set CONFIG_JFFS2_SUMMARY=y CONFIG_JFFS2_FS_XATTR=y CONFIG_JFFS2_FS_POSIX_ACL=y CONFIG_JFFS2_FS_SECURITY=y # CONFIG_JFFS2_COMPRESSION_OPTIONS is not set CONFIG_JFFS2_ZLIB=y # CONFIG_JFFS2_LZO is not set CONFIG_JFFS2_RTIME=y # CONFIG_JFFS2_RUBIN is not set CONFIG_UBIFS_FS=m CONFIG_UBIFS_FS_XATTR=y # CONFIG_UBIFS_FS_ADVANCED_COMPR is not set CONFIG_UBIFS_FS_LZO=y CONFIG_UBIFS_FS_ZLIB=y # CONFIG_UBIFS_FS_DEBUG is not set CONFIG_LOGFS=m CONFIG_CRAMFS=m CONFIG_SQUASHFS=m CONFIG_SQUASHFS_XATTR=y CONFIG_SQUASHFS_LZO=y CONFIG_SQUASHFS_XZ=y # CONFIG_SQUASHFS_EMBEDDED is not set CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 CONFIG_VXFS_FS=m CONFIG_MINIX_FS=m CONFIG_OMFS_FS=m # CONFIG_HPFS_FS is not set CONFIG_QNX4FS_FS=m CONFIG_ROMFS_FS=m CONFIG_ROMFS_BACKED_BY_BLOCK=y # CONFIG_ROMFS_BACKED_BY_MTD is not set # CONFIG_ROMFS_BACKED_BY_BOTH is not set CONFIG_ROMFS_ON_BLOCK=y CONFIG_SYSV_FS=m CONFIG_UFS_FS=m # CONFIG_UFS_FS_WRITE is not set # CONFIG_UFS_DEBUG is not set CONFIG_EXOFS_FS=m # CONFIG_EXOFS_DEBUG is not set CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=m CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_NFS_V4=y CONFIG_NFS_V4_1=y CONFIG_PNFS_FILE_LAYOUT=m CONFIG_NFS_FSCACHE=y # CONFIG_NFS_USE_LEGACY_DNS is not set CONFIG_NFS_USE_KERNEL_DNS=y # CONFIG_NFS_USE_NEW_IDMAPPER is not set CONFIG_NFSD=m # CONFIG_NFSD_DEPRECATED is not set CONFIG_NFSD_V2_ACL=y CONFIG_NFSD_V3=y CONFIG_NFSD_V3_ACL=y CONFIG_NFSD_V4=y CONFIG_LOCKD=m CONFIG_LOCKD_V4=y CONFIG_NFS_ACL_SUPPORT=m CONFIG_NFS_COMMON=y CONFIG_SUNRPC=m CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_XPRT_RDMA=m CONFIG_RPCSEC_GSS_KRB5=m CONFIG_CEPH_FS=m CONFIG_CIFS=m CONFIG_CIFS_STATS=y # CONFIG_CIFS_STATS2 is not set CONFIG_CIFS_WEAK_PW_HASH=y CONFIG_CIFS_UPCALL=y CONFIG_CIFS_XATTR=y CONFIG_CIFS_POSIX=y # CONFIG_CIFS_DEBUG2 is not set CONFIG_CIFS_DFS_UPCALL=y CONFIG_CIFS_FSCACHE=y CONFIG_CIFS_ACL=y CONFIG_CIFS_EXPERIMENTAL=y CONFIG_NCP_FS=m CONFIG_NCPFS_PACKET_SIGNING=y CONFIG_NCPFS_IOCTL_LOCKING=y CONFIG_NCPFS_STRONG=y CONFIG_NCPFS_NFS_NS=y CONFIG_NCPFS_OS2_NS=y CONFIG_NCPFS_SMALLDOS=y CONFIG_NCPFS_NLS=y CONFIG_NCPFS_EXTRAS=y CONFIG_CODA_FS=m # CONFIG_AFS_FS is not set CONFIG_9P_FS=m CONFIG_9P_FSCACHE=y CONFIG_9P_FS_POSIX_ACL=y # # Partition Types # CONFIG_PARTITION_ADVANCED=y # CONFIG_ACORN_PARTITION is not set CONFIG_OSF_PARTITION=y CONFIG_AMIGA_PARTITION=y # CONFIG_ATARI_PARTITION is not set CONFIG_MAC_PARTITION=y CONFIG_MSDOS_PARTITION=y CONFIG_BSD_DISKLABEL=y CONFIG_MINIX_SUBPARTITION=y CONFIG_SOLARIS_X86_PARTITION=y CONFIG_UNIXWARE_DISKLABEL=y # CONFIG_LDM_PARTITION is not set CONFIG_SGI_PARTITION=y # CONFIG_ULTRIX_PARTITION is not set CONFIG_SUN_PARTITION=y CONFIG_KARMA_PARTITION=y CONFIG_EFI_PARTITION=y # CONFIG_SYSV68_PARTITION is not set CONFIG_NLS=y CONFIG_NLS_DEFAULT="utf8" CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_CODEPAGE_737=m CONFIG_NLS_CODEPAGE_775=m CONFIG_NLS_CODEPAGE_850=m CONFIG_NLS_CODEPAGE_852=m CONFIG_NLS_CODEPAGE_855=m CONFIG_NLS_CODEPAGE_857=m CONFIG_NLS_CODEPAGE_860=m CONFIG_NLS_CODEPAGE_861=m CONFIG_NLS_CODEPAGE_862=m CONFIG_NLS_CODEPAGE_863=m CONFIG_NLS_CODEPAGE_864=m CONFIG_NLS_CODEPAGE_865=m CONFIG_NLS_CODEPAGE_866=m CONFIG_NLS_CODEPAGE_869=m CONFIG_NLS_CODEPAGE_936=m CONFIG_NLS_CODEPAGE_950=m CONFIG_NLS_CODEPAGE_932=m CONFIG_NLS_CODEPAGE_949=m CONFIG_NLS_CODEPAGE_874=m CONFIG_NLS_ISO8859_8=m CONFIG_NLS_CODEPAGE_1250=m CONFIG_NLS_CODEPAGE_1251=m CONFIG_NLS_ASCII=y CONFIG_NLS_ISO8859_1=m CONFIG_NLS_ISO8859_2=m CONFIG_NLS_ISO8859_3=m CONFIG_NLS_ISO8859_4=m CONFIG_NLS_ISO8859_5=m CONFIG_NLS_ISO8859_6=m CONFIG_NLS_ISO8859_7=m CONFIG_NLS_ISO8859_9=m CONFIG_NLS_ISO8859_13=m CONFIG_NLS_ISO8859_14=m CONFIG_NLS_ISO8859_15=m CONFIG_NLS_KOI8_R=m CONFIG_NLS_KOI8_U=m CONFIG_NLS_UTF8=m CONFIG_DLM=m CONFIG_DLM_DEBUG=y # # Kernel hacking # CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_PRINTK_TIME=y # CONFIG_ENABLE_WARN_DEPRECATED is not set CONFIG_ENABLE_MUST_CHECK=y CONFIG_FRAME_WARN=2048 CONFIG_MAGIC_SYSRQ=y CONFIG_STRIP_ASM_SYMS=y CONFIG_UNUSED_SYMBOLS=y CONFIG_DEBUG_FS=y CONFIG_HEADERS_CHECK=y CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_SHIRQ=y CONFIG_LOCKUP_DETECTOR=y CONFIG_HARDLOCKUP_DETECTOR=y # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0 # CONFIG_DETECT_HUNG_TASK is not set CONFIG_SCHED_DEBUG=y CONFIG_SCHEDSTATS=y CONFIG_TIMER_STATS=y # CONFIG_DEBUG_OBJECTS is not set # CONFIG_SLUB_DEBUG_ON is not set # CONFIG_SLUB_STATS is not set # CONFIG_DEBUG_RT_MUTEXES is not set # CONFIG_RT_MUTEX_TESTER is not set # CONFIG_DEBUG_SPINLOCK is not set # CONFIG_DEBUG_MUTEXES is not set CONFIG_BKL=y # CONFIG_DEBUG_LOCK_ALLOC is not set # CONFIG_PROVE_LOCKING is not set CONFIG_SPARSE_RCU_POINTER=y # CONFIG_LOCK_STAT is not set CONFIG_DEBUG_SPINLOCK_SLEEP=y # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set CONFIG_STACKTRACE=y # CONFIG_DEBUG_KOBJECT is not set CONFIG_DEBUG_BUGVERBOSE=y CONFIG_DEBUG_INFO=y # CONFIG_DEBUG_INFO_REDUCED is not set # CONFIG_DEBUG_VM is not set # CONFIG_DEBUG_VIRTUAL is not set # CONFIG_DEBUG_WRITECOUNT is not set CONFIG_DEBUG_MEMORY_INIT=y CONFIG_DEBUG_LIST=y # CONFIG_TEST_LIST_SORT is not set # CONFIG_DEBUG_SG is not set # CONFIG_DEBUG_NOTIFIERS is not set # CONFIG_DEBUG_CREDENTIALS is not set CONFIG_ARCH_WANT_FRAME_POINTERS=y CONFIG_FRAME_POINTER=y CONFIG_BOOT_PRINTK_DELAY=y # CONFIG_RCU_TORTURE_TEST is not set # CONFIG_RCU_CPU_STALL_DETECTOR is not set # CONFIG_KPROBES_SANITY_TEST is not set # CONFIG_BACKTRACE_SELF_TEST is not set # CONFIG_DEBUG_BLOCK_EXT_DEVT is not set # CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set # CONFIG_LKDTM is not set # CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set # CONFIG_FAULT_INJECTION is not set CONFIG_LATENCYTOP=y # CONFIG_SYSCTL_SYSCALL_CHECK is not set # CONFIG_DEBUG_PAGEALLOC is not set CONFIG_USER_STACKTRACE_SUPPORT=y CONFIG_NOP_TRACER=y CONFIG_HAVE_FTRACE_NMI_ENTER=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y CONFIG_HAVE_SYSCALL_TRACEPOINTS=y CONFIG_HAVE_C_RECORDMCOUNT=y CONFIG_TRACER_MAX_TRACE=y CONFIG_RING_BUFFER=y CONFIG_FTRACE_NMI_ENTER=y CONFIG_EVENT_TRACING=y CONFIG_EVENT_POWER_TRACING_DEPRECATED=y CONFIG_CONTEXT_SWITCH_TRACER=y CONFIG_RING_BUFFER_ALLOW_SWAP=y CONFIG_TRACING=y CONFIG_GENERIC_TRACER=y CONFIG_TRACING_SUPPORT=y CONFIG_FTRACE=y CONFIG_FUNCTION_TRACER=y CONFIG_FUNCTION_GRAPH_TRACER=y # CONFIG_IRQSOFF_TRACER is not set CONFIG_SCHED_TRACER=y CONFIG_FTRACE_SYSCALLS=y CONFIG_BRANCH_PROFILE_NONE=y # CONFIG_PROFILE_ANNOTATED_BRANCHES is not set # CONFIG_PROFILE_ALL_BRANCHES is not set CONFIG_STACK_TRACER=y CONFIG_BLK_DEV_IO_TRACE=y CONFIG_KPROBE_EVENT=y CONFIG_DYNAMIC_FTRACE=y CONFIG_FUNCTION_PROFILER=y CONFIG_FTRACE_MCOUNT_RECORD=y # CONFIG_FTRACE_STARTUP_TEST is not set # CONFIG_MMIOTRACE is not set CONFIG_RING_BUFFER_BENCHMARK=m CONFIG_PROVIDE_OHCI1394_DMA_INIT=y # CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set CONFIG_BUILD_DOCSRC=y CONFIG_DYNAMIC_DEBUG=y # CONFIG_DMA_API_DEBUG is not set CONFIG_ATOMIC64_SELFTEST=y CONFIG_ASYNC_RAID6_TEST=m # CONFIG_SAMPLES is not set CONFIG_HAVE_ARCH_KGDB=y CONFIG_KGDB=y CONFIG_KGDB_SERIAL_CONSOLE=y CONFIG_KGDB_TESTS=y # CONFIG_KGDB_TESTS_ON_BOOT is not set CONFIG_KGDB_LOW_LEVEL_TRAP=y CONFIG_KGDB_KDB=y CONFIG_KDB_KEYBOARD=y CONFIG_HAVE_ARCH_KMEMCHECK=y CONFIG_TEST_KSTRTOX=m CONFIG_STRICT_DEVMEM=y # CONFIG_X86_VERBOSE_BOOTUP is not set CONFIG_EARLY_PRINTK=y CONFIG_EARLY_PRINTK_DBGP=y CONFIG_DEBUG_STACKOVERFLOW=y # CONFIG_DEBUG_STACK_USAGE is not set # CONFIG_DEBUG_PER_CPU_MAPS is not set # CONFIG_X86_PTDUMP is not set CONFIG_DEBUG_RODATA=y CONFIG_DEBUG_RODATA_TEST=y # CONFIG_DEBUG_SET_MODULE_RONX is not set CONFIG_DEBUG_NX_TEST=m # CONFIG_IOMMU_DEBUG is not set # CONFIG_IOMMU_STRESS is not set CONFIG_HAVE_MMIOTRACE_SUPPORT=y CONFIG_X86_DECODER_SELFTEST=y CONFIG_IO_DELAY_TYPE_0X80=0 CONFIG_IO_DELAY_TYPE_0XED=1 CONFIG_IO_DELAY_TYPE_UDELAY=2 CONFIG_IO_DELAY_TYPE_NONE=3 CONFIG_IO_DELAY_0X80=y # CONFIG_IO_DELAY_0XED is not set # CONFIG_IO_DELAY_UDELAY is not set # CONFIG_IO_DELAY_NONE is not set CONFIG_DEFAULT_IO_DELAY_TYPE=0 CONFIG_DEBUG_BOOT_PARAMS=y # CONFIG_CPA_DEBUG is not set CONFIG_OPTIMIZE_INLINING=y # CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set # # Security options # CONFIG_KEYS=y CONFIG_TRUSTED_KEYS=m CONFIG_ENCRYPTED_KEYS=m CONFIG_KEYS_DEBUG_PROC_KEYS=y # CONFIG_SECURITY_DMESG_RESTRICT is not set CONFIG_SECURITY=y CONFIG_SECURITYFS=y CONFIG_SECURITY_NETWORK=y CONFIG_SECURITY_NETWORK_XFRM=y # CONFIG_SECURITY_PATH is not set CONFIG_INTEL_TXT=y CONFIG_LSM_MMAP_MIN_ADDR=65536 CONFIG_SECURITY_SELINUX=y CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1 CONFIG_SECURITY_SELINUX_DISABLE=y CONFIG_SECURITY_SELINUX_DEVELOP=y CONFIG_SECURITY_SELINUX_AVC_STATS=y CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1 # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set # CONFIG_SECURITY_SMACK is not set # CONFIG_SECURITY_TOMOYO is not set # CONFIG_SECURITY_APPARMOR is not set CONFIG_IMA=y CONFIG_IMA_MEASURE_PCR_IDX=10 CONFIG_IMA_AUDIT=y CONFIG_IMA_LSM_RULES=y CONFIG_DEFAULT_SECURITY_SELINUX=y # CONFIG_DEFAULT_SECURITY_DAC is not set CONFIG_DEFAULT_SECURITY="selinux" CONFIG_XOR_BLOCKS=m CONFIG_ASYNC_CORE=m CONFIG_ASYNC_MEMCPY=m CONFIG_ASYNC_XOR=m CONFIG_ASYNC_PQ=m CONFIG_ASYNC_RAID6_RECOV=m CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y CONFIG_CRYPTO=y # # Crypto core or helper # CONFIG_CRYPTO_ALGAPI=y CONFIG_CRYPTO_ALGAPI2=y CONFIG_CRYPTO_AEAD=y CONFIG_CRYPTO_AEAD2=y CONFIG_CRYPTO_BLKCIPHER=y CONFIG_CRYPTO_BLKCIPHER2=y CONFIG_CRYPTO_HASH=y CONFIG_CRYPTO_HASH2=y CONFIG_CRYPTO_RNG=y CONFIG_CRYPTO_RNG2=y CONFIG_CRYPTO_PCOMP=m CONFIG_CRYPTO_PCOMP2=y CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_MANAGER2=y CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y CONFIG_CRYPTO_GF128MUL=m CONFIG_CRYPTO_NULL=m CONFIG_CRYPTO_PCRYPT=m CONFIG_CRYPTO_WORKQUEUE=y CONFIG_CRYPTO_CRYPTD=y CONFIG_CRYPTO_AUTHENC=m CONFIG_CRYPTO_TEST=m # # Authenticated Encryption with Associated Data # CONFIG_CRYPTO_CCM=m CONFIG_CRYPTO_GCM=m CONFIG_CRYPTO_SEQIV=y # # Block modes # CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_CTR=y CONFIG_CRYPTO_CTS=m CONFIG_CRYPTO_ECB=y CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_XTS=m CONFIG_CRYPTO_FPU=y # # Hash modes # CONFIG_CRYPTO_HMAC=y CONFIG_CRYPTO_XCBC=m CONFIG_CRYPTO_VMAC=m # # Digest # CONFIG_CRYPTO_CRC32C=y CONFIG_CRYPTO_CRC32C_INTEL=m CONFIG_CRYPTO_GHASH=m CONFIG_CRYPTO_MD4=m CONFIG_CRYPTO_MD5=y CONFIG_CRYPTO_MICHAEL_MIC=m CONFIG_CRYPTO_RMD128=m CONFIG_CRYPTO_RMD160=m CONFIG_CRYPTO_RMD256=m CONFIG_CRYPTO_RMD320=m CONFIG_CRYPTO_SHA1=y CONFIG_CRYPTO_SHA256=m CONFIG_CRYPTO_SHA512=m CONFIG_CRYPTO_TGR192=m CONFIG_CRYPTO_WP512=m CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m # # Ciphers # CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_AES_X86_64=y CONFIG_CRYPTO_AES_NI_INTEL=y CONFIG_CRYPTO_ANUBIS=m CONFIG_CRYPTO_ARC4=m CONFIG_CRYPTO_BLOWFISH=m CONFIG_CRYPTO_CAMELLIA=m CONFIG_CRYPTO_CAST5=m CONFIG_CRYPTO_CAST6=m CONFIG_CRYPTO_DES=m CONFIG_CRYPTO_FCRYPT=m CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SALSA20_X86_64=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_TWOFISH_COMMON=m CONFIG_CRYPTO_TWOFISH_X86_64=m # # Compression # CONFIG_CRYPTO_DEFLATE=m CONFIG_CRYPTO_ZLIB=m CONFIG_CRYPTO_LZO=m # # Random Number Generation # CONFIG_CRYPTO_ANSI_CPRNG=m CONFIG_CRYPTO_USER_API=y CONFIG_CRYPTO_USER_API_HASH=y CONFIG_CRYPTO_USER_API_SKCIPHER=y CONFIG_CRYPTO_HW=y CONFIG_CRYPTO_DEV_PADLOCK=m CONFIG_CRYPTO_DEV_PADLOCK_AES=m CONFIG_CRYPTO_DEV_PADLOCK_SHA=m CONFIG_CRYPTO_DEV_HIFN_795X=m CONFIG_CRYPTO_DEV_HIFN_795X_RNG=y CONFIG_HAVE_KVM=y CONFIG_HAVE_KVM_IRQCHIP=y CONFIG_HAVE_KVM_EVENTFD=y CONFIG_KVM_APIC_ARCHITECTURE=y CONFIG_KVM_MMIO=y CONFIG_KVM_ASYNC_PF=y CONFIG_VIRTUALIZATION=y CONFIG_KVM=m CONFIG_KVM_INTEL=m CONFIG_KVM_AMD=m CONFIG_KVM_MMU_AUDIT=y CONFIG_VHOST_NET=m CONFIG_VIRTIO=y CONFIG_VIRTIO_RING=y CONFIG_VIRTIO_PCI=y CONFIG_VIRTIO_BALLOON=m CONFIG_BINARY_PRINTF=y # # Library routines # CONFIG_RAID6_PQ=m CONFIG_BITREVERSE=y CONFIG_GENERIC_FIND_FIRST_BIT=y CONFIG_GENERIC_FIND_NEXT_BIT=y CONFIG_GENERIC_FIND_LAST_BIT=y CONFIG_CRC_CCITT=m CONFIG_CRC16=y CONFIG_CRC_T10DIF=y CONFIG_CRC_ITU_T=m CONFIG_CRC32=y CONFIG_CRC7=m CONFIG_LIBCRC32C=m CONFIG_ZLIB_INFLATE=y CONFIG_ZLIB_DEFLATE=m CONFIG_LZO_COMPRESS=y CONFIG_LZO_DECOMPRESS=y CONFIG_XZ_DEC=y CONFIG_XZ_DEC_X86=y CONFIG_XZ_DEC_POWERPC=y CONFIG_XZ_DEC_IA64=y CONFIG_XZ_DEC_ARM=y CONFIG_XZ_DEC_ARMTHUMB=y CONFIG_XZ_DEC_SPARC=y CONFIG_XZ_DEC_BCJ=y # CONFIG_XZ_DEC_TEST is not set CONFIG_DECOMPRESS_GZIP=y CONFIG_DECOMPRESS_BZIP2=y CONFIG_DECOMPRESS_LZMA=y CONFIG_DECOMPRESS_XZ=y CONFIG_DECOMPRESS_LZO=y CONFIG_GENERIC_ALLOCATOR=y CONFIG_REED_SOLOMON=m CONFIG_REED_SOLOMON_DEC16=y CONFIG_TEXTSEARCH=y CONFIG_TEXTSEARCH_KMP=m CONFIG_TEXTSEARCH_BM=m CONFIG_TEXTSEARCH_FSM=m CONFIG_BTREE=y CONFIG_HAS_IOMEM=y CONFIG_HAS_IOPORT=y CONFIG_HAS_DMA=y CONFIG_CHECK_SIGNATURE=y CONFIG_NLATTR=y CONFIG_LRU_CACHE=m CONFIG_AVERAGE=y CONFIG_PCI_DEFAULT_USE_CRS=y ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 17:18 ` Mel Gorman @ 2011-05-03 9:54 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-05-03 9:54 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote: > > > This is the output of perf record -g -a -f sleep 5 > > > > > > (hopefully the list won't choke) > > > > Um, this one actually shows kswapd > > > > James > > > > --- > > > > # Events: 6K cycles > > # > > # Overhead Command Shared Object Symbol > > # ........ ........... ................... ....................................... > > # > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > > | > > --- shrink_slab > > | > > |--99.91%-- kswapd > > | kthread > > | kernel_thread_helper > > --0.09%-- [...] > > > > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use > compaction instead of lumpy reclaim" is related unless we are seeing > two problems that happen to manifest in a similar manner. That is a distinct possibility. > > However, there were a number of changes made to dcache in particular > for 2.6.38. Specifically thinks like dentry_kill use trylock and is > happy to loop around if it fails to acquire anything. See things like > this for example; > [ text deleted ] > Way hey, cgroups are also in the mix. How jolly. > > Is systemd a common element of the machines hitting this bug by any > chance? Not in my case, using upstart on my machine. > Colin ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-03 9:54 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-05-03 9:54 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote: > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote: > > On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote: > > > This is the output of perf record -g -a -f sleep 5 > > > > > > (hopefully the list won't choke) > > > > Um, this one actually shows kswapd > > > > James > > > > --- > > > > # Events: 6K cycles > > # > > # Overhead Command Shared Object Symbol > > # ........ ........... ................... ....................................... > > # > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab > > | > > --- shrink_slab > > | > > |--99.91%-- kswapd > > | kthread > > | kernel_thread_helper > > --0.09%-- [...] > > > > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use > compaction instead of lumpy reclaim" is related unless we are seeing > two problems that happen to manifest in a similar manner. That is a distinct possibility. > > However, there were a number of changes made to dcache in particular > for 2.6.38. Specifically thinks like dentry_kill use trylock and is > happy to loop around if it fails to acquire anything. See things like > this for example; > [ text deleted ] > Way hey, cgroups are also in the mix. How jolly. > > Is systemd a common element of the machines hitting this bug by any > chance? Not in my case, using upstart on my machine. > Colin -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-28 15:08 ` Mel Gorman @ 2011-04-28 17:10 ` Colin Ian King -1 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 17:10 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote: [ text deleted ] > Another consequence of this patch is that when high order allocations > are in progress (is the test case fork heavy in any way for > example? alternatively, it might be something in the storage stack > that requires high-order allocs) we are no longer necessarily going > to sleep because of should_reclaim_continue() check. This could > explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is > set (does unsetting CONFIG_COMPACTION help). If the bug only triggers > for CONFIG_COMPACTION, does the following *untested* patch help any? Afraid to report this patch didn't help either. > > (as a warning, I'm offline Friday until Tuesday morning. I'll try > check mail over the weekend but it's unlikely I'll find a terminal > or be allowed to use it without an ass kicking) Ditto, me, to, I will pick this up Tuesday. > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 148c6e6..c74a501 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone, > return false; > > /* > - * If we failed to reclaim and have scanned the full list, stop. > - * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far > - * faster but obviously would be less likely to succeed > - * allocation. If this is desirable, use GFP_REPEAT to decide > - * if both reclaimed and scanned should be checked or just > - * reclaimed > + * For direct reclaimers > + * If we failed to reclaim and have scanned the full list, stop. > + * The caller will check congestion and sleep if necessary until > + * some IO completes. > + * For kswapd > + * Check just nr_reclaimed. If we are failing to reclaim, we > + * want to stop this reclaim loop, increase the priority and > + * go to sleep if necessary to allow IO a change to complete. > + * This avoids kswapd going into a busy loop in shrink_zone() > */ > - if (!nr_reclaimed && !nr_scanned) > - return false; > + if (!nr_reclaimed) { > + if (current_is_kswapd()) > + return false; > + else if (!nr_scanned) > + return false; > + } > > /* > * If we have not reclaimed enough pages for compaction and the > @@ -1924,8 +1931,13 @@ restart: > > /* reclaim/compaction might need reclaim to continue */ > if (should_continue_reclaim(zone, nr_reclaimed, > - sc->nr_scanned - nr_scanned, sc)) > + sc->nr_scanned - nr_scanned, sc)) { > + /* Throttle direct reclaimers if congested */ > + if (!current_is_kswapd()) > + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10); > + > goto restart; > + } > > throttle_vm_writeout(sc->gfp_mask); > } ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 17:10 ` Colin Ian King 0 siblings, 0 replies; 138+ messages in thread From: Colin Ian King @ 2011-04-28 17:10 UTC (permalink / raw) To: Mel Gorman Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4, mgorman On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote: [ text deleted ] > Another consequence of this patch is that when high order allocations > are in progress (is the test case fork heavy in any way for > example? alternatively, it might be something in the storage stack > that requires high-order allocs) we are no longer necessarily going > to sleep because of should_reclaim_continue() check. This could > explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is > set (does unsetting CONFIG_COMPACTION help). If the bug only triggers > for CONFIG_COMPACTION, does the following *untested* patch help any? Afraid to report this patch didn't help either. > > (as a warning, I'm offline Friday until Tuesday morning. I'll try > check mail over the weekend but it's unlikely I'll find a terminal > or be allowed to use it without an ass kicking) Ditto, me, to, I will pick this up Tuesday. > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 148c6e6..c74a501 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone, > return false; > > /* > - * If we failed to reclaim and have scanned the full list, stop. > - * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far > - * faster but obviously would be less likely to succeed > - * allocation. If this is desirable, use GFP_REPEAT to decide > - * if both reclaimed and scanned should be checked or just > - * reclaimed > + * For direct reclaimers > + * If we failed to reclaim and have scanned the full list, stop. > + * The caller will check congestion and sleep if necessary until > + * some IO completes. > + * For kswapd > + * Check just nr_reclaimed. If we are failing to reclaim, we > + * want to stop this reclaim loop, increase the priority and > + * go to sleep if necessary to allow IO a change to complete. > + * This avoids kswapd going into a busy loop in shrink_zone() > */ > - if (!nr_reclaimed && !nr_scanned) > - return false; > + if (!nr_reclaimed) { > + if (current_is_kswapd()) > + return false; > + else if (!nr_scanned) > + return false; > + } > > /* > * If we have not reclaimed enough pages for compaction and the > @@ -1924,8 +1931,13 @@ restart: > > /* reclaim/compaction might need reclaim to continue */ > if (should_continue_reclaim(zone, nr_reclaimed, > - sc->nr_scanned - nr_scanned, sc)) > + sc->nr_scanned - nr_scanned, sc)) { > + /* Throttle direct reclaimers if congested */ > + if (!current_is_kswapd()) > + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10); > + > goto restart; > + } > > throttle_vm_writeout(sc->gfp_mask); > } -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 17:21 ` James Bottomley @ 2011-04-28 0:37 ` Dave Chinner -1 siblings, 0 replies; 138+ messages in thread From: Dave Chinner @ 2011-04-28 0:37 UTC (permalink / raw) To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel On Wed, Apr 27, 2011 at 12:21:42PM -0500, James Bottomley wrote: > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > case better. > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > This is the trace with sysrq-l and sysrq-w > > The repro this time doesn't have a soft lockup, just the tar is hung and > one of my CPUs is in 99% system. > > James > > --- > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > [ 454.753048] Call Trace: > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d > [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde > [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13 Looks like it is blocked waiting for the journal. > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > [ 454.839453] Call Trace: > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 which is blocked waiting for _data_ IO completion. > [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 > [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 > [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 > [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 > [ 454.899530] Call Trace: > [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e > [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 > [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a > [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b And tar is blocked waiting for the journal, too. Looks like a case of a ordered mode data writeback journal stall (the same basic problem as the ext3 fsync sync-the-world issue). Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-28 0:37 ` Dave Chinner 0 siblings, 0 replies; 138+ messages in thread From: Dave Chinner @ 2011-04-28 0:37 UTC (permalink / raw) To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel On Wed, Apr 27, 2011 at 12:21:42PM -0500, James Bottomley wrote: > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held > > case better. > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > This is the trace with sysrq-l and sysrq-w > > The repro this time doesn't have a soft lockup, just the tar is hung and > one of my CPUs is in 99% system. > > James > > --- > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > [ 454.753048] Call Trace: > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d > [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde > [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13 Looks like it is blocked waiting for the journal. > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > [ 454.839453] Call Trace: > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 which is blocked waiting for _data_ IO completion. > [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 > [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 > [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 > [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 > [ 454.899530] Call Trace: > [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e > [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 > [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a > [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b And tar is blocked waiting for the journal, too. Looks like a case of a ordered mode data writeback journal stall (the same basic problem as the ext3 fsync sync-the-world issue). Cheers, Dave. -- Dave Chinner david@fromorbit.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-27 17:21 ` James Bottomley @ 2011-04-29 10:23 ` Sedat Dilek -1 siblings, 0 replies; 138+ messages in thread From: Sedat Dilek @ 2011-04-29 10:23 UTC (permalink / raw) To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel Sorry for the top-posting! But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. First, I have seen negative effects on my UP-system was when playing with linux-next [2]. It was not clear what the origin was and the the side-effects were somehow "bizarre". The issue could be easily reproduced by tar-ing the kernel build-dir to an external USB-hdd. The issue kept RCU and TIP folks really busy. Before stepping 4 weeks in the dark, give it a try and let me know in case of success. For the systemd/cgroups part of the discussion (yes, I followed this thread in parallel): The patch from [4] might be interesting (untested here). Hope this helps you. Have fun! - Sedat - [1] http://git.us.kernel.org/?p=linux/kernel/git/tip/linux-2.6-tip.git;a=commit;h=ce31332d3c77532d6ea97ddcb475a2b02dd358b4 [2] http://lkml.org/lkml/2011/3/25/97 [3] http://lkml.org/lkml/2011/4/28/444 [4] https://patchwork.kernel.org/patch/738921/ On Wed, Apr 27, 2011 at 7:21 PM, James Bottomley <James.Bottomley@hansenpartnership.com> wrote: > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: >> Ok, I'd try turning it on so we catch the sleeping with a spinlock held >> case better. > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > This is the trace with sysrq-l and sysrq-w > > The repro this time doesn't have a soft lockup, just the tar is hung and > one of my CPUs is in 99% system. > > James > > --- > > > [ 453.351255] SysRq : Show backtrace of all active CPUs > [ 453.352601] sending NMI to all CPUs: > [ 453.353849] NMI backtrace for cpu 3 > [ 453.355545] CPU 3 > [ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.363188] > [ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38 > [ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246 > [ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001 > [ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820 > [ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad > [ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000 > [ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820 > [ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 > [ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000) > [ 453.392649] Stack: > [ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5 > [ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000 > [ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89 > [ 453.400015] Call Trace: > [ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] > [ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 > [ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 > [ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17 > [ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f > [ 453.421534] Call Trace: > [ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] > [ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 > [ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 > [ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 453.439654] NMI backtrace for cpu 2 > [ 453.441508] CPU 2 > [ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.449517] > [ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046 > [ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 > [ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640 > [ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5 > [ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030 > [ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002 > [ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 > [ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700) > [ 453.479024] Stack: > [ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290 > [ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170 > [ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868 > [ 453.486518] Call Trace: > [ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 453.499958] Call Trace: > [ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.509598] NMI backtrace for cpu 1 > [ 453.511390] CPU 1 > [ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.519288] > [ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 > [ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 > [ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 > [ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 > [ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 > [ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001 > [ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 > [ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) > [ 453.549274] Stack: > [ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 > [ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 > [ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 > [ 453.557156] Call Trace: > [ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 453.570292] Call Trace: > [ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.579811] NMI backtrace for cpu 0 > [ 453.581279] CPU 0 > [ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.587576] > [ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14 > [ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883 > [ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040 > [ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d > [ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000 > [ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d > [ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001 > [ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 > [ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0 > [ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) > [ 453.611134] Stack: > [ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb > [ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002 > [ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77 > [ 453.617101] Call Trace: > [ 453.618492] <IRQ> > [ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 > [ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 > [ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11 > [ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e > [ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 453.649695] <EOI> > [ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55 > [ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2 > [ 453.663728] Call Trace: > [ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 > [ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 > [ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11 > [ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e > [ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 454.680802] SysRq : Show Blocked State > [ 454.683427] task PC stack pid father > [ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000 > [ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000 > [ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40 > [ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db > [ 454.696969] Call Trace: > [ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde > [ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c > [ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17 > [ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101 > [ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 > [ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f > [ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c > [ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 > [ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3 > [ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353 > [ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121 > [ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea > [ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe > [ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a > [ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18 > [ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > [ 454.753048] Call Trace: > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d > [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde > [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > [ 454.839453] Call Trace: > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 > [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 > [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 > [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 > [ 454.899530] Call Trace: > [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e > [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 > [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a > [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000 > [ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000 > [ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40 > [ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8 > [ 454.975323] Call Trace: > [ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12 > [ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b > [ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68 > [ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43 > [ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640 > [ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff > [ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63 > [ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 > [ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f > [ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8 > [ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277 > [ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a > [ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0 > [ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 > [ 455.027487] ktime : 455879.226285 > [ 455.030142] sched_clk : 455024.886257 > [ 455.032786] cpu_clk : 455024.886397 > [ 455.035352] jiffies : 4295123185 > [ 455.037904] sched_clock_stable : 1 > [ 455.040413] > [ 455.042892] sysctl_sched > [ 455.045306] .sysctl_sched_latency : 18.000000 > [ 455.047775] .sysctl_sched_min_granularity : 2.250000 > [ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000 > [ 455.052643] .sysctl_sched_child_runs_first : 0 > [ 455.055034] .sysctl_sched_features : 7279 > [ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic) > [ 455.059829] > [ 455.059830] cpu#0, 2491.994 MHz > [ 455.064443] .nr_running : 0 > [ 455.066757] .load : 0 > [ 455.069054] .nr_switches : 146510 > [ 455.071353] .nr_load_updates : 233084 > [ 455.073642] .nr_uninterruptible : 2 > [ 455.075894] .next_balance : 4295.122831 > [ 455.078152] .curr->pid : 0 > [ 455.080396] .clock : 454680.481348 > [ 455.082634] .cpu_load[0] : 0 > [ 455.084867] .cpu_load[1] : 0 > [ 455.087065] .cpu_load[2] : 0 > [ 455.089233] .cpu_load[3] : 0 > [ 455.091390] .cpu_load[4] : 0 > [ 455.093499] .yld_count : 0 > [ 455.095605] .sched_switch : 0 > [ 455.097667] .sched_count : 149062 > [ 455.099765] .sched_goidle : 62756 > [ 455.101781] .avg_idle : 1000000 > [ 455.103807] .ttwu_count : 77219 > [ 455.105958] .ttwu_local : 74144 > [ 455.107957] .bkl_count : 0 > [ 455.109914] > [ 455.109915] cfs_rq[0]:/ > [ 455.113642] .exec_clock : 20017.048374 > [ 455.115515] .MIN_vruntime : 0.000001 > [ 455.117353] .min_vruntime : 28900.800090 > [ 455.119185] .max_vruntime : 0.000001 > [ 455.121028] .spread : 0.000000 > [ 455.122820] .spread0 : 0.000000 > [ 455.124581] .nr_spread_over : 54 > [ 455.126318] .nr_running : 0 > [ 455.128045] .load : 0 > [ 455.129743] .load_avg : 0.000000 > [ 455.131414] .load_period : 0.000000 > [ 455.133052] .load_contrib : 0 > [ 455.134692] .load_tg : 0 > [ 455.136305] > [ 455.136306] runnable tasks: > [ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.136310] ---------------------------------------------------------------------------------------------------------- > [ 455.142856] > [ 455.142857] cpu#1, 2491.994 MHz > [ 455.146215] .nr_running : 0 > [ 455.147931] .load : 0 > [ 455.149648] .nr_switches : 62175 > [ 455.151348] .nr_load_updates : 217324 > [ 455.153045] .nr_uninterruptible : 2 > [ 455.154747] .next_balance : 4295.123306 > [ 455.156457] .curr->pid : 0 > [ 455.158149] .clock : 455157.364642 > [ 455.159867] .cpu_load[0] : 0 > [ 455.161594] .cpu_load[1] : 0 > [ 455.163283] .cpu_load[2] : 0 > [ 455.164954] .cpu_load[3] : 0 > [ 455.166575] .cpu_load[4] : 0 > [ 455.168185] .yld_count : 60 > [ 455.169791] .sched_switch : 0 > [ 455.171401] .sched_count : 62899 > [ 455.172984] .sched_goidle : 27394 > [ 455.174580] .avg_idle : 1000000 > [ 455.176171] .ttwu_count : 30510 > [ 455.177739] .ttwu_local : 25277 > [ 455.179292] .bkl_count : 0 > [ 455.180882] > [ 455.180883] cfs_rq[1]:/ > [ 455.183954] .exec_clock : 10655.021809 > [ 455.185550] .MIN_vruntime : 0.000001 > [ 455.187141] .min_vruntime : 19718.135550 > [ 455.188771] .max_vruntime : 0.000001 > [ 455.190407] .spread : 0.000000 > [ 455.192016] .spread0 : -9182.664540 > [ 455.193634] .nr_spread_over : 80 > [ 455.195242] .nr_running : 0 > [ 455.196848] .load : 0 > [ 455.198441] .load_avg : 0.000000 > [ 455.200059] .load_period : 0.000000 > [ 455.201654] .load_contrib : 0 > [ 455.203240] .load_tg : 0 > [ 455.204836] > [ 455.204837] runnable tasks: > [ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.204841] ---------------------------------------------------------------------------------------------------------- > [ 455.211403] > [ 455.211404] cpu#2, 2491.994 MHz > [ 455.214958] .nr_running : 0 > [ 455.216771] .load : 0 > [ 455.218564] .nr_switches : 37444 > [ 455.220394] .nr_load_updates : 219042 > [ 455.222202] .nr_uninterruptible : 0 > [ 455.224039] .next_balance : 4295.123375 > [ 455.225897] .curr->pid : 0 > [ 455.227769] .clock : 455227.204963 > [ 455.229681] .cpu_load[0] : 0 > [ 455.231548] .cpu_load[1] : 0 > [ 455.233353] .cpu_load[2] : 0 > [ 455.235059] .cpu_load[3] : 0 > [ 455.236676] .cpu_load[4] : 0 > [ 455.238280] .yld_count : 0 > [ 455.239879] .sched_switch : 0 > [ 455.241471] .sched_count : 37815 > [ 455.243075] .sched_goidle : 16831 > [ 455.244674] .avg_idle : 1000000 > [ 455.246270] .ttwu_count : 18348 > [ 455.247849] .ttwu_local : 16899 > [ 455.249430] .bkl_count : 0 > [ 455.250992] > [ 455.250993] cfs_rq[2]:/ > [ 455.254040] .exec_clock : 6758.351942 > [ 455.255630] .MIN_vruntime : 0.000001 > [ 455.257236] .min_vruntime : 13719.942866 > [ 455.258861] .max_vruntime : 0.000001 > [ 455.260497] .spread : 0.000000 > [ 455.262122] .spread0 : -15180.857224 > [ 455.263753] .nr_spread_over : 21 > [ 455.265389] .nr_running : 0 > [ 455.267018] .load : 0 > [ 455.268637] .load_avg : 0.000000 > [ 455.270292] .load_period : 0.000000 > [ 455.271911] .load_contrib : 0 > [ 455.273530] .load_tg : 0 > [ 455.275163] > [ 455.275165] runnable tasks: > [ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.275169] ---------------------------------------------------------------------------------------------------------- > [ 455.281810] > [ 455.281811] cpu#3, 2491.994 MHz > [ 455.285349] .nr_running : 4 > [ 455.287149] .load : 2048 > [ 455.288952] .nr_switches : 30358 > [ 455.290787] .nr_load_updates : 223302 > [ 455.292599] .nr_uninterruptible : 1 > [ 455.294419] .next_balance : 4295.123506 > [ 455.296250] .curr->pid : 46 > [ 455.298090] .clock : 431792.394314 > [ 455.299953] .cpu_load[0] : 2048 > [ 455.301810] .cpu_load[1] : 2048 > [ 455.303610] .cpu_load[2] : 2048 > [ 455.305311] .cpu_load[3] : 2048 > [ 455.306926] .cpu_load[4] : 2048 > [ 455.308531] .yld_count : 63 > [ 455.310130] .sched_switch : 0 > [ 455.311723] .sched_count : 30824 > [ 455.313315] .sched_goidle : 13491 > [ 455.314904] .avg_idle : 1000000 > [ 455.316481] .ttwu_count : 13858 > [ 455.318057] .ttwu_local : 12507 > [ 455.319629] .bkl_count : 0 > [ 455.321187] > [ 455.321188] cfs_rq[3]:/ > [ 455.324213] .exec_clock : 5334.144946 > [ 455.325795] .MIN_vruntime : 13295.262803 > [ 455.327401] .min_vruntime : 13302.523317 > [ 455.329046] .max_vruntime : 13295.262803 > [ 455.330656] .spread : 0.000000 > [ 455.332297] .spread0 : -15598.276773 > [ 455.333925] .nr_spread_over : 117 > [ 455.335577] .nr_running : 2 > [ 455.337204] .load : 2048 > [ 455.338823] .load_avg : 0.000000 > [ 455.340476] .load_period : 0.000000 > [ 455.342094] .load_contrib : 0 > [ 455.343702] .load_tg : 0 > [ 455.345302] > [ 455.345303] rt_rq[3]:/ > [ 455.348416] .rt_nr_running : 1 > [ 455.350006] .rt_throttled : 0 > [ 455.351587] .rt_time : 0.000000 > [ 455.353173] .rt_runtime : 950.000000 > [ 455.354759] > [ 455.354760] runnable tasks: > [ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.354764] ---------------------------------------------------------------------------------------------------------- > [ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 / > [ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 / > [ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 / > [ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 / > [ 455.369317] > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-29 10:23 ` Sedat Dilek 0 siblings, 0 replies; 138+ messages in thread From: Sedat Dilek @ 2011-04-29 10:23 UTC (permalink / raw) To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel Sorry for the top-posting! But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. First, I have seen negative effects on my UP-system was when playing with linux-next [2]. It was not clear what the origin was and the the side-effects were somehow "bizarre". The issue could be easily reproduced by tar-ing the kernel build-dir to an external USB-hdd. The issue kept RCU and TIP folks really busy. Before stepping 4 weeks in the dark, give it a try and let me know in case of success. For the systemd/cgroups part of the discussion (yes, I followed this thread in parallel): The patch from [4] might be interesting (untested here). Hope this helps you. Have fun! - Sedat - [1] http://git.us.kernel.org/?p=linux/kernel/git/tip/linux-2.6-tip.git;a=commit;h=ce31332d3c77532d6ea97ddcb475a2b02dd358b4 [2] http://lkml.org/lkml/2011/3/25/97 [3] http://lkml.org/lkml/2011/4/28/444 [4] https://patchwork.kernel.org/patch/738921/ On Wed, Apr 27, 2011 at 7:21 PM, James Bottomley <James.Bottomley@hansenpartnership.com> wrote: > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: >> Ok, I'd try turning it on so we catch the sleeping with a spinlock held >> case better. > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? > > This is the trace with sysrq-l and sysrq-w > > The repro this time doesn't have a soft lockup, just the tar is hung and > one of my CPUs is in 99% system. > > James > > --- > > > [ 453.351255] SysRq : Show backtrace of all active CPUs > [ 453.352601] sending NMI to all CPUs: > [ 453.353849] NMI backtrace for cpu 3 > [ 453.355545] CPU 3 > [ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.363188] > [ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38 > [ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246 > [ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001 > [ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820 > [ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad > [ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000 > [ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820 > [ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 > [ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000) > [ 453.392649] Stack: > [ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5 > [ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000 > [ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89 > [ 453.400015] Call Trace: > [ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] > [ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 > [ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 > [ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17 > [ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f > [ 453.421534] Call Trace: > [ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915] > [ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0 > [ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166 > [ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798 > [ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 > [ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 453.439654] NMI backtrace for cpu 2 > [ 453.441508] CPU 2 > [ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.449517] > [ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046 > [ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 > [ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640 > [ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5 > [ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030 > [ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002 > [ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 > [ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700) > [ 453.479024] Stack: > [ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290 > [ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170 > [ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868 > [ 453.486518] Call Trace: > [ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 453.499958] Call Trace: > [ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.509598] NMI backtrace for cpu 1 > [ 453.511390] CPU 1 > [ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.519288] > [ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100 > [ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046 > [ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 > [ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640 > [ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5 > [ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010 > [ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001 > [ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000 > [ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0 > [ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700) > [ 453.549274] Stack: > [ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290 > [ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170 > [ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868 > [ 453.557156] Call Trace: > [ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42 > [ 453.570292] Call Trace: > [ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 > [ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225 > [ 453.579811] NMI backtrace for cpu 0 > [ 453.581279] CPU 0 > [ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] > [ 453.587576] > [ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO > [ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14 > [ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883 > [ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040 > [ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d > [ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000 > [ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d > [ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001 > [ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000 > [ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0 > [ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020) > [ 453.611134] Stack: > [ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb > [ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002 > [ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77 > [ 453.617101] Call Trace: > [ 453.618492] <IRQ> > [ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 > [ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 > [ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11 > [ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e > [ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 453.649695] <EOI> > [ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55 > [ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2 > [ 453.663728] Call Trace: > [ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12 > [ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74 > [ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11 > [ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e > [ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88 > [ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10 > [ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c > [ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e > [ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc > [ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f > [ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94 > [ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87 > [ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e > [ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f > [ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab > [ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36 > [ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198 > [ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56 > [ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29 > [ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0 > [ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e > [ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5 > [ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13 > [ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd > [ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100 > [ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100 > [ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166 > [ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf > [ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74 > [ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9 > [ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3 > [ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140 > [ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111 > [ 454.680802] SysRq : Show Blocked State > [ 454.683427] task PC stack pid father > [ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000 > [ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000 > [ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40 > [ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db > [ 454.696969] Call Trace: > [ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde > [ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c > [ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17 > [ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101 > [ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 > [ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f > [ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c > [ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 > [ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3 > [ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353 > [ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121 > [ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea > [ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe > [ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a > [ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18 > [ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000 > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000 > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40 > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3 > [ 454.753048] Call Trace: > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0 > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2 > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04 > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2 > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6 > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7 > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a > [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc > [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161 > [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b > [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335 > [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b > [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f > [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d > [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde > [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5 > [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d > [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000 > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000 > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40 > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0 > [ 454.839453] Call Trace: > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12 > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74 > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163 > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26 > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194 > [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 > [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a > [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a > [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10 > [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c > [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10 > [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 > [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13 > [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000 > [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282 > [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40 > [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3 > [ 454.899530] Call Trace: > [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78 > [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d > [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c > [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42 > [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66 > [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83 > [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c > [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175 > [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e > [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204 > [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a > [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272 > [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba > [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234 > [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff > [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217 > [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4 > [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3 > [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e > [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b > [ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000 > [ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000 > [ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40 > [ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8 > [ 454.975323] Call Trace: > [ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12 > [ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e > [ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b > [ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f > [ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68 > [ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d > [ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43 > [ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640 > [ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff > [ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63 > [ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33 > [ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd > [ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0 > [ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f > [ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8 > [ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277 > [ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a > [ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0 > [ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1 > [ 455.027487] ktime : 455879.226285 > [ 455.030142] sched_clk : 455024.886257 > [ 455.032786] cpu_clk : 455024.886397 > [ 455.035352] jiffies : 4295123185 > [ 455.037904] sched_clock_stable : 1 > [ 455.040413] > [ 455.042892] sysctl_sched > [ 455.045306] .sysctl_sched_latency : 18.000000 > [ 455.047775] .sysctl_sched_min_granularity : 2.250000 > [ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000 > [ 455.052643] .sysctl_sched_child_runs_first : 0 > [ 455.055034] .sysctl_sched_features : 7279 > [ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic) > [ 455.059829] > [ 455.059830] cpu#0, 2491.994 MHz > [ 455.064443] .nr_running : 0 > [ 455.066757] .load : 0 > [ 455.069054] .nr_switches : 146510 > [ 455.071353] .nr_load_updates : 233084 > [ 455.073642] .nr_uninterruptible : 2 > [ 455.075894] .next_balance : 4295.122831 > [ 455.078152] .curr->pid : 0 > [ 455.080396] .clock : 454680.481348 > [ 455.082634] .cpu_load[0] : 0 > [ 455.084867] .cpu_load[1] : 0 > [ 455.087065] .cpu_load[2] : 0 > [ 455.089233] .cpu_load[3] : 0 > [ 455.091390] .cpu_load[4] : 0 > [ 455.093499] .yld_count : 0 > [ 455.095605] .sched_switch : 0 > [ 455.097667] .sched_count : 149062 > [ 455.099765] .sched_goidle : 62756 > [ 455.101781] .avg_idle : 1000000 > [ 455.103807] .ttwu_count : 77219 > [ 455.105958] .ttwu_local : 74144 > [ 455.107957] .bkl_count : 0 > [ 455.109914] > [ 455.109915] cfs_rq[0]:/ > [ 455.113642] .exec_clock : 20017.048374 > [ 455.115515] .MIN_vruntime : 0.000001 > [ 455.117353] .min_vruntime : 28900.800090 > [ 455.119185] .max_vruntime : 0.000001 > [ 455.121028] .spread : 0.000000 > [ 455.122820] .spread0 : 0.000000 > [ 455.124581] .nr_spread_over : 54 > [ 455.126318] .nr_running : 0 > [ 455.128045] .load : 0 > [ 455.129743] .load_avg : 0.000000 > [ 455.131414] .load_period : 0.000000 > [ 455.133052] .load_contrib : 0 > [ 455.134692] .load_tg : 0 > [ 455.136305] > [ 455.136306] runnable tasks: > [ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.136310] ---------------------------------------------------------------------------------------------------------- > [ 455.142856] > [ 455.142857] cpu#1, 2491.994 MHz > [ 455.146215] .nr_running : 0 > [ 455.147931] .load : 0 > [ 455.149648] .nr_switches : 62175 > [ 455.151348] .nr_load_updates : 217324 > [ 455.153045] .nr_uninterruptible : 2 > [ 455.154747] .next_balance : 4295.123306 > [ 455.156457] .curr->pid : 0 > [ 455.158149] .clock : 455157.364642 > [ 455.159867] .cpu_load[0] : 0 > [ 455.161594] .cpu_load[1] : 0 > [ 455.163283] .cpu_load[2] : 0 > [ 455.164954] .cpu_load[3] : 0 > [ 455.166575] .cpu_load[4] : 0 > [ 455.168185] .yld_count : 60 > [ 455.169791] .sched_switch : 0 > [ 455.171401] .sched_count : 62899 > [ 455.172984] .sched_goidle : 27394 > [ 455.174580] .avg_idle : 1000000 > [ 455.176171] .ttwu_count : 30510 > [ 455.177739] .ttwu_local : 25277 > [ 455.179292] .bkl_count : 0 > [ 455.180882] > [ 455.180883] cfs_rq[1]:/ > [ 455.183954] .exec_clock : 10655.021809 > [ 455.185550] .MIN_vruntime : 0.000001 > [ 455.187141] .min_vruntime : 19718.135550 > [ 455.188771] .max_vruntime : 0.000001 > [ 455.190407] .spread : 0.000000 > [ 455.192016] .spread0 : -9182.664540 > [ 455.193634] .nr_spread_over : 80 > [ 455.195242] .nr_running : 0 > [ 455.196848] .load : 0 > [ 455.198441] .load_avg : 0.000000 > [ 455.200059] .load_period : 0.000000 > [ 455.201654] .load_contrib : 0 > [ 455.203240] .load_tg : 0 > [ 455.204836] > [ 455.204837] runnable tasks: > [ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.204841] ---------------------------------------------------------------------------------------------------------- > [ 455.211403] > [ 455.211404] cpu#2, 2491.994 MHz > [ 455.214958] .nr_running : 0 > [ 455.216771] .load : 0 > [ 455.218564] .nr_switches : 37444 > [ 455.220394] .nr_load_updates : 219042 > [ 455.222202] .nr_uninterruptible : 0 > [ 455.224039] .next_balance : 4295.123375 > [ 455.225897] .curr->pid : 0 > [ 455.227769] .clock : 455227.204963 > [ 455.229681] .cpu_load[0] : 0 > [ 455.231548] .cpu_load[1] : 0 > [ 455.233353] .cpu_load[2] : 0 > [ 455.235059] .cpu_load[3] : 0 > [ 455.236676] .cpu_load[4] : 0 > [ 455.238280] .yld_count : 0 > [ 455.239879] .sched_switch : 0 > [ 455.241471] .sched_count : 37815 > [ 455.243075] .sched_goidle : 16831 > [ 455.244674] .avg_idle : 1000000 > [ 455.246270] .ttwu_count : 18348 > [ 455.247849] .ttwu_local : 16899 > [ 455.249430] .bkl_count : 0 > [ 455.250992] > [ 455.250993] cfs_rq[2]:/ > [ 455.254040] .exec_clock : 6758.351942 > [ 455.255630] .MIN_vruntime : 0.000001 > [ 455.257236] .min_vruntime : 13719.942866 > [ 455.258861] .max_vruntime : 0.000001 > [ 455.260497] .spread : 0.000000 > [ 455.262122] .spread0 : -15180.857224 > [ 455.263753] .nr_spread_over : 21 > [ 455.265389] .nr_running : 0 > [ 455.267018] .load : 0 > [ 455.268637] .load_avg : 0.000000 > [ 455.270292] .load_period : 0.000000 > [ 455.271911] .load_contrib : 0 > [ 455.273530] .load_tg : 0 > [ 455.275163] > [ 455.275165] runnable tasks: > [ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.275169] ---------------------------------------------------------------------------------------------------------- > [ 455.281810] > [ 455.281811] cpu#3, 2491.994 MHz > [ 455.285349] .nr_running : 4 > [ 455.287149] .load : 2048 > [ 455.288952] .nr_switches : 30358 > [ 455.290787] .nr_load_updates : 223302 > [ 455.292599] .nr_uninterruptible : 1 > [ 455.294419] .next_balance : 4295.123506 > [ 455.296250] .curr->pid : 46 > [ 455.298090] .clock : 431792.394314 > [ 455.299953] .cpu_load[0] : 2048 > [ 455.301810] .cpu_load[1] : 2048 > [ 455.303610] .cpu_load[2] : 2048 > [ 455.305311] .cpu_load[3] : 2048 > [ 455.306926] .cpu_load[4] : 2048 > [ 455.308531] .yld_count : 63 > [ 455.310130] .sched_switch : 0 > [ 455.311723] .sched_count : 30824 > [ 455.313315] .sched_goidle : 13491 > [ 455.314904] .avg_idle : 1000000 > [ 455.316481] .ttwu_count : 13858 > [ 455.318057] .ttwu_local : 12507 > [ 455.319629] .bkl_count : 0 > [ 455.321187] > [ 455.321188] cfs_rq[3]:/ > [ 455.324213] .exec_clock : 5334.144946 > [ 455.325795] .MIN_vruntime : 13295.262803 > [ 455.327401] .min_vruntime : 13302.523317 > [ 455.329046] .max_vruntime : 13295.262803 > [ 455.330656] .spread : 0.000000 > [ 455.332297] .spread0 : -15598.276773 > [ 455.333925] .nr_spread_over : 117 > [ 455.335577] .nr_running : 2 > [ 455.337204] .load : 2048 > [ 455.338823] .load_avg : 0.000000 > [ 455.340476] .load_period : 0.000000 > [ 455.342094] .load_contrib : 0 > [ 455.343702] .load_tg : 0 > [ 455.345302] > [ 455.345303] rt_rq[3]:/ > [ 455.348416] .rt_nr_running : 1 > [ 455.350006] .rt_throttled : 0 > [ 455.351587] .rt_time : 0.000000 > [ 455.353173] .rt_runtime : 950.000000 > [ 455.354759] > [ 455.354760] runnable tasks: > [ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep > [ 455.354764] ---------------------------------------------------------------------------------------------------------- > [ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 / > [ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 / > [ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 / > [ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 / > [ 455.369317] > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-29 10:23 ` Sedat Dilek @ 2011-04-29 15:37 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 15:37 UTC (permalink / raw) To: sedat.dilek; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > First, I have seen negative effects on my UP-system was when playing > with linux-next [2]. > It was not clear what the origin was and the the side-effects were > somehow "bizarre". > The issue could be easily reproduced by tar-ing the kernel build-dir > to an external USB-hdd. > The issue kept RCU and TIP folks really busy. > Before stepping 4 weeks in the dark, give it a try and let me know in > case of success. Well, it's highly unlikely because that's a 2.6.39 artifact and the bug showed up in 2.6.38 ... I tried it just in case with no effect, so we know it isn't the cause. > For the systemd/cgroups part of the discussion (yes, I followed this > thread in parallel): > The patch from [4] might be interesting (untested here). What makes you think that? This patch is about rcu problems in cgroup manipulation through the mounted cgroup control filesystems. The bug in question manifests when all of that is quiescent, so it's not likely to be related. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-29 15:37 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 15:37 UTC (permalink / raw) To: sedat.dilek; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > First, I have seen negative effects on my UP-system was when playing > with linux-next [2]. > It was not clear what the origin was and the the side-effects were > somehow "bizarre". > The issue could be easily reproduced by tar-ing the kernel build-dir > to an external USB-hdd. > The issue kept RCU and TIP folks really busy. > Before stepping 4 weeks in the dark, give it a try and let me know in > case of success. Well, it's highly unlikely because that's a 2.6.39 artifact and the bug showed up in 2.6.38 ... I tried it just in case with no effect, so we know it isn't the cause. > For the systemd/cgroups part of the discussion (yes, I followed this > thread in parallel): > The patch from [4] might be interesting (untested here). What makes you think that? This patch is about rcu problems in cgroup manipulation through the mounted cgroup control filesystems. The bug in question manifests when all of that is quiescent, so it's not likely to be related. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-29 15:37 ` James Bottomley @ 2011-04-29 16:31 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 16:31 UTC (permalink / raw) To: sedat.dilek Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote: > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > > First, I have seen negative effects on my UP-system was when playing > > with linux-next [2]. > > It was not clear what the origin was and the the side-effects were > > somehow "bizarre". > > The issue could be easily reproduced by tar-ing the kernel build-dir > > to an external USB-hdd. > > The issue kept RCU and TIP folks really busy. > > Before stepping 4 weeks in the dark, give it a try and let me know in > > case of success. > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug > showed up in 2.6.38 ... I tried it just in case with no effect, so we > know it isn't the cause. Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but it does seem to prevent non-PREEMPT from locking up totally (at least it survives three back to back untar runs). It's probable it alters the memory pin conditions that cause the spin, so it's masking the problem rather than fixing it. The question now is, what causes it in 2.6.38 stable? James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-29 16:31 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 16:31 UTC (permalink / raw) To: sedat.dilek Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote: > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > > First, I have seen negative effects on my UP-system was when playing > > with linux-next [2]. > > It was not clear what the origin was and the the side-effects were > > somehow "bizarre". > > The issue could be easily reproduced by tar-ing the kernel build-dir > > to an external USB-hdd. > > The issue kept RCU and TIP folks really busy. > > Before stepping 4 weeks in the dark, give it a try and let me know in > > case of success. > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug > showed up in 2.6.38 ... I tried it just in case with no effect, so we > know it isn't the cause. Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but it does seem to prevent non-PREEMPT from locking up totally (at least it survives three back to back untar runs). It's probable it alters the memory pin conditions that cause the spin, so it's masking the problem rather than fixing it. The question now is, what causes it in 2.6.38 stable? James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-29 16:31 ` James Bottomley @ 2011-04-29 18:02 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 18:02 UTC (permalink / raw) To: sedat.dilek Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote: > On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote: > > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > > > First, I have seen negative effects on my UP-system was when playing > > > with linux-next [2]. > > > It was not clear what the origin was and the the side-effects were > > > somehow "bizarre". > > > The issue could be easily reproduced by tar-ing the kernel build-dir > > > to an external USB-hdd. > > > The issue kept RCU and TIP folks really busy. > > > Before stepping 4 weeks in the dark, give it a try and let me know in > > > case of success. > > > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug > > showed up in 2.6.38 ... I tried it just in case with no effect, so we > > know it isn't the cause. > > Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but > it does seem to prevent non-PREEMPT from locking up totally (at least it > survives three back to back untar runs). > > It's probable it alters the memory pin conditions that cause the spin, > so it's masking the problem rather than fixing it. Confirmed ... it's just harder to reproduce with the hrtimers init fix. The problem definitely still exists (I had to load up the system more before doing the tar). This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w doesn't complete for an unknown reason James --- [ 575.083025] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 575.083043] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 575.083251] CPU 2 [ 575.083255] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 575.083430] [ 575.083437] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 575.083455] RIP: 0010:[<ffffffffa007ff8f>] [<ffffffffa007ff8f>] i915_gem_inactive_shrink+0x6c/0x194 [i915] [ 575.083497] RSP: 0018:ffff88006dfb7d50 EFLAGS: 00000206 [ 575.083504] RAX: ffff88005f018200 RBX: 00000000000000c0 RCX: 0000000000000000 [ 575.083512] RDX: ffff88005f01bab0 RSI: 0000000000000000 RDI: ffff880037885820 [ 575.083529] RBP: ffff88006dfb7d90 R08: 0000000000000000 R09: 0000000000017131 [ 575.083535] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff81482e0e [ 575.083541] R13: ffff88006dfb7cf0 R14: ffff88006dfb7cf8 R15: ffffffff810dd44d [ 575.083548] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 575.083555] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 575.083572] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 575.083579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 575.083587] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 575.083596] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000) [ 575.083602] Stack: [ 575.083608] ffff88006dfb7d90 ffff880037709638 ffff88006dfb7d60 ffff8800377095f0 [ 575.083626] 0000000000000000 0000000000000000 00000000000000d0 000000000005103b [ 575.083643] ffff88006dfb7de0 ffffffff810e1d89 000000000000003d 0000000000000080 [ 575.083660] Call Trace: [ 575.083676] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166 [ 575.083686] [<ffffffff810e49cc>] kswapd+0x533/0x798 [ 575.083698] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 575.083712] [<ffffffff8106e157>] kthread+0x84/0x8c [ 575.083726] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 575.083738] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 575.083750] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 575.083756] Code: e4 48 89 45 c8 75 37 48 8b 43 48 45 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00 [ 575.083887] 05 b0 00 00 00 48 39 d8 0f 18 0a 75 e1 e9 da 00 00 00 4c 89 [ 575.083949] Call Trace: [ 575.083957] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166 [ 575.083966] [<ffffffff810e49cc>] kswapd+0x533/0x798 [ 575.083976] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 575.083986] [<ffffffff8106e157>] kthread+0x84/0x8c [ 575.083998] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 575.084010] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 575.084030] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 658.883363] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 658.883382] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 658.883574] CPU 2 [ 658.883579] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 658.883749] [ 658.883757] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 658.883776] RIP: 0010:[<ffffffff81080bc0>] [<ffffffff81080bc0>] arch_local_irq_restore+0xc/0xd [ 658.883797] RSP: 0018:ffff88006dfb7d98 EFLAGS: 00000246 [ 658.883805] RAX: ffff8801005f9e68 RBX: ffff8801005e6e00 RCX: 0000000000004ae4 [ 658.883813] RDX: ffff88006dfb8000 RSI: 0000000000000246 RDI: 0000000000000246 [ 658.883821] RBP: ffff88006dfb7da0 R08: ffff8801005f9e70 R09: 00000000000007eb [ 658.883828] R10: 0000000000000000 R11: 00000000000000cd R12: ffffffff81482e0e [ 658.883835] R13: 0000000000016fd3 R14: 000000000000a731 R15: ffffffff810ddb31 [ 658.883844] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 658.883853] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 658.883860] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 658.883867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 658.883875] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 658.883883] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000) [ 658.883890] Stack: [ 658.883895] ffffffff8147bc8c ffff88006dfb7de0 ffffffff8106e7c3 ffff880000000002 [ 658.883913] 0000000000000000 ffff8801005e6000 ffff8801005f9e68 0000000000000002 [ 658.883929] 0000000000000000 ffff88006dfb7ee0 ffffffff810e45f0 0000000000000003 [ 658.883947] Call Trace: [ 658.883961] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 658.883976] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78 [ 658.883988] [<ffffffff810e45f0>] kswapd+0x157/0x798 [ 658.884000] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 658.884009] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 658.884020] [<ffffffff8106e157>] kthread+0x84/0x8c [ 658.884033] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 658.884045] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 658.884057] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 658.884062] Code: 1f 44 00 00 66 ff 05 30 16 98 00 fb 66 0f 1f 44 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 5d c3 55 48 89 e5 57 9d 0f 1f 44 00 00 5d <c3> 55 48 89 e5 fa 66 0f 1f 44 00 00 5d c3 55 48 89 e5 50 9c 58 [ 658.884238] Call Trace: [ 658.884248] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 658.884260] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78 [ 658.884269] [<ffffffff810e45f0>] kswapd+0x157/0x798 [ 658.884280] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 658.884289] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 658.884299] [<ffffffff8106e157>] kthread+0x84/0x8c [ 658.884311] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 658.884322] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 658.884333] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 690.767137] SysRq : Show backtrace of all active CPUs [ 690.767158] sending NMI to all CPUs: [ 690.767171] NMI backtrace for cpu 2 [ 690.767188] CPU 2 [ 690.767196] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 690.767457] [ 690.767468] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 690.767493] RIP: 0010:[<ffffffff810e3980>] [<ffffffff810e3980>] shrink_zone+0x221/0x4b3 [ 690.767519] RSP: 0018:ffff88006dfb7d30 EFLAGS: 00000206 [ 690.767529] RAX: 0000000000000000 RBX: ffff88006dfb7e58 RCX: 000000000000000c [ 690.767539] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 690.767550] RBP: ffff88006dfb7de0 R08: 0000000000000001 R09: 00000000000007eb [ 690.767561] R10: 0000000000000002 R11: 00000000000000f5 R12: ffff8801005e6000 [ 690.767572] R13: ffff8801005e64d8 R14: 0000000000001572 R15: 00000000ffffffff [ 690.767584] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 690.767597] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 690.767607] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 690.767618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 690.767628] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 690.767640] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000) [ 690.767650] Stack: [ 690.767657] 0000000000000001 000000000000008c 000000000000003c 0000000000000000 [ 690.767683] ffffffffffffffff ffff8801005e6480 000000000000000c 0000000c81118591 [ 690.767717] 0000000000000000 0000000000000000 ffff880100530380 0000000000000000 [ 690.767742] Call Trace: [ 690.767764] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae [ 690.767779] [<ffffffff810e49a6>] kswapd+0x50d/0x798 [ 690.767794] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 690.767811] [<ffffffff8106e157>] kthread+0x84/0x8c [ 690.767830] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 690.767846] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 690.767861] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 690.767868] Code: e2 ff ff 83 7d 80 00 44 8b 85 50 ff ff ff 74 1b 8a 4d 8c 31 f6 41 83 ff 01 40 0f 96 c6 31 d2 48 d3 e8 48 0f af 44 f5 c0 49 f7 f6 [ 690.768028] 89 c1 48 8d 51 04 49 03 44 d5 00 48 83 f8 1f 49 89 44 d5 00 [ 690.768113] Call Trace: [ 690.768126] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae [ 690.768139] [<ffffffff810e49a6>] kswapd+0x50d/0x798 [ 690.768154] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 690.768168] [<ffffffff8106e157>] kthread+0x84/0x8c [ 690.768183] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 690.768199] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 690.768213] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 690.768223] NMI backtrace for cpu 3 [ 690.768236] CPU 3 [ 690.768244] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 690.768832] [ 690.768843] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 690.768867] RIP: 0010:[<ffffffff81275b36>] [<ffffffff81275b36>] intel_idle+0xaa/0x100 [ 690.768890] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046 [ 690.768900] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 [ 690.768912] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640 [ 690.768923] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003c2 [ 690.768934] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030 [ 690.768946] R13: 12191803fa451798 R14: 0000000000000004 R15: 0000000000000003 [ 690.768959] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 [ 690.768971] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 690.768982] CR2: 0000000000429bf0 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 690.768993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 690.769005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 690.769017] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700) [ 690.769027] Stack: [ 690.769034] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1280 0000000000011280 [ 690.769063] ffff8800715dfeb8 000000038139c77a ffffe8ffffcc0170 ffffe8ffffcc0170 [ 690.769092] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b668 [ 690.769121] Call Trace: [ 690.769137] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 690.769153] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166 [ 690.769173] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 690.769191] [<ffffffff8146ac57>] start_secondary+0x223/0x225 [ 690.769203] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 0b e0 ff 4c 29 e8 48 89 c7 e8 ab 2b de ff 4c 69 e0 40 42 [ 690.769477] Call Trace: [ 690.769491] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 690.769506] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166 [ 690.769522] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 690.770451] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 690.771380] [<ffffffff81483f73>] ? smp_apic_timer_interrupt+0x7e/0x8c [ 692.887516] SysRq : Show Blocked State [ 692.887534] task PC stack pid father [ 692.887551] fsnotify_mark D 0000000000000000 0 49 2 0x00000000 [ 692.887567] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000 [ 692.887585] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40 [ 692.887601] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0 [ 692.887618] Call Trace: [ 692.887635] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde [ 692.887650] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 692.887662] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 692.887673] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 692.887683] [<ffffffff8147a814>] wait_for_common+0xac/0x101 [ 692.887692] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 [ 692.887702] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c [ 692.887711] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f [ 692.887720] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c [ 692.887730] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 [ 692.887740] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89 [ 692.887751] [<ffffffff81072780>] synchronize_srcu+0x15/0x17 [ 692.887762] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165 [ 692.887774] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.887783] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c [ 692.887793] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.887806] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.887818] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.887828] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.887840] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000 [ 692.887854] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000 [ 692.887871] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40 [ 692.887888] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3 [ 692.887905] Call Trace: [ 692.887917] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 692.887928] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 692.887939] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 692.887949] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 692.887960] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 [ 692.887971] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 692.887982] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.887992] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 692.888003] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 692.888012] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 692.888023] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 692.888033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888043] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 692.888053] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.888065] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.888076] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.888087] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.888095] flush-253:1 D 0000000000000000 0 439 2 0x00000000 [ 692.888117] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000 [ 692.888135] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40 [ 692.888153] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798 [ 692.888170] Call Trace: [ 692.888180] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 692.888190] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 692.888200] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888209] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 692.888219] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 692.888227] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 692.888238] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320 [ 692.888248] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 692.888257] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389 [ 692.888267] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543 [ 692.888276] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0 [ 692.888287] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 692.888298] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120 [ 692.888308] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 692.888319] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 692.888331] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 692.888341] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 692.888351] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 692.888361] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 692.888371] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 692.888381] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 692.888392] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 692.888402] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 692.888413] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 692.888423] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.888434] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.888445] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.888455] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.888472] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000 [ 692.888485] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001 [ 692.888501] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40 [ 692.888517] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3 [ 692.888533] Call Trace: [ 692.888547] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 692.888559] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888568] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 692.888578] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 692.888587] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 692.888597] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 692.888607] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888617] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 692.888627] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.888637] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.888648] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.888659] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.888665] NetworkManager D 0000000000000000 0 809 1 0x00000000 [ 692.888682] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000 [ 692.888697] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40 [ 692.888713] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3 [ 692.888729] Call Trace: [ 692.888738] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 692.888753] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888761] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 692.888770] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 692.888780] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 692.888790] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41 [ 692.888799] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e [ 692.888808] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 692.888818] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 692.888827] [<ffffffff811320c7>] touch_atime+0x10e/0x131 [ 692.888840] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640 [ 692.888851] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff [ 692.888860] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d [ 692.888872] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 692.888884] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 692.888894] [<ffffffff8111f311>] vfs_read+0xa9/0xf0 [ 692.888902] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e [ 692.888912] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 692.888923] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000 [ 692.888935] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000 [ 692.888950] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40 [ 692.888966] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3 [ 692.888986] Call Trace: [ 692.888994] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 692.889005] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.889013] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 692.889022] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 692.889031] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 692.889041] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 692.889050] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 692.889059] [<ffffffff81131f95>] file_update_time+0xef/0x113 [ 692.889070] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272 [ 692.889086] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0 [ 692.889099] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba [ 692.889115] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234 [ 692.889125] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff [ 692.889133] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217 [ 692.889144] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4 [ 692.889155] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 692.889164] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 692.889172] [<ffffffff8111f221>] vfs_write+0xac/0xf3 [ 692.889181] [<ffffffff8111f410>] sys_write+0x4a/0x6e [ 692.889191] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 692.889199] flush-253:2 D 0000000000000000 0 883 2 0x00000000 [ 692.889211] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000 [ 692.889446] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40 [ 692.889462] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718 [ 692.889478] Call Trace: [ 692.889488] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 692.889496] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 692.889507] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.889516] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 692.889525] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 692.889536] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19 [ 692.889547] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19 [ 692.889555] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 692.889566] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29 [ 692.889576] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29 [ 692.889587] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87 [ 692.889597] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 692.889606] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320 [ 692.889616] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d [ 692.889625] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389 [ 692.889636] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 692.889645] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 692.889656] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 692.889666] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 692.889677] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 692.889687] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 692.889697] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 692.889706] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 692.889717] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 692.889727] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 692.889736] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 692.889746] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde [ 692.889756] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 692.889766] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 692.889776] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.889787] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.889798] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.889809] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.889815] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000 [ 692.889827] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000 [ 692.889843] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40 [ 692.889860] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3 [ 692.889876] Call Trace: [ 692.889884] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 692.889895] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.889904] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 692.889913] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 692.889923] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 692.889934] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e [ 692.889944] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b [ 692.889954] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8 [ 692.889965] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33 [ 692.889975] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a [ 692.889984] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 692.889998] Xorg D 0000000000000000 0 1187 1183 0x00400084 [ 692.890012] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000 [ 692.890029] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40 [ 692.890046] ffffffff81a0b020 ffff88006d581700 ffff88007171ba38 nc: Write error: No route to host [ 709.230547] SysRq : Show Blocked State [ 709.230567] task PC stack pid father [ 709.230584] fsnotify_mark D 0000000000000000 0 49 2 0x00000000 [ 709.230600] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000 [ 709.230621] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40 [ 709.230638] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0 [ 709.230664] Call Trace: [ 709.230683] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde [ 709.230697] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 709.230710] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 709.230721] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 709.230731] [<ffffffff8147a814>] wait_for_common+0xac/0x101 [ 709.230740] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 [ 709.230751] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c [ 709.230761] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f [ 709.230769] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c [ 709.230780] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 [ 709.230790] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89 [ 709.230800] [<ffffffff81072780>] synchronize_srcu+0x15/0x17 [ 709.230810] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165 [ 709.230822] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.230831] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c [ 709.230842] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.230854] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.230866] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.230876] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.230889] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000 [ 709.230902] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000 [ 709.230920] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40 [ 709.230937] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3 [ 709.230955] Call Trace: [ 709.230967] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 709.230978] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 709.230989] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 709.230999] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 709.231011] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 [ 709.231021] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 709.231033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231043] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 709.231055] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 709.231064] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 709.231075] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 709.231086] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231096] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 709.231106] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.231117] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.231129] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.231139] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.231147] flush-253:1 D 0000000000000000 0 439 2 0x00000000 [ 709.231161] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000 [ 709.231178] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40 [ 709.231195] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798 [ 709.231220] Call Trace: [ 709.231230] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 709.231240] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 709.231251] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231260] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 709.231270] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 709.231279] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 709.231290] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320 [ 709.231300] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 709.231309] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389 [ 709.231318] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543 [ 709.231328] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0 [ 709.231338] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 709.231350] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120 [ 709.231360] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 709.231372] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 709.231383] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 709.231393] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 709.231404] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 709.231414] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 709.231424] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 709.231434] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 709.231445] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 709.231455] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 709.231465] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 709.231475] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.231486] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.231497] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.231513] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.231521] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000 [ 709.231534] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001 [ 709.231550] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40 [ 709.231567] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3 [ 709.231587] Call Trace: [ 709.231597] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 709.231609] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231618] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 709.231628] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 709.231638] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 709.231652] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 709.231664] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231674] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 709.231684] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.231695] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.231706] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.231716] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.231727] NetworkManager D 0000000000000000 0 809 1 0x00000000 [ 709.231739] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000 [ 709.231755] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40 [ 709.231772] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3 [ 709.231787] Call Trace: [ 709.231801] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 709.231812] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231821] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 709.231829] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 709.231839] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 709.231850] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41 [ 709.231859] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e [ 709.231868] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 709.231878] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 709.231887] [<ffffffff811320c7>] touch_atime+0x10e/0x131 [ 709.231899] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640 [ 709.231911] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff [ 709.231919] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d [ 709.231935] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 709.231944] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 709.231953] [<ffffffff8111f311>] vfs_read+0xa9/0xf0 [ 709.231962] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e [ 709.231972] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 709.231983] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000 [ 709.231995] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000 [ 709.232011] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40 [ 709.232027] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3 [ 709.232047] Call Trace: [ 709.232055] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 709.232066] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.232075] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 709.232083] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 709.232093] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 709.232102] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 709.232112] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 709.232121] [<ffffffff81131f95>] file_update_time+0xef/0x113 [ 709.232136] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272 [ 709.232148] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0 [ 709.232159] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba [ 709.232171] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234 [ 709.232181] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff [ 709.232189] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217 [ 709.232200] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4 [ 709.232210] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 709.232219] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 709.232228] [<ffffffff8111f221>] vfs_write+0xac/0xf3 [ 709.232237] [<ffffffff8111f410>] sys_write+0x4a/0x6e [ 709.232246] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 709.232255] flush-253:2 D 0000000000000000 0 883 2 0x00000000 [ 709.232267] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000 [ 709.232500] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40 [ 709.232516] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718 [ 709.232532] Call Trace: [ 709.232542] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 709.232550] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 709.232561] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.232570] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 709.232579] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 709.232589] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19 [ 709.232601] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19 [ 709.232609] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 709.232620] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29 [ 709.232630] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29 [ 709.232641] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87 [ 709.232656] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 709.232666] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320 [ 709.232676] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d [ 709.232685] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389 [ 709.232696] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 709.232706] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 709.232716] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 709.232728] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 709.232738] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 709.232748] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 709.232758] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 709.232768] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 709.232779] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 709.232789] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 709.232799] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 709.232808] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde [ 709.232819] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 709.232829] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 709.232839] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.232850] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.232861] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.232872] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.232878] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000 [ 709.232891] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000 [ 709.232908] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40 [ 709.232925] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3 [ 709.232942] Call Trace: [ 709.232951] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 709.232962] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.232971] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 709.232979] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 709.232989] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 709.233000] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e [ 709.233010] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b [ 709.233021] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8 [ 709.233032] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33 [ 709.233042] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a [ 709.233051] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 709.233066] Xorg D 0000000000000000 0 1187 1183 0x00400084 [ 709.233080] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000 [ 709.233097] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40 [ 709.233114] ffffffff81a0b020 ffff88006d581700[ 709.235257] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 709.235532] .MIN_vruntime : 0.000001 [ 709.235656] [ 709.236129] .se->load.weight : 2 [ 709.236374] .se->load.weight : 2 [ 709.236431] ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-04-29 18:02 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-04-29 18:02 UTC (permalink / raw) To: sedat.dilek Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote: > On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote: > > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > > > First, I have seen negative effects on my UP-system was when playing > > > with linux-next [2]. > > > It was not clear what the origin was and the the side-effects were > > > somehow "bizarre". > > > The issue could be easily reproduced by tar-ing the kernel build-dir > > > to an external USB-hdd. > > > The issue kept RCU and TIP folks really busy. > > > Before stepping 4 weeks in the dark, give it a try and let me know in > > > case of success. > > > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug > > showed up in 2.6.38 ... I tried it just in case with no effect, so we > > know it isn't the cause. > > Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but > it does seem to prevent non-PREEMPT from locking up totally (at least it > survives three back to back untar runs). > > It's probable it alters the memory pin conditions that cause the spin, > so it's masking the problem rather than fixing it. Confirmed ... it's just harder to reproduce with the hrtimers init fix. The problem definitely still exists (I had to load up the system more before doing the tar). This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w doesn't complete for an unknown reason James --- [ 575.083025] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 575.083043] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 575.083251] CPU 2 [ 575.083255] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 575.083430] [ 575.083437] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 575.083455] RIP: 0010:[<ffffffffa007ff8f>] [<ffffffffa007ff8f>] i915_gem_inactive_shrink+0x6c/0x194 [i915] [ 575.083497] RSP: 0018:ffff88006dfb7d50 EFLAGS: 00000206 [ 575.083504] RAX: ffff88005f018200 RBX: 00000000000000c0 RCX: 0000000000000000 [ 575.083512] RDX: ffff88005f01bab0 RSI: 0000000000000000 RDI: ffff880037885820 [ 575.083529] RBP: ffff88006dfb7d90 R08: 0000000000000000 R09: 0000000000017131 [ 575.083535] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff81482e0e [ 575.083541] R13: ffff88006dfb7cf0 R14: ffff88006dfb7cf8 R15: ffffffff810dd44d [ 575.083548] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 575.083555] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 575.083572] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 575.083579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 575.083587] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 575.083596] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000) [ 575.083602] Stack: [ 575.083608] ffff88006dfb7d90 ffff880037709638 ffff88006dfb7d60 ffff8800377095f0 [ 575.083626] 0000000000000000 0000000000000000 00000000000000d0 000000000005103b [ 575.083643] ffff88006dfb7de0 ffffffff810e1d89 000000000000003d 0000000000000080 [ 575.083660] Call Trace: [ 575.083676] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166 [ 575.083686] [<ffffffff810e49cc>] kswapd+0x533/0x798 [ 575.083698] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 575.083712] [<ffffffff8106e157>] kthread+0x84/0x8c [ 575.083726] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 575.083738] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 575.083750] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 575.083756] Code: e4 48 89 45 c8 75 37 48 8b 43 48 45 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00 [ 575.083887] 05 b0 00 00 00 48 39 d8 0f 18 0a 75 e1 e9 da 00 00 00 4c 89 [ 575.083949] Call Trace: [ 575.083957] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166 [ 575.083966] [<ffffffff810e49cc>] kswapd+0x533/0x798 [ 575.083976] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 575.083986] [<ffffffff8106e157>] kthread+0x84/0x8c [ 575.083998] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 575.084010] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 575.084030] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 658.883363] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46] [ 658.883382] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 658.883574] CPU 2 [ 658.883579] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 658.883749] [ 658.883757] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 658.883776] RIP: 0010:[<ffffffff81080bc0>] [<ffffffff81080bc0>] arch_local_irq_restore+0xc/0xd [ 658.883797] RSP: 0018:ffff88006dfb7d98 EFLAGS: 00000246 [ 658.883805] RAX: ffff8801005f9e68 RBX: ffff8801005e6e00 RCX: 0000000000004ae4 [ 658.883813] RDX: ffff88006dfb8000 RSI: 0000000000000246 RDI: 0000000000000246 [ 658.883821] RBP: ffff88006dfb7da0 R08: ffff8801005f9e70 R09: 00000000000007eb [ 658.883828] R10: 0000000000000000 R11: 00000000000000cd R12: ffffffff81482e0e [ 658.883835] R13: 0000000000016fd3 R14: 000000000000a731 R15: ffffffff810ddb31 [ 658.883844] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 658.883853] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 658.883860] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 658.883867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 658.883875] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 658.883883] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000) [ 658.883890] Stack: [ 658.883895] ffffffff8147bc8c ffff88006dfb7de0 ffffffff8106e7c3 ffff880000000002 [ 658.883913] 0000000000000000 ffff8801005e6000 ffff8801005f9e68 0000000000000002 [ 658.883929] 0000000000000000 ffff88006dfb7ee0 ffffffff810e45f0 0000000000000003 [ 658.883947] Call Trace: [ 658.883961] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 658.883976] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78 [ 658.883988] [<ffffffff810e45f0>] kswapd+0x157/0x798 [ 658.884000] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 658.884009] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 658.884020] [<ffffffff8106e157>] kthread+0x84/0x8c [ 658.884033] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 658.884045] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 658.884057] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 658.884062] Code: 1f 44 00 00 66 ff 05 30 16 98 00 fb 66 0f 1f 44 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 5d c3 55 48 89 e5 57 9d 0f 1f 44 00 00 5d <c3> 55 48 89 e5 fa 66 0f 1f 44 00 00 5d c3 55 48 89 e5 50 9c 58 [ 658.884238] Call Trace: [ 658.884248] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 658.884260] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78 [ 658.884269] [<ffffffff810e45f0>] kswapd+0x157/0x798 [ 658.884280] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 658.884289] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 658.884299] [<ffffffff8106e157>] kthread+0x84/0x8c [ 658.884311] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 658.884322] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 658.884333] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 690.767137] SysRq : Show backtrace of all active CPUs [ 690.767158] sending NMI to all CPUs: [ 690.767171] NMI backtrace for cpu 2 [ 690.767188] CPU 2 [ 690.767196] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 690.767457] [ 690.767468] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 690.767493] RIP: 0010:[<ffffffff810e3980>] [<ffffffff810e3980>] shrink_zone+0x221/0x4b3 [ 690.767519] RSP: 0018:ffff88006dfb7d30 EFLAGS: 00000206 [ 690.767529] RAX: 0000000000000000 RBX: ffff88006dfb7e58 RCX: 000000000000000c [ 690.767539] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 690.767550] RBP: ffff88006dfb7de0 R08: 0000000000000001 R09: 00000000000007eb [ 690.767561] R10: 0000000000000002 R11: 00000000000000f5 R12: ffff8801005e6000 [ 690.767572] R13: ffff8801005e64d8 R14: 0000000000001572 R15: 00000000ffffffff [ 690.767584] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000 [ 690.767597] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 690.767607] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 690.767618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 690.767628] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 690.767640] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000) [ 690.767650] Stack: [ 690.767657] 0000000000000001 000000000000008c 000000000000003c 0000000000000000 [ 690.767683] ffffffffffffffff ffff8801005e6480 000000000000000c 0000000c81118591 [ 690.767717] 0000000000000000 0000000000000000 ffff880100530380 0000000000000000 [ 690.767742] Call Trace: [ 690.767764] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae [ 690.767779] [<ffffffff810e49a6>] kswapd+0x50d/0x798 [ 690.767794] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 690.767811] [<ffffffff8106e157>] kthread+0x84/0x8c [ 690.767830] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 690.767846] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 690.767861] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 690.767868] Code: e2 ff ff 83 7d 80 00 44 8b 85 50 ff ff ff 74 1b 8a 4d 8c 31 f6 41 83 ff 01 40 0f 96 c6 31 d2 48 d3 e8 48 0f af 44 f5 c0 49 f7 f6 [ 690.768028] 89 c1 48 8d 51 04 49 03 44 d5 00 48 83 f8 1f 49 89 44 d5 00 [ 690.768113] Call Trace: [ 690.768126] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae [ 690.768139] [<ffffffff810e49a6>] kswapd+0x50d/0x798 [ 690.768154] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3 [ 690.768168] [<ffffffff8106e157>] kthread+0x84/0x8c [ 690.768183] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 690.768199] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 690.768213] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 690.768223] NMI backtrace for cpu 3 [ 690.768236] CPU 3 [ 690.768244] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 690.768832] [ 690.768843] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO [ 690.768867] RIP: 0010:[<ffffffff81275b36>] [<ffffffff81275b36>] intel_idle+0xaa/0x100 [ 690.768890] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046 [ 690.768900] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001 [ 690.768912] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640 [ 690.768923] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003c2 [ 690.768934] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030 [ 690.768946] R13: 12191803fa451798 R14: 0000000000000004 R15: 0000000000000003 [ 690.768959] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000 [ 690.768971] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 690.768982] CR2: 0000000000429bf0 CR3: 0000000001a03000 CR4: 00000000000406e0 [ 690.768993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 690.769005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 690.769017] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700) [ 690.769027] Stack: [ 690.769034] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1280 0000000000011280 [ 690.769063] ffff8800715dfeb8 000000038139c77a ffffe8ffffcc0170 ffffe8ffffcc0170 [ 690.769092] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b668 [ 690.769121] Call Trace: [ 690.769137] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 690.769153] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166 [ 690.769173] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 690.769191] [<ffffffff8146ac57>] start_secondary+0x223/0x225 [ 690.769203] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 0b e0 ff 4c 29 e8 48 89 c7 e8 ab 2b de ff 4c 69 e0 40 42 [ 690.769477] Call Trace: [ 690.769491] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 690.769506] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166 [ 690.769522] [<ffffffff81008321>] cpu_idle+0xa5/0xdf [ 690.770451] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45 [ 690.771380] [<ffffffff81483f73>] ? smp_apic_timer_interrupt+0x7e/0x8c [ 692.887516] SysRq : Show Blocked State [ 692.887534] task PC stack pid father [ 692.887551] fsnotify_mark D 0000000000000000 0 49 2 0x00000000 [ 692.887567] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000 [ 692.887585] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40 [ 692.887601] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0 [ 692.887618] Call Trace: [ 692.887635] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde [ 692.887650] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 692.887662] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 692.887673] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 692.887683] [<ffffffff8147a814>] wait_for_common+0xac/0x101 [ 692.887692] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 [ 692.887702] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c [ 692.887711] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f [ 692.887720] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c [ 692.887730] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 [ 692.887740] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89 [ 692.887751] [<ffffffff81072780>] synchronize_srcu+0x15/0x17 [ 692.887762] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165 [ 692.887774] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.887783] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c [ 692.887793] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.887806] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.887818] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.887828] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.887840] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000 [ 692.887854] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000 [ 692.887871] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40 [ 692.887888] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3 [ 692.887905] Call Trace: [ 692.887917] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 692.887928] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 692.887939] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 692.887949] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 692.887960] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 [ 692.887971] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 692.887982] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.887992] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 692.888003] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 692.888012] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 692.888023] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 692.888033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888043] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 692.888053] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.888065] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.888076] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.888087] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.888095] flush-253:1 D 0000000000000000 0 439 2 0x00000000 [ 692.888117] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000 [ 692.888135] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40 [ 692.888153] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798 [ 692.888170] Call Trace: [ 692.888180] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 692.888190] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 692.888200] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888209] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 692.888219] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 692.888227] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 692.888238] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320 [ 692.888248] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 692.888257] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389 [ 692.888267] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543 [ 692.888276] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0 [ 692.888287] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 692.888298] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120 [ 692.888308] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 692.888319] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 692.888331] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 692.888341] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 692.888351] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 692.888361] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 692.888371] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 692.888381] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 692.888392] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 692.888402] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 692.888413] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 692.888423] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.888434] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.888445] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.888455] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.888472] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000 [ 692.888485] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001 [ 692.888501] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40 [ 692.888517] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3 [ 692.888533] Call Trace: [ 692.888547] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 692.888559] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888568] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 692.888578] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 692.888587] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 692.888597] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 692.888607] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888617] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 692.888627] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.888637] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.888648] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.888659] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.888665] NetworkManager D 0000000000000000 0 809 1 0x00000000 [ 692.888682] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000 [ 692.888697] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40 [ 692.888713] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3 [ 692.888729] Call Trace: [ 692.888738] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 692.888753] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.888761] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 692.888770] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 692.888780] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 692.888790] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41 [ 692.888799] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e [ 692.888808] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 692.888818] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 692.888827] [<ffffffff811320c7>] touch_atime+0x10e/0x131 [ 692.888840] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640 [ 692.888851] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff [ 692.888860] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d [ 692.888872] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 692.888884] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 692.888894] [<ffffffff8111f311>] vfs_read+0xa9/0xf0 [ 692.888902] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e [ 692.888912] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 692.888923] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000 [ 692.888935] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000 [ 692.888950] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40 [ 692.888966] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3 [ 692.888986] Call Trace: [ 692.888994] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 692.889005] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.889013] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 692.889022] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 692.889031] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 692.889041] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 692.889050] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 692.889059] [<ffffffff81131f95>] file_update_time+0xef/0x113 [ 692.889070] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272 [ 692.889086] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0 [ 692.889099] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba [ 692.889115] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234 [ 692.889125] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff [ 692.889133] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217 [ 692.889144] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4 [ 692.889155] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 692.889164] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 692.889172] [<ffffffff8111f221>] vfs_write+0xac/0xf3 [ 692.889181] [<ffffffff8111f410>] sys_write+0x4a/0x6e [ 692.889191] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 692.889199] flush-253:2 D 0000000000000000 0 883 2 0x00000000 [ 692.889211] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000 [ 692.889446] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40 [ 692.889462] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718 [ 692.889478] Call Trace: [ 692.889488] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 692.889496] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 692.889507] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.889516] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 692.889525] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 692.889536] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19 [ 692.889547] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19 [ 692.889555] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 692.889566] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29 [ 692.889576] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29 [ 692.889587] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87 [ 692.889597] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 692.889606] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320 [ 692.889616] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d [ 692.889625] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389 [ 692.889636] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 692.889645] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 692.889656] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 692.889666] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 692.889677] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 692.889687] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 692.889697] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 692.889706] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 692.889717] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 692.889727] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 692.889736] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 692.889746] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde [ 692.889756] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 692.889766] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 692.889776] [<ffffffff8106e157>] kthread+0x84/0x8c [ 692.889787] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 692.889798] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 692.889809] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 692.889815] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000 [ 692.889827] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000 [ 692.889843] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40 [ 692.889860] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3 [ 692.889876] Call Trace: [ 692.889884] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 692.889895] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 692.889904] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 692.889913] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 692.889923] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 692.889934] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e [ 692.889944] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b [ 692.889954] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8 [ 692.889965] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33 [ 692.889975] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a [ 692.889984] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 692.889998] Xorg D 0000000000000000 0 1187 1183 0x00400084 [ 692.890012] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000 [ 692.890029] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40 [ 692.890046] ffffffff81a0b020 ffff88006d581700 ffff88007171ba38 nc: Write error: No route to host [ 709.230547] SysRq : Show Blocked State [ 709.230567] task PC stack pid father [ 709.230584] fsnotify_mark D 0000000000000000 0 49 2 0x00000000 [ 709.230600] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000 [ 709.230621] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40 [ 709.230638] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0 [ 709.230664] Call Trace: [ 709.230683] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde [ 709.230697] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 709.230710] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 709.230721] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 709.230731] [<ffffffff8147a814>] wait_for_common+0xac/0x101 [ 709.230740] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226 [ 709.230751] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c [ 709.230761] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f [ 709.230769] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c [ 709.230780] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43 [ 709.230790] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89 [ 709.230800] [<ffffffff81072780>] synchronize_srcu+0x15/0x17 [ 709.230810] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165 [ 709.230822] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.230831] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c [ 709.230842] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.230854] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.230866] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.230876] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.230889] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000 [ 709.230902] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000 [ 709.230920] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40 [ 709.230937] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3 [ 709.230955] Call Trace: [ 709.230967] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 709.230978] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61 [ 709.230989] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd [ 709.230999] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 709.231011] [<ffffffff8100804e>] ? load_TLS+0x10/0x14 [ 709.231021] [<ffffffff81008714>] ? __switch_to+0xc6/0x220 [ 709.231033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231043] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 709.231055] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 709.231064] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 709.231075] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 709.231086] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231096] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 709.231106] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.231117] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.231129] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.231139] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.231147] flush-253:1 D 0000000000000000 0 439 2 0x00000000 [ 709.231161] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000 [ 709.231178] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40 [ 709.231195] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798 [ 709.231220] Call Trace: [ 709.231230] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 709.231240] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 709.231251] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231260] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 709.231270] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 709.231279] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 709.231290] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320 [ 709.231300] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 709.231309] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389 [ 709.231318] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543 [ 709.231328] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0 [ 709.231338] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 709.231350] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120 [ 709.231360] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 709.231372] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 709.231383] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 709.231393] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 709.231404] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 709.231414] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 709.231424] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6 [ 709.231434] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 709.231445] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 709.231455] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 709.231465] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 709.231475] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.231486] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.231497] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.231513] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.231521] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000 [ 709.231534] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001 [ 709.231550] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40 [ 709.231567] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3 [ 709.231587] Call Trace: [ 709.231597] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194 [ 709.231609] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231618] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52 [ 709.231628] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [ 709.231638] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81 [ 709.231652] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a [ 709.231664] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231674] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10 [ 709.231684] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.231695] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.231706] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.231716] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.231727] NetworkManager D 0000000000000000 0 809 1 0x00000000 [ 709.231739] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000 [ 709.231755] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40 [ 709.231772] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3 [ 709.231787] Call Trace: [ 709.231801] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 709.231812] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.231821] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 709.231829] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 709.231839] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 709.231850] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41 [ 709.231859] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e [ 709.231868] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 709.231878] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 709.231887] [<ffffffff811320c7>] touch_atime+0x10e/0x131 [ 709.231899] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640 [ 709.231911] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff [ 709.231919] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d [ 709.231935] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 709.231944] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 709.231953] [<ffffffff8111f311>] vfs_read+0xa9/0xf0 [ 709.231962] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e [ 709.231972] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 709.231983] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000 [ 709.231995] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000 [ 709.232011] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40 [ 709.232027] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3 [ 709.232047] Call Trace: [ 709.232055] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 709.232066] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.232075] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 709.232083] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 709.232093] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 709.232102] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c [ 709.232112] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175 [ 709.232121] [<ffffffff81131f95>] file_update_time+0xef/0x113 [ 709.232136] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272 [ 709.232148] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0 [ 709.232159] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba [ 709.232171] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234 [ 709.232181] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff [ 709.232189] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217 [ 709.232200] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4 [ 709.232210] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33 [ 709.232219] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 709.232228] [<ffffffff8111f221>] vfs_write+0xac/0xf3 [ 709.232237] [<ffffffff8111f410>] sys_write+0x4a/0x6e [ 709.232246] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 709.232255] flush-253:2 D 0000000000000000 0 883 2 0x00000000 [ 709.232267] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000 [ 709.232500] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40 [ 709.232516] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718 [ 709.232532] Call Trace: [ 709.232542] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e [ 709.232550] [<ffffffff81217900>] get_request_wait+0x102/0x18b [ 709.232561] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.232570] [<ffffffff81218641>] __make_request+0x18a/0x2b8 [ 709.232579] [<ffffffff81217338>] generic_make_request+0x2a9/0x323 [ 709.232589] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19 [ 709.232601] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19 [ 709.232609] [<ffffffff81217490>] submit_bio+0xde/0xfd [ 709.232620] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29 [ 709.232630] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29 [ 709.232641] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87 [ 709.232656] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58 [ 709.232666] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320 [ 709.232676] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d [ 709.232685] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389 [ 709.232696] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd [ 709.232706] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d [ 709.232716] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a [ 709.232728] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc [ 709.232738] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161 [ 709.232748] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b [ 709.232758] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335 [ 709.232768] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b [ 709.232779] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f [ 709.232789] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1 [ 709.232799] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d [ 709.232808] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde [ 709.232819] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5 [ 709.232829] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d [ 709.232839] [<ffffffff8106e157>] kthread+0x84/0x8c [ 709.232850] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10 [ 709.232861] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148 [ 709.232872] [<ffffffff81483560>] ? gs_change+0x13/0x13 [ 709.232878] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000 [ 709.232891] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000 [ 709.232908] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40 [ 709.232925] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3 [ 709.232942] Call Trace: [ 709.232951] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465 [ 709.232962] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a [ 709.232971] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda [ 709.232979] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15 [ 709.232989] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120 [ 709.233000] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e [ 709.233010] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b [ 709.233021] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8 [ 709.233032] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33 [ 709.233042] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a [ 709.233051] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b [ 709.233066] Xorg D 0000000000000000 0 1187 1183 0x00400084 [ 709.233080] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000 [ 709.233097] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40 [ 709.233114] ffffffff81a0b020 ffff88006d581700[ 709.235257] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd [ 709.235532] .MIN_vruntime : 0.000001 [ 709.235656] [ 709.236129] .se->load.weight : 2 [ 709.236374] .se->load.weight : 2 [ 709.236431] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. 2011-04-29 18:02 ` James Bottomley @ 2011-05-02 20:04 ` James Bottomley -1 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-02 20:04 UTC (permalink / raw) To: sedat.dilek Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner On Fri, 2011-04-29 at 13:02 -0500, James Bottomley wrote: > On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote: > > On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote: > > > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > > > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > > > > First, I have seen negative effects on my UP-system was when playing > > > > with linux-next [2]. > > > > It was not clear what the origin was and the the side-effects were > > > > somehow "bizarre". > > > > The issue could be easily reproduced by tar-ing the kernel build-dir > > > > to an external USB-hdd. > > > > The issue kept RCU and TIP folks really busy. > > > > Before stepping 4 weeks in the dark, give it a try and let me know in > > > > case of success. > > > > > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug > > > showed up in 2.6.38 ... I tried it just in case with no effect, so we > > > know it isn't the cause. > > > > Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but > > it does seem to prevent non-PREEMPT from locking up totally (at least it > > survives three back to back untar runs). > > > > It's probable it alters the memory pin conditions that cause the spin, > > so it's masking the problem rather than fixing it. > > Confirmed ... it's just harder to reproduce with the hrtimers init fix. > The problem definitely still exists (I had to load up the system more > before doing the tar). > > This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w > doesn't complete for an unknown reason As a follow on to this, there's a shrink_zone(0, ...) in the cgroup path. This causes it to scan all memory exhaustively (generating quite a lot of work). The comment above it implies it's some type of hack for cgroup accounting, but reducing it to DEF_PRIORITY makes the hang go away (verified on both 2.6.39-rc4 and 2.6.38.4). Note that I still get soft lockups in kswapd0, but they no longer hang the box. I'll prepare a patch for the next round of debate. James ^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. @ 2011-05-02 20:04 ` James Bottomley 0 siblings, 0 replies; 138+ messages in thread From: James Bottomley @ 2011-05-02 20:04 UTC (permalink / raw) To: sedat.dilek Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner On Fri, 2011-04-29 at 13:02 -0500, James Bottomley wrote: > On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote: > > On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote: > > > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote: > > > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try. > > > > First, I have seen negative effects on my UP-system was when playing > > > > with linux-next [2]. > > > > It was not clear what the origin was and the the side-effects were > > > > somehow "bizarre". > > > > The issue could be easily reproduced by tar-ing the kernel build-dir > > > > to an external USB-hdd. > > > > The issue kept RCU and TIP folks really busy. > > > > Before stepping 4 weeks in the dark, give it a try and let me know in > > > > case of success. > > > > > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug > > > showed up in 2.6.38 ... I tried it just in case with no effect, so we > > > know it isn't the cause. > > > > Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but > > it does seem to prevent non-PREEMPT from locking up totally (at least it > > survives three back to back untar runs). > > > > It's probable it alters the memory pin conditions that cause the spin, > > so it's masking the problem rather than fixing it. > > Confirmed ... it's just harder to reproduce with the hrtimers init fix. > The problem definitely still exists (I had to load up the system more > before doing the tar). > > This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w > doesn't complete for an unknown reason As a follow on to this, there's a shrink_zone(0, ...) in the cgroup path. This causes it to scan all memory exhaustively (generating quite a lot of work). The comment above it implies it's some type of hack for cgroup accounting, but reducing it to DEF_PRIORITY makes the hang go away (verified on both 2.6.39-rc4 and 2.6.38.4). Note that I still get soft lockups in kswapd0, but they no longer hang the box. I'll prepare a patch for the next round of debate. James -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 138+ messages in thread
end of thread, other threads:[~2011-05-11 16:47 UTC | newest] Thread overview: 138+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-04-27 16:09 [BUG] fatal hang untarring 90GB file, possibly writeback related James Bottomley 2011-04-27 16:09 ` James Bottomley 2011-04-27 16:33 ` Chris Mason 2011-04-27 16:33 ` Chris Mason 2011-04-27 16:50 ` James Bottomley 2011-04-27 16:50 ` James Bottomley 2011-04-27 16:50 ` James Bottomley 2011-04-27 16:54 ` Chris Mason 2011-04-27 16:54 ` Chris Mason 2011-04-27 17:21 ` James Bottomley 2011-04-27 17:21 ` James Bottomley 2011-04-27 17:21 ` James Bottomley 2011-04-27 17:34 ` Chris Mason 2011-04-27 17:34 ` Chris Mason 2011-04-27 17:50 ` James Bottomley 2011-04-27 17:50 ` James Bottomley 2011-04-27 18:25 ` Colin Ian King 2011-04-27 18:25 ` Colin Ian King 2011-04-28 15:57 ` James Bottomley 2011-04-28 15:57 ` James Bottomley 2011-04-27 20:05 ` James Bottomley 2011-04-27 20:05 ` James Bottomley 2011-04-28 11:36 ` Colin Ian King 2011-04-28 11:36 ` Colin Ian King 2011-04-28 12:29 ` Chris Mason 2011-04-28 12:29 ` Chris Mason 2011-04-28 13:42 ` Colin Ian King 2011-04-28 13:42 ` Colin Ian King 2011-04-28 13:45 ` Chris Mason 2011-04-28 13:45 ` Chris Mason 2011-04-28 14:01 ` Colin Ian King 2011-04-28 14:04 ` Chris Mason 2011-04-28 14:04 ` Chris Mason 2011-04-28 15:23 ` Colin Ian King 2011-04-28 14:25 ` Jan Kara 2011-04-28 14:25 ` Jan Kara 2011-04-28 14:33 ` Jan Kara 2011-04-28 14:33 ` Jan Kara 2011-04-28 14:58 ` Colin Ian King 2011-04-28 22:40 ` Jan Kara 2011-04-28 22:40 ` Jan Kara 2011-04-28 22:44 ` James Bottomley 2011-04-28 22:44 ` James Bottomley 2011-05-03 18:55 ` Colin Ian King 2011-05-03 18:55 ` Colin Ian King 2011-04-28 16:11 ` James Bottomley 2011-04-28 16:11 ` James Bottomley 2011-04-28 14:49 ` James Bottomley 2011-04-28 14:49 ` James Bottomley 2011-04-28 13:52 ` Jan Kara 2011-04-28 13:52 ` Jan Kara 2011-04-28 14:07 ` Mel Gorman 2011-04-28 14:07 ` Mel Gorman 2011-04-28 14:25 ` James Bottomley 2011-04-28 14:25 ` James Bottomley 2011-04-28 15:08 ` Mel Gorman 2011-04-28 15:08 ` Mel Gorman 2011-04-28 16:01 ` James Bottomley 2011-04-28 16:01 ` James Bottomley 2011-04-28 16:50 ` James Bottomley 2011-04-28 16:50 ` James Bottomley 2011-04-28 16:56 ` James Bottomley 2011-04-28 16:56 ` James Bottomley 2011-04-28 17:18 ` Mel Gorman 2011-04-28 17:18 ` Mel Gorman 2011-04-28 18:30 ` James Bottomley 2011-04-28 18:30 ` James Bottomley 2011-04-28 19:21 ` Mel Gorman 2011-04-28 19:21 ` Mel Gorman 2011-04-28 19:59 ` James Bottomley 2011-04-28 19:59 ` James Bottomley 2011-04-28 20:27 ` Mel Gorman 2011-04-28 20:27 ` Mel Gorman 2011-04-29 15:02 ` James Bottomley 2011-04-29 15:02 ` James Bottomley 2011-04-28 21:12 ` James Bottomley 2011-04-28 21:12 ` James Bottomley 2011-04-28 22:43 ` James Bottomley 2011-04-28 22:43 ` James Bottomley 2011-05-03 9:13 ` Mel Gorman 2011-05-03 9:13 ` Mel Gorman 2011-05-03 14:13 ` James Bottomley 2011-05-03 14:13 ` James Bottomley 2011-05-03 14:22 ` James Bottomley 2011-05-06 7:42 ` Mel Gorman 2011-05-06 7:42 ` Mel Gorman 2011-05-06 8:07 ` Mel Gorman 2011-05-09 18:16 ` James Bottomley 2011-05-09 18:16 ` James Bottomley 2011-05-10 10:21 ` Mel Gorman 2011-05-10 10:21 ` Mel Gorman 2011-05-10 10:33 ` Pekka Enberg 2011-05-10 10:33 ` Pekka Enberg 2011-05-10 14:01 ` James Bottomley 2011-05-10 14:01 ` James Bottomley 2011-05-10 14:35 ` Mel Gorman 2011-05-10 14:35 ` Mel Gorman 2011-05-10 15:29 ` James Bottomley 2011-05-10 15:29 ` James Bottomley 2011-05-10 15:57 ` James Bottomley 2011-05-10 15:57 ` James Bottomley 2011-05-10 17:05 ` James Bottomley 2011-05-10 17:05 ` James Bottomley 2011-05-10 17:17 ` Mel Gorman 2011-05-10 17:17 ` Mel Gorman 2011-05-10 17:29 ` James Bottomley 2011-05-10 17:29 ` James Bottomley 2011-05-10 21:08 ` Raghavendra D Prabhu 2011-05-11 9:16 ` Mel Gorman 2011-05-11 9:16 ` Mel Gorman 2011-05-06 11:42 ` Mel Gorman 2011-05-06 11:42 ` Mel Gorman 2011-05-06 15:44 ` Mel Gorman 2011-05-06 15:44 ` Mel Gorman 2011-05-06 19:14 ` James Bottomley 2011-05-06 19:14 ` James Bottomley 2011-05-06 19:37 ` Mel Gorman 2011-05-06 19:37 ` Mel Gorman 2011-05-10 5:37 ` Colin Ian King 2011-05-10 5:37 ` Colin Ian King 2011-05-10 5:37 ` Colin Ian King 2011-05-06 15:58 ` James Bottomley 2011-05-03 9:54 ` Colin Ian King 2011-05-03 9:54 ` Colin Ian King 2011-04-28 17:10 ` Colin Ian King 2011-04-28 17:10 ` Colin Ian King 2011-04-28 0:37 ` Dave Chinner 2011-04-28 0:37 ` Dave Chinner 2011-04-29 10:23 ` Sedat Dilek 2011-04-29 10:23 ` Sedat Dilek 2011-04-29 15:37 ` James Bottomley 2011-04-29 15:37 ` James Bottomley 2011-04-29 16:31 ` James Bottomley 2011-04-29 16:31 ` James Bottomley 2011-04-29 18:02 ` James Bottomley 2011-04-29 18:02 ` James Bottomley 2011-05-02 20:04 ` James Bottomley 2011-05-02 20:04 ` James Bottomley
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.