hang in writeback code on nfsv4 mount

hang in writeback code on nfsv4 mount

[J. Bruce Fields]

As of 253c34e9b10c30d3064be654b5b78fbc1a8b1896 "writeback: prevent
unnecessary bdi threads wakeups", any nfs mount hangs for me.  Is this a
known issue?

--b.

INFO: task mount.nfs4:3812 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mount.nfs4    D 0000000000000000  2880  3812   3811 0x00000000
 ffff88001ed25a28 0000000000000046 ffff88001ed25fd8 ffff88001ed25fd8
 ffff88001ed24000 ffff88001ed24000 ffff88001ed24000 ffff88001f9503a0
 ffff88001ed25fd8 ffff88001f9503a8 ffff88001ed24000 ffff88001ed25fd8
Call Trace:
 [<ffffffff819608dd>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a23c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff81963970>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a51d>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc50>] wait_for_common+0x120/0x190
 [<ffffffff81033bb0>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fd9d>] wait_for_completion+0x1d/0x20
 [<ffffffff8105951a>] kthread_stop+0x4a/0x150
 [<ffffffff81061980>] ? thaw_process+0x70/0x80
 [<ffffffff810cc5aa>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229cd9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee7d4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee8c6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d2c9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810ed955>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edec9>] deactivate_super+0x49/0x70
 [<ffffffff811081a4>] mntput_no_expire+0x84/0xe0
 [<ffffffff811083ff>] release_mounts+0x9f/0xc0
 [<ffffffff81108485>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cb66>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cecf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122cfd2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edbc8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810edda2>] do_kern_mount+0x52/0x130
 [<ffffffff81963d6a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108dae>] do_mount+0x26e/0x7f0
 [<ffffffff81106a4a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811093c8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
1 lock held by mount.nfs4/3812:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edec1>] deactivate_super+0x41/0x70

RE: hang in writeback code on nfsv4 mount

[Artem.Bityutskiy]

> As of 253c34e9b10c30d3064be654b5b78fbc1a8b1896 "writeback: prevent
> unnecessary bdi threads wakeups", any nfs mount hangs for me.  Is this a
> known issue?

I'll try to reproduce this and take a look. No, I did not see this. Is the commit you point
the result of git bisect ?

Artem.

Re: hang in writeback code on nfsv4 mount

[Artem Bityutskiy]

On Wed, 2010-08-25 at 04:34 +0200, ext J. Bruce Fields wrote:
> As of 253c34e9b10c30d3064be654b5b78fbc1a8b1896 "writeback: prevent
> unnecessary bdi threads wakeups", any nfs mount hangs for me.  Is this a
> known issue?
> 
> --b.
> 
> INFO: task mount.nfs4:3812 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> mount.nfs4    D 0000000000000000  2880  3812   3811 0x00000000
>  ffff88001ed25a28 0000000000000046 ffff88001ed25fd8 ffff88001ed25fd8
>  ffff88001ed24000 ffff88001ed24000 ffff88001ed24000 ffff88001f9503a0
>  ffff88001ed25fd8 ffff88001f9503a8 ffff88001ed24000 ffff88001ed25fd8
> Call Trace:
>  [<ffffffff819608dd>] schedule_timeout+0x1cd/0x2e0
>  [<ffffffff8106a23c>] ? mark_held_locks+0x6c/0xa0
>  [<ffffffff81963970>] ? _raw_spin_unlock_irq+0x30/0x60
>  [<ffffffff8106a51d>] ? trace_hardirqs_on_caller+0x14d/0x190
>  [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
>  [<ffffffff8195fc50>] wait_for_common+0x120/0x190
>  [<ffffffff81033bb0>] ? default_wake_function+0x0/0x20
>  [<ffffffff8195fd9d>] wait_for_completion+0x1d/0x20
>  [<ffffffff8105951a>] kthread_stop+0x4a/0x150
>  [<ffffffff81061980>] ? thaw_process+0x70/0x80
>  [<ffffffff810cc5aa>] bdi_unregister+0x10a/0x1a0
>  [<ffffffff81229cd9>] nfs_put_super+0x19/0x20
>  [<ffffffff810ee7d4>] generic_shutdown_super+0x54/0xe0
>  [<ffffffff810ee8c6>] kill_anon_super+0x16/0x60
>  [<ffffffff8122d2c9>] nfs4_kill_super+0x39/0x90
>  [<ffffffff810ed955>] deactivate_locked_super+0x45/0x60
>  [<ffffffff810edec9>] deactivate_super+0x49/0x70
>  [<ffffffff811081a4>] mntput_no_expire+0x84/0xe0
>  [<ffffffff811083ff>] release_mounts+0x9f/0xc0
>  [<ffffffff81108485>] put_mnt_ns+0x65/0x80
>  [<ffffffff8122cb66>] nfs_follow_remote_path+0x1e6/0x420
>  [<ffffffff8122cecf>] nfs4_try_mount+0x6f/0xd0
>  [<ffffffff8122cfd2>] nfs4_get_sb+0xa2/0x360
>  [<ffffffff810edbc8>] vfs_kern_mount+0x88/0x1f0
>  [<ffffffff810edda2>] do_kern_mount+0x52/0x130
>  [<ffffffff81963d6a>] ? _lock_kernel+0x6a/0x170
>  [<ffffffff81108dae>] do_mount+0x26e/0x7f0
>  [<ffffffff81106a4a>] ? copy_mount_options+0xea/0x190
>  [<ffffffff811093c8>] sys_mount+0x98/0xf0
>  [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
> 1 lock held by mount.nfs4/3812:
>  #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edec1>] deactivate_super+0x41/0x70

I've tried both v2.6.36-rc2 and commit
253c34e9b10c30d3064be654b5b78fbc1a8b1896 [1] - and I can mount an NFS
share with no problems:

sudo mount -t nfs4 sauron:/home/dedekind/ /mnt/sauron_home/

works fine. Any hints about how to reproduce this are welcome.

I'll try to look at the code and figure out why this could happen.

So, does the mount at some point succeed? Or it is blocked forever? And
sysrq-t output would be useful to look at as well.

Also, it is strange that 'sys_mount()' involves 'nfs4_kill_super()' - is
this normal or this is an error path?

[1]: the kernel tree does not compile on this commit, and I applied
patch on top to solve the compilation issue:
387ac089361fbe5ef287e6950c5c40f6b18e5c55 "block: fix missing export of
blk_types.h"

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: hang in writeback code on nfsv4 mount

[Bryan Schumaker]

I'm seeing this too.  I am able to mount without a problem and the hang occurs during unmount. 

-Bryan

On 08/25/2010 02:32 AM, Artem Bityutskiy wrote:
> On Wed, 2010-08-25 at 04:34 +0200, ext J. Bruce Fields wrote:
>> As of 253c34e9b10c30d3064be654b5b78fbc1a8b1896 "writeback: prevent
>> unnecessary bdi threads wakeups", any nfs mount hangs for me.  Is this a
>> known issue?
>>
>> --b.
>>
>> INFO: task mount.nfs4:3812 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> mount.nfs4    D 0000000000000000  2880  3812   3811 0x00000000
>>  ffff88001ed25a28 0000000000000046 ffff88001ed25fd8 ffff88001ed25fd8
>>  ffff88001ed24000 ffff88001ed24000 ffff88001ed24000 ffff88001f9503a0
>>  ffff88001ed25fd8 ffff88001f9503a8 ffff88001ed24000 ffff88001ed25fd8
>> Call Trace:
>>  [<ffffffff819608dd>] schedule_timeout+0x1cd/0x2e0
>>  [<ffffffff8106a23c>] ? mark_held_locks+0x6c/0xa0
>>  [<ffffffff81963970>] ? _raw_spin_unlock_irq+0x30/0x60
>>  [<ffffffff8106a51d>] ? trace_hardirqs_on_caller+0x14d/0x190
>>  [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
>>  [<ffffffff8195fc50>] wait_for_common+0x120/0x190
>>  [<ffffffff81033bb0>] ? default_wake_function+0x0/0x20
>>  [<ffffffff8195fd9d>] wait_for_completion+0x1d/0x20
>>  [<ffffffff8105951a>] kthread_stop+0x4a/0x150
>>  [<ffffffff81061980>] ? thaw_process+0x70/0x80
>>  [<ffffffff810cc5aa>] bdi_unregister+0x10a/0x1a0
>>  [<ffffffff81229cd9>] nfs_put_super+0x19/0x20
>>  [<ffffffff810ee7d4>] generic_shutdown_super+0x54/0xe0
>>  [<ffffffff810ee8c6>] kill_anon_super+0x16/0x60
>>  [<ffffffff8122d2c9>] nfs4_kill_super+0x39/0x90
>>  [<ffffffff810ed955>] deactivate_locked_super+0x45/0x60
>>  [<ffffffff810edec9>] deactivate_super+0x49/0x70
>>  [<ffffffff811081a4>] mntput_no_expire+0x84/0xe0
>>  [<ffffffff811083ff>] release_mounts+0x9f/0xc0
>>  [<ffffffff81108485>] put_mnt_ns+0x65/0x80
>>  [<ffffffff8122cb66>] nfs_follow_remote_path+0x1e6/0x420
>>  [<ffffffff8122cecf>] nfs4_try_mount+0x6f/0xd0
>>  [<ffffffff8122cfd2>] nfs4_get_sb+0xa2/0x360
>>  [<ffffffff810edbc8>] vfs_kern_mount+0x88/0x1f0
>>  [<ffffffff810edda2>] do_kern_mount+0x52/0x130
>>  [<ffffffff81963d6a>] ? _lock_kernel+0x6a/0x170
>>  [<ffffffff81108dae>] do_mount+0x26e/0x7f0
>>  [<ffffffff81106a4a>] ? copy_mount_options+0xea/0x190
>>  [<ffffffff811093c8>] sys_mount+0x98/0xf0
>>  [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
>> 1 lock held by mount.nfs4/3812:
>>  #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edec1>] deactivate_super+0x41/0x70
> 
> I've tried both v2.6.36-rc2 and commit
> 253c34e9b10c30d3064be654b5b78fbc1a8b1896 [1] - and I can mount an NFS
> share with no problems:
> 
> sudo mount -t nfs4 sauron:/home/dedekind/ /mnt/sauron_home/
> 
> works fine. Any hints about how to reproduce this are welcome.
> 
> I'll try to look at the code and figure out why this could happen.
> 
> So, does the mount at some point succeed? Or it is blocked forever? And
> sysrq-t output would be useful to look at as well.
> 
> Also, it is strange that 'sys_mount()' involves 'nfs4_kill_super()' - is
> this normal or this is an error path?
> 
> [1]: the kernel tree does not compile on this commit, and I applied
> patch on top to solve the compilation issue:
> 387ac089361fbe5ef287e6950c5c40f6b18e5c55 "block: fix missing export of
> blk_types.h"
> 

Re: hang in writeback code on nfsv4 mount

[J. Bruce Fields]

On Wed, Aug 25, 2010 at 09:32:25AM +0300, Artem Bityutskiy wrote:
> On Wed, 2010-08-25 at 04:34 +0200, ext J. Bruce Fields wrote:
> > As of 253c34e9b10c30d3064be654b5b78fbc1a8b1896 "writeback: prevent
> > unnecessary bdi threads wakeups", any nfs mount hangs for me.  Is this a
> > known issue?
> > 
> > --b.
> > 
> > INFO: task mount.nfs4:3812 blocked for more than 120 seconds.
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > mount.nfs4    D 0000000000000000  2880  3812   3811 0x00000000
> >  ffff88001ed25a28 0000000000000046 ffff88001ed25fd8 ffff88001ed25fd8
> >  ffff88001ed24000 ffff88001ed24000 ffff88001ed24000 ffff88001f9503a0
> >  ffff88001ed25fd8 ffff88001f9503a8 ffff88001ed24000 ffff88001ed25fd8
> > Call Trace:
> >  [<ffffffff819608dd>] schedule_timeout+0x1cd/0x2e0
> >  [<ffffffff8106a23c>] ? mark_held_locks+0x6c/0xa0
> >  [<ffffffff81963970>] ? _raw_spin_unlock_irq+0x30/0x60
> >  [<ffffffff8106a51d>] ? trace_hardirqs_on_caller+0x14d/0x190
> >  [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
> >  [<ffffffff8195fc50>] wait_for_common+0x120/0x190
> >  [<ffffffff81033bb0>] ? default_wake_function+0x0/0x20
> >  [<ffffffff8195fd9d>] wait_for_completion+0x1d/0x20
> >  [<ffffffff8105951a>] kthread_stop+0x4a/0x150
> >  [<ffffffff81061980>] ? thaw_process+0x70/0x80
> >  [<ffffffff810cc5aa>] bdi_unregister+0x10a/0x1a0
> >  [<ffffffff81229cd9>] nfs_put_super+0x19/0x20
> >  [<ffffffff810ee7d4>] generic_shutdown_super+0x54/0xe0
> >  [<ffffffff810ee8c6>] kill_anon_super+0x16/0x60
> >  [<ffffffff8122d2c9>] nfs4_kill_super+0x39/0x90
> >  [<ffffffff810ed955>] deactivate_locked_super+0x45/0x60
> >  [<ffffffff810edec9>] deactivate_super+0x49/0x70
> >  [<ffffffff811081a4>] mntput_no_expire+0x84/0xe0
> >  [<ffffffff811083ff>] release_mounts+0x9f/0xc0
> >  [<ffffffff81108485>] put_mnt_ns+0x65/0x80
> >  [<ffffffff8122cb66>] nfs_follow_remote_path+0x1e6/0x420
> >  [<ffffffff8122cecf>] nfs4_try_mount+0x6f/0xd0
> >  [<ffffffff8122cfd2>] nfs4_get_sb+0xa2/0x360
> >  [<ffffffff810edbc8>] vfs_kern_mount+0x88/0x1f0
> >  [<ffffffff810edda2>] do_kern_mount+0x52/0x130
> >  [<ffffffff81963d6a>] ? _lock_kernel+0x6a/0x170
> >  [<ffffffff81108dae>] do_mount+0x26e/0x7f0
> >  [<ffffffff81106a4a>] ? copy_mount_options+0xea/0x190
> >  [<ffffffff811093c8>] sys_mount+0x98/0xf0
> >  [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
> > 1 lock held by mount.nfs4/3812:
> >  #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edec1>] deactivate_super+0x41/0x70
> 
> I've tried both v2.6.36-rc2 and commit
> 253c34e9b10c30d3064be654b5b78fbc1a8b1896 [1] - and I can mount an NFS
> share with no problems:
> 
> sudo mount -t nfs4 sauron:/home/dedekind/ /mnt/sauron_home/
> 
> works fine. Any hints about how to reproduce this are welcome.

Huh.  The simple mount hits it every time for me.  I'll investigate some
more.

> I'll try to look at the code and figure out why this could happen.
> 
> So, does the mount at some point succeed? Or it is blocked forever? And
> sysrq-t output would be useful to look at as well.

It's blocked forever as far as I can tell.  I'll get a sysrq-t trace.

> Also, it is strange that 'sys_mount()' involves 'nfs4_kill_super()' - is
> this normal or this is an error path?

NFSv4 uses a temporary private namespace to look up the initial mount
path--see c02d7adf8c5429727a98bad1d039bccad4c61c50 and preceding commits
for explanation.  So this may well be normal (but I haven't looked at it
closely).

Hm, my mount path has a mountpoint in it--if sauron:/home/dedekind/
doesn't, then that's a difference between our setups.

> [1]: the kernel tree does not compile on this commit, and I applied
> patch on top to solve the compilation issue:
> 387ac089361fbe5ef287e6950c5c40f6b18e5c55 "block: fix missing export of
> blk_types.h"

Maybe you only hit that if you do headers_install or headers_check?

--b.

Re: hang in writeback code on nfsv4 mount

[Artem Bityutskiy]

Hi,

On Wed, 2010-08-25 at 11:44 -0400, J. Bruce Fields wrote:
> > sudo mount -t nfs4 sauron:/home/dedekind/ /mnt/sauron_home/
> > 
> > works fine. Any hints about how to reproduce this are welcome.
> 
> Huh.  The simple mount hits it every time for me.  I'll investigate some
> more.

If you have any hints for me, please let me know. Note, I do not know
much about NFS and I very rarely use it. I'm very interested to
investigate this and fix, but I'm going to LinuxCon Brasil soon so not
sure I'll be able to look at this next week. But if there is a way to
reproduce this with a single laptop (without having 2 hosts separate
hosts, may be using localhost?) - I can look at this while being at the
conference as well :-)

> > I'll try to look at the code and figure out why this could happen.
> > 
> > So, does the mount at some point succeed? Or it is blocked forever? And
> > sysrq-t output would be useful to look at as well.
> 
> It's blocked forever as far as I can tell.  I'll get a sysrq-t trace.
> 
> > Also, it is strange that 'sys_mount()' involves 'nfs4_kill_super()' - is
> > this normal or this is an error path?
> 
> NFSv4 uses a temporary private namespace to look up the initial mount
> path--see c02d7adf8c5429727a98bad1d039bccad4c61c50 and preceding commits
> for explanation.  So this may well be normal (but I haven't looked at it
> closely).

Ok, thanks.

> Hm, my mount path has a mountpoint in it--if sauron:/home/dedekind/
> doesn't, then that's a difference between our setups.

Err, you mean that /home/dedekind on my NFS4 server should be a mount
point? Yes, it is an ext4 FS.

> > [1]: the kernel tree does not compile on this commit, and I applied
> > patch on top to solve the compilation issue:
> > 387ac089361fbe5ef287e6950c5c40f6b18e5c55 "block: fix missing export of
> > blk_types.h"
> 
> Maybe you only hit that if you do headers_install or headers_check?

Yes, this goes from headers_check, but I only run make -j11 and it runs
headers_check at the end, I do not do this explicitly.

Anyway, I'll try to reproduce / look at this tomorrow some more. May be
mount, then stress test, then unmount will help. I was distracted today
and did not really spend enough time to this.

Artem.

Re: hang in writeback code on nfsv4 mount

[Artem Bityutskiy]

On Wed, 2010-08-25 at 11:44 -0400, J. Bruce Fields wrote:
> > sudo mount -t nfs4 sauron:/home/dedekind/ /mnt/sauron_home/
> >=20
> > works fine. Any hints about how to reproduce this are welcome.
>=20
> Huh.  The simple mount hits it every time for me.  I'll investigate s=
ome
> more.

Ok, I can reproduce something which looks like memory corruption for me=
:

=46S-Cache: Netfs 'nfs' registered for caching
------------[ cut here ]------------
WARNING: at lib/list_debug.c:51 list_del+0x5e/0x8b()
Hardware name:        =20
list_del corruption. next->prev should be ffff880234e4c320, but was
ffff880234293290
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc ioatdma
iTCO_wdt i2c_i801 dca iTCO_vendor_support i5k_amb i2c_core [last
unloaded: scsi_wait_scan]
Pid: 1094, comm: mount.nfs4 Not tainted 2.6.35+ #6
Call Trace:
 [<ffffffff810466be>] warn_slowpath_common+0x80/0x98
 [<ffffffff811e29c5>] ? percpu_counter_destroy+0x21/0x4b
 [<ffffffff8104676a>] warn_slowpath_fmt+0x41/0x43
 [<ffffffff811e0412>] list_del+0x5e/0x8b
 [<ffffffff811e29ce>] percpu_counter_destroy+0x2a/0x4b
 [<ffffffff810b8280>] bdi_destroy+0x105/0x122
 [<ffffffffa00c4a96>] nfs_free_server+0xa1/0xca [nfs]
 [<ffffffffa00ce6f2>] nfs4_kill_super+0x4f/0x72 [nfs]
 [<ffffffff810e7584>] deactivate_locked_super+0x21/0x41
 [<ffffffff810e79f2>] deactivate_super+0x40/0x45
 [<ffffffff810fc088>] mntput_no_expire+0xb8/0xed
 [<ffffffff810fc157>] release_mounts+0x9a/0xb0
 [<ffffffff810fc1d7>] put_mnt_ns+0x6a/0x7b
 [<ffffffffa00cf2bf>] nfs_follow_remote_path+0x19a/0x296 [nfs]
 [<ffffffffa00cf4d7>] nfs4_try_mount+0x75/0xaf [nfs]
 [<ffffffffa00cf787>] nfs4_get_sb+0x276/0x2ff [nfs]
 [<ffffffff810e77d6>] vfs_kern_mount+0xb8/0x196
 [<ffffffff810e7912>] do_kern_mount+0x48/0xe8
 [<ffffffff810fd984>] do_mount+0x771/0x7e8
 [<ffffffff810fda7e>] sys_mount+0x83/0xbd
 [<ffffffff810089c2>] system_call_fastpath+0x16/0x1b
---[ end trace df91999b0b9e95b4 ]---

--=20
Best Regards,
Artem Bityutskiy (=D0=90=D1=80=D1=82=D1=91=D0=BC =D0=91=D0=B8=D1=82=D1=8E=
=D1=86=D0=BA=D0=B8=D0=B9)

Re: hang in writeback code on nfsv4 mount

[Artem Bityutskiy]

On Thu, 2010-08-26 at 14:24 +0300, Artem Bityutskiy wrote:
> On Wed, 2010-08-25 at 11:44 -0400, J. Bruce Fields wrote:
> > > sudo mount -t nfs4 sauron:/home/dedekind/ /mnt/sauron_home/
> > >=20
> > > works fine. Any hints about how to reproduce this are welcome.
> >=20
> > Huh.  The simple mount hits it every time for me.  I'll investigate=
 some
> > more.
>=20
> Ok, I can reproduce something which looks like memory corruption for =
me:

Err, scratch this please, this was my debugging code which I injected.
Still trying to reproduce...

--=20
Best Regards,
Artem Bityutskiy (=D0=90=D1=80=D1=82=D1=91=D0=BC =D0=91=D0=B8=D1=82=D1=8E=
=D1=86=D0=BA=D0=B8=D0=B9)

Re: hang in writeback code on nfsv4 mount

[Artem Bityutskiy]

On Wed, 2010-08-25 at 04:34 +0200, ext J. Bruce Fields wrote:
> As of 253c34e9b10c30d3064be654b5b78fbc1a8b1896 "writeback: prevent
> unnecessary bdi threads wakeups", any nfs mount hangs for me.  Is this a
> known issue?
> 
> --b.
> 
> INFO: task mount.nfs4:3812 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> mount.nfs4    D 0000000000000000  2880  3812   3811 0x00000000
>  ffff88001ed25a28 0000000000000046 ffff88001ed25fd8 ffff88001ed25fd8
>  ffff88001ed24000 ffff88001ed24000 ffff88001ed24000 ffff88001f9503a0
>  ffff88001ed25fd8 ffff88001f9503a8 ffff88001ed24000 ffff88001ed25fd8
> Call Trace:
>  [<ffffffff819608dd>] schedule_timeout+0x1cd/0x2e0
>  [<ffffffff8106a23c>] ? mark_held_locks+0x6c/0xa0
>  [<ffffffff81963970>] ? _raw_spin_unlock_irq+0x30/0x60
>  [<ffffffff8106a51d>] ? trace_hardirqs_on_caller+0x14d/0x190
>  [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
>  [<ffffffff8195fc50>] wait_for_common+0x120/0x190
>  [<ffffffff81033bb0>] ? default_wake_function+0x0/0x20
>  [<ffffffff8195fd9d>] wait_for_completion+0x1d/0x20
>  [<ffffffff8105951a>] kthread_stop+0x4a/0x150
>  [<ffffffff81061980>] ? thaw_process+0x70/0x80
>  [<ffffffff810cc5aa>] bdi_unregister+0x10a/0x1a0
>  [<ffffffff81229cd9>] nfs_put_super+0x19/0x20
>  [<ffffffff810ee7d4>] generic_shutdown_super+0x54/0xe0
>  [<ffffffff810ee8c6>] kill_anon_super+0x16/0x60
>  [<ffffffff8122d2c9>] nfs4_kill_super+0x39/0x90
>  [<ffffffff810ed955>] deactivate_locked_super+0x45/0x60
>  [<ffffffff810edec9>] deactivate_super+0x49/0x70
>  [<ffffffff811081a4>] mntput_no_expire+0x84/0xe0
>  [<ffffffff811083ff>] release_mounts+0x9f/0xc0
>  [<ffffffff81108485>] put_mnt_ns+0x65/0x80
>  [<ffffffff8122cb66>] nfs_follow_remote_path+0x1e6/0x420
>  [<ffffffff8122cecf>] nfs4_try_mount+0x6f/0xd0
>  [<ffffffff8122cfd2>] nfs4_get_sb+0xa2/0x360
>  [<ffffffff810edbc8>] vfs_kern_mount+0x88/0x1f0
>  [<ffffffff810edda2>] do_kern_mount+0x52/0x130
>  [<ffffffff81963d6a>] ? _lock_kernel+0x6a/0x170
>  [<ffffffff81108dae>] do_mount+0x26e/0x7f0
>  [<ffffffff81106a4a>] ? copy_mount_options+0xea/0x190
>  [<ffffffff811093c8>] sys_mount+0x98/0xf0
>  [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
> 1 lock held by mount.nfs4/3812:
>  #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edec1>] deactivate_super+0x41/0x70

Bruce, I can reproduce this by doing mount/fsstress/unmount, but it
takes few hours for me. I added few printks and tried to reproduce this
issue, but hit another issue. I am not sure the root cause for your
issue is the same or not, but here is the bugfix anyway.

Too bad I have to fly and cannot continue investigating. But I'll start
this again as soon as I can.

Jens, please, consider taking this to 2.6.36.

>From aa87b87c8555151bb8e083d0dba3fd68c5b0d811 Mon Sep 17 00:00:00 2001
From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
Date: Fri, 27 Aug 2010 08:21:18 +0300
Subject: [PATCH] writeback: do not lose wakeup events when forking bdi threads

This patch fixes the following issue:

INFO: task mount.nfs4:1120 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mount.nfs4    D 00000000fffc6a21     0  1120   1119 0x00000000
 ffff880235643948 0000000000000046 ffffffff00000000 ffffffff00000000
 ffff880235643fd8 ffff880235314760 00000000001d44c0 ffff880235643fd8
 00000000001d44c0 00000000001d44c0 00000000001d44c0 00000000001d44c0
Call Trace:
 [<ffffffff813bc747>] schedule_timeout+0x34/0xf1
 [<ffffffff813bc530>] ? wait_for_common+0x3f/0x130
 [<ffffffff8106b50b>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffff813bc5c3>] wait_for_common+0xd2/0x130
 [<ffffffff8104159c>] ? default_wake_function+0x0/0xf
 [<ffffffff813beaa0>] ? _raw_spin_unlock+0x26/0x2a
 [<ffffffff813bc6bb>] wait_for_completion+0x18/0x1a
 [<ffffffff81101a03>] sync_inodes_sb+0xca/0x1bc
 [<ffffffff811056a6>] __sync_filesystem+0x47/0x7e
 [<ffffffff81105798>] sync_filesystem+0x47/0x4b
 [<ffffffff810e7ffd>] generic_shutdown_super+0x22/0xd2
 [<ffffffff810e80f8>] kill_anon_super+0x11/0x4f
 [<ffffffffa00d06d7>] nfs4_kill_super+0x3f/0x72 [nfs]
 [<ffffffff810e7b68>] deactivate_locked_super+0x21/0x41
 [<ffffffff810e7fd6>] deactivate_super+0x40/0x45
 [<ffffffff810fc66c>] mntput_no_expire+0xb8/0xed
 [<ffffffff810fc73b>] release_mounts+0x9a/0xb0
 [<ffffffff810fc7bb>] put_mnt_ns+0x6a/0x7b
 [<ffffffffa00d0fb2>] nfs_follow_remote_path+0x19a/0x296 [nfs]
 [<ffffffffa00d11ca>] nfs4_try_mount+0x75/0xaf [nfs]
 [<ffffffffa00d1790>] nfs4_get_sb+0x276/0x2ff [nfs]
 [<ffffffff810e7dba>] vfs_kern_mount+0xb8/0x196
 [<ffffffff810e7ef6>] do_kern_mount+0x48/0xe8
 [<ffffffff810fdf68>] do_mount+0x771/0x7e8
 [<ffffffff810fe062>] sys_mount+0x83/0xbd
 [<ffffffff810089c2>] system_call_fastpath+0x16/0x1b

The reason of this hang was a race condition: when the flusher thread is
forking a bdi thread, we use 'kthread_run()', so we run it _before_ we make it
visible in 'bdi->wb.task'. The bdi thread runs, does all works, and goes sleep.
'bdi->wb.task' is still NULL. And this is a dangerous time window.

If at this time someone queues a work for this bdi, he does not see the bdi
thread and wakes up the forker thread instead! But the forker has already
forked this bdi thread, but just did not make it visible yet!

The result is that we lose the wake up event for this bdi thread and the NFS4
code waits forever.

To fix the problem, we should use 'ktrhead_create()' for creating bdi threads,
then make them visible in 'bdi->wb.task', and only after this wake them up.
This is exactly what this patch does.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 mm/backing-dev.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index a9a08d8..a9e0ec2 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -414,8 +414,8 @@ static int bdi_forker_thread(void *ptr)
 		switch (action) {
 		case FORK_THREAD:
 			__set_current_state(TASK_RUNNING);
-			task = kthread_run(bdi_writeback_thread, &bdi->wb, "flush-%s",
-					   dev_name(bdi->dev));
+			task = kthread_create(bdi_writeback_thread, &bdi->wb,
+					      "flush-%s", dev_name(bdi->dev));
 			if (IS_ERR(task)) {
 				/*
 				 * If thread creation fails, force writeout of
@@ -426,10 +426,13 @@ static int bdi_forker_thread(void *ptr)
 				/*
 				 * The spinlock makes sure we do not lose
 				 * wake-ups when racing with 'bdi_queue_work()'.
+				 * And as soon as the bdi thread is visible, we
+				 * can start it.
 				 */
 				spin_lock(&bdi->wb_lock);
 				bdi->wb.task = task;
 				spin_unlock(&bdi->wb_lock);
+				wake_up_process(task);
 			}
 			break;
 
-- 
1.7.1.1


-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: hang in writeback code on nfsv4 mount

[Jens Axboe]

On 2010-08-27 08:13, Artem Bityutskiy wrote:
> On Wed, 2010-08-25 at 04:34 +0200, ext J. Bruce Fields wrote:
>> As of 253c34e9b10c30d3064be654b5b78fbc1a8b1896 "writeback: prevent
>> unnecessary bdi threads wakeups", any nfs mount hangs for me.  Is this a
>> known issue?
>>
>> --b.
>>
>> INFO: task mount.nfs4:3812 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> mount.nfs4    D 0000000000000000  2880  3812   3811 0x00000000
>>  ffff88001ed25a28 0000000000000046 ffff88001ed25fd8 ffff88001ed25fd8
>>  ffff88001ed24000 ffff88001ed24000 ffff88001ed24000 ffff88001f9503a0
>>  ffff88001ed25fd8 ffff88001f9503a8 ffff88001ed24000 ffff88001ed25fd8
>> Call Trace:
>>  [<ffffffff819608dd>] schedule_timeout+0x1cd/0x2e0
>>  [<ffffffff8106a23c>] ? mark_held_locks+0x6c/0xa0
>>  [<ffffffff81963970>] ? _raw_spin_unlock_irq+0x30/0x60
>>  [<ffffffff8106a51d>] ? trace_hardirqs_on_caller+0x14d/0x190
>>  [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
>>  [<ffffffff8195fc50>] wait_for_common+0x120/0x190
>>  [<ffffffff81033bb0>] ? default_wake_function+0x0/0x20
>>  [<ffffffff8195fd9d>] wait_for_completion+0x1d/0x20
>>  [<ffffffff8105951a>] kthread_stop+0x4a/0x150
>>  [<ffffffff81061980>] ? thaw_process+0x70/0x80
>>  [<ffffffff810cc5aa>] bdi_unregister+0x10a/0x1a0
>>  [<ffffffff81229cd9>] nfs_put_super+0x19/0x20
>>  [<ffffffff810ee7d4>] generic_shutdown_super+0x54/0xe0
>>  [<ffffffff810ee8c6>] kill_anon_super+0x16/0x60
>>  [<ffffffff8122d2c9>] nfs4_kill_super+0x39/0x90
>>  [<ffffffff810ed955>] deactivate_locked_super+0x45/0x60
>>  [<ffffffff810edec9>] deactivate_super+0x49/0x70
>>  [<ffffffff811081a4>] mntput_no_expire+0x84/0xe0
>>  [<ffffffff811083ff>] release_mounts+0x9f/0xc0
>>  [<ffffffff81108485>] put_mnt_ns+0x65/0x80
>>  [<ffffffff8122cb66>] nfs_follow_remote_path+0x1e6/0x420
>>  [<ffffffff8122cecf>] nfs4_try_mount+0x6f/0xd0
>>  [<ffffffff8122cfd2>] nfs4_get_sb+0xa2/0x360
>>  [<ffffffff810edbc8>] vfs_kern_mount+0x88/0x1f0
>>  [<ffffffff810edda2>] do_kern_mount+0x52/0x130
>>  [<ffffffff81963d6a>] ? _lock_kernel+0x6a/0x170
>>  [<ffffffff81108dae>] do_mount+0x26e/0x7f0
>>  [<ffffffff81106a4a>] ? copy_mount_options+0xea/0x190
>>  [<ffffffff811093c8>] sys_mount+0x98/0xf0
>>  [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
>> 1 lock held by mount.nfs4/3812:
>>  #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edec1>] deactivate_super+0x41/0x70
> 
> Bruce, I can reproduce this by doing mount/fsstress/unmount, but it
> takes few hours for me. I added few printks and tried to reproduce this
> issue, but hit another issue. I am not sure the root cause for your
> issue is the same or not, but here is the bugfix anyway.
> 
> Too bad I have to fly and cannot continue investigating. But I'll start
> this again as soon as I can.
> 
> Jens, please, consider taking this to 2.6.36.

Thanks, applied!

-- 
Jens Axboe

Re: hang in writeback code on nfsv4 mount

[Artem Bityutskiy]

On Fri, 2010-08-27 at 09:13 +0300, Artem Bityutskiy wrote:
> From aa87b87c8555151bb8e083d0dba3fd68c5b0d811 Mon Sep 17 00:00:00 2001
> From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
> Date: Fri, 27 Aug 2010 08:21:18 +0300
> Subject: [PATCH] writeback: do not lose wakeup events when forking bdi threads

Bruce, FYI, with this patch I still cannot reproduce the issue, but
usually it happens faster. Again, too bad I cannot leave the test
running for days, because we are relocating and I have to pack all my
stuff. But when I come back from the conference, if the issue is still
there, I'll continue.

Please, try the patch I sent.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: hang in writeback code on nfsv4 mount

[Bryan Schumaker]

 The patch fixed my unmount hang.

-Bryan

On 08/27/2010 05:36 AM, Artem Bityutskiy wrote:
> On Fri, 2010-08-27 at 09:13 +0300, Artem Bityutskiy wrote:
>> From aa87b87c8555151bb8e083d0dba3fd68c5b0d811 Mon Sep 17 00:00:00 2001
>> From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
>> Date: Fri, 27 Aug 2010 08:21:18 +0300
>> Subject: [PATCH] writeback: do not lose wakeup events when forking bdi threads
> Bruce, FYI, with this patch I still cannot reproduce the issue, but
> usually it happens faster. Again, too bad I cannot leave the test
> running for days, because we are relocating and I have to pack all my
> stuff. But when I come back from the conference, if the issue is still
> there, I'll continue.
>
> Please, try the patch I sent.
>

Re: hang in writeback code on nfsv4 mount

[J. Bruce Fields]

On Fri, Aug 27, 2010 at 09:06:59AM -0400, Bryan Schumaker wrote:
>  The patch fixed my unmount hang.

For me, after applying the patch to the latest upstream, the behavior is
still the same: a hang on mount; blocked-task warning and sysrq-t dump
follow.

--b.

INFO: task mount.nfs4:4314 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
 ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
 ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
 ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc80>] wait_for_common+0x120/0x190
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
 [<ffffffff810595fa>] kthread_stop+0x4a/0x150
 [<ffffffff81061a60>] ? thaw_process+0x70/0x80
 [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edfb9>] deactivate_super+0x49/0x70
 [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
 [<ffffffff811084ef>] release_mounts+0x9f/0xc0
 [<ffffffff81108575>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810ede92>] do_kern_mount+0x52/0x130
 [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
 [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811094b8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
1 lock held by mount.nfs4/4314:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70

---

SysRq : Show State
  task                        PC stack   pid father
init          S 0000000000000000  4024     1      0 0x00000000
 ffff88001f4bd8f8 0000000000000046 ffff88001f4bdfd8 ffff88001f4bdfd8
 ffff88001f4bc000 ffff88001f4bc000 ffff88001f4bc000 ffff88001f4ba3a0
 ffff88001f4bdfd8 ffff88001f4ba3a8 ffff88001f4bc000 ffff88001f4bdfd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce3e3>] ? might_fault+0xc3/0xd0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
kthreadd      S 0000000000000000  6072     2      0 0x00000000
 ffff88001f4c1f10 0000000000000046 ffff88001f4c1fd8 ffff88001f4c1fd8
 ffff88001f4c0000 ffff88001f4c0000 ffff88001f4c0000 ffff88001f4be3e0
 ffff88001f4c1fd8 ffff88001f4be3e8 ffff88001f4c0000 ffff88001f4c1fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810595a6>] kthreadd+0xf6/0x100
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff810594b0>] ? kthreadd+0x0/0x100
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ksoftirqd/0   S 0000000000000000  6600     3      2 0x00000000
 ffff88001f4c7e80 0000000000000046 ffff88001f4c7fd8 ffff88001f4c7fd8
 ffff88001f4c6000 ffff88001f4c6000 ffff88001f4c6000 ffff88001f4c43a0
 ffff88001f4c7fd8 ffff88001f4c43a8 ffff88001f4c6000 ffff88001f4c7fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8103f427>] run_ksoftirqd+0x117/0x190
 [<ffffffff8103f310>] ? run_ksoftirqd+0x0/0x190
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/0:0   S 0000000000000000  6248     4      2 0x00000000
 ffff88001f4cbe30 0000000000000046 ffff88001f4cbfd8 ffff88001f4cbfd8
 ffff88001f4ca000 ffff88001f4ca000 ffff88001f4ca000 ffff88001f4c83e0
 ffff88001f4cbfd8 ffff88001f4c83e8 ffff88001f4ca000 ffff88001f4cbfd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/u:0   S 0000000000000000  3928     5      2 0x00000000
 ffff88001f4ede30 0000000000000046 ffff88001f4edfd8 ffff88001f4edfd8
 ffff88001f4ec000 ffff88001f4ec000 ffff88001f4ec000 ffff88001f4ea3a0
 ffff88001f4edfd8 ffff88001f4ea3a8 ffff88001f4ec000 ffff88001f4edfd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khelper       S 0000000000000000  6840     6      2 0x00000000
 ffff88001f4f3e30 0000000000000046 ffff88001f4f3fd8 ffff88001f4f3fd8
 ffff88001f4f2000 ffff88001f4f2000 ffff88001f4f2000 ffff88001f4f03e0
 ffff88001f4f3fd8 ffff88001f4f03e8 ffff88001f4f2000 ffff88001f4f3fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/u:1   S 0000000000000000  5672     7      2 0x00000000
 ffff88001f4f9e30 0000000000000046 ffff88001f4f9fd8 ffff88001f4f9fd8
 ffff88001f4f8000 ffff88001f4f8000 ffff88001f4f8000 ffff88001f4f63a0
 ffff88001f4f9fd8 ffff88001f4f63a8 ffff88001f4f8000 ffff88001f4f9fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
sync_supers   S 0000000000000000  6968   266      2 0x00000000
 ffff88001f55be90 0000000000000046 ffff88001f55bfd8 ffff88001f55bfd8
 ffff88001f55a000 ffff88001f55a000 ffff88001f55a000 ffff88001f5583e0
 ffff88001f55bfd8 ffff88001f5583e8 ffff88001f55a000 ffff88001f55bfd8
Call Trace:
 [<ffffffff810cc1c0>] ? bdi_sync_supers+0x0/0x50
 [<ffffffff810cc1f4>] bdi_sync_supers+0x34/0x50
 [<ffffffff810cc1c0>] ? bdi_sync_supers+0x0/0x50
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
bdi-default   S 0000000000000000  6312   268      2 0x00000000
 ffff88001f589d30 0000000000000046 ffff88001f589fd8 ffff88001f589fd8
 ffff88001f588000 ffff88001f588000 ffff88001f588000 ffff88001f51c3e0
 ffff88001f589fd8 ffff88001f51c3e8 ffff88001f588000 ffff88001f589fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819608b5>] schedule_timeout+0x175/0x2e0
 [<ffffffff81048900>] ? process_timeout+0x0/0x10
 [<ffffffff810ccec2>] bdi_forker_thread+0x212/0x4d0
 [<ffffffff810cccb0>] ? bdi_forker_thread+0x0/0x4d0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kblockd       S 0000000000000000  6840   270      2 0x00000000
 ffff88001f5c7e30 0000000000000046 ffff88001f5c7fd8 ffff88001f5c7fd8
 ffff88001f5c6000 ffff88001f5c6000 ffff88001f5c6000 ffff88001f5143e0
 ffff88001f5c7fd8 ffff88001f5143e8 ffff88001f5c6000 ffff88001f5c7fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kacpid        S 0000000000000000  6840   272      2 0x00000000
 ffff88001f5c5e30 0000000000000046 ffff88001f5c5fd8 ffff88001f5c5fd8
 ffff88001f5c4000 ffff88001f5c4000 ffff88001f5c4000 ffff88001f5543a0
 ffff88001f5c5fd8 ffff88001f5543a8 ffff88001f5c4000 ffff88001f5c5fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kacpi_notify  S 0000000000000000  6840   273      2 0x00000000
 ffff88001f557e30 0000000000000046 ffff88001f557fd8 ffff88001f557fd8
 ffff88001f556000 ffff88001f556000 ffff88001f556000 ffff88001f55c3a0
 ffff88001f557fd8 ffff88001f55c3a8 ffff88001f556000 ffff88001f557fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kacpi_hotplug S 0000000000000000  6840   274      2 0x00000000
 ffff88001f55fe30 0000000000000046 ffff88001f55ffd8 ffff88001f55ffd8
 ffff88001f55e000 ffff88001f55e000 ffff88001f55e000 ffff88001f5a03e0
 ffff88001f55ffd8 ffff88001f5a03e8 ffff88001f55e000 ffff88001f55ffd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ata_sff       S 0000000000000000  6840   346      2 0x00000000
 ffff88001f603e30 0000000000000046 ffff88001f603fd8 ffff88001f603fd8
 ffff88001f602000 ffff88001f602000 ffff88001f602000 ffff88001f5b03a0
 ffff88001f603fd8 ffff88001f5b03a8 ffff88001f602000 ffff88001f603fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khubd         S 0000000000000000  3416   352      2 0x00000000
 ffff88001f631d50 0000000000000046 ffff88001f631fd8 ffff88001f631fd8
 ffff88001f630000 ffff88001f630000 ffff88001f630000 ffff88001f5ae3e0
 ffff88001f631fd8 ffff88001f5ae3e8 ffff88001f630000 ffff88001f631fd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff81725fda>] hub_thread+0xdfa/0x1180
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff817251e0>] ? hub_thread+0x0/0x1180
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kseriod       S 0000000000000000  5176   355      2 0x00000000
 ffff88001f63fe20 0000000000000046 ffff88001f63ffd8 ffff88001f63ffd8
 ffff88001f63e000 ffff88001f63e000 ffff88001f63e000 ffff88001f63c3a0
 ffff88001f63ffd8 ffff88001f63c3a8 ffff88001f63e000 ffff88001f63ffd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff817516df>] serio_thread+0x32f/0x360
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff817513b0>] ? serio_thread+0x0/0x360
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khpsbpkt      S 0000000000000000  6768   453      2 0x00000000
 ffff88001f667e70 0000000000000046 ffff88001f667fd8 ffff88001f667fd8
 ffff88001f666000 ffff88001f666000 ffff88001f666000 ffff88001f6643a0
 ffff88001f667fd8 ffff88001f6643a8 ffff88001f666000 ffff88001f667fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff816f6d7a>] hpsbpkt_thread+0x13a/0x140
 [<ffffffff816f6c40>] ? hpsbpkt_thread+0x0/0x140
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
rpciod        S 0000000000000000  6840   461      2 0x00000000
 ffff88001f67de30 0000000000000046 ffff88001f67dfd8 ffff88001f67dfd8
 ffff88001f67c000 ffff88001f67c000 ffff88001f67c000 ffff88001f6843a0
 ffff88001f67dfd8 ffff88001f6843a8 ffff88001f67c000 ffff88001f67dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/0:1   S 0000000000000000  4824   462      2 0x00000000
 ffff88001f625e30 0000000000000046 ffff88001f625fd8 ffff88001f625fd8
 ffff88001f624000 ffff88001f624000 ffff88001f624000 ffff88001f6803e0
 ffff88001f625fd8 ffff88001f6803e8 ffff88001f624000 ffff88001f625fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khungtaskd    S 0000000000000000  5936   484      2 0x00000000
 ffff88001f723d90 0000000000000046 ffff88001f723fd8 ffff88001f723fd8
 ffff88001f722000 ffff88001f722000 ffff88001f722000 ffff88001e5ea3a0
 ffff88001f723fd8 ffff88001e5ea3a8 ffff88001f722000 ffff88001f723fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819608b5>] schedule_timeout+0x175/0x2e0
 [<ffffffff81048900>] ? process_timeout+0x0/0x10
 [<ffffffff81960a7d>] schedule_timeout_interruptible+0x1d/0x20
 [<ffffffff8108707a>] watchdog+0x5a/0x300
 [<ffffffff8108709e>] ? watchdog+0x7e/0x300
 [<ffffffff81087020>] ? watchdog+0x0/0x300
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kswapd0       S 0000000000000000  6288   485      2 0x00000000
 ffff88001f711de0 0000000000000046 ffff88001f711fd8 ffff88001f711fd8
 ffff88001f710000 ffff88001f710000 ffff88001f710000 ffff88001e5e63e0
 ffff88001f711fd8 ffff88001e5e63e8 ffff88001f710000 ffff88001f711fd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff810c45a4>] kswapd+0x304/0x420
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810c42a0>] ? kswapd+0x0/0x420
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
fsnotify_mark S 0000000000000000  6312   486      2 0x00000000
 ffff88001f715e20 0000000000000046 ffff88001f715fd8 ffff88001f715fd8
 ffff88001f714000 ffff88001f714000 ffff88001f714000 ffff88001e5e23a0
 ffff88001f715fd8 ffff88001e5e23a8 ffff88001f714000 ffff88001f715fd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff81123561>] fsnotify_mark_destroy+0x151/0x190
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81123410>] ? fsnotify_mark_destroy+0x0/0x190
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
aio           S 0000000000000000  6840   487      2 0x00000000
 ffff88001f719e30 0000000000000046 ffff88001f719fd8 ffff88001f719fd8
 ffff88001f718000 ffff88001f718000 ffff88001f718000 ffff88001e5de3e0
 ffff88001f719fd8 ffff88001e5de3e8 ffff88001f718000 ffff88001f719fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
nfsiod        S 0000000000000000  6840   491      2 0x00000000
 ffff88001e5cbe30 0000000000000046 ffff88001e5cbfd8 ffff88001e5cbfd8
 ffff88001e5ca000 ffff88001e5ca000 ffff88001e5ca000 ffff88001f70c3e0
 ffff88001e5cbfd8 ffff88001f70c3e8 ffff88001e5ca000 ffff88001e5cbfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jfsIO         S 0000000000000000  6800   492      2 0x00000000
 ffff88001f913e90 0000000000000046 ffff88001f913fd8 ffff88001f913fd8
 ffff88001f912000 ffff88001f912000 ffff88001f912000 ffff88001f6983a0
 ffff88001f913fd8 ffff88001f6983a8 ffff88001f912000 ffff88001f913fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff812b7750>] ? jfsIOWait+0x0/0xc0
 [<ffffffff812b7800>] jfsIOWait+0xb0/0xc0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jfsCommit     S 0000000000000000  6656   493      2 0x00000000
 ffff88001f915e30 0000000000000046 ffff88001f915fd8 ffff88001f915fd8
 ffff88001f914000 ffff88001f914000 ffff88001f914000 ffff88001f7083a0
 ffff88001f915fd8 ffff88001f7083a8 ffff88001f914000 ffff88001f915fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff812bc7c7>] jfs_lazycommit+0x267/0x290
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff812bc560>] ? jfs_lazycommit+0x0/0x290
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jfsSync       S 0000000000000000  6768   494      2 0x00000000
 ffff88001f919e70 0000000000000046 ffff88001f919fd8 ffff88001f919fd8
 ffff88001f918000 ffff88001f918000 ffff88001f918000 ffff88001f6cc3e0
 ffff88001f919fd8 ffff88001f6cc3e8 ffff88001f918000 ffff88001f919fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff812bc3e8>] jfs_sync+0x228/0x240
 [<ffffffff812bc1c0>] ? jfs_sync+0x0/0x240
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfs_mru_cache S 0000000000000000  6840   495      2 0x00000000
 ffff88001f6a1e30 0000000000000046 ffff88001f6a1fd8 ffff88001f6a1fd8
 ffff88001f6a0000 ffff88001f6a0000 ffff88001f6a0000 ffff88001f6703a0
 ffff88001f6a1fd8 ffff88001f6703a8 ffff88001f6a0000 ffff88001f6a1fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfslogd       S 0000000000000000  6840   496      2 0x00000000
 ffff88001f71de30 0000000000000046 ffff88001f71dfd8 ffff88001f71dfd8
 ffff88001f71c000 ffff88001f71c000 ffff88001f71c000 ffff88001f6783e0
 ffff88001f71dfd8 ffff88001f6783e8 ffff88001f71c000 ffff88001f71dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfsdatad      S 0000000000000000  6840   497      2 0x00000000
 ffff88001f91be30 0000000000000046 ffff88001f91bfd8 ffff88001f91bfd8
 ffff88001f91a000 ffff88001f91a000 ffff88001f91a000 ffff88001e5da3a0
 ffff88001f91bfd8 ffff88001e5da3a8 ffff88001f91a000 ffff88001f91bfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfsconvertd   S 0000000000000000  6840   498      2 0x00000000
 ffff88001f91de30 0000000000000046 ffff88001f91dfd8 ffff88001f91dfd8
 ffff88001f91c000 ffff88001f91c000 ffff88001f91c000 ffff88001f68c3a0
 ffff88001f91dfd8 ffff88001f68c3a8 ffff88001f91c000 ffff88001f91dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ocfs2_wq      S 0000000000000000  6840   499      2 0x00000000
 ffff88001f91fe30 0000000000000046 ffff88001f91ffd8 ffff88001f91ffd8
 ffff88001f91e000 ffff88001f91e000 ffff88001f91e000 ffff88001f6a83e0
 ffff88001f91ffd8 ffff88001f6a83e8 ffff88001f91e000 ffff88001f91ffd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
o2quot        S 0000000000000000  6840   500      2 0x00000000
 ffff88001f923e30 0000000000000046 ffff88001f923fd8 ffff88001f923fd8
 ffff88001f922000 ffff88001f922000 ffff88001f922000 ffff88001f7003e0
 ffff88001f923fd8 ffff88001f7003e8 ffff88001f922000 ffff88001f923fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
user_dlm      S 0000000000000000  6840   502      2 0x00000000
 ffff88001f92be30 0000000000000046 ffff88001f92bfd8 ffff88001f92bfd8
 ffff88001f92a000 ffff88001f92a000 ffff88001f92a000 ffff88001f9283a0
 ffff88001f92bfd8 ffff88001f9283a8 ffff88001f92a000 ffff88001f92bfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
crypto        S 0000000000000000  6840   505      2 0x00000000
 ffff88001f945e30 0000000000000046 ffff88001f945fd8 ffff88001f945fd8
 ffff88001f944000 ffff88001f944000 ffff88001f944000 ffff88001f9423e0
 ffff88001f945fd8 ffff88001f9423e8 ffff88001f944000 ffff88001f945fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
vballoon      S 0000000000000000  6576   573      2 0x00000000
 ffff88001f9e1df0 0000000000000046 ffff88001f9e1fd8 ffff88001f9e1fd8
 ffff88001f9e0000 ffff88001f9e0000 ffff88001f9e0000 ffff88001f9ea3e0
 ffff88001f9e1fd8 ffff88001f9ea3e8 ffff88001f9e0000 ffff88001f9e1fd8
Call Trace:
 [<ffffffff8155d850>] ? vp_get+0x50/0x70
 [<ffffffff8155ed90>] balloon+0x320/0x3f0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8155ea70>] ? balloon+0x0/0x3f0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
scsi_eh_0     S 0000000000000000  5688  1169      2 0x00000000
 ffff88001fbd7e20 0000000000000046 ffff88001fbd7fd8 ffff88001fbd7fd8
 ffff88001fbd6000 ffff88001fbd6000 ffff88001fbd6000 ffff88001fb8e3e0
 ffff88001fbd7fd8 ffff88001fb8e3e8 ffff88001fbd6000 ffff88001fbd7fd8
Call Trace:
 [<ffffffff81684967>] scsi_error_handler+0x97/0x610
 [<ffffffff816848d0>] ? scsi_error_handler+0x0/0x610
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
scsi_eh_1     S 0000000000000000  5000  1172      2 0x00000000
 ffff88001fbefe20 0000000000000046 ffff88001fbeffd8 ffff88001fbeffd8
 ffff88001fbee000 ffff88001fbee000 ffff88001fbee000 ffff88001fbec3e0
 ffff88001fbeffd8 ffff88001fbec3e8 ffff88001fbee000 ffff88001fbeffd8
Call Trace:
 [<ffffffff81684967>] scsi_error_handler+0x97/0x610
 [<ffffffff816848d0>] ? scsi_error_handler+0x0/0x610
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kpsmoused     S 0000000000000000  6840  1222      2 0x00000000
 ffff88001fba1e30 0000000000000046 ffff88001fba1fd8 ffff88001fba1fd8
 ffff88001fba0000 ffff88001fba0000 ffff88001fba0000 ffff88001f9de3a0
 ffff88001fba1fd8 ffff88001f9de3a8 ffff88001fba0000 ffff88001fba1fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kstriped      S 0000000000000000  6840  1230      2 0x00000000
 ffff88001fbf5e30 0000000000000046 ffff88001fbf5fd8 ffff88001fbf5fd8
 ffff88001fbf4000 ffff88001fbf4000 ffff88001fbf4000 ffff88001fb6a3a0
 ffff88001fbf5fd8 ffff88001fb6a3a8 ffff88001fbf4000 ffff88001fbf5fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kmpathd       S 0000000000000000  6840  1232      2 0x00000000
 ffff88001fb23e30 0000000000000046 ffff88001fb23fd8 ffff88001fb23fd8
 ffff88001fb22000 ffff88001fb22000 ffff88001fb22000 ffff88001e1223a0
 ffff88001fb23fd8 ffff88001e1223a8 ffff88001fb22000 ffff88001fb23fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kmpath_handle S 0000000000000000  6840  1233      2 0x00000000
 ffff88001f9d5e30 0000000000000046 ffff88001f9d5fd8 ffff88001f9d5fd8
 ffff88001f9d4000 ffff88001f9d4000 ffff88001f9d4000 ffff88001e1263e0
 ffff88001f9d5fd8 ffff88001e1263e8 ffff88001f9d4000 ffff88001f9d5fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ksnapd        S 0000000000000000  6840  1234      2 0x00000000
 ffff88001f9cbe30 0000000000000046 ffff88001f9cbfd8 ffff88001f9cbfd8
 ffff88001f9ca000 ffff88001f9ca000 ffff88001f9ca000 ffff88001fbf63a0
 ffff88001f9cbfd8 ffff88001fbf63a8 ffff88001f9ca000 ffff88001f9cbfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
edac-poller   S 0000000000000000  6840  1237      2 0x00000000
 ffff88001f9dde30 0000000000000046 ffff88001f9ddfd8 ffff88001f9ddfd8
 ffff88001f9dc000 ffff88001f9dc000 ffff88001f9dc000 ffff88001f9d63e0
 ffff88001f9ddfd8 ffff88001f9d63e8 ffff88001f9dc000 ffff88001f9ddfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ib_mcast      S 0000000000000000  6840  1239      2 0x00000000
 ffff88001fb95e30 0000000000000046 ffff88001fb95fd8 ffff88001fb95fd8
 ffff88001fb94000 ffff88001fb94000 ffff88001fb94000 ffff88001fb663e0
 ffff88001fb95fd8 ffff88001fb663e8 ffff88001fb94000 ffff88001fb95fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ib_cm         S 0000000000000000  6840  1241      2 0x00000000
 ffff88001fbe1e30 0000000000000046 ffff88001fbe1fd8 ffff88001fbe1fd8
 ffff88001fbe0000 ffff88001fbe0000 ffff88001fbe0000 ffff88001fb963e0
 ffff88001fbe1fd8 ffff88001fb963e8 ffff88001fbe0000 ffff88001fbe1fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
iw_cm_wq      S 0000000000000000  6840  1242      2 0x00000000
 ffff88001e133e30 0000000000000046 ffff88001e133fd8 ffff88001e133fd8
 ffff88001e132000 ffff88001e132000 ffff88001e132000 ffff88001fb9c3a0
 ffff88001e133fd8 ffff88001fb9c3a8 ffff88001e132000 ffff88001e133fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ib_addr       S 0000000000000000  6840  1243      2 0x00000000
 ffff88001e135e30 0000000000000046 ffff88001e135fd8 ffff88001e135fd8
 ffff88001e134000 ffff88001e134000 ffff88001e134000 ffff88001fbf23e0
 ffff88001e135fd8 ffff88001fbf23e8 ffff88001e134000 ffff88001e135fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
rdma_cm       S 0000000000000000  6840  1244      2 0x00000000
 ffff88001e13be30 0000000000000046 ffff88001e13bfd8 ffff88001e13bfd8
 ffff88001e13a000 ffff88001e13a000 ffff88001e13a000 ffff88001e1383e0
 ffff88001e13bfd8 ffff88001e1383e8 ffff88001e13a000 ffff88001e13bfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
usbhid_resume S 0000000000000000  6840  1273      2 0x00000000
 ffff88001e199e30 0000000000000046 ffff88001e199fd8 ffff88001e199fd8
 ffff88001e198000 ffff88001e198000 ffff88001e198000 ffff88001e1963a0
 ffff88001e199fd8 ffff88001e1963a8 ffff88001e198000 ffff88001e199fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kdmflush      S 0000000000000000  6840  2899      2 0x00000000
 ffff88001d967e30 0000000000000046 ffff88001d967fd8 ffff88001d967fd8
 ffff88001d966000 ffff88001d966000 ffff88001d966000 ffff88001d98c3e0
 ffff88001d967fd8 ffff88001d98c3e8 ffff88001d966000 ffff88001d967fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kdmflush      S 0000000000000000  6840  2909      2 0x00000000
 ffff88001d9b7e30 0000000000000046 ffff88001d9b7fd8 ffff88001d9b7fd8
 ffff88001d9b6000 ffff88001d9b6000 ffff88001d9b6000 ffff88001d9883a0
 ffff88001d9b7fd8 ffff88001d9883a8 ffff88001d9b6000 ffff88001d9b7fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jbd2/dm-0-8   S 0000000000000000  4752  2919      2 0x00000000
 ffff88001d8e9e30 0000000000000046 ffff88001d8e9fd8 ffff88001d8e9fd8
 ffff88001d8e8000 ffff88001d8e8000 ffff88001d8e8000 ffff88001d9983a0
 ffff88001d8e9fd8 ffff88001d9983a8 ffff88001d8e8000 ffff88001d8e9fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8120b434>] kjournald2+0x284/0x3b0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8120b1b0>] ? kjournald2+0x0/0x3b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ext4-dio-unwr S 0000000000000000  6840  2920      2 0x00000000
 ffff88001d97de30 0000000000000046 ffff88001d97dfd8 ffff88001d97dfd8
 ffff88001d97c000 ffff88001d97c000 ffff88001d97c000 ffff88001d9943e0
 ffff88001d97dfd8 ffff88001d9943e8 ffff88001d97c000 ffff88001d97dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
udevd         S 0000000000000000  5552  2990      1 0x00000000
 ffff88001f70fa68 0000000000000046 ffff88001f70ffd8 ffff88001f70ffd8
 ffff88001f70e000 ffff88001f70e000 ffff88001f70e000 ffff88001fb923a0
 ffff88001f70ffd8 ffff88001fb923a8 ffff88001f70e000 ffff88001f70ffd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff818b28bf>] ? unix_peer_get+0x3f/0x50
 [<ffffffff818b3410>] ? unix_dgram_poll+0x110/0x1b0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff81121ccd>] ? fsnotify+0x7d/0x340
 [<ffffffff8182cf2a>] ? sockfd_lookup_light+0x2a/0x90
 [<ffffffff8182d833>] ? sys_recvfrom+0xb3/0x120
 [<ffffffff81002508>] ? sysret_check+0x26/0x60
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
jbd2/vda1-8   S 0000000000000000  6824  3258      2 0x00000000
 ffff88001f673e30 0000000000000046 ffff88001f673fd8 ffff88001f673fd8
 ffff88001f672000 ffff88001f672000 ffff88001f672000 ffff88001e1923e0
 ffff88001f673fd8 ffff88001e1923e8 ffff88001f672000 ffff88001f673fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8120b434>] kjournald2+0x284/0x3b0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8120b1b0>] ? kjournald2+0x0/0x3b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ext4-dio-unwr S 0000000000000000  6840  3259      2 0x00000000
 ffff88001d965e30 0000000000000046 ffff88001d965fd8 ffff88001d965fd8
 ffff88001d964000 ffff88001d964000 ffff88001d964000 ffff88001eb163e0
 ffff88001d965fd8 ffff88001eb163e8 ffff88001d964000 ffff88001d965fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
udevd         S 0000000000000000  5184  3323   2990 0x00000000
 ffff88001ea0da18 0000000000000046 ffff88001ea0dfd8 ffff88001ea0dfd8
 ffff88001ea0c000 ffff88001ea0c000 ffff88001ea0c000 ffff88001e9b03a0
 ffff88001ea0dfd8 ffff88001e9b03a8 ffff88001ea0c000 ffff88001ea0dfd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff8102c717>] ? wakeup_preempt_entity+0x47/0x50
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff81963a12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff818309f6>] ? sock_def_readable+0x96/0xe0
 [<ffffffff81830960>] ? sock_def_readable+0x0/0xe0
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff818b5681>] ? unix_dgram_sendmsg+0x5b1/0x660
 [<ffffffff8182d472>] ? sys_sendmsg+0x172/0x290
 [<ffffffff8104a5e3>] ? sigprocmask+0x43/0x100
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819639ab>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff810fd36c>] sys_ppoll+0x4c/0x180
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
udevd         S 0000000000000000  5184  3324   2990 0x00000000
 ffff88001e985a18 0000000000000046 ffff88001e985fd8 ffff88001e985fd8
 ffff88001e984000 ffff88001e984000 ffff88001e984000 ffff88001eac03e0
 ffff88001e985fd8 ffff88001eac03e8 ffff88001e984000 ffff88001e985fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810b4420>] ? find_get_page+0x0/0x180
 [<ffffffff81830a2d>] ? sock_def_readable+0xcd/0xe0
 [<ffffffff81830960>] ? sock_def_readable+0x0/0xe0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810ceb58>] ? __do_fault+0x138/0x4e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810d0e0a>] ? handle_mm_fault+0x17a/0x830
 [<ffffffff8104a5e3>] ? sigprocmask+0x43/0x100
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819639ab>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff810fd36c>] sys_ppoll+0x4c/0x180
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
flush-252:0   S 0000000000000000  4136  3456      2 0x00000000
 ffff88001e49bd80 0000000000000046 ffff88001e49bfd8 ffff88001e49bfd8
 ffff88001e49a000 ffff88001e49a000 ffff88001e49a000 ffff88001eb5a3e0
 ffff88001e49bfd8 ffff88001eb5a3e8 ffff88001e49a000 ffff88001e49bfd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819608b5>] schedule_timeout+0x175/0x2e0
 [<ffffffff81048900>] ? process_timeout+0x0/0x10
 [<ffffffff811109f3>] bdi_writeback_thread+0x193/0x2f0
 [<ffffffff81110860>] ? bdi_writeback_thread+0x0/0x2f0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
rsyslogd      S 0000000000000000  5248  3520      1 0x00000000
 ffff88001ee1b8f8 0000000000000046 ffff88001ee1bfd8 ffff88001ee1bfd8
 ffff88001ee1a000 ffff88001ee1a000 ffff88001ee1a000 ffff88001d9163e0
 ffff88001ee1bfd8 ffff88001d9163e8 ffff88001ee1a000 ffff88001ee1bfd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810d1377>] ? handle_mm_fault+0x6e7/0x830
 [<ffffffff810d12b3>] ? handle_mm_fault+0x623/0x830
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff8106bf79>] ? __lock_acquire+0x399/0x1bc0
 [<ffffffff810b8ea3>] ? __rmqueue+0xd3/0x4f0
 [<ffffffff8106bf79>] ? __lock_acquire+0x399/0x1bc0
 [<ffffffff810b4420>] ? find_get_page+0x0/0x180
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810ceb58>] ? __do_fault+0x138/0x4e0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rsyslogd      S 0000000000000000  5184  3522      1 0x00000000
 ffff88001eb158f8 0000000000000046 ffff88001eb15fd8 ffff88001eb15fd8
 ffff88001eb14000 ffff88001eb14000 ffff88001eb14000 ffff88001ed203e0
 ffff88001eb15fd8 ffff88001ed203e8 ffff88001eb14000 ffff88001eb15fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff818b28bf>] ? unix_peer_get+0x3f/0x50
 [<ffffffff818b3410>] ? unix_dgram_poll+0x110/0x1b0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff814ebc8d>] ? rb_insert_color+0x9d/0x160
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81032530>] ? finish_task_switch.clone.3+0x0/0xd0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff819639ab>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81032530>] ? finish_task_switch.clone.3+0x0/0xd0
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rsyslogd      S 0000000000000000  6296  3523      1 0x00000000
 ffff88001d8dfe08 0000000000000046 ffff88001d8dffd8 ffff88001d8dffd8
 ffff88001d8de000 ffff88001d8de000 ffff88001d8de000 ffff88001e1403e0
 ffff88001d8dffd8 ffff88001e1403e8 ffff88001d8de000 ffff88001d8dffd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff8103a901>] do_syslog+0x491/0x510
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff811489d0>] ? kmsg_read+0x0/0x70
 [<ffffffff81148a07>] kmsg_read+0x37/0x70
 [<ffffffff8113dd35>] proc_reg_read+0x85/0xc0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rsyslogd      S 0000000000000000  5648  4401      1 0x00000000
 ffff88001e9c9c28 0000000000000046 ffff88001e9c9fd8 ffff88001e9c9fd8
 ffff88001e9c8000 ffff88001e9c8000 ffff88001e9c8000 ffff88001f9563e0
 ffff88001e9c9fd8 ffff88001f9563e8 ffff88001e9c8000 ffff88001e9c9fd8
Call Trace:
 [<ffffffff8106f3c2>] futex_wait_queue_me+0xc2/0x100
 [<ffffffff810707ef>] futex_wait+0x1df/0x350
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81071b21>] do_futex+0x121/0xb20
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff8107259b>] sys_futex+0x7b/0x180
 [<ffffffff8106a500>] ? trace_hardirqs_on_caller+0x50/0x190
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpcbind       S 0000000000000000  5184  3531      1 0x00000000
 ffff88001f587a68 0000000000000046 ffff88001f587fd8 ffff88001f587fd8
 ffff88001f586000 ffff88001f586000 ffff88001f586000 ffff88001f7343e0
 ffff88001f587fd8 ffff88001f7343e8 ffff88001f586000 ffff88001f587fd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81838a7a>] ? datagram_poll+0xea/0x100
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff81840230>] ? dev_queue_xmit+0x0/0x680
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff814f772f>] ? debug_check_no_obj_freed+0x18f/0x210
 [<ffffffff810e5a56>] ? kfree_debugcheck+0x16/0x40
 [<ffffffff810e603d>] ? kmem_cache_free+0xcd/0x190
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff810fe975>] ? __d_free+0x45/0x60
 [<ffffffff810fe9e8>] ? d_free+0x58/0x60
 [<ffffffff810feb4c>] ? d_kill+0x4c/0x60
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fccad>] ? poll_select_set_timeout+0x8d/0xa0
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpc.statd     S 0000000000000000  5248  3544      1 0x00000000
 ffff88001e4998f8 0000000000000046 ffff88001e499fd8 ffff88001e499fd8
 ffff88001e498000 ffff88001e498000 ffff88001e498000 ffff88001e9923e0
 ffff88001e499fd8 ffff88001e9923e8 ffff88001e498000 ffff88001e499fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81838a7a>] ? datagram_poll+0xea/0x100
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8105e3a3>] ? up_read+0x23/0x40
 [<ffffffff81966ea0>] ? do_page_fault+0x190/0x4e0
 [<ffffffff810fe9e8>] ? d_free+0x58/0x60
 [<ffffffff810feb4c>] ? d_kill+0x4c/0x60
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mdadm         S 0000000000000000  5336  3562      1 0x00000000
 ffff88001eadf8f8 0000000000000046 ffff88001eadffd8 ffff88001eadffd8
 ffff88001eade000 ffff88001eade000 ffff88001eade000 ffff88001ea463a0
 ffff88001eadffd8 ffff88001ea463a8 ffff88001eade000 ffff88001eadffd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff8113d88b>] ? pde_users_dec+0x4b/0x60
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810badef>] ? get_page_from_freelist+0x2ff/0x690
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810b4420>] ? find_get_page+0x0/0x180
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpc.gssd      S 0000000000000000  5552  3606      1 0x00000000
 ffff88001edf3a68 0000000000000046 ffff88001edf3fd8 ffff88001edf3fd8
 ffff88001edf2000 ffff88001edf2000 ffff88001edf2000 ffff88001da683e0
 ffff88001edf3fd8 ffff88001da683e8 ffff88001edf2000 ffff88001edf3fd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810ff312>] ? d_rehash+0x32/0x60
 [<ffffffff8110114c>] ? d_alloc+0x14c/0x1f0
 [<ffffffff8108b563>] ? __call_rcu+0xc3/0x180
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff810f5d21>] ? path_put+0x31/0x40
 [<ffffffff814f7628>] ? debug_check_no_obj_freed+0x88/0x210
 [<ffffffff810f9d9b>] ? do_filp_open+0x11b/0x690
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpc.svcgssd   S 0000000000000000  5248  3640      1 0x00000000
 ffff88001e409a68 0000000000000046 ffff88001e409fd8 ffff88001e409fd8
 ffff88001e408000 ffff88001e408000 ffff88001e408000 ffff88001e1463e0
 ffff88001e409fd8 ffff88001e1463e8 ffff88001e408000 ffff88001e409fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff8113d88b>] ? pde_users_dec+0x4b/0x60
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810ceb58>] ? __do_fault+0x138/0x4e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810b483b>] ? unlock_page+0x2b/0x40
 [<ffffffff810cede1>] ? __do_fault+0x3c1/0x4e0
 [<ffffffff810d0e0a>] ? handle_mm_fault+0x17a/0x830
 [<ffffffff8105e3a3>] ? up_read+0x23/0x40
 [<ffffffff81966ea0>] ? do_page_fault+0x190/0x4e0
 [<ffffffff810e5a56>] ? kfree_debugcheck+0x16/0x40
 [<ffffffff810e603d>] ? kmem_cache_free+0xcd/0x190
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8196406b>] ? retint_swapgs+0xe/0x13
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sshd          S 0000000000000000  5184  3650      1 0x00000000
 ffff88001e1a38f8 0000000000000046 ffff88001e1a3fd8 ffff88001e1a3fd8
 ffff88001e1a2000 ffff88001e1a2000 ffff88001e1a2000 ffff88001ebfe3a0
 ffff88001e1a3fd8 ffff88001ebfe3a8 ffff88001e1a2000 ffff88001e1a3fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81963a12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810fe975>] ? __d_free+0x45/0x60
 [<ffffffff810fe9e8>] ? d_free+0x58/0x60
 [<ffffffff810feb4c>] ? d_kill+0x4c/0x60
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
ntpd          S 0000000000000000  5056  3658      1 0x00000000
 ffff88001eb5d8f8 0000000000000046 ffff88001eb5dfd8 ffff88001eb5dfd8
 ffff88001eb5c000 ffff88001eb5c000 ffff88001eb5c000 ffff88001da043e0
 ffff88001eb5dfd8 ffff88001da043e8 ffff88001eb5c000 ffff88001eb5dfd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81838a7a>] ? datagram_poll+0xea/0x100
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81838938>] ? verify_iovec+0x88/0xe0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sendmail      S 0000000000000000  5184  3674      1 0x00000000
 ffff88001eb278f8 0000000000000046 ffff88001eb27fd8 ffff88001eb27fd8
 ffff88001eb26000 ffff88001eb26000 ffff88001eb26000 ffff88001eab63a0
 ffff88001eb27fd8 ffff88001eab63a8 ffff88001eb26000 ffff88001eb27fd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8110abe1>] ? seq_open+0x91/0x140
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sendmail      S 0000000000000000  5296  3682      1 0x00000000
 ffff88001eab9f68 0000000000000046 ffff88001eab9fd8 ffff88001eab9fd8
 ffff88001eab8000 ffff88001eab8000 ffff88001eab8000 ffff88001ed983a0
 ffff88001eab9fd8 ffff88001ed983a8 ffff88001eab8000 ffff88001eab9fd8
Call Trace:
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff8104a22d>] sys_pause+0x1d/0x30
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
crond         S 0000000000000000  5296  3691      1 0x00000000
 ffff88001d93fe78 0000000000000046 ffff88001d93ffd8 ffff88001d93ffd8
 ffff88001d93e000 ffff88001d93e000 ffff88001d93e000 ffff88001f7aa3e0
 ffff88001d93ffd8 ffff88001f7aa3e8 ffff88001d93e000 ffff88001d93ffd8
Call Trace:
 [<ffffffff81961c5c>] do_nanosleep+0x8c/0xc0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff8105dfde>] hrtimer_nanosleep+0xae/0x150
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff8105e104>] sys_nanosleep+0x84/0x90
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5040  3704      1 0x00000000
 ffff88001e2a1ce8 0000000000000046 ffff88001e2a1fd8 ffff88001e2a1fd8
 ffff88001e2a0000 ffff88001e2a0000 ffff88001e2a0000 ffff88001e14e3a0
 ffff88001e2a1fd8 ffff88001e14e3a8 ffff88001e2a0000 ffff88001e2a1fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  4976  3706      1 0x00000000
 ffff88001e891ce8 0000000000000046 ffff88001e891fd8 ffff88001e891fd8
 ffff88001e890000 ffff88001e890000 ffff88001e890000 ffff88001ea2c3e0
 ffff88001e891fd8 ffff88001ea2c3e8 ffff88001e890000 ffff88001e891fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3708      1 0x00000000
 ffff88001eba7ce8 0000000000000046 ffff88001eba7fd8 ffff88001eba7fd8
 ffff88001eba6000 ffff88001eba6000 ffff88001eba6000 ffff88001eaac3a0
 ffff88001eba7fd8 ffff88001eaac3a8 ffff88001eba6000 ffff88001eba7fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3710      1 0x00000000
 ffff88001e8abce8 0000000000000046 ffff88001e8abfd8 ffff88001e8abfd8
 ffff88001e8aa000 ffff88001e8aa000 ffff88001e8aa000 ffff88001ecf83e0
 ffff88001e8abfd8 ffff88001ecf83e8 ffff88001e8aa000 ffff88001e8abfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3712      1 0x00000000
 ffff88001e8adce8 0000000000000046 ffff88001e8adfd8 ffff88001e8adfd8
 ffff88001e8ac000 ffff88001e8ac000 ffff88001e8ac000 ffff88001ebd63e0
 ffff88001e8adfd8 ffff88001ebd63e8 ffff88001e8ac000 ffff88001e8adfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3714      1 0x00000000
 ffff88001ea85ce8 0000000000000046 ffff88001ea85fd8 ffff88001ea85fd8
 ffff88001ea84000 ffff88001ea84000 ffff88001ea84000 ffff88001eb443a0
 ffff88001ea85fd8 ffff88001eb443a8 ffff88001ea84000 ffff88001ea85fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sshd          S 0000000000000000  5152  3742   3650 0x00000000
 ffff88001e9818f8 0000000000000046 ffff88001e981fd8 ffff88001e981fd8
 ffff88001e980000 ffff88001e980000 ffff88001e980000 ffff88001e13c3e0
 ffff88001e981fd8 ffff88001e13c3e8 ffff88001e980000 ffff88001e981fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8196395e>] ? _raw_spin_unlock_bh+0x3e/0x50
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8105e8c1>] ? __srcu_read_unlock+0x41/0x70
 [<ffffffff81121e39>] ? fsnotify+0x1e9/0x340
 [<ffffffff81121ccd>] ? fsnotify+0x7d/0x340
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
bash          S 0000000000000000  5216  3750   3742 0x00000000
 ffff88001f6afe78 0000000000000046 ffff88001f6affd8 ffff88001f6affd8
 ffff88001f6ae000 ffff88001f6ae000 ffff88001f6ae000 ffff88001e8b43a0
 ffff88001f6affd8 ffff88001e8b43a8 ffff88001f6ae000 ffff88001f6affd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8103c916>] do_wait+0x1c6/0x280
 [<ffffffff8103ca72>] sys_wait4+0xa2/0x100
 [<ffffffff8103adc0>] ? child_wait_callback+0x0/0x70
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
lease_tests   S 0000000000000000  5968  4309      1 0x00000000
 ffff88001e26bcd8 0000000000000046 ffff88001e26bfd8 ffff88001e26bfd8
 ffff88001e26a000 ffff88001e26a000 ffff88001e26a000 ffff88001e8d63a0
 ffff88001e26bfd8 ffff88001e8d63a8 ffff88001e26a000 ffff88001e26bfd8
Call Trace:
 [<ffffffff810f477a>] pipe_wait+0x5a/0x80
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810f5235>] pipe_read+0x3b5/0x4a0
 [<ffffffff810eaf1a>] do_sync_read+0xda/0x120
 [<ffffffff810ff4aa>] ? dput+0xaa/0x160
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff814961ca>] ? security_file_permission+0x6a/0x80
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mount         S 0000000000000000  5056  4313   3750 0x00000000
 ffff88001edf1e78 0000000000000046 ffff88001edf1fd8 ffff88001edf1fd8
 ffff88001edf0000 ffff88001edf0000 ffff88001edf0000 ffff88001ed583a0
 ffff88001edf1fd8 ffff88001ed583a8 ffff88001edf0000 ffff88001edf1fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8103c916>] do_wait+0x1c6/0x280
 [<ffffffff8103ca72>] sys_wait4+0xa2/0x100
 [<ffffffff8103adc0>] ? child_wait_callback+0x0/0x70
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
 ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
 ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
 ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc80>] wait_for_common+0x120/0x190
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
 [<ffffffff810595fa>] kthread_stop+0x4a/0x150
 [<ffffffff81061a60>] ? thaw_process+0x70/0x80
 [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edfb9>] deactivate_super+0x49/0x70
 [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
 [<ffffffff811084ef>] release_mounts+0x9f/0xc0
 [<ffffffff81108575>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810ede92>] do_kern_mount+0x52/0x130
 [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
 [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811094b8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
nfsv4.0-svc   S 0000000000000000  6592  4315      2 0x00000000
 ffff88001f7a1d50 0000000000000046 ffff88001f7a1fd8 ffff88001f7a1fd8
 ffff88001f7a0000 ffff88001f7a0000 ffff88001f7a0000 ffff88001ed5a3e0
 ffff88001f7a1fd8 ffff88001ed5a3e8 ffff88001f7a0000 ffff88001f7a1fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8103fba3>] ? local_bh_enable_ip+0x83/0x110
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff81910d7f>] ? svc_recv+0x5bf/0x8a0
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8103fba3>] ? local_bh_enable_ip+0x83/0x110
 [<ffffffff81910d88>] svc_recv+0x5c8/0x8a0
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff812556b0>] ? nfs4_callback_svc+0x0/0xb0
 [<ffffffff812556f2>] nfs4_callback_svc+0x42/0xb0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
flush-0:19    S 0000000000000000  6280  4316      2 0x00000000
 ffff88001d8f5e20 0000000000000046 ffff88001d8f5fd8 ffff88001d8f5fd8
 ffff88001d8f4000 ffff88001d8f4000 ffff88001d8f4000 ffff88001e8c43e0
 ffff88001d8f5fd8 ffff88001e8c43e8 ffff88001d8f4000 ffff88001d8f5fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff811109b9>] bdi_writeback_thread+0x159/0x2f0
 [<ffffffff81110860>] ? bdi_writeback_thread+0x0/0x2f0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
anacron       S 0000000000000000  5248  4381      1 0x00000000
 ffff88001e46ff38 0000000000000046 ffff88001e46ffd8 ffff88001e46ffd8
 ffff88001e46e000 ffff88001e46e000 ffff88001e46e000 ffff88001f9483a0
 ffff88001e46ffd8 ffff88001f9483a8 ffff88001e46e000 ffff88001e46ffd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8104a4f8>] sys_rt_sigsuspend+0xb8/0xe0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sshd          S 0000000000000000  5152  4433   3650 0x00000000
 ffff88001e1a98f8 0000000000000046 ffff88001e1a9fd8 ffff88001e1a9fd8
 ffff88001e1a8000 ffff88001e1a8000 ffff88001e1a8000 ffff88001d8c63a0
 ffff88001e1a9fd8 ffff88001d8c63a8 ffff88001e1a8000 ffff88001e1a9fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8102fb52>] ? __wake_up+0x32/0x70
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8105e8c1>] ? __srcu_read_unlock+0x41/0x70
 [<ffffffff81121e39>] ? fsnotify+0x1e9/0x340
 [<ffffffff81121ccd>] ? fsnotify+0x7d/0x340
 [<ffffffff8156ae1e>] ? tty_ldisc_deref+0xe/0x10
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
bash          R  running task     5008  4441   4433 0x00000000
 ffff88001e8cfad8 ffffffff81039743 ffffffff8208d355 ffffffff819671fe
 ffff88001e8cfb08 ffffffff8208d355 ffff88001e8cfb78 ffffffff81039cdc
 ffff88001e8cfb88 0000000000000092 ffff88001e8cfbe8 0000000000000092
Call Trace:
 [<ffffffff81039743>] ? release_console_sem+0x1f3/0x230
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff81039cdc>] ? vprintk+0x26c/0x4d0
 [<ffffffff814eec6f>] ? string+0x4f/0xf0
 [<ffffffff814eec6f>] ? string+0x4f/0xf0
 [<ffffffff8195f915>] ? printk+0x68/0x6a
 [<ffffffff8195f915>] ? printk+0x68/0x6a
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff81075410>] ? is_module_text_address+0x30/0x60
 [<ffffffff81056878>] ? __kernel_text_address+0x58/0x80
 [<ffffffff81005b15>] ? print_context_stack+0x85/0x140
 [<ffffffff81004a4c>] ? dump_trace+0x18c/0x390
 [<ffffffff8100589c>] show_trace_log_lvl+0x5c/0x80
 [<ffffffff81004511>] show_stack_log_lvl+0xc1/0x1b0
 [<ffffffff8100583c>] show_stack+0x1c/0x20
 [<ffffffff810306fc>] sched_show_task+0xac/0x110
 [<ffffffff81035d8b>] show_state_filter+0x7b/0xd0
 [<ffffffff8157c2d0>] sysrq_handle_showstate+0x10/0x20
 [<ffffffff8157c731>] __handle_sysrq+0x131/0x1a0
 [<ffffffff8157c7a0>] ? write_sysrq_trigger+0x0/0x50
 [<ffffffff8157c7ee>] write_sysrq_trigger+0x4e/0x50
 [<ffffffff8113dc75>] proc_reg_write+0x85/0xc0
 [<ffffffff810eb8f7>] vfs_write+0xc7/0x170
 [<ffffffff810eba91>] sys_write+0x51/0x90
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
Sched Debug Version: v0.09, 2.6.36-rc2-00238-g0e8f983 #11
now at 828420.724224 msecs
  .jiffies                                 : 4295099271
  .sysctl_sched_latency                    : 6.000000
  .sysctl_sched_min_granularity            : 2.000000
  .sysctl_sched_wakeup_granularity         : 1.000000
  .sysctl_sched_child_runs_first           : 0
  .sysctl_sched_features                   : 15471
  .sysctl_sched_tunable_scaling            : 1 (logaritmic)

cpu#0, 2394.006 MHz
  .nr_running                    : 2
  .load                          : 2048
  .nr_switches                   : 51559
  .nr_load_updates               : 206975
  .nr_uninterruptible            : 1
  .next_balance                  : 0.000000
  .curr->pid                     : 4441
  .clock                         : 828415.137767
  .cpu_load[0]                   : 0
  .cpu_load[1]                   : 0
  .cpu_load[2]                   : 0
  .cpu_load[3]                   : 0
  .cpu_load[4]                   : 0
  .yld_count                     : 1741
  .sched_switch                  : 0
  .sched_count                   : 142628
  .sched_goidle                  : 93415
  .ttwu_count                    : 0
  .ttwu_local                    : 0
  .bkl_count                     : 192

cfs_rq[0]:
  .exec_clock                    : 17547.448581
  .MIN_vruntime                  : 16937.936514
  .min_vruntime                  : 16937.933504
  .max_vruntime                  : 16937.936514
  .spread                        : 0.000000
  .spread0                       : 0.000000
  .nr_running                    : 2
  .load                          : 2048
  .nr_spread_over                : 12

rt_rq[0]:
  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
            task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
----------------------------------------------------------------------------------------------------------
R           bash  4441     16934.933504        33   120     16934.933504        22.132725      3682.649290


Showing all locks held in the system:
1 lock held by mingetty/3704:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3706:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3708:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3710:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3712:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3714:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mount.nfs4/4314:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70
2 locks held by bash/4441:
 #0:  (sysrq_key_table_lock){......}, at: [<ffffffff8157c635>] __handle_sysrq+0x35/0x1a0
 #1:  (tasklist_lock){.+.+..}, at: [<ffffffff8106988e>] debug_show_all_locks+0x3e/0x1c0

=============================================

INFO: task mount.nfs4:4314 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
 ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
 ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
 ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc80>] wait_for_common+0x120/0x190
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
 [<ffffffff810595fa>] kthread_stop+0x4a/0x150
 [<ffffffff81061a60>] ? thaw_process+0x70/0x80
 [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edfb9>] deactivate_super+0x49/0x70
 [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
 [<ffffffff811084ef>] release_mounts+0x9f/0xc0
 [<ffffffff81108575>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810ede92>] do_kern_mount+0x52/0x130
 [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
 [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811094b8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
1 lock held by mount.nfs4/4314:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70

RE: hang in writeback code on nfsv4 mount

[Artem.Bityutskiy]

I need to look more. But so far I do not really understand what could make kthread_stop() wait
forever. I looked into the code, and thought may be barriers are missing there, but I is unlikely
this generic type of code would have bugs.

I cannot see what happens from the code and need to experiment with printks and other debugging
things. Will do as soon as I have a possibility.
________________________________________
From: ext J. Bruce Fields [bfields@fieldses.org]
Sent: Friday, August 27, 2010 6:09 PM
To: Bryan Schumaker
Cc: Bityutskiy Artem (Nokia-MS/Helsinki); Jens Axboe; linux-nfs@vger.kernel.org; Trond Myklebust; Christoph Hellwig
Subject: Re: hang in writeback code on nfsv4 mount

On Fri, Aug 27, 2010 at 09:06:59AM -0400, Bryan Schumaker wrote:
>  The patch fixed my unmount hang.

For me, after applying the patch to the latest upstream, the behavior is
still the same: a hang on mount; blocked-task warning and sysrq-t dump
follow.

--b.

INFO: task mount.nfs4:4314 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
 ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
 ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
 ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc80>] wait_for_common+0x120/0x190
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
 [<ffffffff810595fa>] kthread_stop+0x4a/0x150
 [<ffffffff81061a60>] ? thaw_process+0x70/0x80
 [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edfb9>] deactivate_super+0x49/0x70
 [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
 [<ffffffff811084ef>] release_mounts+0x9f/0xc0
 [<ffffffff81108575>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810ede92>] do_kern_mount+0x52/0x130
 [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
 [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811094b8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
1 lock held by mount.nfs4/4314:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70

---

SysRq : Show State
  task                        PC stack   pid father
init          S 0000000000000000  4024     1      0 0x00000000
 ffff88001f4bd8f8 0000000000000046 ffff88001f4bdfd8 ffff88001f4bdfd8
 ffff88001f4bc000 ffff88001f4bc000 ffff88001f4bc000 ffff88001f4ba3a0
 ffff88001f4bdfd8 ffff88001f4ba3a8 ffff88001f4bc000 ffff88001f4bdfd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce3e3>] ? might_fault+0xc3/0xd0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
kthreadd      S 0000000000000000  6072     2      0 0x00000000
 ffff88001f4c1f10 0000000000000046 ffff88001f4c1fd8 ffff88001f4c1fd8
 ffff88001f4c0000 ffff88001f4c0000 ffff88001f4c0000 ffff88001f4be3e0
 ffff88001f4c1fd8 ffff88001f4be3e8 ffff88001f4c0000 ffff88001f4c1fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810595a6>] kthreadd+0xf6/0x100
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff810594b0>] ? kthreadd+0x0/0x100
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ksoftirqd/0   S 0000000000000000  6600     3      2 0x00000000
 ffff88001f4c7e80 0000000000000046 ffff88001f4c7fd8 ffff88001f4c7fd8
 ffff88001f4c6000 ffff88001f4c6000 ffff88001f4c6000 ffff88001f4c43a0
 ffff88001f4c7fd8 ffff88001f4c43a8 ffff88001f4c6000 ffff88001f4c7fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8103f427>] run_ksoftirqd+0x117/0x190
 [<ffffffff8103f310>] ? run_ksoftirqd+0x0/0x190
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/0:0   S 0000000000000000  6248     4      2 0x00000000
 ffff88001f4cbe30 0000000000000046 ffff88001f4cbfd8 ffff88001f4cbfd8
 ffff88001f4ca000 ffff88001f4ca000 ffff88001f4ca000 ffff88001f4c83e0
 ffff88001f4cbfd8 ffff88001f4c83e8 ffff88001f4ca000 ffff88001f4cbfd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/u:0   S 0000000000000000  3928     5      2 0x00000000
 ffff88001f4ede30 0000000000000046 ffff88001f4edfd8 ffff88001f4edfd8
 ffff88001f4ec000 ffff88001f4ec000 ffff88001f4ec000 ffff88001f4ea3a0
 ffff88001f4edfd8 ffff88001f4ea3a8 ffff88001f4ec000 ffff88001f4edfd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khelper       S 0000000000000000  6840     6      2 0x00000000
 ffff88001f4f3e30 0000000000000046 ffff88001f4f3fd8 ffff88001f4f3fd8
 ffff88001f4f2000 ffff88001f4f2000 ffff88001f4f2000 ffff88001f4f03e0
 ffff88001f4f3fd8 ffff88001f4f03e8 ffff88001f4f2000 ffff88001f4f3fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/u:1   S 0000000000000000  5672     7      2 0x00000000
 ffff88001f4f9e30 0000000000000046 ffff88001f4f9fd8 ffff88001f4f9fd8
 ffff88001f4f8000 ffff88001f4f8000 ffff88001f4f8000 ffff88001f4f63a0
 ffff88001f4f9fd8 ffff88001f4f63a8 ffff88001f4f8000 ffff88001f4f9fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
sync_supers   S 0000000000000000  6968   266      2 0x00000000
 ffff88001f55be90 0000000000000046 ffff88001f55bfd8 ffff88001f55bfd8
 ffff88001f55a000 ffff88001f55a000 ffff88001f55a000 ffff88001f5583e0
 ffff88001f55bfd8 ffff88001f5583e8 ffff88001f55a000 ffff88001f55bfd8
Call Trace:
 [<ffffffff810cc1c0>] ? bdi_sync_supers+0x0/0x50
 [<ffffffff810cc1f4>] bdi_sync_supers+0x34/0x50
 [<ffffffff810cc1c0>] ? bdi_sync_supers+0x0/0x50
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
bdi-default   S 0000000000000000  6312   268      2 0x00000000
 ffff88001f589d30 0000000000000046 ffff88001f589fd8 ffff88001f589fd8
 ffff88001f588000 ffff88001f588000 ffff88001f588000 ffff88001f51c3e0
 ffff88001f589fd8 ffff88001f51c3e8 ffff88001f588000 ffff88001f589fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819608b5>] schedule_timeout+0x175/0x2e0
 [<ffffffff81048900>] ? process_timeout+0x0/0x10
 [<ffffffff810ccec2>] bdi_forker_thread+0x212/0x4d0
 [<ffffffff810cccb0>] ? bdi_forker_thread+0x0/0x4d0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kblockd       S 0000000000000000  6840   270      2 0x00000000
 ffff88001f5c7e30 0000000000000046 ffff88001f5c7fd8 ffff88001f5c7fd8
 ffff88001f5c6000 ffff88001f5c6000 ffff88001f5c6000 ffff88001f5143e0
 ffff88001f5c7fd8 ffff88001f5143e8 ffff88001f5c6000 ffff88001f5c7fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kacpid        S 0000000000000000  6840   272      2 0x00000000
 ffff88001f5c5e30 0000000000000046 ffff88001f5c5fd8 ffff88001f5c5fd8
 ffff88001f5c4000 ffff88001f5c4000 ffff88001f5c4000 ffff88001f5543a0
 ffff88001f5c5fd8 ffff88001f5543a8 ffff88001f5c4000 ffff88001f5c5fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kacpi_notify  S 0000000000000000  6840   273      2 0x00000000
 ffff88001f557e30 0000000000000046 ffff88001f557fd8 ffff88001f557fd8
 ffff88001f556000 ffff88001f556000 ffff88001f556000 ffff88001f55c3a0
 ffff88001f557fd8 ffff88001f55c3a8 ffff88001f556000 ffff88001f557fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kacpi_hotplug S 0000000000000000  6840   274      2 0x00000000
 ffff88001f55fe30 0000000000000046 ffff88001f55ffd8 ffff88001f55ffd8
 ffff88001f55e000 ffff88001f55e000 ffff88001f55e000 ffff88001f5a03e0
 ffff88001f55ffd8 ffff88001f5a03e8 ffff88001f55e000 ffff88001f55ffd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ata_sff       S 0000000000000000  6840   346      2 0x00000000
 ffff88001f603e30 0000000000000046 ffff88001f603fd8 ffff88001f603fd8
 ffff88001f602000 ffff88001f602000 ffff88001f602000 ffff88001f5b03a0
 ffff88001f603fd8 ffff88001f5b03a8 ffff88001f602000 ffff88001f603fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khubd         S 0000000000000000  3416   352      2 0x00000000
 ffff88001f631d50 0000000000000046 ffff88001f631fd8 ffff88001f631fd8
 ffff88001f630000 ffff88001f630000 ffff88001f630000 ffff88001f5ae3e0
 ffff88001f631fd8 ffff88001f5ae3e8 ffff88001f630000 ffff88001f631fd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff81725fda>] hub_thread+0xdfa/0x1180
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff817251e0>] ? hub_thread+0x0/0x1180
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kseriod       S 0000000000000000  5176   355      2 0x00000000
 ffff88001f63fe20 0000000000000046 ffff88001f63ffd8 ffff88001f63ffd8
 ffff88001f63e000 ffff88001f63e000 ffff88001f63e000 ffff88001f63c3a0
 ffff88001f63ffd8 ffff88001f63c3a8 ffff88001f63e000 ffff88001f63ffd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff817516df>] serio_thread+0x32f/0x360
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff817513b0>] ? serio_thread+0x0/0x360
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khpsbpkt      S 0000000000000000  6768   453      2 0x00000000
 ffff88001f667e70 0000000000000046 ffff88001f667fd8 ffff88001f667fd8
 ffff88001f666000 ffff88001f666000 ffff88001f666000 ffff88001f6643a0
 ffff88001f667fd8 ffff88001f6643a8 ffff88001f666000 ffff88001f667fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff816f6d7a>] hpsbpkt_thread+0x13a/0x140
 [<ffffffff816f6c40>] ? hpsbpkt_thread+0x0/0x140
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
rpciod        S 0000000000000000  6840   461      2 0x00000000
 ffff88001f67de30 0000000000000046 ffff88001f67dfd8 ffff88001f67dfd8
 ffff88001f67c000 ffff88001f67c000 ffff88001f67c000 ffff88001f6843a0
 ffff88001f67dfd8 ffff88001f6843a8 ffff88001f67c000 ffff88001f67dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kworker/0:1   S 0000000000000000  4824   462      2 0x00000000
 ffff88001f625e30 0000000000000046 ffff88001f625fd8 ffff88001f625fd8
 ffff88001f624000 ffff88001f624000 ffff88001f624000 ffff88001f6803e0
 ffff88001f625fd8 ffff88001f6803e8 ffff88001f624000 ffff88001f625fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81055458>] worker_thread+0x1f8/0x390
 [<ffffffff81055260>] ? worker_thread+0x0/0x390
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
khungtaskd    S 0000000000000000  5936   484      2 0x00000000
 ffff88001f723d90 0000000000000046 ffff88001f723fd8 ffff88001f723fd8
 ffff88001f722000 ffff88001f722000 ffff88001f722000 ffff88001e5ea3a0
 ffff88001f723fd8 ffff88001e5ea3a8 ffff88001f722000 ffff88001f723fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819608b5>] schedule_timeout+0x175/0x2e0
 [<ffffffff81048900>] ? process_timeout+0x0/0x10
 [<ffffffff81960a7d>] schedule_timeout_interruptible+0x1d/0x20
 [<ffffffff8108707a>] watchdog+0x5a/0x300
 [<ffffffff8108709e>] ? watchdog+0x7e/0x300
 [<ffffffff81087020>] ? watchdog+0x0/0x300
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kswapd0       S 0000000000000000  6288   485      2 0x00000000
 ffff88001f711de0 0000000000000046 ffff88001f711fd8 ffff88001f711fd8
 ffff88001f710000 ffff88001f710000 ffff88001f710000 ffff88001e5e63e0
 ffff88001f711fd8 ffff88001e5e63e8 ffff88001f710000 ffff88001f711fd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff810c45a4>] kswapd+0x304/0x420
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810c42a0>] ? kswapd+0x0/0x420
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
fsnotify_mark S 0000000000000000  6312   486      2 0x00000000
 ffff88001f715e20 0000000000000046 ffff88001f715fd8 ffff88001f715fd8
 ffff88001f714000 ffff88001f714000 ffff88001f714000 ffff88001e5e23a0
 ffff88001f715fd8 ffff88001e5e23a8 ffff88001f714000 ffff88001f715fd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff81123561>] fsnotify_mark_destroy+0x151/0x190
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81123410>] ? fsnotify_mark_destroy+0x0/0x190
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
aio           S 0000000000000000  6840   487      2 0x00000000
 ffff88001f719e30 0000000000000046 ffff88001f719fd8 ffff88001f719fd8
 ffff88001f718000 ffff88001f718000 ffff88001f718000 ffff88001e5de3e0
 ffff88001f719fd8 ffff88001e5de3e8 ffff88001f718000 ffff88001f719fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
nfsiod        S 0000000000000000  6840   491      2 0x00000000
 ffff88001e5cbe30 0000000000000046 ffff88001e5cbfd8 ffff88001e5cbfd8
 ffff88001e5ca000 ffff88001e5ca000 ffff88001e5ca000 ffff88001f70c3e0
 ffff88001e5cbfd8 ffff88001f70c3e8 ffff88001e5ca000 ffff88001e5cbfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jfsIO         S 0000000000000000  6800   492      2 0x00000000
 ffff88001f913e90 0000000000000046 ffff88001f913fd8 ffff88001f913fd8
 ffff88001f912000 ffff88001f912000 ffff88001f912000 ffff88001f6983a0
 ffff88001f913fd8 ffff88001f6983a8 ffff88001f912000 ffff88001f913fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff812b7750>] ? jfsIOWait+0x0/0xc0
 [<ffffffff812b7800>] jfsIOWait+0xb0/0xc0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jfsCommit     S 0000000000000000  6656   493      2 0x00000000
 ffff88001f915e30 0000000000000046 ffff88001f915fd8 ffff88001f915fd8
 ffff88001f914000 ffff88001f914000 ffff88001f914000 ffff88001f7083a0
 ffff88001f915fd8 ffff88001f7083a8 ffff88001f914000 ffff88001f915fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff812bc7c7>] jfs_lazycommit+0x267/0x290
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff812bc560>] ? jfs_lazycommit+0x0/0x290
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jfsSync       S 0000000000000000  6768   494      2 0x00000000
 ffff88001f919e70 0000000000000046 ffff88001f919fd8 ffff88001f919fd8
 ffff88001f918000 ffff88001f918000 ffff88001f918000 ffff88001f6cc3e0
 ffff88001f919fd8 ffff88001f6cc3e8 ffff88001f918000 ffff88001f919fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff812bc3e8>] jfs_sync+0x228/0x240
 [<ffffffff812bc1c0>] ? jfs_sync+0x0/0x240
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfs_mru_cache S 0000000000000000  6840   495      2 0x00000000
 ffff88001f6a1e30 0000000000000046 ffff88001f6a1fd8 ffff88001f6a1fd8
 ffff88001f6a0000 ffff88001f6a0000 ffff88001f6a0000 ffff88001f6703a0
 ffff88001f6a1fd8 ffff88001f6703a8 ffff88001f6a0000 ffff88001f6a1fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfslogd       S 0000000000000000  6840   496      2 0x00000000
 ffff88001f71de30 0000000000000046 ffff88001f71dfd8 ffff88001f71dfd8
 ffff88001f71c000 ffff88001f71c000 ffff88001f71c000 ffff88001f6783e0
 ffff88001f71dfd8 ffff88001f6783e8 ffff88001f71c000 ffff88001f71dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfsdatad      S 0000000000000000  6840   497      2 0x00000000
 ffff88001f91be30 0000000000000046 ffff88001f91bfd8 ffff88001f91bfd8
 ffff88001f91a000 ffff88001f91a000 ffff88001f91a000 ffff88001e5da3a0
 ffff88001f91bfd8 ffff88001e5da3a8 ffff88001f91a000 ffff88001f91bfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
xfsconvertd   S 0000000000000000  6840   498      2 0x00000000
 ffff88001f91de30 0000000000000046 ffff88001f91dfd8 ffff88001f91dfd8
 ffff88001f91c000 ffff88001f91c000 ffff88001f91c000 ffff88001f68c3a0
 ffff88001f91dfd8 ffff88001f68c3a8 ffff88001f91c000 ffff88001f91dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ocfs2_wq      S 0000000000000000  6840   499      2 0x00000000
 ffff88001f91fe30 0000000000000046 ffff88001f91ffd8 ffff88001f91ffd8
 ffff88001f91e000 ffff88001f91e000 ffff88001f91e000 ffff88001f6a83e0
 ffff88001f91ffd8 ffff88001f6a83e8 ffff88001f91e000 ffff88001f91ffd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
o2quot        S 0000000000000000  6840   500      2 0x00000000
 ffff88001f923e30 0000000000000046 ffff88001f923fd8 ffff88001f923fd8
 ffff88001f922000 ffff88001f922000 ffff88001f922000 ffff88001f7003e0
 ffff88001f923fd8 ffff88001f7003e8 ffff88001f922000 ffff88001f923fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
user_dlm      S 0000000000000000  6840   502      2 0x00000000
 ffff88001f92be30 0000000000000046 ffff88001f92bfd8 ffff88001f92bfd8
 ffff88001f92a000 ffff88001f92a000 ffff88001f92a000 ffff88001f9283a0
 ffff88001f92bfd8 ffff88001f9283a8 ffff88001f92a000 ffff88001f92bfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
crypto        S 0000000000000000  6840   505      2 0x00000000
 ffff88001f945e30 0000000000000046 ffff88001f945fd8 ffff88001f945fd8
 ffff88001f944000 ffff88001f944000 ffff88001f944000 ffff88001f9423e0
 ffff88001f945fd8 ffff88001f9423e8 ffff88001f944000 ffff88001f945fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
vballoon      S 0000000000000000  6576   573      2 0x00000000
 ffff88001f9e1df0 0000000000000046 ffff88001f9e1fd8 ffff88001f9e1fd8
 ffff88001f9e0000 ffff88001f9e0000 ffff88001f9e0000 ffff88001f9ea3e0
 ffff88001f9e1fd8 ffff88001f9ea3e8 ffff88001f9e0000 ffff88001f9e1fd8
Call Trace:
 [<ffffffff8155d850>] ? vp_get+0x50/0x70
 [<ffffffff8155ed90>] balloon+0x320/0x3f0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8155ea70>] ? balloon+0x0/0x3f0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
scsi_eh_0     S 0000000000000000  5688  1169      2 0x00000000
 ffff88001fbd7e20 0000000000000046 ffff88001fbd7fd8 ffff88001fbd7fd8
 ffff88001fbd6000 ffff88001fbd6000 ffff88001fbd6000 ffff88001fb8e3e0
 ffff88001fbd7fd8 ffff88001fb8e3e8 ffff88001fbd6000 ffff88001fbd7fd8
Call Trace:
 [<ffffffff81684967>] scsi_error_handler+0x97/0x610
 [<ffffffff816848d0>] ? scsi_error_handler+0x0/0x610
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
scsi_eh_1     S 0000000000000000  5000  1172      2 0x00000000
 ffff88001fbefe20 0000000000000046 ffff88001fbeffd8 ffff88001fbeffd8
 ffff88001fbee000 ffff88001fbee000 ffff88001fbee000 ffff88001fbec3e0
 ffff88001fbeffd8 ffff88001fbec3e8 ffff88001fbee000 ffff88001fbeffd8
Call Trace:
 [<ffffffff81684967>] scsi_error_handler+0x97/0x610
 [<ffffffff816848d0>] ? scsi_error_handler+0x0/0x610
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kpsmoused     S 0000000000000000  6840  1222      2 0x00000000
 ffff88001fba1e30 0000000000000046 ffff88001fba1fd8 ffff88001fba1fd8
 ffff88001fba0000 ffff88001fba0000 ffff88001fba0000 ffff88001f9de3a0
 ffff88001fba1fd8 ffff88001f9de3a8 ffff88001fba0000 ffff88001fba1fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kstriped      S 0000000000000000  6840  1230      2 0x00000000
 ffff88001fbf5e30 0000000000000046 ffff88001fbf5fd8 ffff88001fbf5fd8
 ffff88001fbf4000 ffff88001fbf4000 ffff88001fbf4000 ffff88001fb6a3a0
 ffff88001fbf5fd8 ffff88001fb6a3a8 ffff88001fbf4000 ffff88001fbf5fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kmpathd       S 0000000000000000  6840  1232      2 0x00000000
 ffff88001fb23e30 0000000000000046 ffff88001fb23fd8 ffff88001fb23fd8
 ffff88001fb22000 ffff88001fb22000 ffff88001fb22000 ffff88001e1223a0
 ffff88001fb23fd8 ffff88001e1223a8 ffff88001fb22000 ffff88001fb23fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kmpath_handle S 0000000000000000  6840  1233      2 0x00000000
 ffff88001f9d5e30 0000000000000046 ffff88001f9d5fd8 ffff88001f9d5fd8
 ffff88001f9d4000 ffff88001f9d4000 ffff88001f9d4000 ffff88001e1263e0
 ffff88001f9d5fd8 ffff88001e1263e8 ffff88001f9d4000 ffff88001f9d5fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ksnapd        S 0000000000000000  6840  1234      2 0x00000000
 ffff88001f9cbe30 0000000000000046 ffff88001f9cbfd8 ffff88001f9cbfd8
 ffff88001f9ca000 ffff88001f9ca000 ffff88001f9ca000 ffff88001fbf63a0
 ffff88001f9cbfd8 ffff88001fbf63a8 ffff88001f9ca000 ffff88001f9cbfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
edac-poller   S 0000000000000000  6840  1237      2 0x00000000
 ffff88001f9dde30 0000000000000046 ffff88001f9ddfd8 ffff88001f9ddfd8
 ffff88001f9dc000 ffff88001f9dc000 ffff88001f9dc000 ffff88001f9d63e0
 ffff88001f9ddfd8 ffff88001f9d63e8 ffff88001f9dc000 ffff88001f9ddfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ib_mcast      S 0000000000000000  6840  1239      2 0x00000000
 ffff88001fb95e30 0000000000000046 ffff88001fb95fd8 ffff88001fb95fd8
 ffff88001fb94000 ffff88001fb94000 ffff88001fb94000 ffff88001fb663e0
 ffff88001fb95fd8 ffff88001fb663e8 ffff88001fb94000 ffff88001fb95fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ib_cm         S 0000000000000000  6840  1241      2 0x00000000
 ffff88001fbe1e30 0000000000000046 ffff88001fbe1fd8 ffff88001fbe1fd8
 ffff88001fbe0000 ffff88001fbe0000 ffff88001fbe0000 ffff88001fb963e0
 ffff88001fbe1fd8 ffff88001fb963e8 ffff88001fbe0000 ffff88001fbe1fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
iw_cm_wq      S 0000000000000000  6840  1242      2 0x00000000
 ffff88001e133e30 0000000000000046 ffff88001e133fd8 ffff88001e133fd8
 ffff88001e132000 ffff88001e132000 ffff88001e132000 ffff88001fb9c3a0
 ffff88001e133fd8 ffff88001fb9c3a8 ffff88001e132000 ffff88001e133fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ib_addr       S 0000000000000000  6840  1243      2 0x00000000
 ffff88001e135e30 0000000000000046 ffff88001e135fd8 ffff88001e135fd8
 ffff88001e134000 ffff88001e134000 ffff88001e134000 ffff88001fbf23e0
 ffff88001e135fd8 ffff88001fbf23e8 ffff88001e134000 ffff88001e135fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
rdma_cm       S 0000000000000000  6840  1244      2 0x00000000
 ffff88001e13be30 0000000000000046 ffff88001e13bfd8 ffff88001e13bfd8
 ffff88001e13a000 ffff88001e13a000 ffff88001e13a000 ffff88001e1383e0
 ffff88001e13bfd8 ffff88001e1383e8 ffff88001e13a000 ffff88001e13bfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
usbhid_resume S 0000000000000000  6840  1273      2 0x00000000
 ffff88001e199e30 0000000000000046 ffff88001e199fd8 ffff88001e199fd8
 ffff88001e198000 ffff88001e198000 ffff88001e198000 ffff88001e1963a0
 ffff88001e199fd8 ffff88001e1963a8 ffff88001e198000 ffff88001e199fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kdmflush      S 0000000000000000  6840  2899      2 0x00000000
 ffff88001d967e30 0000000000000046 ffff88001d967fd8 ffff88001d967fd8
 ffff88001d966000 ffff88001d966000 ffff88001d966000 ffff88001d98c3e0
 ffff88001d967fd8 ffff88001d98c3e8 ffff88001d966000 ffff88001d967fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
kdmflush      S 0000000000000000  6840  2909      2 0x00000000
 ffff88001d9b7e30 0000000000000046 ffff88001d9b7fd8 ffff88001d9b7fd8
 ffff88001d9b6000 ffff88001d9b6000 ffff88001d9b6000 ffff88001d9883a0
 ffff88001d9b7fd8 ffff88001d9883a8 ffff88001d9b6000 ffff88001d9b7fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8196933c>] ? kprobe_flush_task+0xbc/0xe0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
jbd2/dm-0-8   S 0000000000000000  4752  2919      2 0x00000000
 ffff88001d8e9e30 0000000000000046 ffff88001d8e9fd8 ffff88001d8e9fd8
 ffff88001d8e8000 ffff88001d8e8000 ffff88001d8e8000 ffff88001d9983a0
 ffff88001d8e9fd8 ffff88001d9983a8 ffff88001d8e8000 ffff88001d8e9fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8120b434>] kjournald2+0x284/0x3b0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8120b1b0>] ? kjournald2+0x0/0x3b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ext4-dio-unwr S 0000000000000000  6840  2920      2 0x00000000
 ffff88001d97de30 0000000000000046 ffff88001d97dfd8 ffff88001d97dfd8
 ffff88001d97c000 ffff88001d97c000 ffff88001d97c000 ffff88001d9943e0
 ffff88001d97dfd8 ffff88001d9943e8 ffff88001d97c000 ffff88001d97dfd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
udevd         S 0000000000000000  5552  2990      1 0x00000000
 ffff88001f70fa68 0000000000000046 ffff88001f70ffd8 ffff88001f70ffd8
 ffff88001f70e000 ffff88001f70e000 ffff88001f70e000 ffff88001fb923a0
 ffff88001f70ffd8 ffff88001fb923a8 ffff88001f70e000 ffff88001f70ffd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff818b28bf>] ? unix_peer_get+0x3f/0x50
 [<ffffffff818b3410>] ? unix_dgram_poll+0x110/0x1b0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff81121ccd>] ? fsnotify+0x7d/0x340
 [<ffffffff8182cf2a>] ? sockfd_lookup_light+0x2a/0x90
 [<ffffffff8182d833>] ? sys_recvfrom+0xb3/0x120
 [<ffffffff81002508>] ? sysret_check+0x26/0x60
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
jbd2/vda1-8   S 0000000000000000  6824  3258      2 0x00000000
 ffff88001f673e30 0000000000000046 ffff88001f673fd8 ffff88001f673fd8
 ffff88001f672000 ffff88001f672000 ffff88001f672000 ffff88001e1923e0
 ffff88001f673fd8 ffff88001e1923e8 ffff88001f672000 ffff88001f673fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8120b434>] kjournald2+0x284/0x3b0
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8120b1b0>] ? kjournald2+0x0/0x3b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
ext4-dio-unwr S 0000000000000000  6840  3259      2 0x00000000
 ffff88001d965e30 0000000000000046 ffff88001d965fd8 ffff88001d965fd8
 ffff88001d964000 ffff88001d964000 ffff88001d964000 ffff88001eb163e0
 ffff88001d965fd8 ffff88001eb163e8 ffff88001d964000 ffff88001d965fd8
Call Trace:
 [<ffffffff81031ce9>] ? set_user_nice+0xc9/0x140
 [<ffffffff81055202>] rescuer_thread+0x152/0x1b0
 [<ffffffff810550b0>] ? rescuer_thread+0x0/0x1b0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
udevd         S 0000000000000000  5184  3323   2990 0x00000000
 ffff88001ea0da18 0000000000000046 ffff88001ea0dfd8 ffff88001ea0dfd8
 ffff88001ea0c000 ffff88001ea0c000 ffff88001ea0c000 ffff88001e9b03a0
 ffff88001ea0dfd8 ffff88001e9b03a8 ffff88001ea0c000 ffff88001ea0dfd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff8102c717>] ? wakeup_preempt_entity+0x47/0x50
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff81963a12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff818309f6>] ? sock_def_readable+0x96/0xe0
 [<ffffffff81830960>] ? sock_def_readable+0x0/0xe0
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff818b5681>] ? unix_dgram_sendmsg+0x5b1/0x660
 [<ffffffff8182d472>] ? sys_sendmsg+0x172/0x290
 [<ffffffff8104a5e3>] ? sigprocmask+0x43/0x100
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819639ab>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff810fd36c>] sys_ppoll+0x4c/0x180
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
udevd         S 0000000000000000  5184  3324   2990 0x00000000
 ffff88001e985a18 0000000000000046 ffff88001e985fd8 ffff88001e985fd8
 ffff88001e984000 ffff88001e984000 ffff88001e984000 ffff88001eac03e0
 ffff88001e985fd8 ffff88001eac03e8 ffff88001e984000 ffff88001e985fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810b4420>] ? find_get_page+0x0/0x180
 [<ffffffff81830a2d>] ? sock_def_readable+0xcd/0xe0
 [<ffffffff81830960>] ? sock_def_readable+0x0/0xe0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810ceb58>] ? __do_fault+0x138/0x4e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810d0e0a>] ? handle_mm_fault+0x17a/0x830
 [<ffffffff8104a5e3>] ? sigprocmask+0x43/0x100
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819639ab>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff810fd36c>] sys_ppoll+0x4c/0x180
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
flush-252:0   S 0000000000000000  4136  3456      2 0x00000000
 ffff88001e49bd80 0000000000000046 ffff88001e49bfd8 ffff88001e49bfd8
 ffff88001e49a000 ffff88001e49a000 ffff88001e49a000 ffff88001eb5a3e0
 ffff88001e49bfd8 ffff88001eb5a3e8 ffff88001e49a000 ffff88001e49bfd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff819608b5>] schedule_timeout+0x175/0x2e0
 [<ffffffff81048900>] ? process_timeout+0x0/0x10
 [<ffffffff811109f3>] bdi_writeback_thread+0x193/0x2f0
 [<ffffffff81110860>] ? bdi_writeback_thread+0x0/0x2f0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
rsyslogd      S 0000000000000000  5248  3520      1 0x00000000
 ffff88001ee1b8f8 0000000000000046 ffff88001ee1bfd8 ffff88001ee1bfd8
 ffff88001ee1a000 ffff88001ee1a000 ffff88001ee1a000 ffff88001d9163e0
 ffff88001ee1bfd8 ffff88001d9163e8 ffff88001ee1a000 ffff88001ee1bfd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810d1377>] ? handle_mm_fault+0x6e7/0x830
 [<ffffffff810d12b3>] ? handle_mm_fault+0x623/0x830
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff8106bf79>] ? __lock_acquire+0x399/0x1bc0
 [<ffffffff810b8ea3>] ? __rmqueue+0xd3/0x4f0
 [<ffffffff8106bf79>] ? __lock_acquire+0x399/0x1bc0
 [<ffffffff810b4420>] ? find_get_page+0x0/0x180
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810ceb58>] ? __do_fault+0x138/0x4e0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rsyslogd      S 0000000000000000  5184  3522      1 0x00000000
 ffff88001eb158f8 0000000000000046 ffff88001eb15fd8 ffff88001eb15fd8
 ffff88001eb14000 ffff88001eb14000 ffff88001eb14000 ffff88001ed203e0
 ffff88001eb15fd8 ffff88001ed203e8 ffff88001eb14000 ffff88001eb15fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff818b28bf>] ? unix_peer_get+0x3f/0x50
 [<ffffffff818b3410>] ? unix_dgram_poll+0x110/0x1b0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff814ebc8d>] ? rb_insert_color+0x9d/0x160
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81032530>] ? finish_task_switch.clone.3+0x0/0xd0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff819639ab>] ? _raw_spin_unlock_irq+0x3b/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81032530>] ? finish_task_switch.clone.3+0x0/0xd0
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rsyslogd      S 0000000000000000  6296  3523      1 0x00000000
 ffff88001d8dfe08 0000000000000046 ffff88001d8dffd8 ffff88001d8dffd8
 ffff88001d8de000 ffff88001d8de000 ffff88001d8de000 ffff88001e1403e0
 ffff88001d8dffd8 ffff88001e1403e8 ffff88001d8de000 ffff88001d8dffd8
Call Trace:
 [<ffffffff81059c2d>] ? prepare_to_wait+0x4d/0x80
 [<ffffffff8103a901>] do_syslog+0x491/0x510
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff811489d0>] ? kmsg_read+0x0/0x70
 [<ffffffff81148a07>] kmsg_read+0x37/0x70
 [<ffffffff8113dd35>] proc_reg_read+0x85/0xc0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rsyslogd      S 0000000000000000  5648  4401      1 0x00000000
 ffff88001e9c9c28 0000000000000046 ffff88001e9c9fd8 ffff88001e9c9fd8
 ffff88001e9c8000 ffff88001e9c8000 ffff88001e9c8000 ffff88001f9563e0
 ffff88001e9c9fd8 ffff88001f9563e8 ffff88001e9c8000 ffff88001e9c9fd8
Call Trace:
 [<ffffffff8106f3c2>] futex_wait_queue_me+0xc2/0x100
 [<ffffffff810707ef>] futex_wait+0x1df/0x350
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81071b21>] do_futex+0x121/0xb20
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff8107259b>] sys_futex+0x7b/0x180
 [<ffffffff8106a500>] ? trace_hardirqs_on_caller+0x50/0x190
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpcbind       S 0000000000000000  5184  3531      1 0x00000000
 ffff88001f587a68 0000000000000046 ffff88001f587fd8 ffff88001f587fd8
 ffff88001f586000 ffff88001f586000 ffff88001f586000 ffff88001f7343e0
 ffff88001f587fd8 ffff88001f7343e8 ffff88001f586000 ffff88001f587fd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81838a7a>] ? datagram_poll+0xea/0x100
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff81840230>] ? dev_queue_xmit+0x0/0x680
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff814f772f>] ? debug_check_no_obj_freed+0x18f/0x210
 [<ffffffff810e5a56>] ? kfree_debugcheck+0x16/0x40
 [<ffffffff810e603d>] ? kmem_cache_free+0xcd/0x190
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff810fe975>] ? __d_free+0x45/0x60
 [<ffffffff810fe9e8>] ? d_free+0x58/0x60
 [<ffffffff810feb4c>] ? d_kill+0x4c/0x60
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fccad>] ? poll_select_set_timeout+0x8d/0xa0
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpc.statd     S 0000000000000000  5248  3544      1 0x00000000
 ffff88001e4998f8 0000000000000046 ffff88001e499fd8 ffff88001e499fd8
 ffff88001e498000 ffff88001e498000 ffff88001e498000 ffff88001e9923e0
 ffff88001e499fd8 ffff88001e9923e8 ffff88001e498000 ffff88001e499fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81838a7a>] ? datagram_poll+0xea/0x100
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8105e3a3>] ? up_read+0x23/0x40
 [<ffffffff81966ea0>] ? do_page_fault+0x190/0x4e0
 [<ffffffff810fe9e8>] ? d_free+0x58/0x60
 [<ffffffff810feb4c>] ? d_kill+0x4c/0x60
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mdadm         S 0000000000000000  5336  3562      1 0x00000000
 ffff88001eadf8f8 0000000000000046 ffff88001eadffd8 ffff88001eadffd8
 ffff88001eade000 ffff88001eade000 ffff88001eade000 ffff88001ea463a0
 ffff88001eadffd8 ffff88001ea463a8 ffff88001eade000 ffff88001eadffd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff8113d88b>] ? pde_users_dec+0x4b/0x60
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810badef>] ? get_page_from_freelist+0x2ff/0x690
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810b4420>] ? find_get_page+0x0/0x180
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpc.gssd      S 0000000000000000  5552  3606      1 0x00000000
 ffff88001edf3a68 0000000000000046 ffff88001edf3fd8 ffff88001edf3fd8
 ffff88001edf2000 ffff88001edf2000 ffff88001edf2000 ffff88001da683e0
 ffff88001edf3fd8 ffff88001da683e8 ffff88001edf2000 ffff88001edf3fd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810ff312>] ? d_rehash+0x32/0x60
 [<ffffffff8110114c>] ? d_alloc+0x14c/0x1f0
 [<ffffffff8108b563>] ? __call_rcu+0xc3/0x180
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff810f5d21>] ? path_put+0x31/0x40
 [<ffffffff814f7628>] ? debug_check_no_obj_freed+0x88/0x210
 [<ffffffff810f9d9b>] ? do_filp_open+0x11b/0x690
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
rpc.svcgssd   S 0000000000000000  5248  3640      1 0x00000000
 ffff88001e409a68 0000000000000046 ffff88001e409fd8 ffff88001e409fd8
 ffff88001e408000 ffff88001e408000 ffff88001e408000 ffff88001e1463e0
 ffff88001e409fd8 ffff88001e1463e8 ffff88001e408000 ffff88001e409fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81963a85>] ? _raw_spin_unlock+0x35/0x60
 [<ffffffff8113d88b>] ? pde_users_dec+0x4b/0x60
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fd257>] do_sys_poll+0x467/0x530
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff810ceb58>] ? __do_fault+0x138/0x4e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810b483b>] ? unlock_page+0x2b/0x40
 [<ffffffff810cede1>] ? __do_fault+0x3c1/0x4e0
 [<ffffffff810d0e0a>] ? handle_mm_fault+0x17a/0x830
 [<ffffffff8105e3a3>] ? up_read+0x23/0x40
 [<ffffffff81966ea0>] ? do_page_fault+0x190/0x4e0
 [<ffffffff810e5a56>] ? kfree_debugcheck+0x16/0x40
 [<ffffffff810e603d>] ? kmem_cache_free+0xcd/0x190
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8196406b>] ? retint_swapgs+0xe/0x13
 [<ffffffff810fd51c>] sys_poll+0x7c/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sshd          S 0000000000000000  5184  3650      1 0x00000000
 ffff88001e1a38f8 0000000000000046 ffff88001e1a3fd8 ffff88001e1a3fd8
 ffff88001e1a2000 ffff88001e1a2000 ffff88001e1a2000 ffff88001ebfe3a0
 ffff88001e1a3fd8 ffff88001ebfe3a8 ffff88001e1a2000 ffff88001e1a3fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81963a12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810fe975>] ? __d_free+0x45/0x60
 [<ffffffff810fe9e8>] ? d_free+0x58/0x60
 [<ffffffff810feb4c>] ? d_kill+0x4c/0x60
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
ntpd          S 0000000000000000  5056  3658      1 0x00000000
 ffff88001eb5d8f8 0000000000000046 ffff88001eb5dfd8 ffff88001eb5dfd8
 ffff88001eb5c000 ffff88001eb5c000 ffff88001eb5c000 ffff88001da043e0
 ffff88001eb5dfd8 ffff88001da043e8 ffff88001eb5c000 ffff88001eb5dfd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81838a7a>] ? datagram_poll+0xea/0x100
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81838938>] ? verify_iovec+0x88/0xe0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sendmail      S 0000000000000000  5184  3674      1 0x00000000
 ffff88001eb278f8 0000000000000046 ffff88001eb27fd8 ffff88001eb27fd8
 ffff88001eb26000 ffff88001eb26000 ffff88001eb26000 ffff88001eab63a0
 ffff88001eb27fd8 ffff88001eab63a8 ffff88001eb26000 ffff88001eb27fd8
Call Trace:
 [<ffffffff81961dcf>] schedule_hrtimeout_range_clock+0xaf/0x140
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8110abe1>] ? seq_open+0x91/0x140
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff81063776>] ? ktime_get_ts+0xa6/0xe0
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sendmail      S 0000000000000000  5296  3682      1 0x00000000
 ffff88001eab9f68 0000000000000046 ffff88001eab9fd8 ffff88001eab9fd8
 ffff88001eab8000 ffff88001eab8000 ffff88001eab8000 ffff88001ed983a0
 ffff88001eab9fd8 ffff88001ed983a8 ffff88001eab8000 ffff88001eab9fd8
Call Trace:
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff8104a22d>] sys_pause+0x1d/0x30
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
crond         S 0000000000000000  5296  3691      1 0x00000000
 ffff88001d93fe78 0000000000000046 ffff88001d93ffd8 ffff88001d93ffd8
 ffff88001d93e000 ffff88001d93e000 ffff88001d93e000 ffff88001f7aa3e0
 ffff88001d93ffd8 ffff88001f7aa3e8 ffff88001d93e000 ffff88001d93ffd8
Call Trace:
 [<ffffffff81961c5c>] do_nanosleep+0x8c/0xc0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff8105dfde>] hrtimer_nanosleep+0xae/0x150
 [<ffffffff8105d310>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff8105e104>] sys_nanosleep+0x84/0x90
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5040  3704      1 0x00000000
 ffff88001e2a1ce8 0000000000000046 ffff88001e2a1fd8 ffff88001e2a1fd8
 ffff88001e2a0000 ffff88001e2a0000 ffff88001e2a0000 ffff88001e14e3a0
 ffff88001e2a1fd8 ffff88001e14e3a8 ffff88001e2a0000 ffff88001e2a1fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  4976  3706      1 0x00000000
 ffff88001e891ce8 0000000000000046 ffff88001e891fd8 ffff88001e891fd8
 ffff88001e890000 ffff88001e890000 ffff88001e890000 ffff88001ea2c3e0
 ffff88001e891fd8 ffff88001ea2c3e8 ffff88001e890000 ffff88001e891fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3708      1 0x00000000
 ffff88001eba7ce8 0000000000000046 ffff88001eba7fd8 ffff88001eba7fd8
 ffff88001eba6000 ffff88001eba6000 ffff88001eba6000 ffff88001eaac3a0
 ffff88001eba7fd8 ffff88001eaac3a8 ffff88001eba6000 ffff88001eba7fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3710      1 0x00000000
 ffff88001e8abce8 0000000000000046 ffff88001e8abfd8 ffff88001e8abfd8
 ffff88001e8aa000 ffff88001e8aa000 ffff88001e8aa000 ffff88001ecf83e0
 ffff88001e8abfd8 ffff88001ecf83e8 ffff88001e8aa000 ffff88001e8abfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3712      1 0x00000000
 ffff88001e8adce8 0000000000000046 ffff88001e8adfd8 ffff88001e8adfd8
 ffff88001e8ac000 ffff88001e8ac000 ffff88001e8ac000 ffff88001ebd63e0
 ffff88001e8adfd8 ffff88001ebd63e8 ffff88001e8ac000 ffff88001e8adfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mingetty      S 0000000000000000  5392  3714      1 0x00000000
 ffff88001ea85ce8 0000000000000046 ffff88001ea85fd8 ffff88001ea85fd8
 ffff88001ea84000 ffff88001ea84000 ffff88001ea84000 ffff88001eb443a0
 ffff88001ea85fd8 ffff88001eb443a8 ffff88001ea84000 ffff88001ea85fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff815673a9>] n_tty_read+0x589/0x950
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff81561be6>] tty_read+0xa6/0xf0
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sshd          S 0000000000000000  5152  3742   3650 0x00000000
 ffff88001e9818f8 0000000000000046 ffff88001e981fd8 ffff88001e981fd8
 ffff88001e980000 ffff88001e980000 ffff88001e980000 ffff88001e13c3e0
 ffff88001e981fd8 ffff88001e13c3e8 ffff88001e980000 ffff88001e981fd8
Call Trace:
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff81059ca6>] ? add_wait_queue+0x46/0x60
 [<ffffffff810fe474>] ? __pollwait+0x74/0xf0
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8196395e>] ? _raw_spin_unlock_bh+0x3e/0x50
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8105e8c1>] ? __srcu_read_unlock+0x41/0x70
 [<ffffffff81121e39>] ? fsnotify+0x1e9/0x340
 [<ffffffff81121ccd>] ? fsnotify+0x7d/0x340
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
bash          S 0000000000000000  5216  3750   3742 0x00000000
 ffff88001f6afe78 0000000000000046 ffff88001f6affd8 ffff88001f6affd8
 ffff88001f6ae000 ffff88001f6ae000 ffff88001f6ae000 ffff88001e8b43a0
 ffff88001f6affd8 ffff88001e8b43a8 ffff88001f6ae000 ffff88001f6affd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8103c916>] do_wait+0x1c6/0x280
 [<ffffffff8103ca72>] sys_wait4+0xa2/0x100
 [<ffffffff8103adc0>] ? child_wait_callback+0x0/0x70
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
lease_tests   S 0000000000000000  5968  4309      1 0x00000000
 ffff88001e26bcd8 0000000000000046 ffff88001e26bfd8 ffff88001e26bfd8
 ffff88001e26a000 ffff88001e26a000 ffff88001e26a000 ffff88001e8d63a0
 ffff88001e26bfd8 ffff88001e8d63a8 ffff88001e26a000 ffff88001e26bfd8
Call Trace:
 [<ffffffff810f477a>] pipe_wait+0x5a/0x80
 [<ffffffff81059940>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810f5235>] pipe_read+0x3b5/0x4a0
 [<ffffffff810eaf1a>] do_sync_read+0xda/0x120
 [<ffffffff810ff4aa>] ? dput+0xaa/0x160
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff814961ca>] ? security_file_permission+0x6a/0x80
 [<ffffffff810ebb94>] vfs_read+0xc4/0x170
 [<ffffffff810ebd31>] sys_read+0x51/0x80
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mount         S 0000000000000000  5056  4313   3750 0x00000000
 ffff88001edf1e78 0000000000000046 ffff88001edf1fd8 ffff88001edf1fd8
 ffff88001edf0000 ffff88001edf0000 ffff88001edf0000 ffff88001ed583a0
 ffff88001edf1fd8 ffff88001ed583a8 ffff88001edf0000 ffff88001edf1fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8103c916>] do_wait+0x1c6/0x280
 [<ffffffff8103ca72>] sys_wait4+0xa2/0x100
 [<ffffffff8103adc0>] ? child_wait_callback+0x0/0x70
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
 ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
 ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
 ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc80>] wait_for_common+0x120/0x190
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
 [<ffffffff810595fa>] kthread_stop+0x4a/0x150
 [<ffffffff81061a60>] ? thaw_process+0x70/0x80
 [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edfb9>] deactivate_super+0x49/0x70
 [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
 [<ffffffff811084ef>] release_mounts+0x9f/0xc0
 [<ffffffff81108575>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810ede92>] do_kern_mount+0x52/0x130
 [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
 [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811094b8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
nfsv4.0-svc   S 0000000000000000  6592  4315      2 0x00000000
 ffff88001f7a1d50 0000000000000046 ffff88001f7a1fd8 ffff88001f7a1fd8
 ffff88001f7a0000 ffff88001f7a0000 ffff88001f7a0000 ffff88001ed5a3e0
 ffff88001f7a1fd8 ffff88001ed5a3e8 ffff88001f7a0000 ffff88001f7a1fd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8103fba3>] ? local_bh_enable_ip+0x83/0x110
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff81910d7f>] ? svc_recv+0x5bf/0x8a0
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8103fba3>] ? local_bh_enable_ip+0x83/0x110
 [<ffffffff81910d88>] svc_recv+0x5c8/0x8a0
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff812556b0>] ? nfs4_callback_svc+0x0/0xb0
 [<ffffffff812556f2>] nfs4_callback_svc+0x42/0xb0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
flush-0:19    S 0000000000000000  6280  4316      2 0x00000000
 ffff88001d8f5e20 0000000000000046 ffff88001d8f5fd8 ffff88001d8f5fd8
 ffff88001d8f4000 ffff88001d8f4000 ffff88001d8f4000 ffff88001e8c43e0
 ffff88001d8f5fd8 ffff88001e8c43e8 ffff88001d8f4000 ffff88001d8f5fd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff811109b9>] bdi_writeback_thread+0x159/0x2f0
 [<ffffffff81110860>] ? bdi_writeback_thread+0x0/0x2f0
 [<ffffffff810594a6>] kthread+0x96/0xa0
 [<ffffffff810030f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff810325ab>] ? finish_task_switch.clone.3+0x7b/0xd0
 [<ffffffff8196407e>] ? restore_args+0x0/0x30
 [<ffffffff81059410>] ? kthread+0x0/0xa0
 [<ffffffff810030f0>] ? kernel_thread_helper+0x0/0x10
anacron       S 0000000000000000  5248  4381      1 0x00000000
 ffff88001e46ff38 0000000000000046 ffff88001e46ffd8 ffff88001e46ffd8
 ffff88001e46e000 ffff88001e46e000 ffff88001e46e000 ffff88001f9483a0
 ffff88001e46ffd8 ffff88001f9483a8 ffff88001e46e000 ffff88001e46ffd8
Call Trace:
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8104a4f8>] sys_rt_sigsuspend+0xb8/0xe0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
sshd          S 0000000000000000  5152  4433   3650 0x00000000
 ffff88001e1a98f8 0000000000000046 ffff88001e1a9fd8 ffff88001e1a9fd8
 ffff88001e1a8000 ffff88001e1a8000 ffff88001e1a8000 ffff88001d8c63a0
 ffff88001e1a9fd8 ffff88001d8c63a8 ffff88001e1a8000 ffff88001e1a9fd8
Call Trace:
 [<ffffffff81961e3d>] schedule_hrtimeout_range_clock+0x11d/0x140
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff8106a64d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff81961e73>] schedule_hrtimeout_range+0x13/0x20
 [<ffffffff810fccf8>] poll_schedule_timeout+0x38/0x60
 [<ffffffff810fdbfd>] do_select+0x5ed/0x720
 [<ffffffff810fd610>] ? do_select+0x0/0x720
 [<ffffffff810fe400>] ? __pollwait+0x0/0xf0
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff810fe4f0>] ? pollwake+0x0/0x60
 [<ffffffff8106d9c8>] ? lock_release_non_nested+0xa8/0x330
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810ce39a>] ? might_fault+0x7a/0xd0
 [<ffffffff810fdf97>] core_sys_select+0x267/0x3b0
 [<ffffffff810fdd78>] ? core_sys_select+0x48/0x3b0
 [<ffffffff8102fb52>] ? __wake_up+0x32/0x70
 [<ffffffff81030611>] ? get_parent_ip+0x11/0x50
 [<ffffffff8196728d>] ? sub_preempt_count+0x9d/0xd0
 [<ffffffff8105e8c1>] ? __srcu_read_unlock+0x41/0x70
 [<ffffffff81121e39>] ? fsnotify+0x1e9/0x340
 [<ffffffff81121ccd>] ? fsnotify+0x7d/0x340
 [<ffffffff8156ae1e>] ? tty_ldisc_deref+0xe/0x10
 [<ffffffff810fe337>] sys_select+0x47/0x110
 [<ffffffff81962e69>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
bash          R  running task     5008  4441   4433 0x00000000
 ffff88001e8cfad8 ffffffff81039743 ffffffff8208d355 ffffffff819671fe
 ffff88001e8cfb08 ffffffff8208d355 ffff88001e8cfb78 ffffffff81039cdc
 ffff88001e8cfb88 0000000000000092 ffff88001e8cfbe8 0000000000000092
Call Trace:
 [<ffffffff81039743>] ? release_console_sem+0x1f3/0x230
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff81039cdc>] ? vprintk+0x26c/0x4d0
 [<ffffffff814eec6f>] ? string+0x4f/0xf0
 [<ffffffff814eec6f>] ? string+0x4f/0xf0
 [<ffffffff8195f915>] ? printk+0x68/0x6a
 [<ffffffff8195f915>] ? printk+0x68/0x6a
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff81075410>] ? is_module_text_address+0x30/0x60
 [<ffffffff81056878>] ? __kernel_text_address+0x58/0x80
 [<ffffffff81005b15>] ? print_context_stack+0x85/0x140
 [<ffffffff81004a4c>] ? dump_trace+0x18c/0x390
 [<ffffffff8100589c>] show_trace_log_lvl+0x5c/0x80
 [<ffffffff81004511>] show_stack_log_lvl+0xc1/0x1b0
 [<ffffffff8100583c>] show_stack+0x1c/0x20
 [<ffffffff810306fc>] sched_show_task+0xac/0x110
 [<ffffffff81035d8b>] show_state_filter+0x7b/0xd0
 [<ffffffff8157c2d0>] sysrq_handle_showstate+0x10/0x20
 [<ffffffff8157c731>] __handle_sysrq+0x131/0x1a0
 [<ffffffff8157c7a0>] ? write_sysrq_trigger+0x0/0x50
 [<ffffffff8157c7ee>] write_sysrq_trigger+0x4e/0x50
 [<ffffffff8113dc75>] proc_reg_write+0x85/0xc0
 [<ffffffff810eb8f7>] vfs_write+0xc7/0x170
 [<ffffffff810eba91>] sys_write+0x51/0x90
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
Sched Debug Version: v0.09, 2.6.36-rc2-00238-g0e8f983 #11
now at 828420.724224 msecs
  .jiffies                                 : 4295099271
  .sysctl_sched_latency                    : 6.000000
  .sysctl_sched_min_granularity            : 2.000000
  .sysctl_sched_wakeup_granularity         : 1.000000
  .sysctl_sched_child_runs_first           : 0
  .sysctl_sched_features                   : 15471
  .sysctl_sched_tunable_scaling            : 1 (logaritmic)

cpu#0, 2394.006 MHz
  .nr_running                    : 2
  .load                          : 2048
  .nr_switches                   : 51559
  .nr_load_updates               : 206975
  .nr_uninterruptible            : 1
  .next_balance                  : 0.000000
  .curr->pid                     : 4441
  .clock                         : 828415.137767
  .cpu_load[0]                   : 0
  .cpu_load[1]                   : 0
  .cpu_load[2]                   : 0
  .cpu_load[3]                   : 0
  .cpu_load[4]                   : 0
  .yld_count                     : 1741
  .sched_switch                  : 0
  .sched_count                   : 142628
  .sched_goidle                  : 93415
  .ttwu_count                    : 0
  .ttwu_local                    : 0
  .bkl_count                     : 192

cfs_rq[0]:
  .exec_clock                    : 17547.448581
  .MIN_vruntime                  : 16937.936514
  .min_vruntime                  : 16937.933504
  .max_vruntime                  : 16937.936514
  .spread                        : 0.000000
  .spread0                       : 0.000000
  .nr_running                    : 2
  .load                          : 2048
  .nr_spread_over                : 12

rt_rq[0]:
  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
            task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
----------------------------------------------------------------------------------------------------------
R           bash  4441     16934.933504        33   120     16934.933504        22.132725      3682.649290


Showing all locks held in the system:
1 lock held by mingetty/3704:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3706:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3708:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3710:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3712:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mingetty/3714:
 #0:  (&tty->atomic_read_lock){+.+...}, at: [<ffffffff815672fe>] n_tty_read+0x4de/0x950
1 lock held by mount.nfs4/4314:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70
2 locks held by bash/4441:
 #0:  (sysrq_key_table_lock){......}, at: [<ffffffff8157c635>] __handle_sysrq+0x35/0x1a0
 #1:  (tasklist_lock){.+.+..}, at: [<ffffffff8106988e>] debug_show_all_locks+0x3e/0x1c0

=============================================

INFO: task mount.nfs4:4314 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
 ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
 ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
 ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc80>] wait_for_common+0x120/0x190
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
 [<ffffffff810595fa>] kthread_stop+0x4a/0x150
 [<ffffffff81061a60>] ? thaw_process+0x70/0x80
 [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edfb9>] deactivate_super+0x49/0x70
 [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
 [<ffffffff811084ef>] release_mounts+0x9f/0xc0
 [<ffffffff81108575>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810ede92>] do_kern_mount+0x52/0x130
 [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
 [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811094b8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
1 lock held by mount.nfs4/4314:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70

RE: hang in writeback code on nfsv4 mount

[Artem.Bityutskiy]

> I need to look more. But so far I do not really understand what could make kthread_stop() wait
> forever. I looked into the code, and thought may be barriers are missing there, but I is unlikely
> this generic type of code would have bugs.

> I cannot see what happens from the code and need to experiment with printks and other debugging
> things. Will do as soon as I have a possibility.

Oh, and pardon for top posting, writing from web interface to outlook. Forgot to strip the mail.

Artem.

Re: hang in writeback code on nfsv4 mount

[J. Bruce Fields]

On Fri, Aug 27, 2010 at 06:17:36PM +0200, Artem.Bityutskiy@nokia.com wrote:
> I need to look more. But so far I do not really understand what could make kthread_stop() wait
> forever. I looked into the code, and thought may be barriers are missing there, but I is unlikely
> this generic type of code would have bugs.

Hm, is the way you go to sleep in the writeback thread safe?  Surely
there's a race like:

while(!kthread_should_stop()) {

				kthread_stop():
					kthread->should_stop = 1;
					wake_up_process(k);
					wait_for_completion(&kthread->exited);

	/* oops, lose the wake_up here: */
	set_current_state(TASK_INTERRUPTIBLE);
	...
	/* sleep forever: */
	schedule();

Maybe the following?

--b.

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 7d9d06b..ea76550 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -808,7 +808,7 @@ int bdi_writeback_thread(void *data)
 			wb->last_active = jiffies;
 
 		set_current_state(TASK_INTERRUPTIBLE);
-		if (!list_empty(&bdi->work_list)) {
+		if (!list_empty(&bdi->work_list) || kthread_should_stop()) {
 			__set_current_state(TASK_RUNNING);
 			continue;
 		}

[PATCH] Fix lost wake-up shutting down writeback thread

[J. Bruce Fields]

From: J. Bruce Fields <bfields@redhat.com>

Setting the task state here may cause us to miss the wake up from
kthread_stop(), so we need to recheck kthread_should_stop() or risk
sleeping forever in the following schedule().

Symptom was an indefinite hang on an NFSv4 mount.  (NFSv4 may create
multiple mounts in a temporary namespace while traversing the mount
path, and since the temporary namespace is immediately destroyed, it may
end up destroying a mount very soon after it was created, possibly
making this race more likely.)

INFO: task mount.nfs4:4314 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
 ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
 ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
 ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
Call Trace:
 [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
 [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
 [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
 [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
 [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
 [<ffffffff8195fc80>] wait_for_common+0x120/0x190
 [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
 [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
 [<ffffffff810595fa>] kthread_stop+0x4a/0x150
 [<ffffffff81061a60>] ? thaw_process+0x70/0x80
 [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
 [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
 [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
 [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
 [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
 [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
 [<ffffffff810edfb9>] deactivate_super+0x49/0x70
 [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
 [<ffffffff811084ef>] release_mounts+0x9f/0xc0
 [<ffffffff81108575>] put_mnt_ns+0x65/0x80
 [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
 [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
 [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
 [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
 [<ffffffff810ede92>] do_kern_mount+0x52/0x130
 [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
 [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
 [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
 [<ffffffff811094b8>] sys_mount+0x98/0xf0
 [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
1 lock held by mount.nfs4/4314:
 #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70

Signed-off-by: J. Bruce Fields <bfields@redhat.com>
---
 fs/fs-writeback.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

On Fri, Aug 27, 2010 at 05:06:26PM -0400, J. Bruce Fields wrote:
> Maybe the following?

I confirmed that my hang does go away after doing this.

Please apply if you think this makes sense....

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 7d9d06b..81e086d 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -808,7 +808,7 @@ int bdi_writeback_thread(void *data)
 			wb->last_active = jiffies;
 
 		set_current_state(TASK_INTERRUPTIBLE);
-		if (!list_empty(&bdi->work_list)) {
+		if (!list_empty(&bdi->work_list) || kthread_should_stop()) {
 			__set_current_state(TASK_RUNNING);
 			continue;
 		}
-- 
1.7.0.4

RE: [PATCH] Fix lost wake-up shutting down writeback thread

[Artem.Bityutskiy]

> From: J. Bruce Fields <bfields@redhat.com>
> 
> Setting the task state here may cause us to miss the wake up from
> kthread_stop(), so we need to recheck kthread_should_stop() or risk
> sleeping forever in the following schedule().

Oh yeah, if my sleepy brain does not lie me, that's it! Thanks a lot and sorry
for troubles. I believe the other patch I sent is also a valid bug fix, but fixes a
different issue.

Acked-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

Artem.

Re: [PATCH] Fix lost wake-up shutting down writeback thread

[Jens Axboe]

On 08/27/2010 11:28 PM, J. Bruce Fields wrote:
> From: J. Bruce Fields <bfields@redhat.com>
> 
> Setting the task state here may cause us to miss the wake up from
> kthread_stop(), so we need to recheck kthread_should_stop() or risk
> sleeping forever in the following schedule().
> 
> Symptom was an indefinite hang on an NFSv4 mount.  (NFSv4 may create
> multiple mounts in a temporary namespace while traversing the mount
> path, and since the temporary namespace is immediately destroyed, it may
> end up destroying a mount very soon after it was created, possibly
> making this race more likely.)
> 
> INFO: task mount.nfs4:4314 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> mount.nfs4    D 0000000000000000  2880  4314   4313 0x00000000
>  ffff88001ed6da28 0000000000000046 ffff88001ed6dfd8 ffff88001ed6dfd8
>  ffff88001ed6c000 ffff88001ed6c000 ffff88001ed6c000 ffff88001e5003a0
>  ffff88001ed6dfd8 ffff88001e5003a8 ffff88001ed6c000 ffff88001ed6dfd8
> Call Trace:
>  [<ffffffff8196090d>] schedule_timeout+0x1cd/0x2e0
>  [<ffffffff8106a31c>] ? mark_held_locks+0x6c/0xa0
>  [<ffffffff819639a0>] ? _raw_spin_unlock_irq+0x30/0x60
>  [<ffffffff8106a5fd>] ? trace_hardirqs_on_caller+0x14d/0x190
>  [<ffffffff819671fe>] ? sub_preempt_count+0xe/0xd0
>  [<ffffffff8195fc80>] wait_for_common+0x120/0x190
>  [<ffffffff81033c70>] ? default_wake_function+0x0/0x20
>  [<ffffffff8195fdcd>] wait_for_completion+0x1d/0x20
>  [<ffffffff810595fa>] kthread_stop+0x4a/0x150
>  [<ffffffff81061a60>] ? thaw_process+0x70/0x80
>  [<ffffffff810cc68a>] bdi_unregister+0x10a/0x1a0
>  [<ffffffff81229dc9>] nfs_put_super+0x19/0x20
>  [<ffffffff810ee8c4>] generic_shutdown_super+0x54/0xe0
>  [<ffffffff810ee9b6>] kill_anon_super+0x16/0x60
>  [<ffffffff8122d3b9>] nfs4_kill_super+0x39/0x90
>  [<ffffffff810eda45>] deactivate_locked_super+0x45/0x60
>  [<ffffffff810edfb9>] deactivate_super+0x49/0x70
>  [<ffffffff81108294>] mntput_no_expire+0x84/0xe0
>  [<ffffffff811084ef>] release_mounts+0x9f/0xc0
>  [<ffffffff81108575>] put_mnt_ns+0x65/0x80
>  [<ffffffff8122cc56>] nfs_follow_remote_path+0x1e6/0x420
>  [<ffffffff8122cfbf>] nfs4_try_mount+0x6f/0xd0
>  [<ffffffff8122d0c2>] nfs4_get_sb+0xa2/0x360
>  [<ffffffff810edcb8>] vfs_kern_mount+0x88/0x1f0
>  [<ffffffff810ede92>] do_kern_mount+0x52/0x130
>  [<ffffffff81963d9a>] ? _lock_kernel+0x6a/0x170
>  [<ffffffff81108e9e>] do_mount+0x26e/0x7f0
>  [<ffffffff81106b3a>] ? copy_mount_options+0xea/0x190
>  [<ffffffff811094b8>] sys_mount+0x98/0xf0
>  [<ffffffff810024d8>] system_call_fastpath+0x16/0x1b
> 1 lock held by mount.nfs4/4314:
>  #0:  (&type->s_umount_key#24){+.+...}, at: [<ffffffff810edfb1>] deactivate_super+0x41/0x70
> 
> Signed-off-by: J. Bruce Fields <bfields@redhat.com>
> ---
>  fs/fs-writeback.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> On Fri, Aug 27, 2010 at 05:06:26PM -0400, J. Bruce Fields wrote:
>> Maybe the following?
> 
> I confirmed that my hang does go away after doing this.
> 
> Please apply if you think this makes sense....
> 
> diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
> index 7d9d06b..81e086d 100644
> --- a/fs/fs-writeback.c
> +++ b/fs/fs-writeback.c
> @@ -808,7 +808,7 @@ int bdi_writeback_thread(void *data)
>  			wb->last_active = jiffies;
>  
>  		set_current_state(TASK_INTERRUPTIBLE);
> -		if (!list_empty(&bdi->work_list)) {
> +		if (!list_empty(&bdi->work_list) || kthread_should_stop()) {
>  			__set_current_state(TASK_RUNNING);
>  			continue;
>  		}

Thanks Bruce, I've applied this as well.

-- 
Jens Axboe

RE: [PATCH] Fix lost wake-up shutting down writeback thread

[Artem.Bityutskiy]

> Thanks Bruce, I've applied this as well.

Thanks for picking up! But you could also add my Acked tag, just for the record.

Artem.

Re: [PATCH] Fix lost wake-up shutting down writeback thread

[Jens Axboe]

On 2010-08-29 14:21, Artem.Bityutskiy@nokia.com wrote:
>> Thanks Bruce, I've applied this as well.
> 
> Thanks for picking up! But you could also add my Acked tag, just for the record.

Thanks, added.

-- 
Jens Axboe