From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759202Ab3BWQmn (ORCPT ); Sat, 23 Feb 2013 11:42:43 -0500 Received: from mail-qc0-f182.google.com ([209.85.216.182]:62188 "EHLO mail-qc0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758000Ab3BWQml (ORCPT ); Sat, 23 Feb 2013 11:42:41 -0500 Message-ID: <5128F16E.30804@gmail.com> Date: Sat, 23 Feb 2013 11:42:22 -0500 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130113 Thunderbird/17.0.2 MIME-Version: 1.0 To: Peter Hurley CC: Greg Kroah-Hartman , Jiri Slaby , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-serial@vger.kernel.org, Ilya Zykov , Dave Jones , Michael Ellerman , Shawn Guo Subject: Re: [PATCH v4 00/32] ldisc patchset References: <1360095638-6624-1-git-send-email-peter@hurleysoftware.com> <1361390599-15195-1-git-send-email-peter@hurleysoftware.com> <51261E30.9040907@gmail.com> <1361453910.10685.7.camel@thor.lan> <1361558250.14402.6.camel@thor.lan> <5128DF40.7030003@gmail.com> In-Reply-To: <5128DF40.7030003@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/23/2013 10:24 AM, Sasha Levin wrote: > On 02/22/2013 01:37 PM, Peter Hurley wrote: >> On Thu, 2013-02-21 at 08:38 -0500, Peter Hurley wrote: >>> On Thu, 2013-02-21 at 08:16 -0500, Sasha Levin wrote: >>>> On 02/20/2013 03:02 PM, Peter Hurley wrote: >>>>> Sasha and Dave, my trinity testbeds die in other areas right now; >>>>> I would really appreciate if you would please re-test this series. >>>> >>>> Hi Peter, >>>> >>>> I saw this twice in overnight fuzzing: >>>> >>>> [ 1473.912280] ================================= >>>> [ 1473.913180] [ BUG: bad contention detected! ] >>>> [ 1473.914071] 3.8.0-next-20130220-sasha-00038-g1ad55df-dirty #8 Tainted: G W >>>> [ 1473.915684] --------------------------------- >>>> [ 1473.916549] kworker/1:1/361 is trying to contend lock (&tty->ldisc_sem) at: >>>> [ 1473.918031] [] tty_ldisc_ref+0x1f/0x60 >>>> [ 1473.919060] but there are no locks held! >>> >>> Ahh, of course. That explains why the rwsem trylock doesn't track lock >>> stats -- because by the time lock_contended() is called, up_write() >>> could have just called lockdep_release(), so that it appears as if the >>> lock has been released when in fact it has not but is about to. >>> >>> I'll just remove the lock contention test from the trylocks. >> >> Hi Sasha, >> >> Sorry for the delay. I was actually looking into if I could tickle >> lockdep into just recording the lock contention without testing, but >> unfortunately, changes to where lockdep stores the contention now >> requires the lockdep state to have an existing owner. >> >> So here's the trivial patch: > > Hi Peter, > > After more fuzzing, I'm seeing this sort of hangs (which are new): > > [ 2644.723879] INFO: task trinity:17893 blocked for more than 120 seconds. > [ 2644.727112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 2644.731916] trinity D ffff8800a9c904a8 5192 17893 8043 0x00000000 > [ 2644.733517] ffff88006efb3a78 0000000000000002 ffff8800aa0c3b10 ffff8800bb3d7180 > [ 2644.739350] ffff880019103000 ffff880097a78000 ffff88006efb3a78 00000000001d7180 > [ 2644.741459] ffff880097a78000 ffff88006efb3fd8 00000000001d7180 00000000001d7180 > [ 2644.746590] Call Trace: > [ 2644.747177] [] __schedule+0x2e9/0x3b0 > [ 2644.748294] [] schedule+0x55/0x60 > [ 2644.752382] [] schedule_preempt_disabled+0x13/0x20 > [ 2644.753737] [] __mutex_lock_common+0x34d/0x560 > [ 2644.759037] [] ? ptmx_open+0x83/0x190 > [ 2644.760590] [] ? __mutex_unlock_slowpath+0x185/0x1e0 > [ 2644.762064] [] ? ptmx_open+0x83/0x190 > [ 2644.768967] [] mutex_lock_nested+0x3f/0x50 > [ 2644.770314] [] ptmx_open+0x83/0x190 > [ 2644.771413] [] chrdev_open+0x11e/0x190 > [ 2644.780456] [] ? cdev_put+0x30/0x30 > [ 2644.781421] [] do_dentry_open+0x1f9/0x310 > [ 2644.785550] [] finish_open+0x4c/0x70 > [ 2644.786724] [] do_last+0x61b/0x810 > [ 2644.787676] [] path_openat+0xb9/0x4d0 > [ 2644.791868] [] ? __alloc_fd+0x1e8/0x200 > [ 2644.792817] [] ? lock_release_nested+0xb4/0xf0 > [ 2644.794010] [] ? __lock_release+0xe1/0x100 > [ 2644.797401] [] do_filp_open+0x3d/0xa0 > [ 2644.798467] [] ? __alloc_fd+0x1e8/0x200 > [ 2644.799577] [] do_sys_open+0x12b/0x1d0 > [ 2644.804667] [] sys_open+0x1c/0x20 > [ 2644.805542] [] tracesys+0xe1/0xe6 > [ 2644.822807] 1 lock held by trinity/17893: > [ 2644.823685] #0: (tty_mutex){+.+.+.}, at: [] ptmx_open+0x83/0x190 > > The mutex is 'tty_mutex' at drivers/tty/pty.c:701 . > > I didn't grab sysrq-t this time since it was an overnight run, but I'll > try to grab one when it happens again. Hi Peter, I've also got this one for the first time: [ 484.450139] INFO: task init:1 blocked for more than 120 seconds. [ 484.451130] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.452013] init D 00000000001d7180 2688 1 0 0x00000002 [ 484.452883] ffff8800b9acfba8 0000000000000002 00000000001d7180 ffff8800b9b10048 [ 484.453816] ffff8800b94cb000 ffff8800b9b10000 00000000001d7180 00000000001d7180 [ 484.454529] ffff8800b9b10000 ffff8800b9acffd8 00000000001d7180 00000000001d7180 [ 484.455232] Call Trace: [ 484.455515] [] __schedule+0x2e9/0x3b0 [ 484.456081] [] schedule+0x55/0x60 [ 484.456636] [] schedule_timeout+0x3a/0x370 [ 484.457263] [] ? mark_held_locks+0xf9/0x130 [ 484.457866] [] ? down_failed+0x108/0x200 [ 484.458539] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 484.459223] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 484.459822] [] down_failed+0x131/0x200 [ 484.460512] [] ? tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 484.461376] [] ldsem_down_write+0xd3/0x113 [ 484.461895] [] ? tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 484.462824] [] ? trace_hardirqs_on+0xd/0x10 [ 484.463504] [] tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 484.464258] [] tty_ldisc_hangup+0xd0/0x220 [ 484.464825] [] __tty_hangup+0x137/0x4f0 [ 484.465684] [] disassociate_ctty+0x6c/0x230 [ 484.466607] [] do_exit+0x41c/0x590 [ 484.467419] [] ? syscall_trace_enter+0x24/0x2e0 [ 484.468413] [] do_group_exit+0x8a/0xc0 [ 484.469281] [] sys_exit_group+0x12/0x20 [ 484.470159] [] tracesys+0xe1/0xe6 [ 484.470951] 1 lock held by init/1: [ 484.471513] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 484.473290] INFO: task trinity:9732 blocked for more than 120 seconds. [ 484.474324] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.475540] trinity D 00000000001d7180 4992 9732 1 0x00000000 [ 484.476691] ffff8800915ddc28 0000000000000002 00000000001d7180 ffff8800aa09d208 [ 484.477933] ffff8800b94b3000 ffff880070188000 00000000001d7180 00000000001d7180 [ 484.479180] ffff880070188000 ffff8800915ddfd8 00000000001d7180 00000000001d7180 [ 484.480387] Call Trace: [ 484.480744] [] __schedule+0x2e9/0x3b0 [ 484.481487] [] schedule+0x55/0x60 [ 484.482258] [] schedule_timeout+0x3a/0x370 [ 484.483164] [] ? mark_held_locks+0xf9/0x130 [ 484.484059] [] ? down_failed+0x108/0x200 [ 484.484915] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 484.485866] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 484.486890] [] down_failed+0x131/0x200 [ 484.487714] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.488628] [] ldsem_down_read+0x8c/0x100 [ 484.489522] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.490463] [] ? __lock_release+0xe1/0x100 [ 484.491349] [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.491967] [] tty_write+0x98/0xf0 [ 484.492442] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 484.493000] [] redirected_tty_write+0x95/0xc0 [ 484.493826] [] vfs_write+0xb8/0x180 [ 484.494643] [] sys_write+0x50/0xa0 [ 484.495460] [] tracesys+0xe1/0xe6 [ 484.496398] 1 lock held by trinity/9732: [ 484.497032] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.498606] INFO: task trinity:10089 blocked for more than 120 seconds. [ 484.499640] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.500885] trinity D 00000000001d7180 5224 10089 1 0x00000000 [ 484.502053] ffff88006bb2bc28 0000000000000002 00000000001d7180 ffff8800aa09d208 [ 484.503256] ffff8800b94b3000 ffff880094303000 00000000001d7180 00000000001d7180 [ 484.504450] ffff880094303000 ffff88006bb2bfd8 00000000001d7180 00000000001d7180 [ 484.505657] Call Trace: [ 484.506052] [] __schedule+0x2e9/0x3b0 [ 484.506871] [] schedule+0x55/0x60 [ 484.507650] [] schedule_timeout+0x3a/0x370 [ 484.508542] [] ? mark_held_locks+0xf9/0x130 [ 484.509446] [] ? down_failed+0x108/0x200 [ 484.510312] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 484.511285] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 484.512276] [] down_failed+0x131/0x200 [ 484.512813] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.513459] [] ldsem_down_read+0x8c/0x100 [ 484.514345] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.515263] [] ? __lock_release+0xe1/0x100 [ 484.516135] [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.517049] [] tty_write+0x98/0xf0 [ 484.517842] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 484.518780] [] redirected_tty_write+0x95/0xc0 [ 484.519730] [] vfs_write+0xb8/0x180 [ 484.520590] [] sys_write+0x50/0xa0 [ 484.521108] [] tracesys+0xe1/0xe6 [ 484.521976] 1 lock held by trinity/10089: [ 484.522619] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.524181] INFO: task trinity:10254 blocked for more than 120 seconds. [ 484.525200] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.526405] trinity D 00000000001d7180 5240 10254 1 0x00000000 [ 484.527559] ffff880096219c28 0000000000000002 00000000001d7180 ffff8800aa09cb78 [ 484.528799] ffff8800b94c3000 ffff8800961a8000 00000000001d7180 00000000001d7180 [ 484.530059] ffff8800961a8000 ffff880096219fd8 00000000001d7180 00000000001d7180 [ 484.531205] Call Trace: [ 484.531593] [] __schedule+0x2e9/0x3b0 [ 484.532185] [] schedule+0x55/0x60 [ 484.532964] [] schedule_timeout+0x3a/0x370 [ 484.533865] [] ? mark_held_locks+0xf9/0x130 [ 484.534784] [] ? down_failed+0x108/0x200 [ 484.535669] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 484.536660] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 484.537716] [] down_failed+0x131/0x200 [ 484.538568] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.539511] [] ldsem_down_read+0x8c/0x100 [ 484.540426] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.541321] [] ? __lock_release+0xe1/0x100 [ 484.542097] [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.542754] [] tty_write+0x98/0xf0 [ 484.543299] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 484.543930] [] redirected_tty_write+0x95/0xc0 [ 484.544529] [] vfs_write+0xb8/0x180 [ 484.545109] [] sys_write+0x50/0xa0 [ 484.545590] [] tracesys+0xe1/0xe6 [ 484.546280] 1 lock held by trinity/10254: [ 484.546841] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.547890] INFO: task trinity:10643 blocked for more than 120 seconds. [ 484.548516] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.549320] trinity D 00000000001d7180 5432 10643 1 0x00000000 [ 484.550051] ffff88006adafc28 0000000000000002 00000000001d7180 ffff8800aa09cec0 [ 484.551316] ffff8800b94c0000 ffff880090a6b000 00000000001d7180 00000000001d7180 [ 484.552233] ffff880090a6b000 ffff88006adaffd8 00000000001d7180 00000000001d7180 [ 484.553283] Call Trace: [ 484.553564] [] __schedule+0x2e9/0x3b0 [ 484.554125] [] schedule+0x55/0x60 [ 484.554746] [] schedule_timeout+0x3a/0x370 [ 484.555490] [] ? mark_held_locks+0xf9/0x130 [ 484.556289] [] ? down_failed+0x108/0x200 [ 484.556892] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 484.557576] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 484.558452] [] down_failed+0x131/0x200 [ 484.559045] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.559786] [] ldsem_down_read+0x8c/0x100 [ 484.560742] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.561913] [] ? __lock_release+0xe1/0x100 [ 484.562875] [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.563587] [] tty_write+0x98/0xf0 [ 484.564126] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 484.564938] [] redirected_tty_write+0x95/0xc0 [ 484.565569] [] vfs_write+0xb8/0x180 [ 484.566098] [] sys_write+0x50/0xa0 [ 484.566740] [] tracesys+0xe1/0xe6 [ 484.567436] 1 lock held by trinity/10643: [ 484.567932] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.569211] INFO: task trinity:11088 blocked for more than 120 seconds. [ 484.569980] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.571067] trinity D 00000000001d7180 5608 11088 1 0x00000000 [ 484.572132] ffff88006b197c28 0000000000000002 00000000001d7180 ffff8800aa09cb78 [ 484.573011] ffff8800b94c3000 ffff88009a633000 00000000001d7180 00000000001d7180 [ 484.573876] ffff88009a633000 ffff88006b197fd8 00000000001d7180 00000000001d7180 [ 484.574822] Call Trace: [ 484.575092] [] __schedule+0x2e9/0x3b0 [ 484.575811] [] schedule+0x55/0x60 [ 484.576413] [] schedule_timeout+0x3a/0x370 [ 484.577003] [] ? mark_held_locks+0xf9/0x130 [ 484.577732] [] ? down_failed+0x108/0x200 [ 484.578381] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 484.579070] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 484.579836] [] down_failed+0x131/0x200 [ 484.580600] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.581320] [] ldsem_down_read+0x8c/0x100 [ 484.581947] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 484.582536] [] ? __lock_release+0xe1/0x100 [ 484.583061] [] tty_ldisc_ref_wait+0x1f/0x50 [ 484.583741] [] tty_write+0x98/0xf0 [ 484.584215] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 484.584893] [] redirected_tty_write+0x95/0xc0 [ 484.585491] [] vfs_write+0xb8/0x180 [ 484.586000] [] sys_write+0x50/0xa0 [ 484.586543] [] tracesys+0xe1/0xe6 [ 484.587211] 1 lock held by trinity/11088: [ 484.587599] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 [ 604.580385] INFO: task init:1 blocked for more than 120 seconds. [ 604.581345] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 604.582364] init D 00000000001d7180 2688 1 0 0x00000002 [ 604.583115] ffff8800b9acfba8 0000000000000002 00000000001d7180 ffff8800b9b10048 [ 604.583949] ffff8800b94cb000 ffff8800b9b10000 00000000001d7180 00000000001d7180 [ 604.584652] ffff8800b9b10000 ffff8800b9acffd8 00000000001d7180 00000000001d7180 [ 604.585375] Call Trace: [ 604.585680] [] __schedule+0x2e9/0x3b0 [ 604.586213] [] schedule+0x55/0x60 [ 604.586744] [] schedule_timeout+0x3a/0x370 [ 604.587322] [] ? mark_held_locks+0xf9/0x130 [ 604.587850] [] ? down_failed+0x108/0x200 [ 604.588379] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 604.588991] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 604.589743] [] down_failed+0x131/0x200 [ 604.590598] [] ? tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 604.591651] [] ldsem_down_write+0xd3/0x113 [ 604.592467] [] ? tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 604.593494] [] ? trace_hardirqs_on+0xd/0x10 [ 604.594402] [] tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 604.595445] [] tty_ldisc_hangup+0xd0/0x220 [ 604.596324] [] __tty_hangup+0x137/0x4f0 [ 604.597159] [] disassociate_ctty+0x6c/0x230 [ 604.598047] [] do_exit+0x41c/0x590 [ 604.598872] [] ? syscall_trace_enter+0x24/0x2e0 [ 604.599827] [] do_group_exit+0x8a/0xc0 [ 604.600686] [] sys_exit_group+0x12/0x20 [ 604.601568] [] tracesys+0xe1/0xe6 [ 604.608562] 1 lock held by init/1: [ 604.609053] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_lock_pair_timeout+0xcd/0x120 [ 604.610836] INFO: task trinity:9732 blocked for more than 120 seconds. [ 604.611877] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 604.613063] trinity D 00000000001d7180 4992 9732 1 0x00000000 [ 604.614204] ffff8800915ddc28 0000000000000002 00000000001d7180 ffff8800aa09d208 [ 604.615410] ffff8800b94b3000 ffff880070188000 00000000001d7180 00000000001d7180 [ 604.616604] ffff880070188000 ffff8800915ddfd8 00000000001d7180 00000000001d7180 [ 604.617846] Call Trace: [ 604.618258] [] __schedule+0x2e9/0x3b0 [ 604.619076] [] schedule+0x55/0x60 [ 604.619844] [] schedule_timeout+0x3a/0x370 [ 604.620754] [] ? mark_held_locks+0xf9/0x130 [ 604.621682] [] ? down_failed+0x108/0x200 [ 604.622533] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 604.623475] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 604.624533] [] down_failed+0x131/0x200 [ 604.625372] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 604.626325] [] ldsem_down_read+0x8c/0x100 [ 604.627239] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 604.628193] [] ? __lock_release+0xe1/0x100 [ 604.629107] [] tty_ldisc_ref_wait+0x1f/0x50 [ 604.630082] [] tty_write+0x98/0xf0 [ 604.630856] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 604.631819] [] redirected_tty_write+0x95/0xc0 [ 604.632778] [] vfs_write+0xb8/0x180 [ 604.633595] [] sys_write+0x50/0xa0 [ 604.634422] [] tracesys+0xe1/0xe6 [ 604.635324] 1 lock held by trinity/9732: [ 604.635906] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 [ 604.637466] INFO: task trinity:10089 blocked for more than 120 seconds. [ 604.638496] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 604.639675] trinity D 00000000001d7180 5224 10089 1 0x00000000 [ 604.640798] ffff88006bb2bc28 0000000000000002 00000000001d7180 ffff8800aa09d208 [ 604.641731] ffff8800b94b3000 ffff880094303000 00000000001d7180 00000000001d7180 [ 604.642769] ffff880094303000 ffff88006bb2bfd8 00000000001d7180 00000000001d7180 [ 604.643977] Call Trace: [ 604.644383] [] __schedule+0x2e9/0x3b0 [ 604.645205] [] schedule+0x55/0x60 [ 604.645957] [] schedule_timeout+0x3a/0x370 [ 604.646843] [] ? mark_held_locks+0xf9/0x130 [ 604.647746] [] ? down_failed+0x108/0x200 [ 604.648624] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 604.649605] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 604.650652] [] down_failed+0x131/0x200 [ 604.651445] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 604.652366] [] ldsem_down_read+0x8c/0x100 [ 604.653267] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 604.654222] [] ? __lock_release+0xe1/0x100 [ 604.655118] [] tty_ldisc_ref_wait+0x1f/0x50 [ 604.656040] [] tty_write+0x98/0xf0 [ 604.656848] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 604.657794] [] redirected_tty_write+0x95/0xc0 [ 604.658750] [] vfs_write+0xb8/0x180 [ 604.659597] [] sys_write+0x50/0xa0 [ 604.660412] [] tracesys+0xe1/0xe6 [ 604.661209] 1 lock held by trinity/10089: [ 604.661828] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 [ 604.662732] INFO: task trinity:10254 blocked for more than 120 seconds. [ 604.663310] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 604.664504] trinity D 00000000001d7180 5240 10254 1 0x00000000 [ 604.665679] ffff880096219c28 0000000000000002 00000000001d7180 ffff8800aa09cb78 [ 604.666874] ffff8800b94c3000 ffff8800961a8000 00000000001d7180 00000000001d7180 [ 604.668060] ffff8800961a8000 ffff880096219fd8 00000000001d7180 00000000001d7180 [ 604.669320] Call Trace: [ 604.669713] [] __schedule+0x2e9/0x3b0 [ 604.670542] [] schedule+0x55/0x60 [ 604.671360] [] schedule_timeout+0x3a/0x370 [ 604.672270] [] ? mark_held_locks+0xf9/0x130 [ 604.673161] [] ? down_failed+0x108/0x200 [ 604.674051] [] ? _raw_spin_unlock_irq+0x2b/0x80 [ 604.675022] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 604.676068] [] down_failed+0x131/0x200 [ 604.676940] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 604.677896] [] ldsem_down_read+0x8c/0x100 [ 604.678803] [] ? tty_ldisc_ref_wait+0x1f/0x50 [ 604.679760] [] ? __lock_release+0xe1/0x100 [ 604.680639] [] tty_ldisc_ref_wait+0x1f/0x50 [ 604.681485] [] tty_write+0x98/0xf0 [ 604.682288] [] ? do_raw_spin_unlock+0xc8/0xe0 [ 604.683246] [] redirected_tty_write+0x95/0xc0 [ 604.684206] [] vfs_write+0xb8/0x180 [ 604.685055] [] sys_write+0x50/0xa0 [ 604.685858] [] tracesys+0xe1/0xe6 [ 604.686738] 1 lock held by trinity/10254: [ 604.687366] #0: (&tty->ldisc_sem){++++++}, at: [] tty_ldisc_ref_wait+0x1f/0x50 Thanks, Sasha