From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1755680AbcBBQbM (ORCPT <rfc822;w@1wt.eu>);
	Tue, 2 Feb 2016 11:31:12 -0500
Received: from parrot.pmhahn.de ([88.198.50.102]:35148 "EHLO parrot.pmhahn.de"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1750794AbcBBQbJ (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
	Tue, 2 Feb 2016 11:31:09 -0500
X-Greylist: delayed 320 seconds by postgrey-1.27 at vger.kernel.org; Tue, 02 Feb 2016 11:31:09 EST
To: Sasha Levin <sasha.levin@oracle.com>,
        Hannes Frederic Sowa <hannes@stressinduktion.org>,
        Rainer Weikusat <rweikusat@mobileactivedefense.com>,
        Andrey Vagin <avagin@openvz.org>, Aaron Conole <aconole@bytheb.org>,
        "David S. Miller" <davem@davemloft.net>, linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
From: Philipp Hahn <pmhahn@pmhahn.de>
Subject: Bug 4.1.16: self-detected stall in net/unix/?
X-Enigmail-Draft-Status: N1110
Message-ID: <56B0D88A.1020609@pmhahn.de>
Date: Tue, 2 Feb 2016 17:25:46 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Icedove/38.5.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi,

we recently updated our kernel to 4.1.16 + patch for "unix: properly
account for FDs passed over unix sockets" and have since then
self-detected stalls triggered by the Samba daemon:

> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007] INFO: rcu_sched self-detected stall on CPU { 3}  (t=162780 jiffies g=47565 c=47564 q=1055670)
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007] Task dump for CPU 3:
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007] smbd            R  running task        0  5938      1 0x0000000c
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  0000000000000004 ffffffff81851340 ffffffff810d3c84 000000000000b9cd
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  ffff8801bfd97100 ffffffff81851340 ffffffff81851340 ffffffff818f6c60
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  ffffffff810d7659 0000000000000000 0000000000000000 00001e847fc2f700
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007] Call Trace:
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  <IRQ>  [<ffffffff810d3c84>] ? rcu_dump_cpu_stacks+0x84/0xc0
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff810d7659>] ? rcu_check_callbacks+0x449/0x740
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff810ec7c0>] ? tick_sched_do_timer+0x40/0x40
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff810dcc54>] ? update_process_times+0x34/0x70
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff810ec45c>] ? tick_sched_handle.isra.12+0x2c/0x70
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff810ec809>] ? tick_sched_timer+0x49/0x80
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff810dd57d>] ? __run_hrtimer+0x6d/0x1b0
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff810ddd4d>] ? hrtimer_interrupt+0xed/0x210
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff815a0ed9>] ? smp_apic_timer_interrupt+0x39/0x50
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff8159ef7e>] ? apic_timer_interrupt+0x6e/0x80

> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  <EOI>  [<ffffffff8159de85>] ? _raw_spin_lock+0x35/0x50
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff8153b343>] ? unix_dgram_connect+0x93/0x200
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff8147f248>] ? SYSC_connect+0xe8/0x100
> Feb  1 09:03:14 dcs1 kernel: [ 1152.840007]  [<ffffffff8159e0f2>] ? system_call_fast_compare_end+0xc/0x6b


> Feb  1 11:48:13 ucs22f kernel: [307999.162254] INFO: rcu_sched self-detected stall on CPU { 0}  (t=5250 jiffies                                                                   g=6586733 c=6586732 q=6757)
> Feb  1 11:48:13 ucs22f kernel: [307999.162264] Task dump for CPU 0:
> Feb  1 11:48:13 ucs22f kernel: [307999.162267] smbd            R running      0  4615   4609 0x00000008
> Feb  1 11:48:13 ucs22f kernel: [307999.162272]  00200082 f5863b90 c10b3fe9 c1682cc0 c1682cc0 c1682cc0 f79d2b00 f                                                                  5863bdc
> Feb  1 11:48:13 ucs22f kernel: [307999.162276]  c10b722d c15bd400 00001482 0064816d 0064816c 00001a65 f79cd840 c                                                                  108b166
> Feb  1 11:48:13 ucs22f kernel: [307999.162280]  00000001 f5863bb8 f5863bb8 00000000 c1682cc0 f6808cf0 00001a65 f                                                                  6808cf0
> Feb  1 11:48:13 ucs22f kernel: [307999.162285] Call Trace:
> Feb  1 11:48:13 ucs22f kernel: [307999.162296]  [<c10b3fe9>] ? rcu_dump_cpu_stacks+0x79/0xc0
> Feb  1 11:48:13 ucs22f kernel: [307999.162300]  [<c10b722d>] ? rcu_check_callbacks+0x3cd/0x630
> Feb  1 11:48:13 ucs22f kernel: [307999.162304]  [<c108b166>] ? account_process_tick+0x66/0x160
> Feb  1 11:48:13 ucs22f kernel: [307999.162307]  [<c10bbe4f>] ? update_process_times+0x2f/0x60
> Feb  1 11:48:13 ucs22f kernel: [307999.162310]  [<c10cbf9d>] ? tick_sched_handle.isra.12+0x2d/0x60
> Feb  1 11:48:13 ucs22f kernel: [307999.162328]  [<c10cc210>] ? tick_sched_timer+0x40/0x80
> Feb  1 11:48:13 ucs22f kernel: [307999.162331]  [<c10bc6b0>] ? __remove_hrtimer+0x40/0xa0
> Feb  1 11:48:13 ucs22f kernel: [307999.162334]  [<c10bc97f>] ? __run_hrtimer+0x6f/0x190
> Feb  1 11:48:13 ucs22f kernel: [307999.162337]  [<c10cc1d0>] ? tick_sched_do_timer+0x30/0x30
> Feb  1 11:48:13 ucs22f kernel: [307999.162339]  [<c10bd15f>] ? hrtimer_interrupt+0xef/0x260
> Feb  1 11:48:13 ucs22f kernel: [307999.162343]  [<c119ae3d>] ? getname_kernel+0x2d/0x100
> Feb  1 11:48:13 ucs22f kernel: [307999.162348]  [<c1046f7f>] ? local_apic_timer_interrupt+0x2f/0x60
> Feb  1 11:48:13 ucs22f kernel: [307999.162353]  [<c14e4543>] ? smp_apic_timer_interrupt+0x33/0x50
> Feb  1 11:48:13 ucs22f kernel: [307999.162355]  [<c14e3c7c>] ? apic_timer_interrupt+0x34/0x3c

> Feb  1 11:48:13 ucs22f kernel: [307999.162358]  [<c14e2dc1>] ? _raw_spin_lock+0x51/0x70
> Feb  1 11:48:13 ucs22f kernel: [307999.162362]  [<c148c075>] ? unix_state_double_lock+0x25/0x60
> Feb  1 11:48:13 ucs22f kernel: [307999.162365]  [<c148de10>] ? unix_dgram_connect+0x90/0x1f0
> Feb  1 11:48:13 ucs22f kernel: [307999.162369]  [<c13e4267>] ? SYSC_connect+0xc7/0xe0
> Feb  1 11:48:13 ucs22f kernel: [307999.162371]  [<c13e2931>] ? sock_map_fd+0x41/0x60
> Feb  1 11:48:13 ucs22f kernel: [307999.162374]  [<c13e5014>] ? SYSC_socketcall+0x1b4/0xa20
> Feb  1 11:48:13 ucs22f kernel: [307999.162376]  [<c10c2940>] ? ktime_get+0x50/0x100
> Feb  1 11:48:13 ucs22f kernel: [307999.162379]  [<c10466db>] ? lapic_next_event+0x1b/0x20
> Feb  1 11:48:13 ucs22f kernel: [307999.162381]  [<c10ca0ed>] ? clockevents_program_event+0x9d/0x140
> Feb  1 11:48:13 ucs22f kernel: [307999.162385]  [<c129e068>] ? list_del+0x8/0x20
> Feb  1 11:48:13 ucs22f kernel: [307999.162388]  [<c1097ef7>] ? remove_wait_queue+0x27/0x40
> Feb  1 11:48:13 ucs22f kernel: [307999.162392]  [<c11c8795>] ? inotify_read+0x295/0x340
> Feb  1 11:48:13 ucs22f kernel: [307999.162396]  [<c10acc76>] ? handle_irq_event_percpu+0xa6/0x1a0
> Feb  1 11:48:13 ucs22f kernel: [307999.162399]  [<c11a786f>] ? set_close_on_exec+0x2f/0x60
> Feb  1 11:48:13 ucs22f kernel: [307999.162402]  [<c119d084>] ? do_fcntl+0x2f4/0x4e0
> Feb  1 11:48:13 ucs22f kernel: [307999.162405]  [<c107d6df>] ? commit_creds+0xff/0x1f0
> Feb  1 11:48:13 ucs22f kernel: [307999.162407]  [<c119d380>] ? SyS_fcntl64+0x60/0x100
> Feb  1 11:48:13 ucs22f kernel: [307999.162409]  [<c13e5953>] ? SyS_socketcall+0x13/0x20
> Feb  1 11:48:13 ucs22f kernel: [307999.162412]  [<c14e30db>] ? sysenter_do_call+0x12/0x12

We have not yet been able to reproduce the hang, but going back to our
previous kernel 4.1.12 makes the problem go away.

Is this a known issue or do you have an idea where to look?
What information should I collect next time it happens?

(Can unix_diag.ko with `ss` help?)
What other kernel configs should I enable do debug this dead-lock?

Thanks in advance
Philipp