From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757901AbcASJzj (ORCPT ); Tue, 19 Jan 2016 04:55:39 -0500 Received: from e06smtp10.uk.ibm.com ([195.75.94.106]:36899 "EHLO e06smtp10.uk.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757627AbcASJzg (ORCPT ); Tue, 19 Jan 2016 04:55:36 -0500 X-IBM-Helo: d06dlp01.portsmouth.uk.ibm.com X-IBM-MailFrom: heiko.carstens@de.ibm.com X-IBM-RcptTo: kvm@vger.kernel.org;linux-kernel@vger.kernel.org;linux-s390@vger.kernel.org Date: Tue, 19 Jan 2016 10:55:18 +0100 From: Heiko Carstens To: Christian Borntraeger Cc: Peter Zijlstra , Tejun Heo , "linux-kernel@vger.kernel.org >> Linux Kernel Mailing List" , linux-s390 , KVM list , Oleg Nesterov , "Paul E. McKenney" Subject: Re: regression 4.4: deadlock in with cgroup percpu_rwsem Message-ID: <20160119095518.GC3528@osiris> References: <56978452.6010606@de.ibm.com> <20160114195630.GA3520@mtj.duckdns.org> <5698A023.9070703@de.ibm.com> <56990C9E.7020801@de.ibm.com> <20160118183205.GW6357@twins.programming.kicks-ass.net> <569D3370.6040503@de.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <569D3370.6040503@de.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16011909-0041-0000-0000-000006FB11AA Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jan 18, 2016 at 07:48:16PM +0100, Christian Borntraeger wrote: > On 01/18/2016 07:32 PM, Peter Zijlstra wrote: > > On Fri, Jan 15, 2016 at 04:13:34PM +0100, Christian Borntraeger wrote: > >>> Yes, the deadlock is gone and the system is still running. > >>> After some time I had the following WARN in the logs, though. > >>> Not sure yet if that is related. > >>> > >>> [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current) > >>> [25331.763630] ------------[ cut here ]------------ > >>> [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80 > > > >> I restarted the test with panic_on_warn. Hopefully I can get a dump to check > >> which mutex this was. > > > > Hard to reproduce warnings like this tend to point towards memory > > corruption. Someone stepped on the mutex value and tickles the sanity > > check. > > > > With lockdep and debugging enabled the mutex gets quite a bit bigger, so > > it gets more likely to be hit by 'random' corruption. > > > > The locking in seq_read() seems rather straight forward. > > I was able to reproduce. The dump shows a mutex that has an owner field, which > does not exists as a task so this all looks fishy. The good thing is, that I > can reproduce the issue within some hours. (exact same backtrace). Will add some > more debug data to get a handle where we come from. Did the owner field show to something that still looks like a task_struct? From mboxrd@z Thu Jan 1 00:00:00 1970 From: Heiko Carstens Subject: Re: regression 4.4: deadlock in with cgroup percpu_rwsem Date: Tue, 19 Jan 2016 10:55:18 +0100 Message-ID: <20160119095518.GC3528@osiris> References: <56978452.6010606@de.ibm.com> <20160114195630.GA3520@mtj.duckdns.org> <5698A023.9070703@de.ibm.com> <56990C9E.7020801@de.ibm.com> <20160118183205.GW6357@twins.programming.kicks-ass.net> <569D3370.6040503@de.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <569D3370.6040503@de.ibm.com> Sender: linux-kernel-owner@vger.kernel.org List-Archive: List-Post: To: Christian Borntraeger Cc: Peter Zijlstra , Tejun Heo , "linux-kernel@vger.kernel.org >> Linux Kernel Mailing List" , linux-s390 , KVM list , Oleg Nesterov , "Paul E. McKenney" List-ID: On Mon, Jan 18, 2016 at 07:48:16PM +0100, Christian Borntraeger wrote: > On 01/18/2016 07:32 PM, Peter Zijlstra wrote: > > On Fri, Jan 15, 2016 at 04:13:34PM +0100, Christian Borntraeger wrote: > >>> Yes, the deadlock is gone and the system is still running. > >>> After some time I had the following WARN in the logs, though. > >>> Not sure yet if that is related. > >>> > >>> [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current) > >>> [25331.763630] ------------[ cut here ]------------ > >>> [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80 > > > >> I restarted the test with panic_on_warn. Hopefully I can get a dump to check > >> which mutex this was. > > > > Hard to reproduce warnings like this tend to point towards memory > > corruption. Someone stepped on the mutex value and tickles the sanity > > check. > > > > With lockdep and debugging enabled the mutex gets quite a bit bigger, so > > it gets more likely to be hit by 'random' corruption. > > > > The locking in seq_read() seems rather straight forward. > > I was able to reproduce. The dump shows a mutex that has an owner field, which > does not exists as a task so this all looks fishy. The good thing is, that I > can reproduce the issue within some hours. (exact same backtrace). Will add some > more debug data to get a handle where we come from. Did the owner field show to something that still looks like a task_struct?