From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752226Ab3AWT55 (ORCPT ); Wed, 23 Jan 2013 14:57:57 -0500 Received: from mail-qa0-f43.google.com ([209.85.216.43]:57379 "EHLO mail-qa0-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751028Ab3AWT5u (ORCPT ); Wed, 23 Jan 2013 14:57:50 -0500 Date: Wed, 23 Jan 2013 11:57:40 -0800 From: Tejun Heo To: "Srivatsa S. Bhat" Cc: tglx@linutronix.de, peterz@infradead.org, oleg@redhat.com, paulmck@linux.vnet.ibm.com, rusty@rustcorp.com.au, mingo@kernel.org, akpm@linux-foundation.org, namhyung@kernel.org, rostedt@goodmis.org, wangyun@linux.vnet.ibm.com, xiaoguangrong@linux.vnet.ibm.com, rjw@sisk.pl, sbw@mit.edu, fweisbec@gmail.com, linux@arm.linux.org.uk, nikunj@linux.vnet.ibm.com, linux-pm@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, netdev@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 04/45] percpu_rwlock: Implement the core design of Per-CPU Reader-Writer Locks Message-ID: <20130123195740.GI2373@mtj.dyndns.org> References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> <20130122073347.13822.85876.stgit@srivatsabhat.in.ibm.com> <20130123185522.GG2373@mtj.dyndns.org> <51003B20.2060506@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51003B20.2060506@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello, Srivatsa. On Thu, Jan 24, 2013 at 01:03:52AM +0530, Srivatsa S. Bhat wrote: > Hmm.. I split it up into steps to help explain the reasoning behind > the code sufficiently, rather than spring all of the intricacies at > one go (which would make it very hard to write the changelog/comments > also). The split made it easier for me to document it well in the > changelog, because I could deal with reasonable chunks of code/complexity > at a time. IMHO that helps people reading it for the first time to > understand the logic easily. I don't know. It's a judgement call I guess. I personally would much prefer having ample documentation as comments in the source itself or as a separate Documentation/ file as that's what most people are gonna be looking at to figure out what's going on. Maybe just compact it a bit and add more in-line documentation instead? > > The only two options are either punishing writers or identifying and > > updating all such possible deadlocks. percpu_rwsem does the former, > > right? I don't know how feasible the latter would be. > > I don't think we can avoid looking into all the possible deadlocks, > as long as we use rwlocks inside get/put_online_cpus_atomic() (assuming > rwlocks are fair). Even with Oleg's idea of using synchronize_sched() > at the writer, we still need to take care of locking rules, because the > synchronize_sched() only helps avoid the memory barriers at the reader, > and doesn't help get rid of the rwlocks themselves. Well, percpu_rwlock don't have to use rwlock for the slow path. It can implement its own writer starving locking scheme. It's not like implementing slow path global rwlock logic is difficult. > CPU 0 CPU 1 > > read_lock(&rwlock) > > write_lock(&rwlock) //spins, because CPU 0 > //has acquired the lock for read > > read_lock(&rwlock) > ^^^^^ > What happens here? Does CPU 0 start spinning (and hence deadlock) or will > it continue realizing that it already holds the rwlock for read? I don't think rwlock allows nesting write lock inside read lock. read_lock(); write_lock() will always deadlock. Thanks. -- tejun From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qc0-f174.google.com (mail-qc0-f174.google.com [209.85.216.174]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (not verified)) by ozlabs.org (Postfix) with ESMTPS id E785D2C008C for ; Thu, 24 Jan 2013 06:57:51 +1100 (EST) Received: by mail-qc0-f174.google.com with SMTP id z24so2407532qcq.19 for ; Wed, 23 Jan 2013 11:57:48 -0800 (PST) Sender: Tejun Heo Date: Wed, 23 Jan 2013 11:57:40 -0800 From: Tejun Heo To: "Srivatsa S. Bhat" Subject: Re: [PATCH v5 04/45] percpu_rwlock: Implement the core design of Per-CPU Reader-Writer Locks Message-ID: <20130123195740.GI2373@mtj.dyndns.org> References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> <20130122073347.13822.85876.stgit@srivatsabhat.in.ibm.com> <20130123185522.GG2373@mtj.dyndns.org> <51003B20.2060506@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <51003B20.2060506@linux.vnet.ibm.com> Cc: linux-doc@vger.kernel.org, peterz@infradead.org, fweisbec@gmail.com, linux-kernel@vger.kernel.org, mingo@kernel.org, linux-arch@vger.kernel.org, linux@arm.linux.org.uk, xiaoguangrong@linux.vnet.ibm.com, wangyun@linux.vnet.ibm.com, paulmck@linux.vnet.ibm.com, nikunj@linux.vnet.ibm.com, linux-pm@vger.kernel.org, rusty@rustcorp.com.au, rostedt@goodmis.org, rjw@sisk.pl, namhyung@kernel.org, tglx@linutronix.de, linux-arm-kernel@lists.infradead.org, netdev@vger.kernel.org, oleg@redhat.com, sbw@mit.edu, akpm@linux-foundation.org, linuxppc-dev@lists.ozlabs.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Hello, Srivatsa. On Thu, Jan 24, 2013 at 01:03:52AM +0530, Srivatsa S. Bhat wrote: > Hmm.. I split it up into steps to help explain the reasoning behind > the code sufficiently, rather than spring all of the intricacies at > one go (which would make it very hard to write the changelog/comments > also). The split made it easier for me to document it well in the > changelog, because I could deal with reasonable chunks of code/complexity > at a time. IMHO that helps people reading it for the first time to > understand the logic easily. I don't know. It's a judgement call I guess. I personally would much prefer having ample documentation as comments in the source itself or as a separate Documentation/ file as that's what most people are gonna be looking at to figure out what's going on. Maybe just compact it a bit and add more in-line documentation instead? > > The only two options are either punishing writers or identifying and > > updating all such possible deadlocks. percpu_rwsem does the former, > > right? I don't know how feasible the latter would be. > > I don't think we can avoid looking into all the possible deadlocks, > as long as we use rwlocks inside get/put_online_cpus_atomic() (assuming > rwlocks are fair). Even with Oleg's idea of using synchronize_sched() > at the writer, we still need to take care of locking rules, because the > synchronize_sched() only helps avoid the memory barriers at the reader, > and doesn't help get rid of the rwlocks themselves. Well, percpu_rwlock don't have to use rwlock for the slow path. It can implement its own writer starving locking scheme. It's not like implementing slow path global rwlock logic is difficult. > CPU 0 CPU 1 > > read_lock(&rwlock) > > write_lock(&rwlock) //spins, because CPU 0 > //has acquired the lock for read > > read_lock(&rwlock) > ^^^^^ > What happens here? Does CPU 0 start spinning (and hence deadlock) or will > it continue realizing that it already holds the rwlock for read? I don't think rwlock allows nesting write lock inside read lock. read_lock(); write_lock() will always deadlock. Thanks. -- tejun From mboxrd@z Thu Jan 1 00:00:00 1970 From: tj@kernel.org (Tejun Heo) Date: Wed, 23 Jan 2013 11:57:40 -0800 Subject: [PATCH v5 04/45] percpu_rwlock: Implement the core design of Per-CPU Reader-Writer Locks In-Reply-To: <51003B20.2060506@linux.vnet.ibm.com> References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> <20130122073347.13822.85876.stgit@srivatsabhat.in.ibm.com> <20130123185522.GG2373@mtj.dyndns.org> <51003B20.2060506@linux.vnet.ibm.com> Message-ID: <20130123195740.GI2373@mtj.dyndns.org> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hello, Srivatsa. On Thu, Jan 24, 2013 at 01:03:52AM +0530, Srivatsa S. Bhat wrote: > Hmm.. I split it up into steps to help explain the reasoning behind > the code sufficiently, rather than spring all of the intricacies at > one go (which would make it very hard to write the changelog/comments > also). The split made it easier for me to document it well in the > changelog, because I could deal with reasonable chunks of code/complexity > at a time. IMHO that helps people reading it for the first time to > understand the logic easily. I don't know. It's a judgement call I guess. I personally would much prefer having ample documentation as comments in the source itself or as a separate Documentation/ file as that's what most people are gonna be looking at to figure out what's going on. Maybe just compact it a bit and add more in-line documentation instead? > > The only two options are either punishing writers or identifying and > > updating all such possible deadlocks. percpu_rwsem does the former, > > right? I don't know how feasible the latter would be. > > I don't think we can avoid looking into all the possible deadlocks, > as long as we use rwlocks inside get/put_online_cpus_atomic() (assuming > rwlocks are fair). Even with Oleg's idea of using synchronize_sched() > at the writer, we still need to take care of locking rules, because the > synchronize_sched() only helps avoid the memory barriers at the reader, > and doesn't help get rid of the rwlocks themselves. Well, percpu_rwlock don't have to use rwlock for the slow path. It can implement its own writer starving locking scheme. It's not like implementing slow path global rwlock logic is difficult. > CPU 0 CPU 1 > > read_lock(&rwlock) > > write_lock(&rwlock) //spins, because CPU 0 > //has acquired the lock for read > > read_lock(&rwlock) > ^^^^^ > What happens here? Does CPU 0 start spinning (and hence deadlock) or will > it continue realizing that it already holds the rwlock for read? I don't think rwlock allows nesting write lock inside read lock. read_lock(); write_lock() will always deadlock. Thanks. -- tejun