From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751425AbcGMVBq (ORCPT ); Wed, 13 Jul 2016 17:01:46 -0400 Received: from mail-io0-f173.google.com ([209.85.223.173]:36613 "EHLO mail-io0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751347AbcGMVBe (ORCPT ); Wed, 13 Jul 2016 17:01:34 -0400 MIME-Version: 1.0 In-Reply-To: <20160713205211.GN7094@linux.vnet.ibm.com> References: <20160713182102.GJ4065@mtj.duckdns.org> <20160713183347.GK4065@mtj.duckdns.org> <20160713201823.GB29670@mtj.duckdns.org> <20160713202657.GW30154@twins.programming.kicks-ass.net> <20160713205211.GN7094@linux.vnet.ibm.com> From: Dmitry Shmidt Date: Wed, 13 Jul 2016 14:01:27 -0700 Message-ID: Subject: Re: Severe performance regression w/ 4.4+ on Android due to cgroup locking changes To: Paul McKenney Cc: Peter Zijlstra , Tejun Heo , John Stultz , Ingo Molnar , lkml , Rom Lemarchand , Colin Cross , Todd Kjos , Oleg Nesterov Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jul 13, 2016 at 1:52 PM, Paul E. McKenney wrote: > On Wed, Jul 13, 2016 at 10:26:57PM +0200, Peter Zijlstra wrote: >> On Wed, Jul 13, 2016 at 04:18:23PM -0400, Tejun Heo wrote: >> > Hello, John. >> > >> > On Wed, Jul 13, 2016 at 01:13:11PM -0700, John Stultz wrote: >> > > On Wed, Jul 13, 2016 at 11:33 AM, Tejun Heo wrote: >> > > > On Wed, Jul 13, 2016 at 02:21:02PM -0400, Tejun Heo wrote: >> > > >> One interesting thing to try would be replacing it with a regular >> > > >> non-percpu rwsem and see how it behaves. That should easily tell us >> > > >> whether this is from actual contention or artifacts from percpu_rwsem >> > > >> implementation. >> > > > >> > > > So, something like the following. Can you please see whether this >> > > > makes any difference? >> > > >> > > Yea. So this brings it down for me closer to what we're seeing with >> > > the Dmitry's patch reverting the two problematic commits, usually >> > > 10-50us with one early spike at 18ms. >> > >> > So, it's a percpu rwsem issue then. I haven't really followed the >> > perpcpu rwsem changes closely. Oleg, are multi-milisec delay expected >> > on down write expected with the current implementation of >> > percpu_rwsem? >> >> There is a synchronize_sched() in there, so sorta. That thing is heavily >> geared towards readers, as is the only 'sane' choice for global locks. > > Then one diagnostic step to take would be to replace that > synchronize_sched() with synchronize_sched_expedited(), and see if that > gets rid of the delays. > > Not a particularly real-time-friendly fix, but certainly a good check > on our various assumptions. All delays <200 us, but one that is 3 ms. > Thanx, Paul > > ------------------------------------------------------------------------ > > diff --git a/kernel/rcu/sync.c b/kernel/rcu/sync.c > index be922c9f3d37..211acddc7e21 100644 > --- a/kernel/rcu/sync.c > +++ b/kernel/rcu/sync.c > @@ -38,19 +38,19 @@ static const struct { > #endif > } gp_ops[] = { > [RCU_SYNC] = { > - .sync = synchronize_rcu, > + .sync = synchronize_rcu_expedited, > .call = call_rcu, > .wait = rcu_barrier, > __INIT_HELD(rcu_read_lock_held) > }, > [RCU_SCHED_SYNC] = { > - .sync = synchronize_sched, > + .sync = synchronize_sched_expedited, > .call = call_rcu_sched, > .wait = rcu_barrier_sched, > __INIT_HELD(rcu_read_lock_sched_held) > }, > [RCU_BH_SYNC] = { > - .sync = synchronize_rcu_bh, > + .sync = synchronize_rcu_bh_expedited, > .call = call_rcu_bh, > .wait = rcu_barrier_bh, > __INIT_HELD(rcu_read_lock_bh_held) >