From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752856AbcBFXvq (ORCPT ); Sat, 6 Feb 2016 18:51:46 -0500 Received: from ipmail06.adl2.internode.on.net ([150.101.137.129]:21740 "EHLO ipmail06.adl2.internode.on.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750901AbcBFXvp (ORCPT ); Sat, 6 Feb 2016 18:51:45 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CECADphbZWPBATLHleKAECgw9SbYJphXKdPwEBAQEBAQaLZoVEhAclhWIEAgKBIU0BAQEBAQEHAQEBAUE/hEIBAQQ6HBYKAxAIAxgJJQ8FJQMHGgoJGwOHfA++FgEBAQEBBQIBGQQYhTKEf4hsBZZ1hUyHe4FkjRiKbINSgmUZgVwoLgEEhxUkgRUBAQE Date: Sun, 7 Feb 2016 10:51:40 +1100 From: Dave Chinner To: Waiman Long Cc: Andi Kleen , Ingo Molnar , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , Alexander Viro , linux-fsdevel@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Scott J Norton , Douglas Hatch Subject: Re: [PATCH v2 3/3] vfs: Enable list batching for the superblock's inode list Message-ID: <20160206235140.GH31407@dastard> References: <1454095846-19628-1-git-send-email-Waiman.Long@hpe.com> <1454095846-19628-4-git-send-email-Waiman.Long@hpe.com> <20160130083557.GA31749@gmail.com> <20160201174526.GA3696@two.firstfloor.org> <56AFD649.9030707@hpe.com> <56B28645.7020008@hpe.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56B28645.7020008@hpe.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Feb 03, 2016 at 05:59:17PM -0500, Waiman Long wrote: > On 02/01/2016 05:03 PM, Waiman Long wrote: > >On 02/01/2016 12:45 PM, Andi Kleen wrote: > >>>I'm wondering, why are inode_sb_list_add()/del() even called > >>>for a presumably > >>>reasonably well cached benchmark running on a system with > >>>enough RAM? Are these > >>>perhaps thousands of temporary files, already deleted, and > >>>released when all the > >>>file descriptors are closed as part of sys_exit()? > >>> > >>>If that's the case then I suspect an even bigger win would be > >>>not just to batch > >>>the (sb-)global list fiddling, but to potentially turn the sb > >>>list into a > >>>percpu_alloc() managed set of per CPU lists? It's a bigger > >>>change, but it could > >>We had such a patch in the lock elision patchkit (It avoided a lot > >>of cache line bouncing leading to aborts) > >> > >>https://git.kernel.org/cgit/linux/kernel/git/ak/linux-misc.git/commit/?h=hle315/combined&id=f1cf9e715a40f44086662ae3b29f123cf059cbf4 > >> > >> > >>-Andi > >> > >> > > > >I like your patch though it cannot be applied cleanly for the > >current upstream kernel. I will port it to the current kernel and > >run my microbenchmark to see what performance gain I can get. > > > > Unfortunately, using per-cpu list didn't have the performance > benefit that I expected. I saw maybe 1 or 2% of performance > increase, but nothing significant. I guess the bulk of the > performance improvement in my patch is in the elimination of most of > the cacheline transfer latencies when the lock ownership is passed > from one CPU to another. Those latencies are still there even if we > use the per-cpu list. Now that I look at the above patch, it doesn't get rid of the global list lock. hence it won't change any of the existing global lock cacheline contention. The list structure change to per-cpu is completely irrelevant because it doesn't address the problem being seen. A proper per-cpu list implementation will provide either per-cpu locks or some other mechanism to protect each list and eliminate a large amount of global cacheline bouncing. Given this, I would not use the above patch and results as a reason for saying this approach will not work, or be a better solution.... Cheers, Dave. -- Dave Chinner david@fromorbit.com