From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jesper Dangaard Brouer Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Date: Mon, 7 Sep 2015 11:30:26 +0200 Message-ID: <20150907113026.5bb28ca3@redhat.com> References: <20150903005115.GA27804@redhat.com> <20150903060247.GV1933@devil.localdomain> <20150904032607.GX1933@devil.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: brouer@redhat.com, Dave Chinner , Mike Snitzer , Christoph Lameter , Pekka Enberg , Andrew Morton , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm , "netdev@vger.kernel.org" To: Linus Torvalds Return-path: In-Reply-To: Sender: owner-linux-mm@kvack.org List-Id: netdev.vger.kernel.org On Thu, 3 Sep 2015 20:51:09 -0700 Linus Torvalds wrote: > On Thu, Sep 3, 2015 at 8:26 PM, Dave Chinner wrote: > > > > The double standard is the problem here. No notification, proof, > > discussion or review was needed to turn on slab merging for > > everyone, but you're setting a very high bar to jump if anyone wants > > to turn it off in their code. > > Ehh. You realize that almost the only load that is actually seriously > allocator-limited is networking? > > And slub was beating slab on that? And slub has been doing the merging > since day one. Slab was just changed to try to keep up with the > winning strategy. Sorry, I have to correct you on this. The slub allocator is not as fast as you might think. The slab allocator is actually faster for networking. IP-forwarding, single CPU, single flow UDP (highly tuned): * Allocator slub: 2043575 pps * Allocator slab: 2088295 pps Difference slab faster than slub: * +44720 pps and -10.48ns The slub allocator have a faster "fastpath", if your workload is fast-reusing within the same per-cpu page-slab, but once the workload increases you hit the slowpath, and then slab catches up. Slub looks great in micro-benchmarking. As you can see in patchset: [PATCH 0/3] Network stack, first user of SLAB/kmem_cache bulk free API. http://thread.gmane.org/gmane.linux.kernel.mm/137469/focus=376625 I'm working on speeding up slub to the level of slab. And it seems like I have succeeded with half-a-nanosec 2090522 pps (+2227 pps or 0.51 ns). And with "slab_nomerge" I get even high performance: * slub: bulk-free and slab_nomerge: 2121824 pps * Diff to slub: +78249 and -18.05ns -- Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Linus Torvalds Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Date: Mon, 7 Sep 2015 13:22:13 -0700 Message-ID: References: <20150903005115.GA27804@redhat.com> <20150903060247.GV1933@devil.localdomain> <20150904032607.GX1933@devil.localdomain> <20150907113026.5bb28ca3@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Cc: Dave Chinner , Mike Snitzer , Christoph Lameter , Pekka Enberg , Andrew Morton , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm , "netdev@vger.kernel.org" To: Jesper Dangaard Brouer Return-path: Received: from mail-ig0-f170.google.com ([209.85.213.170]:37835 "EHLO mail-ig0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751162AbbIGUWO (ORCPT ); Mon, 7 Sep 2015 16:22:14 -0400 Received: by igbni9 with SMTP id ni9so63363227igb.0 for ; Mon, 07 Sep 2015 13:22:13 -0700 (PDT) In-Reply-To: <20150907113026.5bb28ca3@redhat.com> Sender: netdev-owner@vger.kernel.org List-ID: On Mon, Sep 7, 2015 at 2:30 AM, Jesper Dangaard Brouer wrote: > > The slub allocator have a faster "fastpath", if your workload is > fast-reusing within the same per-cpu page-slab, but once the workload > increases you hit the slowpath, and then slab catches up. Slub looks > great in micro-benchmarking. > > And with "slab_nomerge" I get even high performance: I think those two are related. Not merging means that effectively the percpu caches end up being bigger (simply because there are more of them), and so it captures more of the fastpath cases. Obviously the percpu queue size is an easy tunable too, but there are real downsides to that too. I suspect your IP forwarding case isn't so different from some of the microbenchmarks, it just has more outstanding work.. And yes, the slow path (ie not hitting in the percpu cache) of SLUB could hopefully be optimizable too, although maybe the bulk patches are the way to go (and unrelated to this thread - at least part of your bulk patches actually got merged last Friday - they were part of Andrew's patch-bomb). Linus From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jesper Dangaard Brouer Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Date: Mon, 7 Sep 2015 23:17:15 +0200 Message-ID: <20150907231715.0a375b40@redhat.com> References: <20150903005115.GA27804@redhat.com> <20150903060247.GV1933@devil.localdomain> <20150904032607.GX1933@devil.localdomain> <20150907113026.5bb28ca3@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Dave Chinner , Mike Snitzer , Christoph Lameter , Pekka Enberg , Andrew Morton , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm , "netdev@vger.kernel.org" , brouer@redhat.com To: Linus Torvalds Return-path: In-Reply-To: Sender: owner-linux-mm@kvack.org List-Id: netdev.vger.kernel.org On Mon, 7 Sep 2015 13:22:13 -0700 Linus Torvalds wrote: > On Mon, Sep 7, 2015 at 2:30 AM, Jesper Dangaard Brouer > wrote: > > > > The slub allocator have a faster "fastpath", if your workload is > > fast-reusing within the same per-cpu page-slab, but once the workload > > increases you hit the slowpath, and then slab catches up. Slub looks > > great in micro-benchmarking. > > > > And with "slab_nomerge" I get even high performance: > > I think those two are related. > > Not merging means that effectively the percpu caches end up being > bigger (simply because there are more of them), and so it captures > more of the fastpath cases. Yes, that was also my theory. As manually tuning the percpu sizes gave me almost the same boost. > Obviously the percpu queue size is an easy tunable too, but there are > real downsides to that too. The easy fix is to introduce a subsystem specific percpu cache that is large enough for our use-case. That seems to be a trend. I'm hoping to come up with something smarter that every subsystem can benefit from. E.g some heuristic that can dynamic adjust SLUB according to the usage pattern. I can imagine something as simple as a counter for every slowpath call, that is only valid as long as the jiffies count matches (reset to zero, and store new jiffies cnt). (But I have not thought this through...) > I suspect your IP forwarding case isn't so > different from some of the microbenchmarks, it just has more > outstanding work.. Yes, I will admit that my testing is very close to micro benchmarking, and it is specifically designed to pressure the system to its limits[1]. Especially the minimum frame size is evil and unrealistic, but the real purpose is preparing the stack for increasing speeds like 100Gbit/s. > And yes, the slow path (ie not hitting in the percpu cache) of SLUB > could hopefully be optimizable too, although maybe the bulk patches > are the way to go (and unrelated to this thread - at least part of > your bulk patches actually got merged last Friday - they were part of > Andrew's patch-bomb). Cool. Yes, it is only part of the bulk patches. The real performance boosters are not in yet (but I need to make them work correctly with memory debugging enabled before they can get merged). At least the main API is in, which allows me to implement use-case easier in other subsystems :-) [1] http://netoptimizer.blogspot.dk/2014/09/packet-per-sec-measurements-for.html -- Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f178.google.com (mail-io0-f178.google.com [209.85.223.178]) by kanga.kvack.org (Postfix) with ESMTP id EA2FD6B0254 for ; Wed, 2 Sep 2015 19:13:45 -0400 (EDT) Received: by ioiz6 with SMTP id z6so38646760ioi.2 for ; Wed, 02 Sep 2015 16:13:45 -0700 (PDT) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com. [2607:f8b0:4001:c06::232]) by mx.google.com with ESMTPS id ri1si3619370igc.93.2015.09.02.16.13.45 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 16:13:45 -0700 (PDT) Received: by iofb144 with SMTP id b144so38889239iof.1 for ; Wed, 02 Sep 2015 16:13:45 -0700 (PDT) MIME-Version: 1.0 Date: Wed, 2 Sep 2015 16:13:44 -0700 Message-ID: Subject: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) From: Linus Torvalds Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Mike Snitzer , Dave Chinner , Christoph Lameter , Pekka Enberg , Andrew Morton , David Rientjes , Joonsoo Kim Cc: "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm On Wed, Sep 2, 2015 at 10:39 AM, Mike Snitzer wrote: > > - last but not least: add SLAB_NO_MERGE flag to mm/slab_common and > disable slab merging for all of DM's slabs (XFS will also use > SLAB_NO_MERGE once merged). So I'm not at all convinced this is the right thing to do. In fact, I'm pretty convinced it shouldn't be done this way. Since those commits were at the top of your tree, I just didn't pull them, but took the rest.. You are basically making this one-sided decision based on your notion of convenience, and just forcing that thing unconditionally on people. Your rationale seems _totally_ bogus: you say that it's to be able to observe the sizes of the dm slabs without using slab debugging. First off, you don't have to enable slab debugging. You can just disable slab merging. It's called "slab_nomerge". It does exactly what you would think it does. And what is it that makes dm slabs such a special little princess? What makes you think that the fact that _you_ want to look at slab statistics means that everybody else suddenly must have separate slabs for dm, and dm only? Or xfs? The other "rationale" was that not merging slabs limits cross-subsystem memory corruption. Again, what the _hell_ is special about device mapper that dm - and only dm - would make this a special thing? That is just pure and utter garbage. Again, we already have that "slab_nomerge" option, exactly so that when odd slab corruption issues happen (they are rare, but they do occasionally happen), you can try that to see if that pinpoints the problem more. And it is *not* limited to some random set of subsystems. Which makes it clearly superior to your broken approach, wouldn't you agree? The only possible true rationale for why dm is special is "because dm is such a buggy piece of sh*t that it's much more likely to have these slab corruption bugs than anything else, so I'm just protecting the rest of the system". Is that really your rationale? Somehow I doubt it. But if it is, you really should have said so. At least then it would make sense why this thing came in through the dm tree, and why dm is so special than it - and only it - would disable slab merging. So I'm not pulling things like this from the device mapper tree. There is just no excuse that I can see for something like SLAB_NO_MERGE to go through the dm tree in the first place, but that's doubly true when the rationale for these things were bogus and had nothing what-so-ever to do with dm. Things like this aren't supposed to come in through random irrelevant trees like this, and with no discussion (at least judging by the commits) with the maintainers of the other pieces of code. If you have issues with slab merging, then those should be discussed as such, not as some magical and bogus dm or xfs special case when they damn well aren't, and damn well will never be. Yes, I'm annoyed. This was not done well. I realize that everybody thinks that _their_ code is so special and exceptional that "obviously" they should be treated specially, but I don't see that that is the case at all in this case. If you want to argue that slab merging should be disabled by default, then that is an argument that I'm willing to believe might be valid ("the downsides are bigger than the upsides"). Or if you are able to explain why dm really _is_ special, that's an option too. But this kind of "random subsystems decide unilaterally to not follow the normal rules" is not acceptable. Not when the "arguments" for it have absolutely nothing in particular to do with that subsystem. Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f179.google.com (mail-ig0-f179.google.com [209.85.213.179]) by kanga.kvack.org (Postfix) with ESMTP id 6AD166B0254 for ; Wed, 2 Sep 2015 20:48:17 -0400 (EDT) Received: by igcpb10 with SMTP id pb10so3053025igc.1 for ; Wed, 02 Sep 2015 17:48:17 -0700 (PDT) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org. [140.211.169.12]) by mx.google.com with ESMTPS id e90si6283302ioj.33.2015.09.02.17.48.16 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 17:48:16 -0700 (PDT) Date: Wed, 2 Sep 2015 17:48:15 -0700 From: Andrew Morton Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Message-Id: <20150902174815.199d51a480a01e9a754367e3@linux-foundation.org> In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Mike Snitzer , Dave Chinner , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm On Wed, 2 Sep 2015 16:13:44 -0700 Linus Torvalds wrote: > On Wed, Sep 2, 2015 at 10:39 AM, Mike Snitzer wrote: > > > > - last but not least: add SLAB_NO_MERGE flag to mm/slab_common and > > disable slab merging for all of DM's slabs (XFS will also use > > SLAB_NO_MERGE once merged). > > So I'm not at all convinced this is the right thing to do. In fact, > I'm pretty convinced it shouldn't be done this way. Since those > commits were at the top of your tree, I just didn't pull them, but > took the rest.. I don't have problems with the patch itself, really. It only affects callers who use SLAB_NO_MERGE and those developers can make their own decisions. It is a bit sad to de-optimise dm for all users for all time in order to make life a bit easier for dm's developers, but maybe that's a decent tradeoff. What I do have a problem with is that afaict the patch appeared on linux-mm for the first time just yesterday. Didn't cc slab developers, it isn't in linux-next, didn't cc linux-kernel or linux-mm or slab/mm developers on the pull request. Bad! I'd like the slab developers to have time to understand and review this change, please. Partly so they have a chance to provide feedback for the usual reasons, but also to help them understand the effect their design choice had on client subystems. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f177.google.com (mail-io0-f177.google.com [209.85.223.177]) by kanga.kvack.org (Postfix) with ESMTP id 07B8E6B0254 for ; Wed, 2 Sep 2015 20:51:19 -0400 (EDT) Received: by ioiz6 with SMTP id z6so40606060ioi.2 for ; Wed, 02 Sep 2015 17:51:18 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id j4si13444128pda.206.2015.09.02.17.51.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 17:51:18 -0700 (PDT) Date: Wed, 2 Sep 2015 20:51:15 -0400 From: Mike Snitzer Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Message-ID: <20150903005115.GA27804@redhat.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Dave Chinner , Christoph Lameter , Pekka Enberg , Andrew Morton , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm On Wed, Sep 02 2015 at 7:13pm -0400, Linus Torvalds wrote: > On Wed, Sep 2, 2015 at 10:39 AM, Mike Snitzer wrote: > > > > - last but not least: add SLAB_NO_MERGE flag to mm/slab_common and > > disable slab merging for all of DM's slabs (XFS will also use > > SLAB_NO_MERGE once merged). > > So I'm not at all convinced this is the right thing to do. In fact, > I'm pretty convinced it shouldn't be done this way. Since those > commits were at the top of your tree, I just didn't pull them, but > took the rest.. OK, thanks. > You are basically making this one-sided decision based on your notion > of convenience, and just forcing that thing unconditionally on people. The switch to slab merging was forced on everyone without proper notice. What I made possible with SLAB_NO_MERGE is for each subsystem to decide if they would prefer to not allow slab merging. > Your rationale seems _totally_ bogus: you say that it's to be able to > observe the sizes of the dm slabs without using slab debugging. > > First off, you don't have to enable slab debugging. You can just > disable slab merging. It's called "slab_nomerge". It does exactly > what you would think it does. I'm well aware of slab_nomerge. I called it out in my commit message. > And what is it that makes dm slabs such a special little princess? > What makes you think that the fact that _you_ want to look at slab > statistics means that everybody else suddenly must have separate slabs > for dm, and dm only? Or xfs? >>From where I sit it is much more useful to have separate slabs. Could be if a case was actually made for slab merging I'd change my view. But as of now these trump the stated benefits of slab merging: 1) useful slab usage stats 2) fault isolation from other subsystems > The other "rationale" was that not merging slabs limits > cross-subsystem memory corruption. Again, what the _hell_ is special > about device mapper that dm - and only dm - would make this a special > thing? That is just pure and utter garbage. Again, we already have > that "slab_nomerge" option, exactly so that when odd slab corruption > issues happen (they are rare, but they do occasionally happen), you > can try that to see if that pinpoints the problem more. And it is > *not* limited to some random set of subsystems. Which makes it clearly > superior to your broken approach, wouldn't you agree? I'm not interested in deciding such things for everyone. I added a flag that enables piecewise enablement of unshared slabs for subsystems that really don't want shared slabs. Aside from improved accounting, the point is to not allow other crap code (e.g. staging or whatever) to impact other subsystems via shared slabs. > The only possible true rationale for why dm is special is "because dm > is such a buggy piece of sh*t that it's much more likely to have these > slab corruption bugs than anything else, so I'm just protecting the > rest of the system". > > Is that really your rationale? Somehow I doubt it. But if it is, you > really should have said so. At least then it would make sense why this > thing came in through the dm tree, and why dm is so special than it - > and only it - would disable slab merging. The 3 lines that added SLAB_NO_MERGE were pretty damn clean. SLAB_NO_MERGE gives subsystems a choice they didn't have before and they frankly probably never knew they had to care about it because they didn't know slabs were being merged. I asked around enough to know I'm not an idiot for having missed the memo on slab merging. Lack of awareness aside, nobody ever _convincingly_ detailed why slab merging was pushed on everyone. Look at the header for commit 12220de ("mm/slab: support slab merge") -- now that is some seriously weak justification! I sought to get more insight on "why slab merging?" and all I found was this in Documentation/vm/slub.txt: " Slab merging ------------ If no debug options are specified then SLUB may merge similar slabs together in order to reduce overhead and increase cache hotness of objects. slabinfo -a displays which slabs were merged together." " I couldn't even find which package provides slabinfo to run slabinfo -a! And the hand-wavvy "reduce overhead and increase cache hotness of objects" frankly sucks. > So I'm not pulling things like this from the device mapper tree. There > is just no excuse that I can see for something like SLAB_NO_MERGE to > go through the dm tree in the first place, but that's doubly true when > the rationale for these things were bogus and had nothing what-so-ever > to do with dm. As DM maintainer I do have a choice about how the subsystem is architected. > Things like this aren't supposed to come in through random irrelevant > trees like this, and with no discussion (at least judging by the > commits) with the maintainers of the other pieces of code. DM is irrelevant now? Because I pissed you off? Or because you trully think that? This is the first and hopefully last time I get flamed by you. I shouldn't have pushed for this change so aggressively. The lack of feedback from mm people shouldn't have been taken by me as implied "we forced it on you a year ago, fuck you". But I'm genuinely _not_ appreciative of this change to shared slabs so I took action to restore what I hold to be the right way to design system software. > If you have issues with slab merging, then those should be discussed > as such, not as some magical and bogus dm or xfs special case when > they damn well aren't, and damn well will never be. > > Yes, I'm annoyed. This was not done well. I realize that everybody > thinks that _their_ code is so special and exceptional that > "obviously" they should be treated specially, but I don't see that > that is the case at all in this case. > > If you want to argue that slab merging should be disabled by default, > then that is an argument that I'm willing to believe might be valid > ("the downsides are bigger than the upsides"). Or if you are able to > explain why dm really _is_ special, that's an option too. But this > kind of "random subsystems decide unilaterally to not follow the > normal rules" is not acceptable. Not when the "arguments" for it have > absolutely nothing in particular to do with that subsystem. DM isn't special. Never intended it to come off like it is. I don't want slab merging but as a middle ground I made it so it is left to each subsystem to decide to use it or not. I clearly was the first to take issue with slab merging by calling it out with patches. In doing so Dave Chinner said he'd rather avoid using shared slabs in XFS. Pretty sure XFS isn't irrelvant yet. I'd wager there would be a flood of other subsystems opting to use SLAB_NO_MERGE. I can appreciate that as something the pro-slab-merge camp would like to avoid (the more that opt-out the more useless slab merging becomes). It is messed up that no _real_ justification was given for slab merging yet it was pushed on everyone. Thankfully it hasn't been unstable (which backs up your point) but I'd still love to understand how it is so beneficial. Is it a significant win? If so where? Or is it a microoptimization at the expense of both accounting and fault isolation? Mike -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by kanga.kvack.org (Postfix) with ESMTP id AF3276B0254 for ; Wed, 2 Sep 2015 20:53:35 -0400 (EDT) Received: by pacwi10 with SMTP id wi10so27769583pac.3 for ; Wed, 02 Sep 2015 17:53:35 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id e4si38443945pdc.222.2015.09.02.17.53.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 17:53:35 -0700 (PDT) Date: Wed, 2 Sep 2015 20:53:33 -0400 From: Mike Snitzer Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Message-ID: <20150903005332.GB27804@redhat.com> References: <20150902174815.199d51a480a01e9a754367e3@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150902174815.199d51a480a01e9a754367e3@linux-foundation.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Linus Torvalds , Dave Chinner , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm On Wed, Sep 02 2015 at 8:48pm -0400, Andrew Morton wrote: > On Wed, 2 Sep 2015 16:13:44 -0700 Linus Torvalds wrote: > > > On Wed, Sep 2, 2015 at 10:39 AM, Mike Snitzer wrote: > > > > > > - last but not least: add SLAB_NO_MERGE flag to mm/slab_common and > > > disable slab merging for all of DM's slabs (XFS will also use > > > SLAB_NO_MERGE once merged). > > > > So I'm not at all convinced this is the right thing to do. In fact, > > I'm pretty convinced it shouldn't be done this way. Since those > > commits were at the top of your tree, I just didn't pull them, but > > took the rest.. > > I don't have problems with the patch itself, really. It only affects > callers who use SLAB_NO_MERGE and those developers can make > their own decisions. > > It is a bit sad to de-optimise dm for all users for all time in order > to make life a bit easier for dm's developers, but maybe that's a > decent tradeoff. > > > What I do have a problem with is that afaict the patch appeared on > linux-mm for the first time just yesterday. Didn't cc slab developers, > it isn't in linux-next, didn't cc linux-kernel or linux-mm or slab/mm > developers on the pull request. Bad! Yeap, noted. Won't happen again. > I'd like the slab developers to have time to understand and review this > change, please. Partly so they have a chance to provide feedback for > the usual reasons, but also to help them understand the effect their > design choice had on client subystems. Sure, sorry to force the issue like I did. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f178.google.com (mail-ig0-f178.google.com [209.85.213.178]) by kanga.kvack.org (Postfix) with ESMTP id BBD696B0254 for ; Wed, 2 Sep 2015 21:21:03 -0400 (EDT) Received: by igcrk20 with SMTP id rk20so38618034igc.1 for ; Wed, 02 Sep 2015 18:21:03 -0700 (PDT) Received: from mail-io0-x22f.google.com (mail-io0-x22f.google.com. [2607:f8b0:4001:c06::22f]) by mx.google.com with ESMTPS id vk4si3889147igb.97.2015.09.02.18.21.03 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 18:21:03 -0700 (PDT) Received: by iofb144 with SMTP id b144so41477845iof.1 for ; Wed, 02 Sep 2015 18:21:03 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20150903005115.GA27804@redhat.com> References: <20150903005115.GA27804@redhat.com> Date: Wed, 2 Sep 2015 18:21:02 -0700 Message-ID: Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) From: Linus Torvalds Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Mike Snitzer Cc: Dave Chinner , Christoph Lameter , Pekka Enberg , Andrew Morton , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm On Wed, Sep 2, 2015 at 5:51 PM, Mike Snitzer wrote: > > What I made possible with SLAB_NO_MERGE is for each subsystem to decide > if they would prefer to not allow slab merging. .. and why is that a choice that even makes sense at that level? Seriously. THAT is the fundamental issue here. There are absolutely zero reasons this is dm-specific, but it is equally true that there are absolutely zero reasons that it is xyzzy-specific, for any random value of 'xyzzy'. And THAT is why I'm fairly convinced that the whole approach is bogus and broken. And note that that bogosity is separate from how this was done. It's a broken approach, but it was also done wrong. Two totally separate issues, but together it sure is annoying. > From where I sit it is much more useful to have separate slabs. Could > be if a case was actually made for slab merging I'd change my view. But > as of now these trump the stated benefits of slab merging: > 1) useful slab usage stats > 2) fault isolation from other subsystems .. and again, absolutely NEITHER of those have anything to do with "subsystem X". Can you really not see how *illogical* it is to make this a subsystem choice? So explain to me why you made it so? > The 3 lines that added SLAB_NO_MERGE were pretty damn clean. No. It really seriously wasn't. The code may be simple, but it sure isn't "pretty damn clean", exactly because I think the whole concept is fundamentally illogical. See above. As I mentioned in my email: if your point is that "slab_nomerge" has the wrong default value, then that is a different discussion, and one that may well be valid. But the whole concept of "random slabs can mark themselves no-merge for no obvious reasons" is broken. That was my argument, and you don't seem to get it. And even if it turns out not to be broken (please explain), it still should have been discussed. > SLAB_NO_MERGE gives subsystems a choice they didn't have before and they > frankly probably never knew they had to care about it because they didn't > know slabs were being merged. I asked around enough to know I'm not an > idiot for having missed the memo on slab merging. Put another way: things have been merged for years, and you didn't even notice. Seriously. I'm not exaggerating about "for years". At least for slub, it's been that way since it was initially merged, back in 2007. Yeah, it may have taken a while for slub to then become one of the major allocators, but it's been the default in at least Fedora for years and years too, afaik, so it's not like slub is something odd and unusual. You seem to argue that "not being aware of it" means that it's surprising and should be disabled. But quite frankly, wouldn't you say that "it hasn't caused any obvious problems" is at _least_ as likely an explanation for you not being aware of it? Because clearly, that lack of statistics and the possible cross-subsystem corruption hasn't actually been a pressing concern in reality. But suddenly it became such a big issue that you just _had_ to fix it, right? After seven years it's suddenly *so* important that dm absolutely has to disable it. And it really had to be dm that did it for its caches, rather than just use "slab_nomerge". Despite there not being anything dm-specific about that choice. Now tell me, what was the rationale for this all again? Because really, I'm not seeing it. And I'm _particularly_ not seeing why it then had to be sneaked in like this. Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f48.google.com (mail-pa0-f48.google.com [209.85.220.48]) by kanga.kvack.org (Postfix) with ESMTP id 56ABF6B0254 for ; Wed, 2 Sep 2015 22:31:29 -0400 (EDT) Received: by pacwi10 with SMTP id wi10so30217923pac.3 for ; Wed, 02 Sep 2015 19:31:29 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id x5si38771934pbt.100.2015.09.02.19.31.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 19:31:28 -0700 (PDT) Date: Wed, 2 Sep 2015 22:31:25 -0400 From: Mike Snitzer Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Message-ID: <20150903023125.GC27804@redhat.com> References: <20150903005115.GA27804@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Heinz Mauelshagen , Andrew Morton , Viresh Kumar , Dave Chinner , Joe Thornber , Pekka Enberg , linux-mm , "dm-devel@redhat.com" , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , David Rientjes , Joonsoo Kim , Christoph Lameter , Alasdair G Kergon On Wed, Sep 02 2015 at 9:21pm -0400, Linus Torvalds wrote: > On Wed, Sep 2, 2015 at 5:51 PM, Mike Snitzer wrote: > > > > What I made possible with SLAB_NO_MERGE is for each subsystem to decide > > if they would prefer to not allow slab merging. > > .. and why is that a choice that even makes sense at that level? > > Seriously. > > THAT is the fundamental issue here. > > There are absolutely zero reasons this is dm-specific, but it is > equally true that there are absolutely zero reasons that it is > xyzzy-specific, for any random value of 'xyzzy'. > > And THAT is why I'm fairly convinced that the whole approach is bogus > and broken. Why do we even have slab creation flags? Andrew seemed much more reasonable about this. > And note that that bogosity is separate from how this was done. It's a > broken approach, but it was also done wrong. Two totally separate > issues, but together it sure is annoying. > > > From where I sit it is much more useful to have separate slabs. Could > > be if a case was actually made for slab merging I'd change my view. But > > as of now these trump the stated benefits of slab merging: > > 1) useful slab usage stats > > 2) fault isolation from other subsystems > > .. and again, absolutely NEITHER of those have anything to do with > "subsystem X". OK, I get that I'm unimportant. You can stop beating me over my irrelevant subsystem maintainer head now... But when longstanding isolation and functionality is removed in the name of microoptimizations its difficult to accept -- even if the realization occurs years after the fact. > Can you really not see how *illogical* it is to make this a subsystem choice? > > So explain to me why you made it so? > > > The 3 lines that added SLAB_NO_MERGE were pretty damn clean. > > No. It really seriously wasn't. > > The code may be simple, but it sure isn't "pretty damn clean", exactly > because I think the whole concept is fundamentally illogical. See > above. Yeah, your circular logic doesn't help me. You defined your argument in terms of unsubstantiated claims of me being illogical. What is illogical about wanting DM to: 1) have useful slab accounting 2) have fault isolation from other slab consumers 3) not impose 1+2 on all other subsystems ? I guess I'm just supposed to accept that slab merging is or isn't. There is no in-between (unless I create a slab with SLAB_DESTROY_BY_RCU) > As I mentioned in my email: if your point is that "slab_nomerge" has > the wrong default value, then that is a different discussion, and one > that may well be valid. > > But the whole concept of "random slabs can mark themselves no-merge > for no obvious reasons" is broken. That was my argument, and you don't > seem to get it. I'm not getting it because I don't understand why you really care. What implied benefits come with slab merging that I'm painfully unaware of? Andrew said DM would miss out on performance benefits. I'd obviously not want to do that; but said performance benefits haven't been made apparent. > And even if it turns out not to be broken (please explain), it still > should have been discussed. See above ;) > > SLAB_NO_MERGE gives subsystems a choice they didn't have before and they > > frankly probably never knew they had to care about it because they didn't > > know slabs were being merged. I asked around enough to know I'm not an > > idiot for having missed the memo on slab merging. > > Put another way: things have been merged for years, and you didn't even notice. > > Seriously. I'm not exaggerating about "for years". At least for slub, > it's been that way since it was initially merged, back in 2007. > Yeah, it may have taken a while for slub to then become one of the > major allocators, but it's been the default in at least Fedora for > years and years too, afaik, so it's not like slub is something odd and > unusual. You're also coming at this from a position that shared slabs are automatically good because they have been around for years. For those years I've not had a need to debug a leak in code I maintain; so I didn't notice slabs were merged. I also haven't observed slab corruption being the cause of crashes in DM, block or SCSI. > You seem to argue that "not being aware of it" means that it's > surprising and should be disabled. But quite frankly, wouldn't you say > that "it hasn't caused any obvious problems" is at _least_ as likely > an explanation for you not being aware of it? Sure. > Because clearly, that lack of statistics and the possible > cross-subsystem corruption hasn't actually been a pressing concern in > reality. Agreed. > But suddenly it became such a big issue that you just _had_ to fix it, > right? After seven years it's suddenly *so* important that dm > absolutely has to disable it. And it really had to be dm that did it > for its caches, rather than just use "slab_nomerge". The ship sailed on disabling it for everyone. It is the new norm. I cannot push RHEL to flip-flop slab characteristics (at least not until the next major release). > Despite there not being anything dm-specific about that choice. > > Now tell me, what was the rationale for this all again? I was the first to want the option to opt-out on a per slab basis. And you're shooting the messenger. Calling me illogical. > Because really, I'm not seeing it. And I'm _particularly_ not seeing > why it then had to be sneaked in like this. And I'm a sneaky too... Sneaking isn't what this was. Apologies if that's how it came off. I can appreciate why you might think that. But like I said to Andrew: won't happen again. I'm off the next 5 days. I don't think either of us care _that_ strongly about this particular issue. I've noted my process flaws. I'll calm down and this will just be some unfortunate thing that happened. But I'd still like some pointers/help on what makes slab merging so beneficial. I'm sure Christoph and others have justification. But if not then yes the default to slab merging probably should be revisited. Mike -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f181.google.com (mail-io0-f181.google.com [209.85.223.181]) by kanga.kvack.org (Postfix) with ESMTP id B31ED6B0254 for ; Wed, 2 Sep 2015 23:10:16 -0400 (EDT) Received: by ioiz6 with SMTP id z6so43357487ioi.2 for ; Wed, 02 Sep 2015 20:10:16 -0700 (PDT) Received: from resqmta-po-09v.sys.comcast.net (resqmta-po-09v.sys.comcast.net. [2001:558:fe16:19:96:114:154:168]) by mx.google.com with ESMTPS id q76si9064983ioe.102.2015.09.02.20.10.15 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Wed, 02 Sep 2015 20:10:15 -0700 (PDT) Date: Wed, 2 Sep 2015 22:10:12 -0500 (CDT) From: Christoph Lameter Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) In-Reply-To: <20150903023125.GC27804@redhat.com> Message-ID: References: <20150903005115.GA27804@redhat.com> <20150903023125.GC27804@redhat.com> Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Mike Snitzer Cc: Linus Torvalds , Heinz Mauelshagen , Andrew Morton , Viresh Kumar , Dave Chinner , Joe Thornber , Pekka Enberg , linux-mm , "dm-devel@redhat.com" , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , David Rientjes , Joonsoo Kim , Alasdair G Kergon On Wed, 2 Sep 2015, Mike Snitzer wrote: > You're also coming at this from a position that shared slabs are > automatically good because they have been around for years. > > For those years I've not had a need to debug a leak in code I maintain; > so I didn't notice slabs were merged. I also haven't observed slab > corruption being the cause of crashes in DM, block or SCSI. Hmmm... Thats unusual. I have seen numerous leaks and corruptions that were debugged using the additional debug code in the slab allocators. Merging and debugging can be switched on at runtime if necessary and then you will have a clear separation to be able to track down the offending code as well as detailed problem reports that help to figure out what was wrong. It is then typically even possible to fix these bugs without getting the subsystem specialists involved. > > Because clearly, that lack of statistics and the possible > > cross-subsystem corruption hasn't actually been a pressing concern in > > reality. > > Agreed. To the effect now that even SLAB has adopted cache merging. > But I'd still like some pointers/help on what makes slab merging so > beneficial. I'm sure Christoph and others have justification. But if > not then yes the default to slab merging probably should be revisited. Well, we have discussed the pros and cons for merging a couple of times but the general consensus was that it is beneficial. Performance on modern cpu is very sensitive to cache footprint and reducing the overhead of meta data for object allocation is a worthwhile goal. Also objects are more likely to be kept cache hot if they can be used by multiple subsystems. Slab merging also helps with reducing fragmentation since the free objects on one page can be used for other purposes. Check out the linux-mm archives for these dissussions. This has been such an advantage that the feature was ported to SLAB (to much more signficant effect than SLUB since SLAB is a pig with metadata per node, per cpu and per kmem_cache). And yes sorry the consequence is now that you do no longer have a choice. Both slab allocators default to merging. SLAB had some difficulty staying competitive in performance without that. Joonsoo Kim made SLAB more competitive last year and one of the optimizations was to also support merging. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f177.google.com (mail-io0-f177.google.com [209.85.223.177]) by kanga.kvack.org (Postfix) with ESMTP id 9B8196B0254 for ; Wed, 2 Sep 2015 23:12:00 -0400 (EDT) Received: by ioiz6 with SMTP id z6so43388293ioi.2 for ; Wed, 02 Sep 2015 20:12:00 -0700 (PDT) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com. [2607:f8b0:4001:c05::231]) by mx.google.com with ESMTPS id q77si20314233ioe.201.2015.09.02.20.11.59 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 20:11:59 -0700 (PDT) Received: by igcrk20 with SMTP id rk20so39999462igc.1 for ; Wed, 02 Sep 2015 20:11:59 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20150903023125.GC27804@redhat.com> References: <20150903005115.GA27804@redhat.com> <20150903023125.GC27804@redhat.com> Date: Wed, 2 Sep 2015 20:11:59 -0700 Message-ID: Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) From: Linus Torvalds Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Mike Snitzer Cc: Heinz Mauelshagen , Andrew Morton , Viresh Kumar , Dave Chinner , Joe Thornber , Pekka Enberg , linux-mm , "dm-devel@redhat.com" , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , David Rientjes , Joonsoo Kim , Christoph Lameter , Alasdair G Kergon On Wed, Sep 2, 2015 at 7:31 PM, Mike Snitzer wrote: > > Why do we even have slab creation flags? Ehh? Because they are meaningful? Things like SLAB_DESTROY_BY_RCU have real semantic meaning. The subsystem that creates the slab *cares*, and it makes sense because that kind of choice really fundamentally is a per-slab choice. >> .. and again, absolutely NEITHER of those have anything to do >> "subsystem X". > > OK, I get that I'm unimportant. You can stop beating me over my > irrelevant subsystem maintainer head now... What the hell is your problem? At no point did I state that you are any less important than anything else. But this isssue is simply not in any way specific to dm. dm is not any less important than anything else, but dm is also not magically *more* important than everything else. Really. Then you seem to take it personally, but please realize that that is *your* issue, not mine. > But when longstanding isolation and functionality is removed in the name > of microoptimizations its difficult to accept -- even if the realization > occurs years after the fact. Bullshit. You didn't notice. For years. It just wasn't important. Just admit it. Those things you now tout as so important are complete non-issues. But more importantly, and this is what you seem to not really get at all, is that it's STILL not dm-specific. If you think that isolation is so important, then tell me why isolation is only important for dm? Why isn't it important for everything else? What makes dm so special? Really. I've asked you three times now, and you seem to not get it, you just think I'm trying to put you in your place. I'm not. I'm asking a serious question: what makes dm so special that it has to have different allocation logic from everything else. And *THAT* is why SLAB_DESTROY_BY_RCU is different from your SLAB_NO_MERGE. Because I can actually answer the question: "What makes sighand_cachep need SLAB_DESTROY_BY_RCU but not other users?" with a real technical reason. > I'm not getting it because I don't understand why you really care. What > implied benefits come with slab merging that I'm painfully unaware of? It does actually have less overhead, for one thing. The separation of slabs doesn't cost you just in the slab data structure itself, but in the memory fragmentation. Having multiple slabs share the backing pool of pages uses less memory. > You're also coming at this from a position that shared slabs are > automatically good because they have been around for years. No, I'm really not. Christ, have you read anything I wrote? I'm ok with discussing the "the defaults should be turned around". But at least we *have* an option to turn that default around, so when people care (because they are trying to chase down a slab corruption issue, for example), they can do so. Your patch actually gets rid of that choice, and forces things the other way around. So I would argue that your patch actually makes things *worse*. It hardcodes an arbitrary choice, and it's not even a choice that makes obvious sense. And no, the memory fragmentation issue isn't just made up. One of the downsides of slab was historically that it used a lot of memory, and to be honest, I suspect the percpu queues have made things worse. At least sharing the backing store minimizes the effect of that somewhat. We used to have numbers for this all, but it's really approaching a decade since the whole initial SLUB vs SLAB things, so I don't know where to point you. But the reason I say "it's not a choice that makes obvious sense" isn't even because I'm convinced that the merging is always the best option. I *am* convinced that it has real upsides, but I also agree that it has downsides. But at least as it is right now, the system admin can make a choice. You arbitrarily wanted to take that choice away for dm, without apparently even knowing what the upsides of merging might be. But the *real* issue I have with it is the completely random "dm is different from everything else" thing. Which is bogus. That's what I wanted to know: what makes dm so special that it should be different from everything else? And apparently you don't have an answer to that. You just took my repeated questioning to mean that you're worthless. That wasn't the intent. It was very much a literal "what's so different about dm that it would act differently from everything else"? > The ship sailed on disabling it for everyone. It is the new norm. I > cannot push RHEL to flip-flop slab characteristics (at least not until > the next major release). But you can. Today. Put "slab_nomerge" on the kernel command line. Really. If you care, you can do that. And if you _don't_ care, then clearly not doing that doesn't hurt either. > I was the first to want the option to opt-out on a per slab basis. And > you're shooting the messenger. Calling me illogical. But the opt-in shouldn't be *you*, it should be the system maintainer who can actually tune for his load, or cares about memory use, or wants to debug, or any number of issues. See? Btw, I do agree that the "all or nothing" approach of "slab_nomerge" isn't optimal. But you made things *worse*. You took a tunable, and made it non-tunable, without apparently even knowing what it tuned for. Sure, it was a damn coarse-grained tunable, but you made *that* worse too, since with your code it's not tunable at all for dm. So your version isn't actually any more "fine-grained". Now, what might be interesting - *if* people actually want to tune just one set of slabs and not another - migth be to extend the "slab_nomerge" option to actually take a pattern of slab names, and match that way. So then you could say "slab_nomerge=dm_* slab_nomerge=xfs*", and you'd not merge dm or xfs slabs. I wouldn't mind that kind of approach at all. But please understand _why_ I wouldn't mind it: I wouldn't mind it exactly because you didn't take tuning choice away from people, but because such a patch would actually give people control of it. And it *wouldn't* be dm-specific, because other people might ask to not merge ext4 slabs or whatever. And for a similar reason, I actually wouldn't mind switching the default around for merging. I'm *not* married to the "we have to merge slab caches by default" model. It used to make sense, and I know I've seen numbers (I'm pretty sure Christoph Lameter had several talks about it back in the days), but things can change. But what doesn't make sense is to make random willy-nilly decisions on a basis that makes no sense. And I do claim that random subsystems just unilaterally deciding that they don't care about system default memory management falls under that "makes no sense" heading. Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) by kanga.kvack.org (Postfix) with ESMTP id 9F5CF6B0254 for ; Thu, 3 Sep 2015 00:55:15 -0400 (EDT) Received: by igbni9 with SMTP id ni9so31043026igb.0 for ; Wed, 02 Sep 2015 21:55:15 -0700 (PDT) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org. [140.211.169.12]) by mx.google.com with ESMTPS id y10si4336079igf.95.2015.09.02.21.55.14 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 21:55:15 -0700 (PDT) Date: Wed, 2 Sep 2015 21:55:12 -0700 From: Andrew Morton Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Message-Id: <20150902215512.9d0d62e74fa2f0a460a42af9@linux-foundation.org> In-Reply-To: References: <20150903005115.GA27804@redhat.com> <20150903023125.GC27804@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Christoph Lameter Cc: Mike Snitzer , Linus Torvalds , Heinz Mauelshagen , Viresh Kumar , Dave Chinner , Joe Thornber , Pekka Enberg , linux-mm , "dm-devel@redhat.com" , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , David Rientjes , Joonsoo Kim , Alasdair G Kergon On Wed, 2 Sep 2015 22:10:12 -0500 (CDT) Christoph Lameter wrote: > > But I'd still like some pointers/help on what makes slab merging so > > beneficial. I'm sure Christoph and others have justification. But if > > not then yes the default to slab merging probably should be revisited. > > ... > > Check out the linux-mm archives for these dissussions. Somewhat OT, but... The question Mike asks should be comprehensively answered right there in the switch-to-merging patch's changelog. The fact that it is not answered in the appropriate place and that we're reduced to vaguely waving at the list archives is a fail. And a lesson! -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) by kanga.kvack.org (Postfix) with ESMTP id 61D8C6B0254 for ; Thu, 3 Sep 2015 02:03:01 -0400 (EDT) Received: by iofb144 with SMTP id b144so46679425iof.1 for ; Wed, 02 Sep 2015 23:03:01 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id bz5si39731601pdb.0.2015.09.02.23.03.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 23:03:00 -0700 (PDT) Date: Thu, 3 Sep 2015 16:02:47 +1000 From: Dave Chinner Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) Message-ID: <20150903060247.GV1933@devil.localdomain> References: <20150903005115.GA27804@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Mike Snitzer , Christoph Lameter , Pekka Enberg , Andrew Morton , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com" , Alasdair G Kergon , Joe Thornber , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , Viresh Kumar , Heinz Mauelshagen , linux-mm On Wed, Sep 02, 2015 at 06:21:02PM -0700, Linus Torvalds wrote: > On Wed, Sep 2, 2015 at 5:51 PM, Mike Snitzer wrote: > > > > What I made possible with SLAB_NO_MERGE is for each subsystem to decide > > if they would prefer to not allow slab merging. > > .. and why is that a choice that even makes sense at that level? > > Seriously. > > THAT is the fundamental issue here. It makes a lot more sense than you think, Linus. One of the reasons slab caches exist is to separate objects of identical characteristics from the heap allocator so that they are all grouped together in memory and so can be allocated/freed efficiently. This helps prevent heap fragmentation, allows objects to pack as tightly together as possible, gives direct measurement of the number of objects, the memory usage, the fragmentation factor, etc. Containment of memory corruption is another historical reason for slab separation (proof: current memory debugging options always causes slab separation). Slab merging is the exact opposite of this - we're taking homogenous objects and mixing them with other homogneous containing different objects with different life times. Indeed, we are even mixing them back into the slabs used for the heap, despite the fact the original purpose of named slabs was to separate allocation from the heap... Don't get me wrong - this isn't necessarily bad - but I'm just pointing out that slab merging is doing the opposite of what slabs were originally intended for. Indeed, a lot of people use slab caches just because it's anice encapsulation, not for any specific performance, visibility or anti-fragmentation purposes. I have no problems with automatically merging slabs created like this. However the fact that we are merging slabs automatically for all slabs now has made me think a bit deeper about the problems that can result from this. > There are absolutely zero reasons this is dm-specific, but it is > equally true that there are absolutely zero reasons that it is > xyzzy-specific, for any random value of 'xyzzy'. Right, it's not xyzzy-specific where 'xyzzy' is a subsystem. The flag application is actually *object specific*. That is, the use of the individual objects that determines whether it should be merged or not. e.g. Slab fragmentation levels are affected more than anything by mixing objects with different life times in the same slab. i.e. if we free all the short lived objects from a page but there is one long lived object on the page then that page is pinned and we free no memory. Do that to enough pages in the slab, and we end up with a badly fragmented slab. With slab merging, we have no control over what slabs are merged. We may be merging slabs with objects that have vastly different life times. Hence merging may actually be making one of the underlying cause of slab fragmentation worse rather than better. It really depends on what slabs get merged together and that's largely random chance - you don't get to pick the size of your structures.... Another contributor to slab fragmentation is when allocation order is very different to object freeing order. Pages in the slab get fill up using an algorithm that optimises for temporal locality. i.e. it will fill a partial page before moving on to the next partial page or allocating a new page. If the freeing of objects doesn't have the same temporal locality as allocation then when the slab grows and shrinks we end up with fragmentation. Mixing different object types into the same pages pretty much guarantees that we'll be mixing objects of different alloc/freeing order. Further, rapid growth and shrinking of a slab cache due to memory demand can cause fragmentation. Caches that have this problem are usually those that have a shrinker associated with them. The shrinker causes objects to have a variable, unpredictable lifetime and hence can break allocation/freeing locality (as per above, even for single object slabs). Minmising the effect of this reclaim fragmentation is often held up as the example of why slab merging is good - the other object types fill all the holes and hence reduces the overall fragmentation of the slab. Further, the density of the reclaimable objects is lower, so the slab doesn't fragment as much. On the surface, this looks like a big win but it's not - it's actually a major problem for slab reclaim and it manifests when there are large bursts of allocation activity followed by sudden reclaim activity. When the slab grows rapidly, we get the majority of objects on a page being of one type, but a couple will be of a different type. Than under memory pressure, the shrinker can then only free the majority of objects on a page, guaranteeing the slab will remain fragmented under memory pressure. Continuing to run the shrinker won't result in any more memory being freed from the merged slab and so we are stuck with unfixable slab fragmentation. However, if the slab with a shrinker only contains one kind of object, when it becomes fragmented due to variable object lifetime, continued memory pressure will cause it to keep shrinking and hence will eventually correct the fragmentation problem. This is a much more robust configuration - the system will self correct without user intervention being necessary. IOWs, slab merging prevents us from implementing effective active fragmentation management algorithms and hence prevents us from reducing slab fragmentation via improved shrinker reclaim algorithms. Simply put: slab merging reduces the effectiveness of shrinker based slab reclaim. A key observation I just made: we are extremely lucky that many of the critical slab caches in the system are not affected by merging. A slab cache with a constructor will not get merged and that means inode caches do not get merged. Hence, despite slab merging being enabled, one of the largest memory consuming slabs in the system does not get merged and hence it means the shrinker has been able to do it's job without interference. hence we've avoided the worst outcome of merging slabs by default by luck rather than good managment. Moving on from fragmentation: Slab caches can also back mempools. mempools ar eused to guarantee forwards progress under memory pressure, so it's important to have visibility into their behaviour. Hence it makes sense to ensure these don't get merged with other slabs so they are accounted accurately and we can see exactly the demand being placed on these critical slabs under heavy memory pressure. I've made use of this several times over the past few years to discover why a system is floundering under heavy memory pressure (e.g. writeback way slower than it should have been because the xfs_ioend mempool was operating in 1-in, 1-out mode)... So, when I said that I could use the SLAB_NO_MERGE for some caches in XFS and acked the patch, I was refering to exactly this sort of usage - the slabs that back mempools and the slabs that have a shrinker for reclaim should have this flag set. 4 of 17 named slabs in XFS need this flag - the rest I don't really care about because their memory usage can be inferred from the shrinkable slab cache sizes. Managing slab caches and fragmentation is anything but simple and there is no one right solution. Slab merging in some cases makes sense, but there are several very good reasons for not merging a slab. The right solution is often difficult for people without object-specific expertise to understand, but that goes for just about everything in the kernel these days. BTW, it is trivial to achieve SLAB_NO_MERGE simply by supplying a dummy constructor to the slab initialisation. I'd much prefer SLAB_NO_MERGE or some variant, though. Cheers, Dave. -- Dave Chinner dchinner@redhat.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) by kanga.kvack.org (Postfix) with ESMTP id 252A86B0254 for ; Thu, 3 Sep 2015 02:09:25 -0400 (EDT) Received: by ioiz6 with SMTP id z6so46519523ioi.2 for ; Wed, 02 Sep 2015 23:09:24 -0700 (PDT) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com. [2607:f8b0:4001:c05::22d]) by mx.google.com with ESMTPS id m185si4603064iom.21.2015.09.02.23.09.24 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 23:09:24 -0700 (PDT) Received: by igcpb10 with SMTP id pb10so6966893igc.1 for ; Wed, 02 Sep 2015 23:09:24 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20150902215512.9d0d62e74fa2f0a460a42af9@linux-foundation.org> References: <20150903005115.GA27804@redhat.com> <20150903023125.GC27804@redhat.com> <20150902215512.9d0d62e74fa2f0a460a42af9@linux-foundation.org> Date: Thu, 3 Sep 2015 09:09:24 +0300 Message-ID: Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) From: Pekka Enberg Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Christoph Lameter , Mike Snitzer , Linus Torvalds , Heinz Mauelshagen , Viresh Kumar , Dave Chinner , Joe Thornber , linux-mm , "dm-devel@redhat.com" , Mikulas Patocka , Vivek Goyal , Sami Tolvanen , David Rientjes , Joonsoo Kim , Alasdair G Kergon Hi Andrew, On Wed, 2 Sep 2015 22:10:12 -0500 (CDT) Christoph Lameter wrote: >> > But I'd still like some pointers/help on what makes slab merging so >> > beneficial. I'm sure Christoph and others have justification. But if >> > not then yes the default to slab merging probably should be revisited. >> >> ... >> >> Check out the linux-mm archives for these dissussions. On Thu, Sep 3, 2015 at 7:55 AM, Andrew Morton wrote: > Somewhat OT, but... The question Mike asks should be comprehensively > answered right there in the switch-to-merging patch's changelog. > > The fact that it is not answered in the appropriate place and that > we're reduced to vaguely waving at the list archives is a fail. And a > lesson! Slab merging is a technique to reduce memory footprint and memory fragmentation. Joonsoo reports 3% slab memory reduction after boot when he added the feature to SLAB: commit 12220dea07f1ac6ac717707104773d771c3f3077 Author: Joonsoo Kim Date: Thu Oct 9 15:26:24 2014 -0700 mm/slab: support slab merge Slab merge is good feature to reduce fragmentation. If new creating slab have similar size and property with exsitent slab, this feature reuse it rather than creating new one. As a result, objects are packed into fewer slabs so that fragmentation is reduced. Below is result of my testing. * After boot, sleep 20; cat /proc/meminfo | grep Slab Slab: 25136 kB Slab: 24364 kB We can save 3% memory used by slab. For supporting this feature in SLAB, we need to implement SLAB specific kmem_cache_flag() and __kmem_cache_alias(), because SLUB implements some SLUB specific processing related to debug flag and object size change on these functions. Signed-off-by: Joonsoo Kim Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds We don't have benchmarks to directly measure its performance impact but you should see its effect via something like netperf that stresses the allocator heavily. The assumed benefit is that you're able to recycle cache hot objects much more efficiently as SKB cache and friends are merged to regular kmalloc caches. In any case, reducing kernel memory footprint already is a big win for various use cases, so keeping slab merging on by default is desirable. - Pekka -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f180.google.com (mail-io0-f180.google.com [209.85.223.180]) by kanga.kvack.org (Postfix) with ESMTP id 8F3756B0254 for ; Thu, 3 Sep 2015 02:13:52 -0400 (EDT) Received: by ioiz6 with SMTP id z6so46604797ioi.2 for ; Wed, 02 Sep 2015 23:13:52 -0700 (PDT) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com. [2607:f8b0:4001:c05::231]) by mx.google.com with ESMTPS id 194si12742497ioo.107.2015.09.02.23.13.52 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Sep 2015 23:13:52 -0700 (PDT) Received: by igbkq10 with SMTP id kq10so41868774igb.0 for ; Wed, 02 Sep 2015 23:13:51 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20150903060247.GV1933@devil.localdomain> References: <20150903005115.GA27804@redhat.com> <20150903060247.GV1933@devil.localdomain> Date: Thu, 3 Sep 2015 09:13:51 +0300 Message-ID: Subject: Re: slab-nomerge (was Re: [git pull] device mapper changes for 4.3) From: Pekka Enberg Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Dave Chinner Cc: Linus Torvalds , Mike Snitzer , Christoph Lameter , Andrew Morton , David Rientjes , Joonsoo Kim , "dm-devel@redhat.com"