From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932366AbcHSXBK (ORCPT ); Fri, 19 Aug 2016 19:01:10 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:30786 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753576AbcHSXBH (ORCPT ); Fri, 19 Aug 2016 19:01:07 -0400 Date: Fri, 19 Aug 2016 15:59:38 -0700 From: Dave Watson To: Josh Triplett CC: Andi Kleen , Linus Torvalds , Mathieu Desnoyers , Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , Andy Lutomirski , "Linux Kernel Mailing List" , Linux API , Paul Turner , Andrew Morton , Russell King , "Thomas Gleixner" , Ingo Molnar , "H. Peter Anvin" , Andrew Hunter , Chris Lameter , Ben Maurer , Steven Rostedt , Catalin Marinas , Will Deacon , Michael Kerrisk Subject: Re: [RFC PATCH v8 1/9] Restartable sequences system call Message-ID: <20160819225938.GC1228@davejwatson-mba.local> References: <1471637274-13583-1-git-send-email-mathieu.desnoyers@efficios.com> <1471637274-13583-2-git-send-email-mathieu.desnoyers@efficios.com> <20160819205611.GF5871@two.firstfloor.org> <20160819212423.623v5leqknfrxhpd@x> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20160819212423.623v5leqknfrxhpd@x> User-Agent: Mutt/1.6.0 (2016-04-01) X-Originating-IP: [2620:10d:c090:180::1:c0f5] X-ClientProxiedBy: BN6PR04CA0001.namprd04.prod.outlook.com (10.172.194.11) To BL2PR15MB0980.namprd15.prod.outlook.com (10.167.116.136) X-MS-Office365-Filtering-Correlation-Id: f54ff877-3206-41c4-a4e4-08d3c88481d2 X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0980;2:+vM0PnZMfJlbfw3eXffgdNv+yZecPKppGnrFwLyOk1+QEEZik8pDrqnfdS9dHr4l6HR+DrcSjUqCsgM6388mG7NJurfzFcE6ozwxAxcnTq4l4LEqb26/NI4uByfJdLaoSV9y6CBaWtWRvK1Chg0DTfGNB9q1J72154lYLp5xOEWz2gmLoiej+YbMij+bpv8j;3:CE/H/gP0DURnPGsgZl+my78972DjW9SIFJG7NWTgSUp0h7965ee88F8CWtFkoHDq5MSU34sbs5LUMDKl2klir/L+2Ah7aHgA6ZeBWAkKF18vg7j8sfCfn4kZhpD+dEn4 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BL2PR15MB0980; X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0980;25:mJsMPAljMV49T4GSeJ123Qxtb7zep7P8CDV/mp/I2Ojn69/GsaT7pyW9APv444ExCNUAWXRpBRN0jns1D8OsRIlC/DNNpkojUrwQ/ID63+o78s8ag0i/G2jvmThHkcJ0rZDKIW4qXuk+qoqaUMwZaxxf2ZkYJp+AyjhM78gvbrVWHsw53SDtUh9Xw67pROt8FKkG3xQkGnsdKeqn6sAPjIt25GrBYE4xr/pCHmYIEmTT1lFRA8gLZo3AlAvRgDAi49tDIFt5oBVpkz/3JSwrBKzxSiuoTd9hOUsh2tw97Z67wU/W7/9WOw81d7DQET6J8cC1fi506rV5gOcSiq5nT5WwBBWaoR8H/wr80l1WgXzkyTeJ+KEVLWXk7ND4eLJwJGqI1W0ChPKpE0/X5M+kqpQ+sYHETSAZ6424ezIqS/GTHkBoPf+ked6fQFnIFZr+HUoA81WJRiVlLk0BsCO/Vkt0ekPR1S1CWVdm+UMeFSSg3NUNXSP6Sc69Vzu+NarscGuqdSWXuV0bb71QB6vUPWbXSx9BB+bxBWaFmGInbNpPqfo8U+xBW+aYbRD+QcqpNHSJS6xUepglpV++7QM4+0xYrrD1EjM3dIL8UbO8QvWDjmzvpyd/xQX6wEqZf4UqmrFVIYxdO0diZI5+1Iugxk09mB/PD9GsyqDpk3cRLGxJ37IjGh2jvERONfP83WvOEjFRPPKya6t9dg8mgojyeIFYRiwsdO40uc/O63HFtwLjkHekch7A344THDGJclW9k8Hf7HtKWWOrakrPVu9LzJLLZx89SsxbWAZ/lnA02HOore/Se6PEQlHRj3l1SXC0hPivFJqzKDICiXdxA0d7KAsPbhYf5wAEZkA71Zk1MXo= X-LD-Processed: 8ae927fe-1255-47a7-a2af-5f3a069daaa2,ExtAddr X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0980;31:vE0vwzj27rlbiqcBZNGZpiU0i0aDiKt787BQVSr5DepOP9/5DDJZM76ySj0dJqewJmCp7IOr/gbhdJQDN3u0KQlcrnKCUap2+JoPgSoI7fc2cOpO21IYao8os0QEdoz4vHj6Fe3hVkwZwG4fxn6clkAObstenlf31QCoAruwdVEgJAQ339DuQ1HhC3Ch9WPmfXvFyflWjfxjzMVEzT+/drTmOVy2v5l54BVDd3ovWEw=;20:wvq89gyNTX7vncHAtBgtQNQeLy3eG8c5VlMZB/i5M5OmEVFY2f+bBxb+vAQYdOjO9XtZQqf7w6+Cjg+1qje47rJq96lMksod6HKDCjKZa52sJ1v/IZpmORD7QXHeSZpeiTEu/v1DDCEIfknDYq1hY5e0hyFBjBYgcyrSuhvv+gA=;4:Q92v8Kn7KT2Sm9hz/GY7VT2O9rdBvMdSNgTuwgG6588PO3LI9M8abyidXAcTy1MhfsRY24vFFFZWKkEy2ExLMvESvpl2sz5TEj2gV5Sw4W/pX2ZE2gczl1tHiZQvbcmQ819q+8+WUpeuOV6C0VdtzgvZMTobczQRm9bJmIB1fgjWbikq3Ly/NdBArXNQM7eyPE2tonpdUIDV0gDCD5yKuuyPLdLUWGMN+lUAxU076ti0UKnaV7uEkJBv8IM0v568hj6VpRDGxJFiXTIIppsKvWhwpbLLj7VIqJEW0mie/iJ8AFAI3F84/peJE0hcwQpAkg59QB4RmuN8anla+zZnSE9vrcqig+ZXOQFyXmQWmzE8dwkjgEjn7ROwRv8lj66JtK/grbFeat8W7YCoG5IQzfWsXeVpVC+00QWIv3V+orRJtcZQn5TvUc/qlvozocCV X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(166708455590820); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:BL2PR15MB0980;BCL:0;PCL:0;RULEID:;SRVR:BL2PR15MB0980; X-Forefront-PRVS: 0039C6E5C5 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6069001)(7916002)(377454003)(24454002)(199003)(189002)(33656002)(76506005)(93886004)(9686002)(110136002)(76176999)(7416002)(54356999)(189998001)(97756001)(42186005)(68736007)(106356001)(47776003)(7846002)(81166006)(5660300001)(4001350100001)(7736002)(98436002)(81156014)(8676002)(305945005)(86362001)(77096005)(97736004)(105586002)(92566002)(23726003)(19580395003)(101416001)(1076002)(83506001)(586003)(6116002)(4326007)(50986999)(2906002)(2950100001)(15975445007)(46406003)(575784001)(50466002)(18370500001)(3826002)(217873001);DIR:OUT;SFP:1102;SCL:1;SRVR:BL2PR15MB0980;H:localhost;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;BL2PR15MB0980;23:g27+HR1dll4X44Mtppj4M74dQZI0gw4BSkmP4gsJ3?= =?us-ascii?Q?7Koy77Xojnj5EZZEEvHpKpZ8tvc7NGXEKJndoudjlwWNTuv0+sWxd2iEjd3v?= =?us-ascii?Q?YlcMhTx7+HivnJlOrHp7BWvjbtfAdanG67jz4aJHTbaeq6jRkJkImr3gUxQl?= =?us-ascii?Q?zn/0131OxmRC4UjYqgTuGUIMqhB3zs14DDZcagtld1TvqqJ9zTAUhaNdUxl2?= =?us-ascii?Q?QR9lZbJbpUoQnvkMET0cdGiIeJIEsyveifp1VM9lT/G2/+fjoK9+PVafNqvg?= =?us-ascii?Q?MzK3DmOyBwzxyaxZBALfGqOUP8js6+BGBdjMxvUx84B+CYPUr7QeEmjSAJ3C?= =?us-ascii?Q?junr+kyYfhYYA2Mw/oBV8kYH6Y7bMn2RNvHPpj6kU1z2r+n0s8exxZ6N6LS7?= =?us-ascii?Q?y/8hvmKhVHUdzZiDnS1ziui/CFtEdQSUgiWL2zAH1v8OLDM4K1zdW0JHEhgV?= =?us-ascii?Q?649duWCcKAMp+f58TdaRylW1EBSzyjiKZjE3ht3PdTY1ASPPUCyAY0/QzqFV?= =?us-ascii?Q?9Trd9o/VJ1HA2yOOhD4I0Tt4FPhpcFYLz52EaBcJ+cjJd9gIpcXqu4FJVFzt?= =?us-ascii?Q?addAixjQIyRKCWeXLixfBqGwZljAasHhNkF/K24s/WW2f5P8J1JNrX5wJ4Ss?= =?us-ascii?Q?F43s6DEBBWwSOloAb1/8u7o9ajKPn5gipORKWHaXTH9eEhZybPjb4H20K0pt?= =?us-ascii?Q?cB+wObhWy70Cn+bSWkqN1y5BdkPyzk4oaMIC5WgBDbdER6p2ac5HWWYOTUB8?= =?us-ascii?Q?s73BSEaludD5abIOxV0p8dRre6mIBhdzRyFPrChK9vzwHHRqGpPThZlB9uj0?= =?us-ascii?Q?nD9pWAwvE8x8378RqC++SErBsSelixI/m3u2dHL+qmfgCMnc1mMYSWKjuhW0?= =?us-ascii?Q?c0B+OknUNcZG7keaFBam/lDUWlihZ/2M+VsdMO7MTzSUvX3/eGtjf9qk4ULS?= =?us-ascii?Q?jf6804VzUdL6vM8pEvHVNPhS5dUGrkXeW0q5hPOH9WpNg0HPzEaduWPJZ5da?= =?us-ascii?Q?dFsiksKocDKlfM1c7fhdyMFEVXpob648oyj1bpWdKRBzpQkMONuZzw/cLrhW?= =?us-ascii?Q?cOF6xkvyUBdz0icIFmKXnzJOATvtDH/sivGISdz0zN9/WyDH91hoBqNKhn57?= =?us-ascii?Q?6/LVwrTzE930M3Pz66SXEVaiIuTRgExFWcS7NyDXz57r6yp79M5I6ApBVZ8i?= =?us-ascii?Q?ZvYcxnnYIoPgPhYWxBTpXV6yA1OkFm7EByaOyQjhWTcnxPO7yofNvAQHo1A9?= =?us-ascii?Q?kTOwZJ0bim5/CD2IzxU9JUbVelPmljY099On03ODWIAP0TpvCYYbE8mK/0TX?= =?us-ascii?Q?c7BkUnGNIQiAG6fGzo9iicdcl+tY6O2YFyPEx9dzm92?= X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0980;6:nMDxiI/wza5WFnERl7UY6tqvyZlRrX1+aCn7hJ1Zj09fWkzlycqufNcth1Ds0HTSI3bmcW7M2lH5aWRcQ8QGmAaBtPW0ED/+m6SlvI6jgZjLBmqYDQpFpGzS4l7AhxdHArQBPgr1rLotTs4VorGSyL25c6yIWwlPRgRHrcqPmfKfYM92gi28XWOw2bU0uQKanUt7nvdAidPceyX5ZZ/eNanmOHWsBWtS4BiXh7SL/U6pRtwtgR3B+A+y/NAW2B9vHwqQineTE+ezEfCSWd4srov6Mjyq5VA6hZdbplw4CR4=;5:IG+pDbEDY2DHcD5WMzEhnG+57TMsG+S8kCJrnSg80AL5CAiXS6WxjDaD0+TG1457/0k8JiAjrDTn3gb1A1wuviCrAfIwLrFcsswpClLYNxJEE97mX9y5dTS7LiYe8EvXTec1ZutNy5kxaLYS+tOTxQ==;24:fcCing1CyrdtwuOoEDCTlS4VEZUg0dcI3FGTkqiJrgFC59/stZZJACDz3Vu13lxqNZA9xUkbeYWME7IedCCeze0zSHlzJjPsbYY5qh3K6+Y=;7:oQFsrhZqzO2fhuM9VgwvzfW0/63bLmSLXMDmiqDaKH/MuiLKXuosM2qv30BYtwbIXpz2O/FYLbeJFdurtfBrdASV5g0IuLSW1UjHtMfZZO4MqFhofVOjx6VhKZgVQRWJudEPGRP8QxcFYukfFs2aAyUPTnFchBsy+YWW/s3jqyNQJa2bkqc+dxTXI/3rHet7bBcG55xqPNXybKMUhRHX71z/5WvjIHEisgqO4PQiC2hQJ2p64ccsrkgJRMFFyVIk SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0980;20:+wHD8vLNsBOVcj4Ov4XNSlmC2Y/otTb4tLULzWF3b/zYTUcxXTbbt2NqMi62vNJjjf8m+v2MZmVmot1LW4s8+Mlc0O2VlQ9lIVxq0IQw5+1UpQZU48tjNRFhh8VkrBx4dyhGdYIZEFlxxLqqbm10pVakFRIeFrDtILr5l7MT3Sg= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Aug 2016 22:59:42.5923 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2PR15MB0980 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-08-19_08:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 08/19/16 02:24 PM, Josh Triplett wrote: > On Fri, Aug 19, 2016 at 01:56:11PM -0700, Andi Kleen wrote: > > > Nobody gets a cpu number just to get a cpu number - it's not a useful > > > thing to benchmark. What does getcpu() so much that we care? > > > > malloc is the primary target I believe. Saves lots of memory to keep > > caches per CPU rather than per thread. > > Also improves locality; that does seem like a good idea. Has anyone > written and tested the corresponding changes to a malloc implementation? > I had modified jemalloc to use rseq instead of per-thread caches, and did some testing on one of our services. Memory usage decreased by ~20% due to fewer caches. Our services generally have lots and lots of idle threads (~400), and we already go through a few hoops to try and flush idle thread caches. Threads are often coming from dependent libraries written by disparate teams, making them harder to reduce to a smaller number. We also have quite a few data structures that are sharded thread-locally only to avoid contention, for example we have extensive statistics code that would also be a prime candidate for rseq . We often have to prune some stats because they're taking up too much memory, rseq would let us fit a bit more in. jemalloc diff here (pretty stale now): https://github.com/djwatson/jemalloc/commit/51f6e6f61b88eee8de981f0f2d52bc48f85e0d01 Original numbers posted here: https://lkml.org/lkml/2015/10/22/588