From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756968AbbIYPBH (ORCPT ); Fri, 25 Sep 2015 11:01:07 -0400 Received: from mail-by2on0057.outbound.protection.outlook.com ([207.46.100.57]:15392 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756865AbbIYPBE (ORCPT ); Fri, 25 Sep 2015 11:01:04 -0400 X-Greylist: delayed 934 seconds by postgrey-1.27 at vger.kernel.org; Fri, 25 Sep 2015 11:01:04 EDT Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Robert.Richter@caviumnetworks.com; Date: Fri, 25 Sep 2015 16:45:16 +0200 From: Robert Richter To: Will Deacon CC: Robert Richter , Catalin Marinas , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Tirumalesh Chalamarla Subject: Re: [PATCH] arm64: Increase the max granular size Message-ID: <20150925144516.GM1820@rric.localdomain> References: <1442944788-17254-1-git-send-email-rric@kernel.org> <20150922182902.GO7356@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20150922182902.GO7356@arm.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Originating-IP: [78.53.81.172] X-ClientProxiedBy: DB5PR06CA0005.eurprd06.prod.outlook.com (25.162.165.15) To CY1PR0701MB1614.namprd07.prod.outlook.com (25.163.20.151) X-Microsoft-Exchange-Diagnostics: 1;CY1PR0701MB1614;2:j2FmJh3kxTYIQgJ8L2qKlbP1hm3Ls2CaZPvjTu4S8f2r64lQmCz0tQZhqFiTUvMiwtwK+PeH1s5u6szZACzUVvwJm1/Gx/V/UcsUeBtTbmZpucNA6trDMfbdBiZQDgwyn+EVNPzJSQIwZw9RueQLxmJu8jPhz8OpDmAul+cluNY=;3:AjHx+4dERndudOmtgeSyDhYKlg/dy9BnVrfU7up8lHiWECWsf8xavxoBhImYHzYPXuB5qFj8ELe7WoWQnnuEs95nL7RH4EOD/Q3F6D9ZE2DaI+U8oU2BJKm2msA3VXWu/jAlQBMuZVu+HFM8vRObww==;25:iGb6F6eyvdNLLC9k5Of0drqFHRsd6yh1kKnts/sQ8wFOg5kuPuPmI/fGQjHYnCSqYNZLDnR0EoyJT9hFVvs/apBZdMyK1mYHTeHhn3jhxtUdX5ceJPQd6gQU8g7dp3u3OLQzsYZVunE9rvBhGPvSEN15EWyJxYwgr13W4CpIGaenz+bHmS6r2KlB8VafksXSi5sFENERkSxSnqs86BVASLWWhvcEoGIvt3RPFAumSHgu0UYeUQK4//djYU0UZFaO X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0701MB1614; X-Microsoft-Exchange-Diagnostics: 1;CY1PR0701MB1614;20:DsqyqccuMNmw2OrwayCi8llPeRskRfkEzBEAzGeYDWBGa6jEfKllGFkKuhaQqJJ4w0vd6RNlO6r9mXFWvA0CYLPaFJAMHPROEgK6jx2M7TmRj4yzlka+DQFndB6egCEij3oOe1iPvHzvhOl3iywGz5ZcLhE/bBCMeKMtEB2c4KaCQqy3IYSoQArFdMBjofV4Baez+4P1BA2lTqXFmYD0PkV3a+klRT2CTlaRW1a4yDdzeRIxCApGegLWmgBxSLPx7ffQkGMGyk+ZMwfCuYmumRXi8FHRjYrU3phJl9XJThkxOK0OQ8JH8hcTI0EworH4XYgu+dM9VIn3bV5F+kkI4/pSZ6yYcfJcw7FoxYZ/wIZVU3zWwUp6USsYJp6qv+dyIADSP82kQAniLIyzaPe7x6DCyVAZb56UW8HMLOPtKIR8XqDoZiO5B36ub/KfJWsM98+8ajNeDPFTTneXf/+wR9Ssl2OQWvzNL7gRjGiIFspW5XMooG4mQVjKKl9h69mvgHkGKfNV+p/cYaJNkmGdr7sTlCltKvX83FBlvro25iw/Jzn0BVvVFiCsJQGP4ThrO9MlMYyByhqEGtxNrQ68Hs5DmqLTHwgDc8TK2f/mRyE= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001);SRVR:CY1PR0701MB1614;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0701MB1614; X-Microsoft-Exchange-Diagnostics: 1;CY1PR0701MB1614;4:5yO1gPXeZa64kf+rEjL14kfTmmqg7BjTp8QlPpdE621DwqhjfQucMtgfv/a+IZX9XzebOoZCd3fsapnug5rayyLI2MwzmlPOIlr7k5k++TFqAroMeUTxtUTLy837V2PaNGvVjij/PyhuSoF9v+4PYBHev7PFItd1jJbNF+Audh0t9zn5HPJ8nCzOnz4E5h6a1m++1JzesCJQxu3orNtXcRk3UzwnKhXviCfW9RU0WZ77hmQ+BBVOSlJmEX/8xo2oknDv9MRkMcIbVtzVib6Qy+GfOsU8UR3YJ3g4ZAjdMBtfFIp2MBsyyhUmi4RW4QmQPZ7mIjp/NIgyrCRTg27DxDTwTOCTDfDzQa8AKVhr2Hk= X-Forefront-PRVS: 07106EF9B9 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6009001)(24454002)(164054003)(199003)(189002)(97756001)(105586002)(40100003)(106356001)(23726002)(189998001)(19580405001)(66066001)(47776003)(77096005)(87976001)(2950100001)(68736005)(42186005)(64706001)(19580395003)(62966003)(83506001)(97736004)(86362001)(5001830100001)(4001350100001)(110136002)(122386002)(77156002)(81156007)(5001960100002)(54356999)(76176999)(4001540100001)(50986999)(92566002)(5001920100001)(33656002)(46102003)(50466002)(5007970100001)(5001860100001)(5004730100002)(101416001)(46406003);DIR:OUT;SFP:1101;SCL:1;SRVR:CY1PR0701MB1614;H:rric.localdomain;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;CY1PR0701MB1614;23:H1hsL4WBVUf0rjiytmIYSJ66/shFzhiwsGX/WbZ?= =?us-ascii?Q?B7XXMCDbhrXngKU0j/Cey4UNxB0QLi1x3ZiamakpWAl2n9RzAYIQg+JsZblA?= =?us-ascii?Q?IlU7wNgEoCAouyNdTVT15B8QCXepCJJIwpLZ0QF8QrZ0jbVL9cPebbdX5T8S?= =?us-ascii?Q?WM6icBXxyboJAGb7Ozwv2XxdsjYuPMjDJpVe/dYhJfbpAab1R34AQfZyCsOQ?= =?us-ascii?Q?5CtgMA7tWkUK9/yCT9tK2uAEOe0zxepyh+kBp5XdWM4SgfL81w7PwMdWetif?= =?us-ascii?Q?46oyixmPmlhVfX4MSzxzu5ifk/jekD1Dv7aBHaP0nVEk0V0miE4RoIc8rBIl?= =?us-ascii?Q?/k1I6SCqAfjNXAHY7HF0MC6MMWitXS3mDOSKjiKYnu0kDXqZGzgmQhLxDfyA?= =?us-ascii?Q?Oldw0LzKUXfF1dr8Of67HtnXJusnIVzrtfUkhjUS45fza+/V3B2JkcCgarhc?= =?us-ascii?Q?EeXulhUlfjtOCiBXKJeXyvgGh1DXuNKRoTzBfAusdeFd6Bn0o3PhQDdh/u2n?= =?us-ascii?Q?zr/NrMrWUAnd36MJL8gmBt8xZRCr2zfUzbwVx+3x7SRbNsHKeHzSkx3c54xc?= =?us-ascii?Q?LpBJr0PizEAEITriD0R+Rederx54GaEorm3tI5NBNUAnYL7wkT33elEhAk9t?= =?us-ascii?Q?VXTW5YUv+2izvFdHhTk4f+gGgJwi9R93gZe98TmDppQ1mkClbrBTsmId7+lz?= =?us-ascii?Q?JfKJUMozE+NZvSS8P7Y7fYlIkrdZAjEcUvs4FlG8Oh9lf8x559JYFcwlCPN6?= =?us-ascii?Q?U8JE5McH8TFzgCiTiHEKcSSR2+O//6tkFoacq6LMS2Ni5P87QQBx9uoV8aEw?= =?us-ascii?Q?PgAW5tcxJHwYSMNS7zzdag674FkhG2iuhwOrOfW4RQ3N3ocLi8ykJr6vcxpi?= =?us-ascii?Q?TbTV7nG90/APyKj9R6kN8EHAli1+jKuxNhErKbRBEH+LihRQk2n/AsdujP7Y?= =?us-ascii?Q?TulqNEdI25QHNCLncL8aij14iJME26xAu7lvZHlsoW+EDmG0oME+hssL1uqm?= =?us-ascii?Q?TLqpHsjGtaYmd+mm1vb/r+8Lx2Jc45uBQf/k9t9FnjWu8bcNnbkGLVBRhvyK?= =?us-ascii?Q?a1woakEwAHQvojqF2uVhrGTbcQsCOd4pAIOXSsH6z6S+bGvphuvg6I709IRV?= =?us-ascii?Q?EXcGSlc1eadddScXx7/dALpHHBVFF4WkaFpLWO/+TbjyHJk1hAyQxKuDg58i?= =?us-ascii?Q?IPa0hprGRNbZi9Mqrcm8fSEz1xPh3vHF9984AOw4bEET8xxp9UNqv6g5xU/i?= =?us-ascii?Q?w4wfFAjXt/Xlzl7uSvf4=3D?= X-Microsoft-Exchange-Diagnostics: 1;CY1PR0701MB1614;5:lqBN7lzWyWRhxYGHyWDIg57Kul9UhLcNHv6n8wLO6I6KGzqBflKJN76/dFSJ0tUd7uSEl1W+slCirka0qxilG0MD8Oie3QAeLH2kNw3SHlxt9Lrn2YfqwU1O+jBcW/1BCeiN3J4nwGt4oDAxYI2WqA==;24:sE8Lo0eLvgAn2PKPpVsXYfOHzVYJskuyfxFrXDL3nKRk4RAQYLaka4TJHvKRsbrStBpCWWyaGE6YmcC2OYy40DUAYQ2H7N0LKAzyUPa30P4=;20:JJJCv2ogZP18MgcBxG6d+UjyxH5qiwJCCWbF+dkhdVWTHXMWcD/qMw1K/OTbVCa2V0FFuhoF3r7d1busC6Xu5Q== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2015 14:45:27.1233 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0701MB1614 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Will, On 22.09.15 19:29:02, Will Deacon wrote: > On Tue, Sep 22, 2015 at 06:59:48PM +0100, Robert Richter wrote: > > From: Tirumalesh Chalamarla > > > > Increase the standard cacheline size to avoid having locks in the same > > cacheline. > > > > Cavium's ThunderX core implements cache lines of 128 byte size. With > > current granulare size of 64 bytes (L1_CACHE_SHIFT=6) two locks could > > share the same cache line leading a performance degradation. > > Increasing the size fixes that. > > Do you have an example of that happening? I did some 'poor man's kernel build all modules benchmarking' and could not find significant performance improvements so far (second part with the patch reverted): build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m10.490s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.747s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.264s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m0.435s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.569s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.274s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m0.507s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m1.551s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.073s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m1.738s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m10.644s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.814s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.315s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.610s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.885s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.281s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.869s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.953s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.787s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.656s I will check what kind of workloads this patch was written for. Tirumalesh, any idea? Thanks, -Robert > > > Increasing the size has no negative impact to cache invalidation on > > systems with a smaller cache line. There is an impact on memory usage, > > but that's not too important for arm64 use cases. > > Do you have any before/after numbers to show the impact of this change > on other supported SoCs? From mboxrd@z Thu Jan 1 00:00:00 1970 From: robert.richter@caviumnetworks.com (Robert Richter) Date: Fri, 25 Sep 2015 16:45:16 +0200 Subject: [PATCH] arm64: Increase the max granular size In-Reply-To: <20150922182902.GO7356@arm.com> References: <1442944788-17254-1-git-send-email-rric@kernel.org> <20150922182902.GO7356@arm.com> Message-ID: <20150925144516.GM1820@rric.localdomain> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Will, On 22.09.15 19:29:02, Will Deacon wrote: > On Tue, Sep 22, 2015 at 06:59:48PM +0100, Robert Richter wrote: > > From: Tirumalesh Chalamarla > > > > Increase the standard cacheline size to avoid having locks in the same > > cacheline. > > > > Cavium's ThunderX core implements cache lines of 128 byte size. With > > current granulare size of 64 bytes (L1_CACHE_SHIFT=6) two locks could > > share the same cache line leading a performance degradation. > > Increasing the size fixes that. > > Do you have an example of that happening? I did some 'poor man's kernel build all modules benchmarking' and could not find significant performance improvements so far (second part with the patch reverted): build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m10.490s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.747s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.264s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m0.435s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.569s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.274s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m0.507s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m1.551s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.073s build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m1.738s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m10.644s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.814s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.315s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.610s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.885s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.281s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.869s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.953s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.787s build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.656s I will check what kind of workloads this patch was written for. Tirumalesh, any idea? Thanks, -Robert > > > Increasing the size has no negative impact to cache invalidation on > > systems with a smaller cache line. There is an impact on memory usage, > > but that's not too important for arm64 use cases. > > Do you have any before/after numbers to show the impact of this change > on other supported SoCs?