From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758839AbcKCQ4Z (ORCPT ); Thu, 3 Nov 2016 12:56:25 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:59097 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754015AbcKCQ4W (ORCPT ); Thu, 3 Nov 2016 12:56:22 -0400 Subject: Re: [PATCH 1/4] block: add scalable completion tracking of requests To: Ming Lei References: <1478034325-28232-1-git-send-email-axboe@fb.com> <1478034325-28232-2-git-send-email-axboe@fb.com> CC: Jens Axboe , Linux Kernel Mailing List , linux-block , Christoph Hellwig From: Jens Axboe Message-ID: Date: Thu, 3 Nov 2016 10:55:42 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [198.233.217.214] X-ClientProxiedBy: CY1PR1101CA0042.namprd11.prod.outlook.com (10.169.17.52) To BN6PR15MB1188.namprd15.prod.outlook.com (10.172.205.142) X-MS-Office365-Filtering-Correlation-Id: 4bb3e105-4ea2-4630-0a06-08d4040a44f2 X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1188;2:ZxtmIud2TODbyevpzgcOS/rToHdiztpUZX4kAQcA0ZsibZbec2UJY9sM8FJMi1PCWqMZKooYFzTdGNef+RHbN37t0pM2rIOraXX3tKm1D/ikf4gbGu6U/KxigiwiSnBbRwuk8n5xRtQXoyjZzailZ58KiwcKv0zDmA7vLtMMsNMDYbvMWp+UH9sdSktc6n2hXKPRAo6+yjOh/nDp+zHz3g==;3:WM5lS2sw2ZD3iv7le3DpylPGMCpuSZknXK8GzRUNJRKoHEhTwCVCFcZ9l742rI1LpYs5L/DbeszihcjZH0QppT9CYV49+Y+ZWbkRVh7lYTUanA6RpKzZ/wNZaR6rMAsIK5bAgecLNhdDJElnO6jHkA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1188; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1188;25:gkyP/WgKi5EN3b0IcxJifhysIfRdT+QBI+LhhV5LQqp5UNcctFMT0s7PlTuOa1rvaW9RwOSvUq/zJi+XzlRMiFhry2AdrJQP50mTyRBeBhA4cIu0KtxCvpe9AdiMNd5B9X8aCE+3LgPcOPno2WPnOIpjNRTBzgFiyhuMJl553GC+bdsr2SoajO/zHtq2GjFv4VveGOzWIr0As+oRo7hxOU8d32qVwY34eFibMduXHMmGMU4O5u94rrHSp3QKcM7dH/4AlVM4lFpdPJlAOF/epRvvmnbQ1OeAshUBaHg96lHZyc+3dU1W3Q55QvIhLqjMCAA4z4lLrd3Ah+UaiSmgBK5QJJW+u+6u+txxNMXnOh2v5T7TScD+mo/RP4xdXbNjirqX4h1bvOCpyTZIQmCgVZ7pEWblzJRApGDAVa2lkyVTwUfe5UAkAjBgqnSdF3DQZ8B8H1LcUWop79NeClnf8dKE0Cx75mbMyWJBgwKxZW/i8r7heGridPqgjacAQ31PaM7RqOI1bnDM12/nulFvico+ur+uYueT/EJ4UE57w1GfyL6pD5tsL0vvVAa1bw/t2n+fxPqWs3ai7CYJFHXPpV4ujz7e/Ow9GY4UFA+EpFxVi/MVV+NA9xRbBGF+b/U4RVUVrTON9rQivV0i4HOm3Wa0Mbr1EuwxphT+6dbOm5fJXezSDp2NNq/YBW/0+AavN0vfXNFl3ip/N10vQVJ+MtHCUizr6CJSpFZqY56Viz0OWDiZ+yALraCfomK9S3/X X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1188;31:abfb//1Zx2bu3oBTBEkcB2l+aNxGyuY2ZbrkzDfY4IqMzYrTrLAaO3VM9bvKuAlFM/cxPhvvrGSoOUoq38k5jhUgUeiF9wuBaDzG9YkPEYsWwdGoX8WBcD9Hm8UAHS4ee7PSYHBrdqVM2KQcZUNqCSVAdbMHGpk0w6nkJRbYB7H/cx85FiF/OMBUTCbOiBtXpnnFSWvbRLV9Zq25Fgr6Tl/1vwAG0vzDPG1fB3OtaTa9GZnxPoeOj/GYi4cmnzTrtgUE9f74upg5BtgGHgHh6w==;20:wYCuIkx2eNhFKX9wNT2tZw2FmrSOliM2/woSqWeyBUT/XSXPFt5N7gMJYowPLz5fT2Kfqa/G0tOePKp2YHdzb0KETG4X3minFHjb/0Sj+LuVYFpM6mQig/nC+8Jlr2AOcKq+nviU6k52n8szMjmkPY4aTbEvIkU1M7w+Oa6oVq0= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(67672495146484)(5213294742642); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046);SRVR:BN6PR15MB1188;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1188; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1188;4:+iz3n7WFOJiylrHxBces6YiGq9LbW4Ar4M9PFEhg37hZKN6sH4Nl0+0KBMOHZT69ZFS2yR09+Tzu7+c4JXCG6JCsRaRuxvkP91HuTSKWMWpUgpmW9uaT6JR+Zyvk7fTbkUviUn9v7RMUM05OTf8RA022uifZKk4JNJX+L5LdFW1YdTE4sYzOGHxBqPU43hwNylcj92v1ms4CUPXPDXeovJ6bD+32LnBZR+UKBbJQIkUp4YoHtl/tvkH3crSnKQAsNoBLpoqf2nIutgUH9MsAo3UTsryxIEY8r7FZgtLoy/wiKnJlx/854iu+/CBX9OeflnHuvCJA/40G1xMUXFytCqNC4PJQdNXjYUOCfeUGtFPzXQxEBwhAjN3638F6W3RAB7uIVNlYXsUebbvHsX28EBPJ+MiHF9RgUt+QfSzy/x3BQh8Ih8/Pj6Z85A3pcvG6GUmEItcWeZPn+2vA3tBOwc/OCgE89eG5uqxGC7Io69E= X-Forefront-PRVS: 011579F31F X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(7916002)(377454003)(24454002)(189002)(199003)(50986999)(106356001)(83506001)(19580395003)(76176999)(19580405001)(54356999)(68736007)(110136003)(65806001)(93886004)(2950100002)(6916009)(97736004)(31686004)(42186005)(189998001)(77096005)(6116002)(101416001)(92566002)(5660300001)(4001350100001)(50466002)(65956001)(66066001)(230700001)(36756003)(33646002)(7846002)(23676002)(81156014)(6666003)(105586002)(81166006)(65826007)(31696002)(305945005)(4326007)(8676002)(2906002)(7736002)(86362001)(47776003)(64126003)(3846002)(586003);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR15MB1188;H:[10.141.86.225];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjE1TUIxMTg4OzIzOklLNmp2Rmp1MDlPZDZxVzVZS2pZT0NWcCtB?= =?utf-8?B?R3NySTB3UHdRbW9Wci9VTUNoOTJOTjlqMVlyd3NNVVozVmoydys3NjNvTHp1?= =?utf-8?B?dUhvbEREcENNNCtzUFdLclJsTjdoaEtWK3RSYWlkUFhNYmNLaUxrZVpzVkZa?= =?utf-8?B?L2M3MDFlRjFHaGhmbFNTUktDU0tyNkFvR1MwTzVyNjdNeURnL1VjbFNUNXht?= =?utf-8?B?Q2pvZGgzYVVyRE9PZVQ2Yk0zazJ4VkFvRjg3a0xGTTFsZmJ6V2Y0VjdtV0Fk?= =?utf-8?B?ZnVBMUZ6Y2ZFQWxIUUlsM004SVltREpKWjJZMXl2MFhLaFE5SkVkNmFKVmwv?= =?utf-8?B?Qmw0QTRyWE5NWnVwejkxMHhLc2crdUNzWXZjYWFwcFBxMmRZaXBBSzRyck9N?= =?utf-8?B?YVJsUjJSZUdmcllIY3JrN0ZNR3piWHRmMS8wM0tFbGRKL0FlRXBoTHN3czZO?= =?utf-8?B?bDZHNmlpVW1kZFVlSmZBK0VoTnU3RWhqRnZzMVZSTVYwYnlXeUZiYVBKc1lw?= =?utf-8?B?b3RmVDhiT3Vic2NRMGVHbkNXeXlXTC9SZVFSQVc3MFlrM0wzS2pYb1lrTFpX?= =?utf-8?B?KzlNSlZNOHUramtBeWFOUFp3T0ZvMnJhek4zbWRUYnVZbThqRVpCR0cvTmM3?= =?utf-8?B?NDJsMWtaUUNJK2Z4aHd5S1RWci9JdjJaVnFscXRlcDJGa01mVWVKUURjVTd6?= =?utf-8?B?dVhsVFlLYzdtdnFCSDB6WVhvd0dVUUVnRU14bjBGL0VuYkZyd3ZPNGxIT3JV?= =?utf-8?B?cGdMZXlxazFRcUE1aGsyczE5NmM2aU1LbnBXZExiT0Z3QXRia1JaMUJXT2VW?= =?utf-8?B?MFNORDNYWlZROG5VMWJFMm1nWmtoWGhVd2laT2FlS05QWE90VnFYdnlyMG5s?= =?utf-8?B?RUFIUUE5bllmNzVsdzExWitsRHNHRllwbjY2a0tRNWo2Y21KUXp6Slp6N0xN?= =?utf-8?B?dEtWdHhiZk9ZMHk2SlZucC9xN0tRWWxvNjljSTFHVWpreVg1MEhoZmlwbXU0?= =?utf-8?B?OVBUVEVyam5lNkhCOVRuaHR2aWpYa1RCd01heFhaWHVhcFNEZWZob0lEak1r?= =?utf-8?B?c01DbC82bTR0RlZzYzA3bTBnQ3lJd0ltemJ3SUlCVWhyQlJib25YZjJhRFdW?= =?utf-8?B?enhnYTB5SEZGVUlhTzkxdkJKUThHTFJONGxIeUhWcVBhL0NkS1dZWldaZk9F?= =?utf-8?B?cldySkc5SWgvdTJxZzdwREx6YWhDVzF0WHlCeUkwcGYxSWpVK2hXMmRYUElG?= =?utf-8?B?SytRdlpUbFhEV3AzQTFFWVBiUm5xazdJeHZ6U01HdVl5UzIrN3ViUFFKNExs?= =?utf-8?B?SkNqMWVCU0RlZ0phN09UUHBhWWpJZVFaWHZGWWNlZ2VtYkthVk5IeGx5VzhE?= =?utf-8?B?dEVtT3R5d2NUenhSMm42ZHQ3MituSmE3aTNMMWtmRTN6bS9MeE5YMU5vVEVJ?= =?utf-8?B?OVliZXg1Nnc3MVgxNWt4b3pJMmtuMUNjcnB6eHQ4Sk4vY3QwRW8xQktqcUw4?= =?utf-8?B?VlBDaC9la0txRUYrMEJwQytxRzNQN1lTOUs4QndiclFxaTRRNUllcC9ITXEy?= =?utf-8?B?cFBDUkt2eVFYZUhFbEExQWxCTVlibURIQmlLSmpuVG11cGxWRWJMWTBWWS9v?= =?utf-8?B?WTVxbGxtRWxtWVA0VUZPSlRkSHJISThPSHlFdWIxeEg5N3laTndEMFFpaEdt?= =?utf-8?B?cGtTMnVlM2RJNXVyMUtYb2NGVEFvazhmTU1IRjNrOE9qbGhVcDhrcWlmSW1K?= =?utf-8?B?aVpzMGY5TWc4Y3BXd1BxVVQ5MTlhNEVydXc2bmtENk03SzQ4d3pkN0piaC9n?= =?utf-8?B?YUpMclRwWDg0NWo3WTIxL21OQzVzVUNGdGY2RU44UE1ibVE9PQ==?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1188;6:79v/n4HmV9RToPHSuLfT4ELsSVNohs0ARo4HhjKzthVo2zFxg2S5AkT3mtsTVOxvrvmaQnUIIF1LeLVKUoNOpsSAJQ0zUgE7e+zWVolmRc73M5IgMBmB5ZAVjRGAfVDtTvswUDq0qOpUwpM/qeb5BnZUrmLUuInC56qXl9nUDE5JRyBZoWCDZhGFtGbye8Nb2ZqltvozqlgJKtFSkN6eMXVb+xGo8VPdLrFgxNLy/AusNevkyftbvLVvFk1WKdpIPy4pTdE9PTp9DodK7MBfihiUOtztz0U2kOZ8jDzee5Tjs8YqZ8PbPfcPdgLeYV7O;5:I3NhNBrQDzuEcbP7X68effh8K8EadUxsXgX1lTCCNFba9Es+kRQGF3FcYRQZF+nYHM6ct7m32CkwPHPwEO3rVK6ElBHZsVVUWGuHXDzLphqicZ2MJfh96bCR6TAxzxDcZxEmblbZwc16JPJc7Yh3BmpECzmu94yMCmn40SfhEVI=;24:TI5XIuTn7Wt5ecYBe+jE0O7XwEc8A2tGSlaD1PXieeDPDN2rprT9+gF7I7/khAg9hDmOagxsDo3YyB64GeG7JBXaF9qta0dECk38t3CJgy8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1188;7:Lg2Md0Wg+zBPcX1O+fBNhXo1IQWD+YBzrTeZ4+O3lx5G01Lc19hO8+gcYkxiLyaZJD+JxgJ5ZJpkqhZ7xSM1EmfEtkkDahq3ORXv+yVmn8+4q9n06FZvvwg0SQAClkc8DPkwA7li6JOi8rJ64t49VBvQwQisWTGKg8uD5rDt5vi9cNXfcb5tI1Kfg59n9kOCk+ddJxqPrCsrdxyAHKRJTYS5+G1Oy4l0EFQz6Rc55RIh4KVcy9uRx7mxnfUMAvL5esH9ZYyNeVKBWl3WjduWwMcJ3p4eKQplxhwWAWOpVTd8HwhPN16GHxpAx6cZwCncB0Rbz4r00iv2Dwo6HwS8XWCR5qP0lNVtF+KOPrpRXDM=;20:vKcdQKJzV2w2pE+598u8ruU6x7OdhtHzbzjZdVst63dzknyYwUBbeGloKVRCekOPWbwFrnMyoFyMVvRGuyWUMt7S/fXGu2q8us8mfXVPTW5oS5oEHcnVxKmLweI5S9oPkVr9yIIz+30Uoba+xfgvGQc5jzq2qMbmL4q8YgsMH9I= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2016 16:55:51.2334 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1188 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-11-03_05:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/03/2016 08:57 AM, Ming Lei wrote: > On Thu, Nov 3, 2016 at 9:38 PM, Jens Axboe wrote: >> On 11/03/2016 05:17 AM, Ming Lei wrote: >>>> >>>> diff --git a/block/blk-core.c b/block/blk-core.c >>>> index 0bfaa54d3e9f..ca77c725b4e5 100644 >>>> --- a/block/blk-core.c >>>> +++ b/block/blk-core.c >>>> @@ -2462,6 +2462,8 @@ void blk_start_request(struct request *req) >>>> { >>>> blk_dequeue_request(req); >>>> >>>> + blk_stat_set_issue_time(&req->issue_stat); >>>> + >>>> /* >>>> * We are now handing the request to the hardware, initialize >>>> * resid_len to full count and add the timeout handler. >>>> @@ -2529,6 +2531,8 @@ bool blk_update_request(struct request *req, int >>>> error, unsigned int nr_bytes) >>>> >>>> trace_block_rq_complete(req->q, req, nr_bytes); >>>> >>>> + blk_stat_add(&req->q->rq_stats[rq_data_dir(req)], req); >>> >>> >>> blk_update_request() is often called lockless, so it isn't good to >>> do it here. >> >> >> It's not really a concern, not for the legacy path here nor the mq one >> where it is per sw context. The collisions are rare enough that it'll > > How do you get the conclusion that the collisions are rare enough > when the counting becomes completely lockless? Of all the races I can spot, it basically boils down to accounting one IO to little or too many. > Even though it is true, the statistics still may become a mess with rare > collisons. How so? Not saying we could not improve it, but we're trading off precision for scalability. My claim is that the existing code is good enough. I've run a TON of testing on it, since I've used it for multiple projects, and it's been solid. >> skew the latencies a bit for that short window, but then go away again. >> I'd much rather take that, than adding locking for this part. > > For legacy case, blk_stat_add() can be moved into blk_finish_request() > for avoiding the collision. Yes, that might be a good idea, since it doesn't cost us anything. For the mq case, I'm hard pressed to think of areas where we could complete IO in parallel on the same software queue. You'll never have a software queue mapped to multiple hardware queues. So we should essentially be serialized. In short, I don't see any problems with this. -- Jens Axboe