From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751274AbdAQCtG (ORCPT ); Mon, 16 Jan 2017 21:49:06 -0500 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:35080 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750774AbdAQCsS (ORCPT ); Mon, 16 Jan 2017 21:48:18 -0500 Subject: Re: [PATCH 6/8] blk-mq-sched: add framework for MQ capable IO schedulers To: Paolo Valente References: <1481933536-12844-1-git-send-email-axboe@fb.com> <1481933536-12844-7-git-send-email-axboe@fb.com> <9936D33B-AE98-46DF-B1DA-D54142A338FD@linaro.org> CC: Jens Axboe , , Linux-Kernal , From: Jens Axboe Message-ID: <8d900827-3569-b695-86d6-3cfe27884d6a@fb.com> Date: Mon, 16 Jan 2017 18:47:14 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [50.233.49.230] X-ClientProxiedBy: SN1PR10CA0027.namprd10.prod.outlook.com (10.162.255.165) To BN6PR15MB1186.namprd15.prod.outlook.com (10.172.205.140) X-MS-Office365-Filtering-Correlation-Id: e80b3fff-973e-4763-b46a-08d43e83273a X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:BN6PR15MB1186; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;3:Eyajv3zRLqZjRkhz5fuFGTokBUcRqSJ2jcwjblDGWuV8OyXInZrNALhjSzohgBiIiFxDQnzDc+CLPgmXkEPIVxJQqNqeB4YfVT3zC5+Rj63Q8Xe6s/J3XvR2pnksfj4nLg3wmx26RO3kxV7sx0UrwE55wCJEs5I5rcmdrOpMXsPTLxkd5BWQAW+pkiLYC1fFKALli6jO8t1VBkHGprPK0VRwC0iOw0VeKX4OoOAA2urri+hp2leFQRRejKKfJuDReb8RapNXKJAmntmt9zoz/w==;25:x4ynfmbyCD6jTyeEYMw9+mncw8cXx8FlAyr/pRVYjt9KRh3m01wYruQf2ac3/WosF4ooRendrZG1w4jN9ExNGbblcdUmVhLdpZzGkx6oB0OVfxkZ/zaGURDck40X8pRGqVLvahFnTlvUABY/KjHyQPwdq9XCLvoKsBzZFmAtCGf0x/TcH9c+MB7GyrEl5qRFOudrThdKg6oMQHVBVPoFwUzXHUSDAhLewFs4xSrdpxA9Jvi8wcwUKxklu0wVX3hL1E3uBTxTEbwwDswI/xNadovqPCx/425tWGIV1jVyWQCeRvElsKE2TkdbdE6Qno+OpbOajsWDPJcmbcPT1Mn63hpgpcej3h3xZu/Ox8fsWF9C7dLGvnyQO7lSlwW+N0fnC8i3rGABh+B9I24bw12E/KDzSL4LvkVVgBhLRw8bDejao5X3xHWJ2K++Kbh0ZkJWvu+2smBlDygeO6YtzcCsuw== X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;31:iwWNt2sudGFimMJVFLrs5j7whz/GhBgTYSBza/53KONXxfi9huTMsitZza9+vQgRCcIbsI5lHjpNcGXg8VXugUYxpGXJMigwS1G6FfsE6GQy5IvmfHXUIOw4uwwAep4r3yFUgKcLQ59C5wN1XkYh5Rqrq3WKQnxtTZkwvR26purNdpmbXxgyGHFNBxHlW3FOIoVJvgqRSd8GViEjHfxNxadmkSg29uuEqinn6CgVA0eYLmIbbbJdZa2ynnd72F7N;20:rs4bZh8fikwJ7fucw4HXtgBE576w8LITWg6zGtWpVq5l7e/17UNPwhp0wM00Tk1DUACTx+rq8hIu3QZPfqxvOSvOFXJMhLpktEMfYbxWMzXZJB+wUm88ECpskK7iPA2+sT4AXrCIGD7S/vyoRYtvvubReHJFpJIl52zTjrLe8qY2use4GRUyB/6tUUXLpDwJQC4/8H4opSL3zrM7JQI0WtqVCEEq2Dl0sKWMrjdaGU6AfQ05Z/G1KMnuT5NGhlj+zyRc0Vj6rCm+W91QCiOm3ao3uftJYVIk2tZqLWUlRfvf7mXZjMnOV09JMZvnwHR0nYt/8LUuzcEweoZ1dLq+lfofSdzfvemynu3PJC9ywaCZaDUOeOLUg+TNmFyxiCGOLc6KEDOhWuB/VIPnsovV+u1pkWoIC97o5p/FiIE5xoL5AQ9QWBSte1N3NLv/0kFZEWv+0WSDJ+GB87NGfZ8M9ngkUjQoWxL4+HTiH0V+HPjsN1+TC/tRYgCgobR1H9hT X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(67672495146484)(211171220733660)(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6041248)(20161123562025)(20161123555025)(20161123564025)(20161123560025)(6072148);SRVR:BN6PR15MB1186;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1186; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;4:BOFsKq6/o6f3tUcludUI/xYEtlzURR+fnTlrUNiKhVHpEdi+JS88WVRarb+9y7b+F5yfN5e1MB+liHcSGrhTDn6gAT7LIR1GB3eGS2xKx0KamVcjPOWuw8daRJlxuZ9WbleQFkEAOPax5rRJMJAViGv8numaOk3lpL2ud1LpgiA8LOX16UnzeJ3FuQxmOCOXJKCsX3mXt/AMpSgMz8Bz7dAoIpFSGRpcP68MnE4SFoNKHGGochnvQOZDbau3/k2mjLpx1yPgcn6nudbWje+8K8fOFe2qvqhuuhAA/l9dU9xHeUyAfSOEeO6/M1KEyUBayx9Q+EEjkVcFa+tHkrMEnyMENlyaSWhY0e3crGOblBHCYUCRXMOloW6XHmIYxdSCkeAGkWdgirPZ3owLC8mNIVDBHaDj8Eh2rbMH1qElSAxnnGxNfRDshw2ik7enm8dY/PtCZfiEyGs4jid/CD5QRxcaKu3i58Gxip8KLxcND61GZEwqqWfrEnO4EX4j8OXFpwzvjAqxFaLZfWjnKmqKNggoEQatIUnV622n7iZbtfd5VyY3+i3JNlN6gzJCtvwuPr9qvY5e8sszAXGqT4NFQk2j7LnYYTDmsOUspMl+MrDrLv9L4VvNjVSIta4iqZz4xtzldEQxRidojVp0yt95u7cB3E89ASQ4D0AXcxpoFIcUhUmmFALaNMODdLA055fF X-Forefront-PRVS: 01901B3451 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(7916002)(39450400003)(199003)(24454002)(377454003)(189002)(3846002)(64126003)(6116002)(93886004)(77096006)(33646002)(4326007)(50986999)(105586002)(23746002)(90366009)(54906002)(106356001)(68736007)(76176999)(54356999)(229853002)(31696002)(6486002)(2906002)(38730400001)(25786008)(230783001)(47776003)(86362001)(101416001)(117156001)(4001350100001)(65956001)(66066001)(97736004)(65826007)(7736002)(31686004)(92566002)(83506001)(6916009)(42186005)(81156014)(50466002)(65806001)(36756003)(8676002)(305945005)(110136003)(81166006)(5660300001)(6666003)(2950100002)(189998001)(230700001)(30001);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR15MB1186;H:[192.168.102.217];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BN6PR15MB1186;23:hCzxKeDrCvT2gclXz/DPelhy1YA4bAmCPh8Ds?= =?Windows-1252?Q?+8q0tgoTqxb5zTnVfeO3hWTEUK08f9xSZ90N1kzE87DtpCypGlMnKzk8?= =?Windows-1252?Q?eHUHOQi+5h1jBRvXO0QUqETcNLC13OgKo+figs4nsioFMKBpVsQTnosW?= =?Windows-1252?Q?2qZmbrlbIQYPTHDaOiXZxrBRuprQ6T9Kjm2xXjjCnQtIukrdFc/RPyDC?= =?Windows-1252?Q?4r0B+kpYIxPq2OdCq6Jdb5Y0Kt/I+GyDHb9zUaerjhBo40lHsTEWKFSt?= =?Windows-1252?Q?nwW6Qie7N2TMr1Hzx5EC0Bl4P0g7+4n3JEGAIMerV7Bc3RQXUOMLwTLT?= =?Windows-1252?Q?afKqDuomqVXwjtA0u8vBsSllhUn/QpvymO7lHV20g9pW4eAoUAOUsbRk?= =?Windows-1252?Q?0ztBuw+26xuI3TEhkvHvMxa3aMZ5+PlTki0MzD4e8HMttFtjoNLKbLHN?= =?Windows-1252?Q?UaDvUwcTln0rswvi4+59fCEIfmgyzM33UlM08pPqsm/tKK1C0BHt4SCS?= =?Windows-1252?Q?VcDFuPE2zXxlWxj1YWoqAKyNR4Z+Sya4alTvhn0zVPtv6SoQt+tcu+fv?= =?Windows-1252?Q?x6Z0bUjXm7X9YDsiXcWB0BK4K+o9sOYSPfiIsXdkIKTCad8GZlqDM2JS?= =?Windows-1252?Q?h1lt2b44hGvGC8kKYaIXpMTXB0R9f18V+rC//y79C3oLCQKiCeusIKtq?= =?Windows-1252?Q?KUGfS+5qQ8G4iCdwP2ug4YsLtgqeM93Rod1sqoyLC/MJuwsaThz9ctj4?= =?Windows-1252?Q?AtvRPpgegSOqmeNVPF5CwNCYnpNrePyVmuPKH0oB2NpUc90sZ4VB+b54?= =?Windows-1252?Q?rbOnrI1d+Uvo1eZifDNqxwcnuKps8cyZv4lu6Vs6msE5gUUrtWYR5cQQ?= =?Windows-1252?Q?pcEWzS5PuS6BQO0Ok//s9/qAstqIS6hJ6DXQQnkn7+pJuZnwrCQ0AWBC?= =?Windows-1252?Q?kDysCnx9OVnqjK8lIJUYuxX2SkAT9oZozM/GMXKHhIaXHOSmBVJ8GNNA?= =?Windows-1252?Q?Ipyh507OMD0zAPg23wFabRzwjDZCxe/XujbSU6pIdx6dTrrEBrjyG7OQ?= =?Windows-1252?Q?0D0KxnRQdhS3GZ8OXxQHu6NgyCbwHTt86KomThtos6qp+fZzB2JVFOme?= =?Windows-1252?Q?NgeC/hiYAkyNJxoTFrEfhMZ5SWERvpJxWE1+S698b85KWhiFLuDbZ+Kv?= =?Windows-1252?Q?8tTcYhZ78wht91l6466a2lFFja2K4Qg0bffsKeMOHp3RE/j7gb99YQ6H?= =?Windows-1252?Q?mlCLsfFVCKuONURmGVMZ7umB+CBEssRaZgrQCw/cHoQRd5LxcUYZMo9m?= =?Windows-1252?Q?Fhp/lUGKoEuufJcleuaf83Jp9IFXtDU3CLFDq4l1hVSsqMR7jW22kZK4?= =?Windows-1252?Q?aT1QPsuAyRZpjWLqQMzJ+KAObZfgO4Smc8YUUG7CymsM9VjVVzw+7nPX?= =?Windows-1252?Q?fA3JcVAM5CVTSZUw41s/5Q9EDZO2pBG+1TmjRGpLePj0NNBMQSyRO84k?= =?Windows-1252?Q?/cbYKpBr1bToGGW2CUlKJwIw4hpZqFZYgKl5AVeKR1luPaCTRo7pq9Cg?= =?Windows-1252?Q?uUOb9uENmdROdE=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;6:jaawDTpLcfSnV6+5t6F0iaworOFgnQYVL6ye+sjoAmbei897njXlm3cOa+DEksSLi0UXAFQ2acs2A5bfGkkZVk2w927jdA+awtEcSj7m4aOkpHVa7SL3bLVGSmAWN3pzTO/QFj31H8jxUhdXSKaf1kw+WYBBFMMrqY52be/950UVgTsGVzrS7VxRtRcE5GCwHTuPdVW4/KMsJggyOHRhqV7vRSA3l+ICdnhIU2UWn/DMV9EacW4wyoCzd2TIUdXgbM4hC255+B59hb55VPZ17zHB/ipJatu4fgjcgiVCNRDMGPhL6LJrPSRs68kdwKO2o7dMk+d+SCEVrNFE0uSo4aZXepZ1tW4pL+B2WnU7iO9QCTevbjcypavc0jdbsPyDSp8dPrkXwW7Cyze4bIQJ7Q3s7orcaIhtc6ypn5tNzmg=;5:ipIaVpSuliX8MNKVCPfADbwOLxDCsb1pKM3GuxiHscM7IyX7cYs89PU7xWqNOB20hmQl0JZBjwPjj/vLej04QFxsZ/BmhUhEB4nzpFDBgK3POi9puzrlQpkaqhilxNqhXypPV6FmMHDTsguHDEsWafUrgduP8p1Sfim2V7p8Tfw=;24:lBztbMxMaY/mQ9bYDCelH2aKoOsQtEL7He+fiZSQ8oMEoxg6Fopm8ro+CmSLjIWxWStE9P4fRXloc0XHuwjUYVE2Cnh3ETCGzb8V+5O6qVc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;7:vUQVm1lkY7hzsv1/tkjwerS5PB0BBUhoU3VUaC044PkuxSQcjqpXTVoEDmpREBIIje34J2T/4/Nafe5TewyeKOtDYrhRQWR0mShgQStJ1wU1zJYMwrHNlWblKVzWhHLPn9qUjm9MmE4GCQtFkoacW3GwkBtlFO0QOT9Q8qlSQAJlv2e784niv8Q7nZmUkPkjSxKJNQKMiS9aqpx1Xld5HZwCxh0WjhMoq9CFp/HlwvGZotzwzI/aY+N/u8fvNtSXt5INw5j+OVP6gYY4+5gJTMKm7Dmhkk/bTJ86Um5tFbNff7OaAd4DyQwiZWbsdx/bEQAOGAL+jw6N2AOGrkzqXpWnNGM7/QY8cDzJDKX/NhGp2dq+YWITp1ZTdEWkIB97UBTzzGv3wZKDU7SF2a/ym+3YHcHqOIVny75AHHrd2sAw0J298ZEHSvcynoSudthtBf2QxuyZUnCLEN+QXBP1zw==;20:U/5iggzMFL2UaBWiyO3IZPito6HP6vRTGTST5PGlCCvdjQmbRDLBhDsOpJVMhLBtEQFg/Q8vE6r/NmwOU9/OlgyEsOigP2R2FE8y+lEynBJjvCk3qgrz2wm2npwQCnw+RT5AeuMgRqUgE8JFqllecXe7Gmmrt55hbfq0XBo/J2s= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2017 02:47:17.3859 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1186 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-01-17_01:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/22/2016 04:13 AM, Paolo Valente wrote: > >> Il giorno 22 dic 2016, alle ore 10:59, Paolo Valente ha scritto: >> >>> >>> Il giorno 17 dic 2016, alle ore 01:12, Jens Axboe ha scritto: >>> >>> This adds a set of hooks that intercepts the blk-mq path of >>> allocating/inserting/issuing/completing requests, allowing >>> us to develop a scheduler within that framework. >>> >>> We reuse the existing elevator scheduler API on the registration >>> side, but augment that with the scheduler flagging support for >>> the blk-mq interfce, and with a separate set of ops hooks for MQ >>> devices. >>> >>> Schedulers can opt in to using shadow requests. Shadow requests >>> are internal requests that the scheduler uses for for the allocate >>> and insert part, which are then mapped to a real driver request >>> at dispatch time. This is needed to separate the device queue depth >>> from the pool of requests that the scheduler has to work with. >>> >>> Signed-off-by: Jens Axboe >>> >> ... >> >>> diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c >>> new file mode 100644 >>> index 000000000000..b7e1839d4785 >>> --- /dev/null >>> +++ b/block/blk-mq-sched.c >> >>> ... >>> +static inline bool >>> +blk_mq_sched_allow_merge(struct request_queue *q, struct request *rq, >>> + struct bio *bio) >>> +{ >>> + struct elevator_queue *e = q->elevator; >>> + >>> + if (e && e->type->ops.mq.allow_merge) >>> + return e->type->ops.mq.allow_merge(q, rq, bio); >>> + >>> + return true; >>> +} >>> + >> >> Something does not seem to add up here: >> e->type->ops.mq.allow_merge may be called only in >> blk_mq_sched_allow_merge, which, in its turn, may be called only in >> blk_mq_attempt_merge, which, finally, may be called only in >> blk_mq_merge_queue_io. Yet the latter may be called only if there is >> no elevator (line 1399 and 1507 in blk-mq.c). >> >> Therefore, e->type->ops.mq.allow_merge can never be called, both if >> there is and if there is not an elevator. Be patient if I'm missing >> something huge, but I thought it was worth reporting this. >> > > Just another detail: if e->type->ops.mq.allow_merge does get invoked > from the above path, then it is invoked of course without the > scheduler lock held. In contrast, if this function gets invoked > from dd_bio_merge, then the scheduler lock is held. But the scheduler controls that itself. So it'd be perfectly fine to have a locked and unlocked variant. The way that's typically done is to have function() grabbing the lock, and __function() is invoked with the lock held. > To handle this opposite alternatives, I don't know whether checking if > the lock is held (and possibly taking it) from inside > e->type->ops.mq.allow_merge is a good solution. In any case, before > possibly trying it, I will wait for some feedback on the main problem, > i.e., on the fact that e->type->ops.mq.allow_merge > seems unreachable in the above path. Checking if a lock is held is NEVER a good idea, as it leads to both bad and incorrect code. If you just check if a lock is held when being called, you don't necessarily know if it was the caller that grabbed it or it just happens to be held by someone else for unrelated reasons. -- Jens Axboe