From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751542AbdAYQNt (ORCPT ); Wed, 25 Jan 2017 11:13:49 -0500 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:60846 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750852AbdAYQNq (ORCPT ); Wed, 25 Jan 2017 11:13:46 -0500 Subject: Re: [PATCHSET v4] blk-mq-scheduling framework To: Paolo Valente References: <1481933536-12844-1-git-send-email-axboe@fb.com> <7A8A5078-E9B8-4EBF-BAB1-9E8EEBF3A043@linaro.org> <37570439-97C8-4009-B143-C5E78789A137@linaro.org> <2028A64C-40E5-403D-B5E9-05863E94B4C5@linaro.org> <59b134de-5d2d-5f16-3118-bdc16bc6e36b@fb.com> <22EDFDE7-71FF-4265-8DDE-B141D3FD26B2@linaro.org> <15FC1715-D77C-4E31-811B-60898C0C6DAA@linaro.org> CC: Jens Axboe , , Linux-Kernal , Omar Sandoval , Linus Walleij , Ulf Hansson , Mark Brown From: Jens Axboe Message-ID: <0e047692-5ff2-d404-a400-8484e060218b@fb.com> Date: Wed, 25 Jan 2017 09:13:35 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <15FC1715-D77C-4E31-811B-60898C0C6DAA@linaro.org> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [216.160.245.98] X-ClientProxiedBy: MWHPR13CA0032.namprd13.prod.outlook.com (10.173.117.146) To DM5PR15MB1196.namprd15.prod.outlook.com (10.173.209.14) X-MS-Office365-Filtering-Correlation-Id: 9ad81453-4c0e-4b2c-6f75-08d4453d2004 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DM5PR15MB1196; X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1196;3:xbCrv70xePaUMH+rwzJ6xSGXz/RrFBtjY6c2RvqwQTCYyEJjQ2aI9eQZQawJjubYuOMg8tJMG13u1wgGMvlIm1636w5jKEr93iVQjJ7SPfelMJT9BdFEhN5iDetOFC3fS7hVnnnyy6JcU8K7GF/1WCcblZdCwbrGKUYJqK4xDJrejsIVFuF0Wt8oeCjiSR2Eo1rwQRuXYAEnM/Gzc2Rzv54ArEolSLAmMQYoj+6YPybn0DNxEMNji9PnRobi5xTczC1JQDdNr4D+tMn/Cs30bg== X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1196;25:yQm+gEvdQf2JzzrjGwVe8hcU7Phg3H7HqUzlS1vB35pbFndCLJnpIY4fv2ydoh+LiSbyEi8KkowHuPNaYlYj9V7rRFKGTmiitlBVXEdREg+MaZrzRkBlWOtBO1Mv1GiDdAYil40gsZleS89l6yHdUkfJHtM/tpORAQIC59vGAVTYd3K3pJKfdsu0x4gRCE5DzlFT26R7IjxlMZRjzjWsWtISBxU1vD78MdnXB7b2YWDMtUpxLYK28OX8gV6GfLymPhYwzJlYVSVSIXksv8UVLcaSuP7xfVA1SuYmMEs37B9JzdRPeSDWIPF7m1grHzDTj1R4QmwthL+tVovIgBl+P/ElEWM79KFjdm9Q7VLPHVP+E8CJdidWdYp7eKL8yQkr+cMmMTrltNTpBSAHK9vO1szWy/MY/EwqFxetTTXpsRZEFsgng3T3IdlXQaRGKOuF7Edk5nmPo1RgooAFPJmBSccvzvnKHxnff7NIx33uJ/OVCqEaew6FXDsq2OwEkLqHF+WrZEiIpEFZAnwqxHFIH6zzVJ5upnhuSkLU42e3v50S8/nNfQMmfFBKNum2aRNlh2hDALh+M6U14iba5QpZrzb8D7eM84e0IuNbRr1w/hSUpBiezXezqZIYshf7omPFqe2JqC1LcVc4Fj3Ywi7QZ8WUxMuwqQ/CV7PqQkmfcveLPW2IIjKZSZsrFu5ZDqAQ72zVjY7Sfo72brstTm4mhZG9Tv74Cln/nWr8SAvYtc4AftDRgW7wQmMjmmbkdAZw X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1196;31:QWbRXU6MHaUjaiobnAHqDOXJ4zl6LBJ1Q7yR9KUQ1ADqgEc2VnxFaUAfjXKNmgg4kk5XTmu5C7H+BzfJxDKpuO+mvXAkQ9VPVYFUQmFtN0RzU6w9An7acdC3T4MliKj0RG7Dj44Ljamdo0wlfg4hzX1ETtYtxnK+DMJ1PDl+MoajISiMFU7KvDk9ZTFRrH4DKK7fhiYO+0xMKwGfJngVt0EkcCxjS2kLDnbR161UuifWF53ZzjF58O4NtfgQEA1E;20:W8OwEHtet0zkzOFJ9dbvJATvsp1qdQXdiHcxAnhd7vSUpPUEStr4dZXToC2lcdm1ElBaOyIi/6i41xNqTAHdI+EIflZDwTesV0FTs+WqXVlk3E800qcf9YLEjc79vr9XCfRcmF+FRXfA16DUT8GDITeY7rqHEXvcRLJx5dbCtAfUccolR0zJr/H40bAFYGLUb0RvipFr5f9OPIbc9pySqCgZiExKda8pXLaY5YtKhBzIVS85QWHJjCVVKycMXXGbBhAE4bBfXZExa1RnY3AT5x+X+Fvm15FuRWuP8LFwlgX9ghMSUtKoTEmAo2T+sSDK03KuAvH2prTkNE9Roiypnp+qDqClZ7mi85HZ8UZsjJm7b7Hf4b6GPM76aHoSWOJ/xeTgXbjJUJFZNkCy3HZ+bAiqXaiM6EEXSacQQpRlrcBztuZpuWcg+rilsOaAofZ4dzf72UKWqjNDPHlPWx+yXVIDbdRFKhO3f/XcKOXu4BVArf7r/uy7wBpbHSlZkqhc X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(67672495146484); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6041248)(20161123560025)(20161123562025)(20161123555025)(20161123564025)(6072148);SRVR:DM5PR15MB1196;BCL:0;PCL:0;RULEID:;SRVR:DM5PR15MB1196; X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1196;4:nS16cAOGovsN4kWraZ5uqhyMtDWX8IPOn0LySii1XnTk7nKJ+oUJlTubS4jN9NwC6RuUxEH40NovxKobwsTSjgeXrZTduXZVq9I6wjI0+ehB745ZaxRugYkkqPKI9RL4MjOflrGh6GNwe+PRsULFJhmyiux/U2pKePt7v/wwL5IE5xPhuNJxba6h0Fz9n6dJtR3a1d5lO7q5SGrTMAVV4lhru7CW2JHbELkr5sDPgTS5esqVPwUXzbpdEYOiTXPU6xbDltt8HfUF5x3mWq+CZXxO0dddlBGi5dGDF/9hTiWRS+9gtsziNfKOUhJoosdpQ9UkIB02dBE9gDwDB++PFQvfChVdli5K7L22ZNxMkSwnERBUSA1FPWgCMKTgnOjylrA/F7hoGEqtwaeOQGcpysIwdziTdYOwWfsdyFzZ6aABjH0e5xs+nyWrFDixbs5SrMfb3bQNjqHchbWm1MSAaA2JMcyotMec02jeD/JmOwPWoAJbjAjC2cMkz/WwOm8kHXXUHUnGIiX0wCQDDLY6GlF2SVsVkxV8zNaZ9tBJEviQO9w9xO4ES2IitNmxP4SwYk6POnd2rsuWGbV3CsLZ6cWFAHBl2Up8HMHY0XRTgBSg4AT9xKbnXMn1iZ8BcxiX X-Forefront-PRVS: 01986AE76B X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6049001)(6009001)(7916002)(39450400003)(189002)(377454003)(199003)(24454002)(3846002)(93886004)(90366009)(65826007)(5660300001)(6116002)(230783001)(68736007)(117156001)(305945005)(54906002)(64126003)(33646002)(50466002)(25786008)(5890100001)(83506001)(189998001)(31696002)(47776003)(65806001)(66066001)(6486002)(65956001)(77096006)(86362001)(230700001)(23746002)(7736002)(6916009)(6666003)(97736004)(36756003)(4326007)(4001350100001)(31686004)(110136003)(101416001)(81156014)(8676002)(2906002)(38730400001)(229853002)(92566002)(53936002)(81166006)(42186005)(76176999)(106356001)(50986999)(2950100002)(105586002)(54356999);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR15MB1196;H:[192.168.1.154];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;DM5PR15MB1196;23:g8PZETaCzdLPABdJ3Dj0HCuW3dKcu1EmkwsmR?= =?Windows-1252?Q?3wTlZ68FdOj9qxdvVYYVxP5ogo5KeO2cPwAZjJbibLqh+zR8DFEeClWU?= =?Windows-1252?Q?osMgvHFAuAj4OFyCHGe29H1Q7eU2M110Ieg5GWFIZ+Sv5ZefCmJdM2Gl?= =?Windows-1252?Q?fsLMjvyNjtrMbksG+tMmGqss1ogQIIwVdU4InGp/xE4QYfOKdxOEX3o4?= =?Windows-1252?Q?+V5y9ZLQIjDXey+C85F2WO7gF6BGHNj1dEj9kpnXPTpdxZBqaBGvn8hF?= =?Windows-1252?Q?gnLtqUlmjIbxhI99nftyg/ksbGe+jEgmqwS9AopL6pgx0mj90CeJdoLc?= =?Windows-1252?Q?nG7OX874pZQEEgrofnomOpwzKfs9n46U1XyRAJkjHucFmR9BoCqFVgUS?= =?Windows-1252?Q?A/1BU0x+Meusz+1tjxPOXHvkgGGvYRCrd/XTZlGls/v6VPsqZQYSwY43?= =?Windows-1252?Q?N4IkOzdd3M8GvZztrG69iHiE2YVdfVN2NktT4c2cHpKAPSQcLpqW97CM?= =?Windows-1252?Q?2cj0mdPxqEvo6YYqCt0w8yc2yTPXNT27iyo9ajheTZoqKK6n3N9Qr5G6?= =?Windows-1252?Q?PyRN9L+oxCpFLWSnd/+fEDtN3VBZZOdJHSR4MaqDNlyM62VrTaWRg/ke?= =?Windows-1252?Q?ZiXr1O9QIyXxyYAaGmDm6oGbwyzsbhXNfamKed1BYpshgI1RLuHbqXh9?= =?Windows-1252?Q?jLM/2LFz/4sRPe4VBjVsYlMlyjvg4rcY07ILrQ7EX9hs7gjYAd8+NYrE?= =?Windows-1252?Q?4nLYS3oU7M3+/duKX+4ZI9Q4Kn0mjfH61EKTo3BE6cHPzpxm65sFTGXi?= =?Windows-1252?Q?OggeMUSIRQ3DnCzwUL02KBPpPOfAlbDIsHz9keZFB6xxFaR5KTC6eCYi?= =?Windows-1252?Q?gCu5KFubbEKq0rt9cDkfWuN8JZXdw/NEGPup+uIN8FMFhscVY2+ACqa/?= =?Windows-1252?Q?St9bz+k1f803O7h59PVt7dSN+yXsRLQkRag1xvMaRATFdwgu+IvL4pX6?= =?Windows-1252?Q?ZjQcS7bT5ylDUVgYuXad/pIAdBro92o+KS4oc2/2Lojy2FmlymLyj+N0?= =?Windows-1252?Q?3y3SCPdXAODkx1KlCBB8i4aY4qsQbymNti3UWbtVSbnbDp4oSXz1bpoN?= =?Windows-1252?Q?uXKTRzk8cueMHqaFISjSScZ0IzzgPzPzUmX8veWxCrM06eGK7p8m55iX?= =?Windows-1252?Q?O3NXMPvIVie/5ACGDAi/pN3MH2zm47Lm9l1F6fmyD9oY8hDMWZCwNSzv?= =?Windows-1252?Q?wa6jMvEDV026V2RrcF2l6yZA1J/R7yI8k/VSUSG1AW1tIAB24Q7T6S/n?= =?Windows-1252?Q?jMWHzquvXnyJnh/oja7vkAJk7NM1E28vIfhs9kyoRui9RRnT1z92gsyu?= =?Windows-1252?Q?iDWXzAZfU/x0fZrncmgvuQIm2krOUgXxEd2WGDXlcL/wgsvXhQAM3aKf?= =?Windows-1252?Q?eHF/nzQNcTRm9uwBMEJB0nZAuI7qDuOqR1Rn6uuVanoF6S/FWqC8AGqu?= =?Windows-1252?Q?ZleX3MRTcXeOqKD/GisBou71OB93p2gtFPybevQ5MDjwSt6gGdvupHKz?= =?Windows-1252?Q?hFpDdyZfy7pm9NtMDQ+IIld6QNrQOkJlLJ1plplDz9FpOB0kT5fLWPQf?= =?Windows-1252?B?Zz09?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1196;6:b+A+btIwYnt6uQ2HTSCH7P09hXGCNoSAOL0C6eR+IlCyIIyf1wwT21XHPgEszsgur3MNC6DyfPj7i46YNST72SEIu/b6I43b23UsYnXhLHY4ID3XrzoPcclnZpJ03/0ri8NRoBXXHGnC+ZNOJgYb317btKhDEwIuQ2ZGVbqcOHdPF5p8BHK7vCDsKz59B2fjxvDONYQxHkFXDpkX/be5+nJzEs3JivUn7pFXaV5ISxOg3HixaQ7UeuFUWiZCOPsB5KuoNzZpzFFlUlw3iXIHsRtnDy3BFc+VmeKUknPFLXLkSl42R5rmgwVBDa30DmHPAw4GdZq1G6SWOpx4JqtQVXVz/fpgyzoHInBTU9BDPbTfOujyKKRZsNm85wq1IPtgK+LYEVMCk/M9Pn7prAPTBl5hsygQ2vBztADLEAcBPbg=;5:RRjgOJqU830GvaBtSMGiIOE8RUCXWmv1SFepXMQ0dsvt1XBLCd1WoUKXMQv7di88yol4quWF3od/Qm/OTV3fU5yJYw560UZy3Zzs9i1Gaz+majwimB+MWoDxr6laj9LtIvAw5bSlYW5oCDRlQZ4yfw==;24:VJsYTgB8/2yOUzNWkij3ECoxH7jY0YJSr7L0D2N0miv7L4USi1ZrfD/mV76S2WfQLsZff4HjFpTla0Rih58hY4Rme0DVTXM6QSf/JbpdQSs= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1196;7:QTfWJ1YNukGFEzAwPtZL2qZ/tv9csHxoxeXmOUib7qgE/TrMo97KcS52d9e0nTPNJzdWDy/mYnzH6cKNVlrw/1IQ/9b6wox3n6GUaJSgzVc5jQ0Que4uz+cjJgDzOSOpOQAx9Aw+Vk4q+r+MEUfsa1tzDoDaIv4+UwWfveJWjDdIB4I4avthu/4iauVTctg8dLG3kyXmJ+IjSO7QX7IeDfV4OWqrhHEy0Zt5Fuuhh/RiKQO6hixiXgtEBCS2Bzhir+YAOFnT1g9CorgOHearrQfcj8d8IEf/jNZ9RG3Echb9pXSGk4SVGckIPWoNQJnhDMyXWTWFRELQomV/6uvu2ry20omHPlnrKyMHMCEpgoSB1cmM/xz/7+6Ze8o+BOjPXiH7F0JPfIK3b6IV8PZpSp89tvz7XrNdVE5csfirXb4EUCMSHbP8Jv72NFuWE508PP8cqGPLLYjAJr/dqzi0jQ==;20:yI7bpI42xXSxXwnGwDW6n/pr6TMDhefg/BfpkH5GJeAFmX3PAaK335vUhpvB/kLlE3zSqqY/oNHOnI/6SvWRXQywbS2rpBh0r2oY7nNsJ94ipfEZ2OIpcQT7BEjbp4HR/DntTnBCGLtTKz+NnHJZEpXccqNMzB3YcJ2qF6WA+hE= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2017 16:13:39.0448 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1196 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-01-25_11:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 01/25/2017 01:46 AM, Paolo Valente wrote: > >> Il giorno 23 gen 2017, alle ore 18:42, Jens Axboe ha scritto: >> >> On 01/23/2017 10:04 AM, Paolo Valente wrote: >>> >>>> Il giorno 18 gen 2017, alle ore 17:21, Jens Axboe ha scritto: >>>> >>>> On 01/18/2017 08:14 AM, Paolo Valente wrote: >>>>> according to the function blk_mq_sched_put_request, the >>>>> mq.completed_request hook seems to always be invoked (if set) for a >>>>> request for which the mq.put_rq_priv is invoked (if set). >>>> >>>> Correct, any request that came out of blk_mq_sched_get_request() >>>> will always have completed called on it, regardless of whether it >>>> had IO started on it or not. >>>> >>> >>> It seems that some request, after being dispatched, happens to have no >>> mq.put_rq_priv invoked on it now or then. Is it expected? If it is, >>> could you point me to the path through which the end of the life of >>> such a request is handled? >> >> I'm guessing that's a flush request. I added RQF_QUEUED to check for >> that, if RQF_QUEUED is set, you know it has come from your get_request >> handler. >> > > Exactly, the completion-without-put_rq_priv pattern seems to occur > only for requests coming from the flusher, precisely because they have > the flag RQF_ELVPRIV unset. Just to understand: why is this flag > unset for these requests, if they do have private elevator (bfq) > data attached? What am I misunderstanding? > > Just to be certain: this should be the only case where the > completed_request hook is invoked while the put_rq_priv is not, right? They must NOT have scheduler data attached. In your get_request function, you must bypass if blk_mq_sched_bypass_insert() returns true. See how mq-deadline does that. This is important, or you will get hangs with flushes as well, since the IO scheduler private data and the flush data is unionized in the request. -- Jens Axboe