From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753698AbbDAOgk (ORCPT ); Wed, 1 Apr 2015 10:36:40 -0400 Received: from mail-pd0-f176.google.com ([209.85.192.176]:35899 "EHLO mail-pd0-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753490AbbDAOgi (ORCPT ); Wed, 1 Apr 2015 10:36:38 -0400 Message-ID: <551C0273.2030004@kernel.dk> Date: Wed, 01 Apr 2015 08:36:35 -0600 From: Jens Axboe User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Marcelo Tosatti , Rik van Riel CC: axboe@kernel.org, fweisbec@redhat.com, mingo@kernel.org, linux-kernel@vger.kernel.org, lcapitulino@redhat.com Subject: Re: [PATCH RFC] nohz,blk-mq: do not create blk-mq workqueues on nohz dedicated CPUs References: <20150331102726.076a6860@annuminas.surriel.com> <20150331231700.GB4161@amt.cnet> In-Reply-To: <20150331231700.GB4161@amt.cnet> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/31/2015 05:17 PM, Marcelo Tosatti wrote: > On Tue, Mar 31, 2015 at 10:27:26AM -0400, Rik van Riel wrote: >> CPUs with nohz_full do not want disruption from timer interrupts, >> or other random system things. This includes block mq work. >> >> There is another issue with block mq vs. realtime tasks that run >> 100% of the time, which is not uncommon on systems that have CPUs >> dedicated to real time use with isolcpus= and nohz_full= >> >> Specifically, on systems like that, a block work item may never >> get to run, which could lead to filesystems getting stuck forever. >> >> We can avoid both issues by not scheduling blk-mq workqueues on >> cpus in nohz_full mode. >> >> Question for Jens: should we try to spread out the load for >> currently offline and nohz CPUs across the remaining CPUs in >> the system, to get the full benefit of blk-mq in these situations? >> >> If so, do you have any preference on how I should implement that? >> >> Cc: Frederic Weisbecker >> Cc: Ingo Molnar >> Cc: Jens Axboe >> Signed-off-by: Rik van Riel >> --- >> block/blk-mq.c | 5 +++++ >> 1 file changed, 5 insertions(+) >> >> diff --git a/block/blk-mq.c b/block/blk-mq.c >> index 4f4bea21052e..1004d6817fa4 100644 >> --- a/block/blk-mq.c >> +++ b/block/blk-mq.c >> @@ -21,6 +21,7 @@ >> #include >> #include >> #include >> +#include >> >> #include >> >> @@ -1760,6 +1761,10 @@ static void blk_mq_init_cpu_queues(struct request_queue *q, >> if (!cpu_online(i)) >> continue; >> >> + /* Do not schedule work on nohz full dedicated CPUs. */ >> + if (tick_nohz_full_cpu(i)) >> + continue; >> + >> hctx = q->mq_ops->map_queue(q, i); >> cpumask_set_cpu(i, hctx->cpumask); >> hctx->nr_ctx++; > > Rik, > > I suppose any bound workqueue queued on isolated CPUs should be moved at > queue time to other CPUs (sacrifficing performance). > > So that by doing "queue_work" on an isolated CPU would move that > work somewhere else. That wont work for blk-mq, we rely on the characteristics of bound workqueues. So it would have to be handled up front, like in the patch I sent out. -- Jens Axboe