From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757228AbdJMIEl (ORCPT ); Fri, 13 Oct 2017 04:04:41 -0400 Received: from mail.sssup.it ([193.205.80.98]:63979 "EHLO mail.santannapisa.it" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755985AbdJMIEM (ORCPT ); Fri, 13 Oct 2017 04:04:12 -0400 X-Greylist: delayed 1679 seconds by postgrey-1.27 at vger.kernel.org; Fri, 13 Oct 2017 04:04:11 EDT Date: Fri, 13 Oct 2017 10:04:04 +0200 From: Luca Abeni To: Mathieu Poirier Cc: Peter Zijlstra , Ingo Molnar , tj@kernel.org, vbabka@suse.cz, Li Zefan , akpm@linux-foundation.org, weiyongjun1@huawei.com, Juri Lelli , Steven Rostedt , Claudio Scordino , Daniel Bristot de Oliveira , "linux-kernel@vger.kernel.org" , Tommaso Cucinotta Subject: Re: [PATCH 0/7] sched/deadline: fix cpusets bandwidth accounting Message-ID: <20171013100404.41cefbe0@luca> In-Reply-To: References: <1502918443-30169-1-git-send-email-mathieu.poirier@linaro.org> <20171011160208.ebt6z6vf5ccefsky@hirez.programming.kicks-ass.net> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.30; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Mathieu, On Thu, 12 Oct 2017 10:57:09 -0600 Mathieu Poirier wrote: [...] > >> Regardless of how we proceed (using existing CPUset list or new ones) we > >> need to deal with DL tasks that span more than one root domain, something > >> that will typically happen after a CPUset operation. For example, if we > >> split the number of available CPUs on a system in two CPUsets and then turn > >> off the 'sched_load_balance' flag on the parent CPUset, DL tasks in the > >> parent CPUset will end up spanning two root domains. > >> > >> One way to deal with this is to prevent CPUset operations from happening > >> when such condition is detected, as enacted in this set. Although simple > >> this approach feels brittle and akin to a "whack-a-mole" game. A better > >> and more reliable approach would be to teach the DL scheduler to deal with > >> tasks that span multiple root domains, a serious and substantial > >> undertaking. > >> > >> I am sending this as a starting point for discussion. I would be grateful > >> if you could take the time to comment on the approach and most importantly > >> provide input on how to deal with the open issue underlined above. > > > > Right, so teaching DEADLINE about arbitrary affinities is 'interesting'. > > > > Although the rules proposed by Tomasso; if found sufficient; would > > greatly simplify things. Also the online semi-partition approach to SMP > > could help with that. > > The "rules" proposed by Tomasso, are you referring to patches or the > deadline/cgroup extension work that he presented at OSPM? No, that is an unrelated thing... Tommaso previously proposed some improvements to the admission control mechanism to take arbitrary affinities into account. I think Tommaso's proposal is similar to what I previously proposed in this thread (to admit a SCHED_DEADLINE task with utilization u = runtime / period and affinity to N runqueues, we can account u / N to each one of the runqueues, and check if the sum of the utilizations on each runqueue is < 1). As previously noticed by Peter, this might have some scalability issues (a naive implementation would lock the root domain while iterating on all the runqueues). Few days ago, I was discussing with Tommaso about a possible solution based on not locking the root domain structure, and eventually using a roll-back strategy if the status of the root domain changes while we are updating it. I think in a previous email you mentioned RCU, which might result in a similar solution. Anyway, I am adding Tommaso in cc so that he can comment more. > I'd also be > interested to know more about this "online semi-partition approach to > SMP" you mentioned. It is basically an implementation (and extension to arbitrary affinities) of this work: http://drops.dagstuhl.de/opus/volltexte/2017/7165/ Luca > Maybe that's a conversation we could have at the > upcoming RT summit in Prague. > > > > > But yes, that's fairly massive surgery. For now I think we'll have to > > live and accept the limitations. So failing the various cpuset > > operations when they violate rules seems fine. Relaxing rules is always > > easier than tightening them (later). > > Agreed. > > > > > One 'series' you might be interested in when respinning these is: > > > > https://lkml.kernel.org/r/20171011094833.pdp4torvotvjdmkt@hirez.programming.kicks-ass.net > > > > By doing synchronous domain rebuild we loose a bunch of funnies. > > Getting rid of the asynchronous nature of the hotplug path would be a > delight - I'll start keeping track of that effort as well. > > Thanks for the review, > Mathieu