From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A9A2C433ED for ; Fri, 21 May 2021 08:16:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 607D46135B for ; Fri, 21 May 2021 08:16:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230341AbhEUIRV (ORCPT ); Fri, 21 May 2021 04:17:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230295AbhEUIRT (ORCPT ); Fri, 21 May 2021 04:17:19 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA419C061763 for ; Fri, 21 May 2021 01:15:56 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id r12so20137741wrp.1 for ; Fri, 21 May 2021 01:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=zT8e7kYg29cZEuKphVTHl2YAXUXxdO63nOwX3vqV14Y=; b=EVReti+ELiL60IzYNerOImeZoNrTo83zejaiKbh35740R15r3ck28CuPj/O/Iwe2wO ZiC1bwucPd/WNnTvpho3Us6X5BZzayEU59+S5QTZWuf0abCU1HO9gFsmdOBaFiJiddK0 4Bq33NVWEXFRAuh7E2azfiWZE4tU2HzwmvFDhfDInI+znqAK4dsbmtfiVL549fVvoNxJ 5UE5jCQhOCkcejAR+thHZ/8rC8EqEA52MMktCRR4xNtMNjp7tfPiFTj/MhqKyS+xkxlL eCx42D0f+ifqe+avdh8Df7FN/u9YxLO5habZhFEWD+CY3SJy0oyXP7ZsPk/ZoyWB5wL6 Vauw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=zT8e7kYg29cZEuKphVTHl2YAXUXxdO63nOwX3vqV14Y=; b=J229NFMWKEz+2Igkk2sqP1Z2T9wT1+tyewGBoKcbNbsHkLAOGmOiPtdVh50Q5yO64n p7ViHMA1YAzQtH3aofiD73jKk13Q374m+LyBN2Bhcy6GdT+5CZS4urFB4ZHzW5hrET+R 76tH77sGWYudbkofttLOy5PUnAivqUCf0O7yQe6iZiN7eg1OrmuzAjbGou9neKcim7lj TwbXk5ZgWfuR3lTp12CnmHdXK2ISuzOVZ2YgmJYL1cO4zojBLU1XNchlKAwaMJMQDE5m kUmZ/EofWqzwd0BqxNugHQ0rd2LqWeuzeDpopooK5+XO5wXXDJqndB++N6yH1vjh9dbb tUwA== X-Gm-Message-State: AOAM533Ky7TVfM1MDqjQPMQ0XRsYnyrmr8V2Khl7gXIwHJChfy3YIVek g9hEosg3JwawKone+ORNPDkbgA== X-Google-Smtp-Source: ABdhPJyaHV6Z9gNyiik1g5x3E8QnahnNsRAei/IuEo3VgJ8sjRloFFLXTBxaK8sGkYW9DAmQ/tNLXw== X-Received: by 2002:adf:fc4a:: with SMTP id e10mr7956183wrs.375.1621584955270; Fri, 21 May 2021 01:15:55 -0700 (PDT) Received: from google.com (105.168.195.35.bc.googleusercontent.com. [35.195.168.105]) by smtp.gmail.com with ESMTPSA id t16sm1261347wrb.66.2021.05.21.01.15.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 01:15:54 -0700 (PDT) Date: Fri, 21 May 2021 08:15:52 +0000 From: Quentin Perret To: Juri Lelli Cc: Will Deacon , Daniel Bristot de Oliveira , linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas , Marc Zyngier , Greg Kroah-Hartman , Peter Zijlstra , Morten Rasmussen , Qais Yousef , Suren Baghdasaryan , Tejun Heo , Johannes Weiner , Ingo Molnar , Vincent Guittot , "Rafael J. Wysocki" , kernel-team@android.com Subject: Re: [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE Message-ID: References: <20210518102833.GA7770@willie-the-truck> <20210518105951.GC7770@willie-the-truck> <20210520101640.GA10065@willie-the-truck> <20210520180138.GA10523@willie-the-truck> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Friday 21 May 2021 at 07:25:51 (+0200), Juri Lelli wrote: > On 20/05/21 19:01, Will Deacon wrote: > > On Thu, May 20, 2021 at 02:38:55PM +0200, Daniel Bristot de Oliveira wrote: > > > On 5/20/21 12:33 PM, Quentin Perret wrote: > > > > On Thursday 20 May 2021 at 11:16:41 (+0100), Will Deacon wrote: > > > >> Ok, thanks for the insight. In which case, I'll go with what we discussed: > > > >> require admission control to be disabled for sched_setattr() but allow > > > >> execve() to a 32-bit task from a 64-bit deadline task with a warning (this > > > >> is probably similar to CPU hotplug?). > > > > > > > > Still not sure that we can let execve go through ... It will break AC > > > > all the same, so it should probably fail as well if AC is on IMO > > > > > > > > > > If the cpumask of the 32-bit task is != of the 64-bit task that is executing it, > > > the admission control needs to be re-executed, and it could fail. So I see this > > > operation equivalent to sched_setaffinity(). This will likely be true for future > > > schedulers that will allow arbitrary affinities (AC should run on affinity > > > change, and could fail). > > > > > > I would vote with Juri: "I'd go with fail hard if AC is on, let it > > > pass if AC is off (supposedly the user knows what to do)," (also hope nobody > > > complains until we add better support for affinity, and use this as a motivation > > > to get back on this front). > > > > I can have a go at implementing it, but I don't think it's a great solution > > and here's why: > > > > Failing an execve() is _very_ likely to be fatal to the application. It's > > also very likely that the task calling execve() doesn't know whether the > > program it's trying to execute is 32-bit or not. Consequently, if we go > > with failing execve() then all that will happen is that people will disable > > admission control altogether. Right, but only on these dumb 32bit asymmetric systems, and only if we care about running 32bits deadline tasks -- which I seriously doubt for the Android use-case. Note that running deadline tasks is also a privileged operation, it can't be done by random apps. > > That has a negative impact on "pure" 64-bit > > applications and so I think we end up with the tail wagging the dog because > > admission control will be disabled for everybody just because there is a > > handful of 32-bit programs which may get executed. I understand that it > > also means that RT throttling would be disabled. > > Completely understand your perplexity. But how can the kernel still give > guarantees to "pure" 64-bit applications if there are 32-bit > applications around that essentially broke admission control when they > were restricted to a subset of cores? > > > Allowing the execve() to continue with a warning is very similar to the > > case in which all the 64-bit CPUs are hot-unplugged at the point of > > execve(), and this is much closer to the illusion that this patch series > > intends to provide. > > So, for hotplug we currently have a check that would make hotplug > operations fail if removing a CPU would mean not enough bandwidth to run > the currently admitted set of DEADLINE tasks. Aha, wasn't aware. Any pointers to that check for my education? > > So, personally speaking, I would prefer the behaviour where we refuse to > > admit 32-bit tasks vioa sched_set_attr() if the root domain contains > > 64-bit CPUs, but we _don't_ fail execve() of a 32-bit program from a > > 64-bit deadline task. > > OK, this is interesting and I guess a very valid alternative. That would > force users to create exclusive domains for 32-bit tasks, right? FWIW this is not practical at all for our use-cases, the implications of splitting the system in independent root-domains are way too important for us to be able to recommend that. Disabling AC, OTOH, sounds simple enough. The RT throttling part is the only 'worrying' part, but even that may not be the end of the world. Thanks! Quentin From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BE98C433ED for ; Fri, 21 May 2021 08:18:28 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C440613AC for ; Fri, 21 May 2021 08:18:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C440613AC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Nvprsjo8xR0PTgkt/3N+zT+3Kca7CkLCIcnc9jRJNqU=; b=OKcco2XL8aolEO3hbjKZqN4fCV ca3QX73sVE9f9RSw9oodZuD8vLSNIdN9JIDlDXR1Jad0EFMs0DhgkedsDZaStGZA5st2QkhCNCSZy CRm9ET6JbfRVIx8y8fym0l3g60wyxlD011OhfpE0KE+hnyZg2eaRNwcdF/fzjxXUjGvBf/IljOwAJ UStCwMSOaZ2FO0+SGKv7H6rhu+0HsBzW0efjL/lSMVmrOWoc11J9hucxs7WLkZWA0v2hJj+M17qma tysckKUoCtE2UvbEEXg+Nrx017KTp5C681okkdYN7XuezRSA3U2UYlh+6X1+hnzEF749T4HjWjr4d i1NW7QqQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lk0KC-004ORe-6K; Fri, 21 May 2021 08:16:04 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lk0K8-004ORP-1v for linux-arm-kernel@desiato.infradead.org; Fri, 21 May 2021 08:16:01 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=In-Reply-To:Content-Type:MIME-Version :References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=zT8e7kYg29cZEuKphVTHl2YAXUXxdO63nOwX3vqV14Y=; b=LHIS6Rq/QF7X3oFN5LBcM/mqdL mVh2Eij2K9oCV08ndilOiPH7WFn0o5vjPT+hSgjfvCVxmBzavXWXw5WAr1x88m4PbcZHvZaCc8YLk Uz08oqGTSoonvIixPDHnrB6Q7wSR6Z+gJE3u9Pbp30BLtMl2oE4ecPaSuyf0P/5WKqsfP1T8Cehog yFUfES4HNcKygTEQFvrqNuGFEUM5AqRRMdAM0NJq8d5Smw0FxqCHOf5cbRbgRK44pGS1Gg80XSDtp 7gdVUsj2vc0Iv2ObLofmfqi+v27TQULUJ3eUiAqFNGah5EW98cnuxmg1MGwRGdPw64D9BnjPS/PrX GrAtwQKA==; Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lk0K4-00Gw6f-Vr for linux-arm-kernel@lists.infradead.org; Fri, 21 May 2021 08:15:58 +0000 Received: by mail-wr1-x430.google.com with SMTP id x7so518595wrt.12 for ; Fri, 21 May 2021 01:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=zT8e7kYg29cZEuKphVTHl2YAXUXxdO63nOwX3vqV14Y=; b=EVReti+ELiL60IzYNerOImeZoNrTo83zejaiKbh35740R15r3ck28CuPj/O/Iwe2wO ZiC1bwucPd/WNnTvpho3Us6X5BZzayEU59+S5QTZWuf0abCU1HO9gFsmdOBaFiJiddK0 4Bq33NVWEXFRAuh7E2azfiWZE4tU2HzwmvFDhfDInI+znqAK4dsbmtfiVL549fVvoNxJ 5UE5jCQhOCkcejAR+thHZ/8rC8EqEA52MMktCRR4xNtMNjp7tfPiFTj/MhqKyS+xkxlL eCx42D0f+ifqe+avdh8Df7FN/u9YxLO5habZhFEWD+CY3SJy0oyXP7ZsPk/ZoyWB5wL6 Vauw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=zT8e7kYg29cZEuKphVTHl2YAXUXxdO63nOwX3vqV14Y=; b=Ezsk2ysVDOucwNLB9sIs5jaOrqqBVkeuxCuNOplP2T4wHLgx8pHJg++XrLR5Em5DqO L+8RTUsIgtCQk1y2wDjt7p5J0AWDUw5lQXTvJ4n4oVNChRY26WoNckiGhVgm9Tj8weNv S30//HqrgzL67FZPQWAOLchJoexSeTOZ9grymb9Ge76S1Ro63vot0RJIsoEd03VZWg3+ yHON0lEuFG8RoRD6U1ekUIsVBW+0NBFwrA2+mf4OBkYXg0Uhuh2GOovTs3mVW1Xjcrps hfi3lqge6IU74ATyxv5kjPplaPNIvn3xRsFbe7ZjQ61W56vbEYL/CxCPZLzU+JBL6P+Q Ygag== X-Gm-Message-State: AOAM533P/VBJJjDk/R4jYwzeKB03z9ntHA+zrLoBcmfnyzA13+WfTEP4 Qwe3Uqq089t6Oi/hz4AKabr6+g== X-Google-Smtp-Source: ABdhPJyaHV6Z9gNyiik1g5x3E8QnahnNsRAei/IuEo3VgJ8sjRloFFLXTBxaK8sGkYW9DAmQ/tNLXw== X-Received: by 2002:adf:fc4a:: with SMTP id e10mr7956183wrs.375.1621584955270; Fri, 21 May 2021 01:15:55 -0700 (PDT) Received: from google.com (105.168.195.35.bc.googleusercontent.com. [35.195.168.105]) by smtp.gmail.com with ESMTPSA id t16sm1261347wrb.66.2021.05.21.01.15.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 01:15:54 -0700 (PDT) Date: Fri, 21 May 2021 08:15:52 +0000 From: Quentin Perret To: Juri Lelli Cc: Will Deacon , Daniel Bristot de Oliveira , linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas , Marc Zyngier , Greg Kroah-Hartman , Peter Zijlstra , Morten Rasmussen , Qais Yousef , Suren Baghdasaryan , Tejun Heo , Johannes Weiner , Ingo Molnar , Vincent Guittot , "Rafael J. Wysocki" , kernel-team@android.com Subject: Re: [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE Message-ID: References: <20210518102833.GA7770@willie-the-truck> <20210518105951.GC7770@willie-the-truck> <20210520101640.GA10065@willie-the-truck> <20210520180138.GA10523@willie-the-truck> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210521_011557_073245_2C733E37 X-CRM114-Status: GOOD ( 40.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Friday 21 May 2021 at 07:25:51 (+0200), Juri Lelli wrote: > On 20/05/21 19:01, Will Deacon wrote: > > On Thu, May 20, 2021 at 02:38:55PM +0200, Daniel Bristot de Oliveira wrote: > > > On 5/20/21 12:33 PM, Quentin Perret wrote: > > > > On Thursday 20 May 2021 at 11:16:41 (+0100), Will Deacon wrote: > > > >> Ok, thanks for the insight. In which case, I'll go with what we discussed: > > > >> require admission control to be disabled for sched_setattr() but allow > > > >> execve() to a 32-bit task from a 64-bit deadline task with a warning (this > > > >> is probably similar to CPU hotplug?). > > > > > > > > Still not sure that we can let execve go through ... It will break AC > > > > all the same, so it should probably fail as well if AC is on IMO > > > > > > > > > > If the cpumask of the 32-bit task is != of the 64-bit task that is executing it, > > > the admission control needs to be re-executed, and it could fail. So I see this > > > operation equivalent to sched_setaffinity(). This will likely be true for future > > > schedulers that will allow arbitrary affinities (AC should run on affinity > > > change, and could fail). > > > > > > I would vote with Juri: "I'd go with fail hard if AC is on, let it > > > pass if AC is off (supposedly the user knows what to do)," (also hope nobody > > > complains until we add better support for affinity, and use this as a motivation > > > to get back on this front). > > > > I can have a go at implementing it, but I don't think it's a great solution > > and here's why: > > > > Failing an execve() is _very_ likely to be fatal to the application. It's > > also very likely that the task calling execve() doesn't know whether the > > program it's trying to execute is 32-bit or not. Consequently, if we go > > with failing execve() then all that will happen is that people will disable > > admission control altogether. Right, but only on these dumb 32bit asymmetric systems, and only if we care about running 32bits deadline tasks -- which I seriously doubt for the Android use-case. Note that running deadline tasks is also a privileged operation, it can't be done by random apps. > > That has a negative impact on "pure" 64-bit > > applications and so I think we end up with the tail wagging the dog because > > admission control will be disabled for everybody just because there is a > > handful of 32-bit programs which may get executed. I understand that it > > also means that RT throttling would be disabled. > > Completely understand your perplexity. But how can the kernel still give > guarantees to "pure" 64-bit applications if there are 32-bit > applications around that essentially broke admission control when they > were restricted to a subset of cores? > > > Allowing the execve() to continue with a warning is very similar to the > > case in which all the 64-bit CPUs are hot-unplugged at the point of > > execve(), and this is much closer to the illusion that this patch series > > intends to provide. > > So, for hotplug we currently have a check that would make hotplug > operations fail if removing a CPU would mean not enough bandwidth to run > the currently admitted set of DEADLINE tasks. Aha, wasn't aware. Any pointers to that check for my education? > > So, personally speaking, I would prefer the behaviour where we refuse to > > admit 32-bit tasks vioa sched_set_attr() if the root domain contains > > 64-bit CPUs, but we _don't_ fail execve() of a 32-bit program from a > > 64-bit deadline task. > > OK, this is interesting and I guess a very valid alternative. That would > force users to create exclusive domains for 32-bit tasks, right? FWIW this is not practical at all for our use-cases, the implications of splitting the system in independent root-domains are way too important for us to be able to recommend that. Disabling AC, OTOH, sounds simple enough. The RT throttling part is the only 'worrying' part, but even that may not be the end of the world. Thanks! Quentin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel