From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86898C47076 for ; Fri, 21 May 2021 17:47:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 607A96135C for ; Fri, 21 May 2021 17:47:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238367AbhEURst (ORCPT ); Fri, 21 May 2021 13:48:49 -0400 Received: from foss.arm.com ([217.140.110.172]:52684 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235764AbhEURsr (ORCPT ); Fri, 21 May 2021 13:48:47 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 316201424; Fri, 21 May 2021 10:47:24 -0700 (PDT) Received: from [192.168.1.16] (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B6F273F73D; Fri, 21 May 2021 10:47:20 -0700 (PDT) Subject: Re: [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE To: Juri Lelli , Quentin Perret Cc: Will Deacon , Daniel Bristot de Oliveira , linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas , Marc Zyngier , Greg Kroah-Hartman , Peter Zijlstra , Morten Rasmussen , Qais Yousef , Suren Baghdasaryan , Tejun Heo , Johannes Weiner , Ingo Molnar , Vincent Guittot , "Rafael J. Wysocki" , kernel-team@android.com References: <20210520101640.GA10065@willie-the-truck> <20210520180138.GA10523@willie-the-truck> <20210521103724.GA11680@willie-the-truck> <3620bad5-2a27-0f9e-f1f0-70036997d33c@arm.com> From: Dietmar Eggemann Message-ID: <1031558c-acc8-d1b2-2964-ed78fd9b22a0@arm.com> Date: Fri, 21 May 2021 19:47:19 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 21/05/2021 16:04, Juri Lelli wrote: > On 21/05/21 13:02, Quentin Perret wrote: > > ... > >> So I think Will has a point since, IIRC, the root domains get rebuilt >> during hotplug. So you can imagine a case with a single root domain, but >> CPUs 4-7 are offline. In this case, sched_setattr() will happily promote >> a task to DL as long as its affinity mask is a superset of the rd span, >> but things may get ugly when CPUs are plugged back in later on. Yeah, that's true. I understand the condition, that the task's affinity mask has to be a superset of the rd span, as that DL AC (i.e DL BW management) can only work correctly if all admitted tasks can run on every CPU in the rd. Like you said, you can already today let tasks with reduced affinity mask pass the DL AC in case you hp out the other CPUs and then trick DL AC by hp in the remaining CPUs and admit more DL tasks. But these steps require a lot of effort to create this false setup. The dedicated rd for 32-bit tasks matching `aarch32_el0` in an exclusive cpuset env seems to be a feasible approach to me. But I also don't see an eminent use case for this. >> This looks like an existing bug though. I just tried the following on a >> system with 4 CPUs: >> >> // Create a task affined to CPU [0-2] >> > while true; do echo "Hi" > /dev/null; done & >> [1] 560 >> > mypid=$! >> > taskset -p 7 $mypid >> pid 560's current affinity mask: f >> pid 560's new affinity mask: 7 >> >> // Try to move it DL, this should fail because of the affinity >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> chrt: failed to set pid 560's policy: Operation not permitted >> >> // Offline CPU 3, so the rd now covers CPUs 0-2 only >> > echo 0 > /sys/devices/system/cpu/cpu3/online >> [ 400.843830] CPU3: shutdown >> [ 400.844100] psci: CPU3 killed (polled 0 ms) >> >> // Try to admit the task again, which now succeeds >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> >> // Plug CPU3 back online >> > echo 1 > /sys/devices/system/cpu/cpu3/online >> [ 408.819337] Detected PIPT I-cache on CPU3 >> [ 408.819642] GICv3: CPU3: found redistributor 3 region 0:0x0000000008100000 >> [ 408.820165] CPU3: Booted secondary processor 0x0000000003 [0x410fd083] >> >> I don't see any easy way to fix this w/o iterating over all deadline >> tasks in the rd when hotplugging a CPU back on, and blocking the hotplug >> operation if it'll cause affinity issues. Urgh. Something like dl_cpu_busy() in cpuset_cpu_inactive() but the other way around in cpuset_cpu_active(). We iterate over all DL tasks in partition_and_rebuild_sched_domains() -> rebuild_root_domains() -> update_tasks_root_domain() -> dl_add_task_root_domain(struct task_struct *p) to recreate DL BW information after CPU hp but this is asynchronously to cpuset_cpu_active(). > > Yeah this looks like a plain existing bug, joy. :) > > We fixed a few around AC lately, but I guess work wasn't complete. > > Thanks, > Juri From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76AF5C4707C for ; Fri, 21 May 2021 17:51:02 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ED2796135C for ; Fri, 21 May 2021 17:51:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED2796135C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:Cc:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kB1QS5diKxmdNDWVtbb1ieccW8bLeq+VmkFwQFUP4n8=; b=KPtKvHnbKfpKquJCKUEdP7IKY3 O0bNL3M+ZqeODzUz6Lr+89f7tkdNTEQrqOR2wKAuaBn3N1ye+e1e5XIj96uLa9EAqGIkgylaxmJxY TBYJa0pwo9PxqcmixVuKeSm74iQGetcywfJlF/rvAP56tHIoJy9T3cqFMTKFpmHMznZy/DT51BEw/ mSYipKbz3sHRF0thZ4SVk9OE7ssdJtw8fnZW+1fKgiN0yOHANGgWFk/t3f7UexD3Ji8/45NeKgQ5R SMTwvGH4tN8/3Zptbc+G3jmcMfTzlzJG6XxO82YFgzD+Zhtk+ZYcBaDFQxsflfS/jGONuEWm6wJ7i leuANr4Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lk9Gw-000bEe-01; Fri, 21 May 2021 17:49:18 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lk9FF-000afw-IU for linux-arm-kernel@desiato.infradead.org; Fri, 21 May 2021 17:47:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To: Subject:Sender:Reply-To:Content-ID:Content-Description; bh=Rc2E62xErvwoGmFjwIIwZ2orQLiiFuLXmki6mhDBQ9o=; b=Ua6guoKu1Ljah6dhgt6m79Y9S5 /bMiosN2VExoIehgIgI6GRhsrTfcDyu6l6TggztJinwPMMLbloLXbJ7A61YBoEt1LK+20TTu2zOMn 7sYavYVT0aFJCPiEyvfB92CQjNXjxSLB0vheHvQBnb9L4b44fwO6jj/DqMclWT0HzOqZXelzsaOut 3WgV/6c3x+oN4vgYXqoZy9eYAJgl1ZaGfywRO9MgkCJLHrbmcTS4DjqFaWjqs8hFQ5oT3WcpmFwSQ p/HKvEdc+jp5+zLwJ/+Sf04Vqaos6FtWOoz8eQY5UG1Z4i9sOM8tYJvEgub/eHyd4wdzb0yvcGwrF WJm4nLaA==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lk9FB-00HKJr-VI for linux-arm-kernel@lists.infradead.org; Fri, 21 May 2021 17:47:32 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 316201424; Fri, 21 May 2021 10:47:24 -0700 (PDT) Received: from [192.168.1.16] (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B6F273F73D; Fri, 21 May 2021 10:47:20 -0700 (PDT) Subject: Re: [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE To: Juri Lelli , Quentin Perret Cc: Will Deacon , Daniel Bristot de Oliveira , linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas , Marc Zyngier , Greg Kroah-Hartman , Peter Zijlstra , Morten Rasmussen , Qais Yousef , Suren Baghdasaryan , Tejun Heo , Johannes Weiner , Ingo Molnar , Vincent Guittot , "Rafael J. Wysocki" , kernel-team@android.com References: <20210520101640.GA10065@willie-the-truck> <20210520180138.GA10523@willie-the-truck> <20210521103724.GA11680@willie-the-truck> <3620bad5-2a27-0f9e-f1f0-70036997d33c@arm.com> From: Dietmar Eggemann Message-ID: <1031558c-acc8-d1b2-2964-ed78fd9b22a0@arm.com> Date: Fri, 21 May 2021 19:47:19 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210521_104730_109341_D28501EB X-CRM114-Status: GOOD ( 20.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 21/05/2021 16:04, Juri Lelli wrote: > On 21/05/21 13:02, Quentin Perret wrote: > > ... > >> So I think Will has a point since, IIRC, the root domains get rebuilt >> during hotplug. So you can imagine a case with a single root domain, but >> CPUs 4-7 are offline. In this case, sched_setattr() will happily promote >> a task to DL as long as its affinity mask is a superset of the rd span, >> but things may get ugly when CPUs are plugged back in later on. Yeah, that's true. I understand the condition, that the task's affinity mask has to be a superset of the rd span, as that DL AC (i.e DL BW management) can only work correctly if all admitted tasks can run on every CPU in the rd. Like you said, you can already today let tasks with reduced affinity mask pass the DL AC in case you hp out the other CPUs and then trick DL AC by hp in the remaining CPUs and admit more DL tasks. But these steps require a lot of effort to create this false setup. The dedicated rd for 32-bit tasks matching `aarch32_el0` in an exclusive cpuset env seems to be a feasible approach to me. But I also don't see an eminent use case for this. >> This looks like an existing bug though. I just tried the following on a >> system with 4 CPUs: >> >> // Create a task affined to CPU [0-2] >> > while true; do echo "Hi" > /dev/null; done & >> [1] 560 >> > mypid=$! >> > taskset -p 7 $mypid >> pid 560's current affinity mask: f >> pid 560's new affinity mask: 7 >> >> // Try to move it DL, this should fail because of the affinity >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> chrt: failed to set pid 560's policy: Operation not permitted >> >> // Offline CPU 3, so the rd now covers CPUs 0-2 only >> > echo 0 > /sys/devices/system/cpu/cpu3/online >> [ 400.843830] CPU3: shutdown >> [ 400.844100] psci: CPU3 killed (polled 0 ms) >> >> // Try to admit the task again, which now succeeds >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> >> // Plug CPU3 back online >> > echo 1 > /sys/devices/system/cpu/cpu3/online >> [ 408.819337] Detected PIPT I-cache on CPU3 >> [ 408.819642] GICv3: CPU3: found redistributor 3 region 0:0x0000000008100000 >> [ 408.820165] CPU3: Booted secondary processor 0x0000000003 [0x410fd083] >> >> I don't see any easy way to fix this w/o iterating over all deadline >> tasks in the rd when hotplugging a CPU back on, and blocking the hotplug >> operation if it'll cause affinity issues. Urgh. Something like dl_cpu_busy() in cpuset_cpu_inactive() but the other way around in cpuset_cpu_active(). We iterate over all DL tasks in partition_and_rebuild_sched_domains() -> rebuild_root_domains() -> update_tasks_root_domain() -> dl_add_task_root_domain(struct task_struct *p) to recreate DL BW information after CPU hp but this is asynchronously to cpuset_cpu_active(). > > Yeah this looks like a plain existing bug, joy. :) > > We fixed a few around AC lately, but I guess work wasn't complete. > > Thanks, > Juri _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel