From: Dietmar Eggemann <dietmar.eggemann@arm.com> To: Juri Lelli <juri.lelli@redhat.com>, Quentin Perret <qperret@google.com> Cc: Will Deacon <will@kernel.org>, Daniel Bristot de Oliveira <bristot@redhat.com>, linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas <catalin.marinas@arm.com>, Marc Zyngier <maz@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Peter Zijlstra <peterz@infradead.org>, Morten Rasmussen <morten.rasmussen@arm.com>, Qais Yousef <qais.yousef@arm.com>, Suren Baghdasaryan <surenb@google.com>, Tejun Heo <tj@kernel.org>, Johannes Weiner <hannes@cmpxchg.org>, Ingo Molnar <mingo@redhat.com>, Vincent Guittot <vincent.guittot@linaro.org>, "Rafael J. Wysocki" <rjw@rjwysocki.net>, kernel-team@android.com Subject: Re: [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE Date: Fri, 21 May 2021 19:47:19 +0200 [thread overview] Message-ID: <1031558c-acc8-d1b2-2964-ed78fd9b22a0@arm.com> (raw) In-Reply-To: <YKe94oTVSbywMw2r@localhost.localdomain> On 21/05/2021 16:04, Juri Lelli wrote: > On 21/05/21 13:02, Quentin Perret wrote: > > ... > >> So I think Will has a point since, IIRC, the root domains get rebuilt >> during hotplug. So you can imagine a case with a single root domain, but >> CPUs 4-7 are offline. In this case, sched_setattr() will happily promote >> a task to DL as long as its affinity mask is a superset of the rd span, >> but things may get ugly when CPUs are plugged back in later on. Yeah, that's true. I understand the condition, that the task's affinity mask has to be a superset of the rd span, as that DL AC (i.e DL BW management) can only work correctly if all admitted tasks can run on every CPU in the rd. Like you said, you can already today let tasks with reduced affinity mask pass the DL AC in case you hp out the other CPUs and then trick DL AC by hp in the remaining CPUs and admit more DL tasks. But these steps require a lot of effort to create this false setup. The dedicated rd for 32-bit tasks matching `aarch32_el0` in an exclusive cpuset env seems to be a feasible approach to me. But I also don't see an eminent use case for this. >> This looks like an existing bug though. I just tried the following on a >> system with 4 CPUs: >> >> // Create a task affined to CPU [0-2] >> > while true; do echo "Hi" > /dev/null; done & >> [1] 560 >> > mypid=$! >> > taskset -p 7 $mypid >> pid 560's current affinity mask: f >> pid 560's new affinity mask: 7 >> >> // Try to move it DL, this should fail because of the affinity >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> chrt: failed to set pid 560's policy: Operation not permitted >> >> // Offline CPU 3, so the rd now covers CPUs 0-2 only >> > echo 0 > /sys/devices/system/cpu/cpu3/online >> [ 400.843830] CPU3: shutdown >> [ 400.844100] psci: CPU3 killed (polled 0 ms) >> >> // Try to admit the task again, which now succeeds >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> >> // Plug CPU3 back online >> > echo 1 > /sys/devices/system/cpu/cpu3/online >> [ 408.819337] Detected PIPT I-cache on CPU3 >> [ 408.819642] GICv3: CPU3: found redistributor 3 region 0:0x0000000008100000 >> [ 408.820165] CPU3: Booted secondary processor 0x0000000003 [0x410fd083] >> >> I don't see any easy way to fix this w/o iterating over all deadline >> tasks in the rd when hotplugging a CPU back on, and blocking the hotplug >> operation if it'll cause affinity issues. Urgh. Something like dl_cpu_busy() in cpuset_cpu_inactive() but the other way around in cpuset_cpu_active(). We iterate over all DL tasks in partition_and_rebuild_sched_domains() -> rebuild_root_domains() -> update_tasks_root_domain() -> dl_add_task_root_domain(struct task_struct *p) to recreate DL BW information after CPU hp but this is asynchronously to cpuset_cpu_active(). > > Yeah this looks like a plain existing bug, joy. :) > > We fixed a few around AC lately, but I guess work wasn't complete. > > Thanks, > Juri
WARNING: multiple messages have this Message-ID (diff)
From: Dietmar Eggemann <dietmar.eggemann@arm.com> To: Juri Lelli <juri.lelli@redhat.com>, Quentin Perret <qperret@google.com> Cc: Will Deacon <will@kernel.org>, Daniel Bristot de Oliveira <bristot@redhat.com>, linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas <catalin.marinas@arm.com>, Marc Zyngier <maz@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Peter Zijlstra <peterz@infradead.org>, Morten Rasmussen <morten.rasmussen@arm.com>, Qais Yousef <qais.yousef@arm.com>, Suren Baghdasaryan <surenb@google.com>, Tejun Heo <tj@kernel.org>, Johannes Weiner <hannes@cmpxchg.org>, Ingo Molnar <mingo@redhat.com>, Vincent Guittot <vincent.guittot@linaro.org>, "Rafael J. Wysocki" <rjw@rjwysocki.net>, kernel-team@android.com Subject: Re: [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE Date: Fri, 21 May 2021 19:47:19 +0200 [thread overview] Message-ID: <1031558c-acc8-d1b2-2964-ed78fd9b22a0@arm.com> (raw) In-Reply-To: <YKe94oTVSbywMw2r@localhost.localdomain> On 21/05/2021 16:04, Juri Lelli wrote: > On 21/05/21 13:02, Quentin Perret wrote: > > ... > >> So I think Will has a point since, IIRC, the root domains get rebuilt >> during hotplug. So you can imagine a case with a single root domain, but >> CPUs 4-7 are offline. In this case, sched_setattr() will happily promote >> a task to DL as long as its affinity mask is a superset of the rd span, >> but things may get ugly when CPUs are plugged back in later on. Yeah, that's true. I understand the condition, that the task's affinity mask has to be a superset of the rd span, as that DL AC (i.e DL BW management) can only work correctly if all admitted tasks can run on every CPU in the rd. Like you said, you can already today let tasks with reduced affinity mask pass the DL AC in case you hp out the other CPUs and then trick DL AC by hp in the remaining CPUs and admit more DL tasks. But these steps require a lot of effort to create this false setup. The dedicated rd for 32-bit tasks matching `aarch32_el0` in an exclusive cpuset env seems to be a feasible approach to me. But I also don't see an eminent use case for this. >> This looks like an existing bug though. I just tried the following on a >> system with 4 CPUs: >> >> // Create a task affined to CPU [0-2] >> > while true; do echo "Hi" > /dev/null; done & >> [1] 560 >> > mypid=$! >> > taskset -p 7 $mypid >> pid 560's current affinity mask: f >> pid 560's new affinity mask: 7 >> >> // Try to move it DL, this should fail because of the affinity >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> chrt: failed to set pid 560's policy: Operation not permitted >> >> // Offline CPU 3, so the rd now covers CPUs 0-2 only >> > echo 0 > /sys/devices/system/cpu/cpu3/online >> [ 400.843830] CPU3: shutdown >> [ 400.844100] psci: CPU3 killed (polled 0 ms) >> >> // Try to admit the task again, which now succeeds >> > chrt -d -T 5000000 -P 16666666 -p 0 $mypid >> >> // Plug CPU3 back online >> > echo 1 > /sys/devices/system/cpu/cpu3/online >> [ 408.819337] Detected PIPT I-cache on CPU3 >> [ 408.819642] GICv3: CPU3: found redistributor 3 region 0:0x0000000008100000 >> [ 408.820165] CPU3: Booted secondary processor 0x0000000003 [0x410fd083] >> >> I don't see any easy way to fix this w/o iterating over all deadline >> tasks in the rd when hotplugging a CPU back on, and blocking the hotplug >> operation if it'll cause affinity issues. Urgh. Something like dl_cpu_busy() in cpuset_cpu_inactive() but the other way around in cpuset_cpu_active(). We iterate over all DL tasks in partition_and_rebuild_sched_domains() -> rebuild_root_domains() -> update_tasks_root_domain() -> dl_add_task_root_domain(struct task_struct *p) to recreate DL BW information after CPU hp but this is asynchronously to cpuset_cpu_active(). > > Yeah this looks like a plain existing bug, joy. :) > > We fixed a few around AC lately, but I guess work wasn't complete. > > Thanks, > Juri _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-05-21 17:47 UTC|newest] Thread overview: 166+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-05-18 9:47 [PATCH v6 00/21] Add support for 32-bit tasks on asymmetric AArch32 systems Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 01/21] arm64: cpuinfo: Split AArch32 registers out into a separate struct Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 10:47 ` Catalin Marinas 2021-05-21 10:47 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 02/21] arm64: Allow mismatched 32-bit EL0 support Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 10:25 ` Catalin Marinas 2021-05-21 10:25 ` Catalin Marinas 2021-05-24 12:05 ` Will Deacon 2021-05-24 12:05 ` Will Deacon 2021-05-24 13:49 ` Catalin Marinas 2021-05-24 13:49 ` Catalin Marinas 2021-05-21 10:41 ` Catalin Marinas 2021-05-21 10:41 ` Catalin Marinas 2021-05-24 12:09 ` Will Deacon 2021-05-24 12:09 ` Will Deacon 2021-05-24 13:46 ` Catalin Marinas 2021-05-24 13:46 ` Catalin Marinas 2021-05-21 15:22 ` Qais Yousef 2021-05-21 15:22 ` Qais Yousef 2021-05-24 20:21 ` Will Deacon 2021-05-24 20:21 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 03/21] KVM: arm64: Kill 32-bit vCPUs on systems with mismatched " Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 10:47 ` Catalin Marinas 2021-05-21 10:47 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 04/21] arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 10:55 ` Catalin Marinas 2021-05-21 10:55 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 05/21] arm64: Advertise CPUs capable of running 32-bit applications in sysfs Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 11:00 ` Catalin Marinas 2021-05-21 11:00 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 06/21] sched: Introduce task_cpu_possible_mask() to limit fallback rq selection Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 16:03 ` Peter Zijlstra 2021-05-21 16:03 ` Peter Zijlstra 2021-05-24 12:17 ` Will Deacon 2021-05-24 12:17 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 07/21] cpuset: Don't use the cpu_possible_mask as a last resort for cgroup v1 Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 17:39 ` Qais Yousef 2021-05-21 17:39 ` Qais Yousef 2021-05-24 20:21 ` Will Deacon 2021-05-24 20:21 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 08/21] cpuset: Honour task_cpu_possible_mask() in guarantee_online_cpus() Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 16:25 ` Qais Yousef 2021-05-21 16:25 ` Qais Yousef 2021-05-24 21:09 ` Will Deacon 2021-05-24 21:09 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 09/21] sched: Reject CPU affinity changes based on task_cpu_possible_mask() Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 10/21] sched: Introduce task_struct::user_cpus_ptr to track requested affinity Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 11/21] sched: Split the guts of sched_setaffinity() into a helper function Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 16:41 ` Qais Yousef 2021-05-21 16:41 ` Qais Yousef 2021-05-24 21:16 ` Will Deacon 2021-05-24 21:16 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 12/21] sched: Allow task CPU affinity to be restricted on asymmetric systems Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 17:11 ` Qais Yousef 2021-05-21 17:11 ` Qais Yousef 2021-05-24 21:43 ` Will Deacon 2021-05-24 21:43 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-18 10:20 ` Quentin Perret 2021-05-18 10:20 ` Quentin Perret 2021-05-18 10:28 ` Will Deacon 2021-05-18 10:28 ` Will Deacon 2021-05-18 10:48 ` Quentin Perret 2021-05-18 10:48 ` Quentin Perret 2021-05-18 10:59 ` Will Deacon 2021-05-18 10:59 ` Will Deacon 2021-05-18 13:19 ` Quentin Perret 2021-05-18 13:19 ` Quentin Perret 2021-05-20 9:13 ` Juri Lelli 2021-05-20 9:13 ` Juri Lelli 2021-05-20 10:16 ` Will Deacon 2021-05-20 10:16 ` Will Deacon 2021-05-20 10:33 ` Quentin Perret 2021-05-20 10:33 ` Quentin Perret 2021-05-20 12:38 ` Juri Lelli 2021-05-20 12:38 ` Juri Lelli 2021-05-20 12:38 ` Daniel Bristot de Oliveira 2021-05-20 12:38 ` Daniel Bristot de Oliveira 2021-05-20 15:06 ` Dietmar Eggemann 2021-05-20 15:06 ` Dietmar Eggemann 2021-05-20 16:00 ` Daniel Bristot de Oliveira 2021-05-20 16:00 ` Daniel Bristot de Oliveira 2021-05-20 17:55 ` Dietmar Eggemann 2021-05-20 17:55 ` Dietmar Eggemann 2021-05-20 18:03 ` Will Deacon 2021-05-20 18:03 ` Will Deacon 2021-05-21 11:26 ` Dietmar Eggemann 2021-05-21 11:26 ` Dietmar Eggemann 2021-05-20 18:01 ` Will Deacon 2021-05-20 18:01 ` Will Deacon 2021-05-21 5:25 ` Juri Lelli 2021-05-21 5:25 ` Juri Lelli 2021-05-21 8:15 ` Quentin Perret 2021-05-21 8:15 ` Quentin Perret 2021-05-21 8:39 ` Juri Lelli 2021-05-21 8:39 ` Juri Lelli 2021-05-21 10:37 ` Will Deacon 2021-05-21 10:37 ` Will Deacon 2021-05-21 11:23 ` Dietmar Eggemann 2021-05-21 11:23 ` Dietmar Eggemann 2021-05-21 13:02 ` Quentin Perret 2021-05-21 13:02 ` Quentin Perret 2021-05-21 14:04 ` Juri Lelli 2021-05-21 14:04 ` Juri Lelli 2021-05-21 17:47 ` Dietmar Eggemann [this message] 2021-05-21 17:47 ` Dietmar Eggemann 2021-05-21 13:00 ` Daniel Bristot de Oliveira 2021-05-21 13:00 ` Daniel Bristot de Oliveira 2021-05-21 13:12 ` Quentin Perret 2021-05-21 13:12 ` Quentin Perret 2021-05-24 20:47 ` Will Deacon 2021-05-24 20:47 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 14/21] freezer: Add frozen_or_skipped() helper function Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 15/21] sched: Defer wakeup in ttwu() for unschedulable frozen tasks Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-18 9:47 ` [PATCH v6 16/21] arm64: Implement task_cpu_possible_mask() Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-24 14:57 ` Catalin Marinas 2021-05-24 14:57 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 17/21] arm64: exec: Adjust affinity for compat tasks with mismatched 32-bit EL0 Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-24 15:02 ` Catalin Marinas 2021-05-24 15:02 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 18/21] arm64: Prevent offlining first CPU with 32-bit EL0 on mismatched system Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-24 15:46 ` Catalin Marinas 2021-05-24 15:46 ` Catalin Marinas 2021-05-24 20:32 ` Will Deacon 2021-05-24 20:32 ` Will Deacon 2021-05-25 9:43 ` Catalin Marinas 2021-05-25 9:43 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 19/21] arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0 Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-24 15:47 ` Catalin Marinas 2021-05-24 15:47 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 20/21] arm64: Remove logic to kill 32-bit tasks on 64-bit-only cores Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-24 15:47 ` Catalin Marinas 2021-05-24 15:47 ` Catalin Marinas 2021-05-18 9:47 ` [PATCH v6 21/21] Documentation: arm64: describe asymmetric 32-bit support Will Deacon 2021-05-18 9:47 ` Will Deacon 2021-05-21 17:37 ` Qais Yousef 2021-05-21 17:37 ` Qais Yousef 2021-05-24 21:46 ` Will Deacon 2021-05-24 21:46 ` Will Deacon 2021-05-24 16:22 ` Catalin Marinas 2021-05-24 16:22 ` Catalin Marinas 2021-05-21 17:45 ` [PATCH v6 00/21] Add support for 32-bit tasks on asymmetric AArch32 systems Qais Yousef 2021-05-21 17:45 ` Qais Yousef 2021-05-24 22:08 ` Will Deacon 2021-05-24 22:08 ` Will Deacon
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1031558c-acc8-d1b2-2964-ed78fd9b22a0@arm.com \ --to=dietmar.eggemann@arm.com \ --cc=bristot@redhat.com \ --cc=catalin.marinas@arm.com \ --cc=gregkh@linuxfoundation.org \ --cc=hannes@cmpxchg.org \ --cc=juri.lelli@redhat.com \ --cc=kernel-team@android.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=maz@kernel.org \ --cc=mingo@redhat.com \ --cc=morten.rasmussen@arm.com \ --cc=peterz@infradead.org \ --cc=qais.yousef@arm.com \ --cc=qperret@google.com \ --cc=rjw@rjwysocki.net \ --cc=surenb@google.com \ --cc=tj@kernel.org \ --cc=vincent.guittot@linaro.org \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.