From: Daniel Jordan <daniel.m.jordan@oracle.com>
To: Alexey Klimov <aklimov@redhat.com>,
linux-kernel@vger.kernel.org, cgroups@vger.kernel.org
Cc: peterz@infradead.org, yury.norov@gmail.com, tglx@linutronix.de,
jobaker@redhat.com, audralmitchel@gmail.com, arnd@arndb.de,
gregkh@linuxfoundation.org, rafael@kernel.org, tj@kernel.org,
lizefan@huawei.com, qais.yousef@arm.com, hannes@cmpxchg.org,
klimov.linux@gmail.com
Subject: Re: [PATCH] cpu/hotplug: wait for cpuset_hotplug_work to finish on cpu onlining
Date: Thu, 04 Feb 2021 20:35:34 -0500 [thread overview]
Message-ID: <87blczz47t.fsf@oracle.com> (raw)
In-Reply-To: <20210204010157.1823669-1-aklimov@redhat.com>
Alexey Klimov <aklimov@redhat.com> writes:
> When a CPU offlined and onlined via device_offline() and device_online()
> the userspace gets uevent notification. If, after receiving "online" uevent,
> userspace executes sched_setaffinity() on some task trying to move it
> to a recently onlined CPU, then it often fails with -EINVAL. Userspace needs
> to wait around 5..30 ms before sched_setaffinity() will succeed for the recently
> onlined CPU after receiving uevent.
>
> If in_mask argument for sched_setaffinity() has only recently onlined CPU,
> it often fails with such flow:
>
> sched_setaffinity()
> cpuset_cpus_allowed()
> guarantee_online_cpus() <-- cs->effective_cpus mask does not
> contain recently onlined cpu
> cpumask_and() <-- final new_mask is empty
> __set_cpus_allowed_ptr()
> cpumask_any_and_distribute() <-- returns dest_cpu equal to nr_cpu_ids
> returns -EINVAL
>
> Cpusets used in guarantee_online_cpus() are updated using workqueue from
> cpuset_update_active_cpus() which in its turn is called from cpu hotplug callback
> sched_cpu_activate() hence it may not be observable by sched_setaffinity() if
> it is called immediately after uevent.
> Out of line uevent can be avoided if we will ensure that cpuset_hotplug_work
> has run to completion using cpuset_wait_for_hotplug() after onlining the
> cpu in cpu_up() and in cpuhp_smt_enable().
Nice writeup. I just have some nits, patch looks ok otherwise.
> @@ -1281,6 +1282,11 @@ static int cpu_up(unsigned int cpu, enum cpuhp_state target)
> err = _cpu_up(cpu, 0, target);
> out:
> cpu_maps_update_done();
> +
> + /* To avoid out of line uevent */
Not sure this will make sense out of context. Maybe,
/*
* Wait for cpuset updates to cpumasks to finish. Later on this path
* may generate uevents whose consumers rely on the updates.
*/
> @@ -2062,8 +2068,6 @@ static void cpuhp_offline_cpu_device(unsigned int cpu)
> struct device *dev = get_cpu_device(cpu);
>
> dev->offline = true;
> }
>
> @@ -2071,14 +2075,18 @@ static void cpuhp_online_cpu_device(unsigned int cpu)
> struct device *dev = get_cpu_device(cpu);
>
> dev->offline = false;
> }
You could get rid of these functions and just put the few remaining bits
in the callers. They each have only one.
prev parent reply other threads:[~2021-02-05 1:36 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-04 1:01 [PATCH] cpu/hotplug: wait for cpuset_hotplug_work to finish on cpu onlining Alexey Klimov
2021-02-04 9:46 ` Peter Zijlstra
2021-02-04 12:50 ` Alexey Klimov
2021-02-04 13:42 ` Peter Zijlstra
2021-02-05 0:39 ` Daniel Jordan
2021-02-11 14:09 ` Alexey Klimov
2021-02-05 11:22 ` Qais Yousef
2021-02-11 13:38 ` Alexey Klimov
2021-02-05 1:35 ` Daniel Jordan [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87blczz47t.fsf@oracle.com \
--to=daniel.m.jordan@oracle.com \
--cc=aklimov@redhat.com \
--cc=arnd@arndb.de \
--cc=audralmitchel@gmail.com \
--cc=cgroups@vger.kernel.org \
--cc=gregkh@linuxfoundation.org \
--cc=hannes@cmpxchg.org \
--cc=jobaker@redhat.com \
--cc=klimov.linux@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lizefan@huawei.com \
--cc=peterz@infradead.org \
--cc=qais.yousef@arm.com \
--cc=rafael@kernel.org \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
--cc=yury.norov@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).