From: Gautham R Shenoy <ego.lkml@gmail.com>
To: Nathan Lynch <nathanl@linux.ibm.com>
Cc: ego@linux.vnet.ibm.com, linuxppc-dev <linuxppc-dev@lists.ozlabs.org>
Subject: Re: [PATCH v2 1/3] powerpc/rtas: use device model APIs and serialization during LPM
Date: Tue, 13 Aug 2019 22:50:03 +0530 [thread overview]
Message-ID: <CAHZ_5WxJcgn4A0Tyq3K+ohKZo89pj4tnF3_L90akWkbQii1eyA@mail.gmail.com> (raw)
In-Reply-To: <20190802192926.19277-2-nathanl@linux.ibm.com>
Hello Nathan,
On Sat, Aug 3, 2019 at 1:06 AM Nathan Lynch <nathanl@linux.ibm.com> wrote:
>
> The LPAR migration implementation and userspace-initiated cpu hotplug
> can interleave their executions like so:
>
> 1. Set cpu 7 offline via sysfs.
>
> 2. Begin a partition migration, whose implementation requires the OS
> to ensure all present cpus are online; cpu 7 is onlined:
>
> rtas_ibm_suspend_me -> rtas_online_cpus_mask -> cpu_up
>
> This sets cpu 7 online in all respects except for the cpu's
> corresponding struct device; dev->offline remains true.
>
> 3. Set cpu 7 online via sysfs. _cpu_up() determines that cpu 7 is
> already online and returns success. The driver core (device_online)
> sets dev->offline = false.
>
> 4. The migration completes and restores cpu 7 to offline state:
>
> rtas_ibm_suspend_me -> rtas_offline_cpus_mask -> cpu_down
>
> This leaves cpu7 in a state where the driver core considers the cpu
> device online, but in all other respects it is offline and
> unused. Attempts to online the cpu via sysfs appear to succeed but the
> driver core actually does not pass the request to the lower-level
> cpuhp support code. This makes the cpu unusable until the cpu device
> is manually set offline and then online again via sysfs.
>
> Instead of directly calling cpu_up/cpu_down, the migration code should
> use the higher-level device core APIs to maintain consistent state and
> serialize operations.
>
> Fixes: 120496ac2d2d ("powerpc: Bring all threads online prior to migration/hibernation")
> Signed-off-by: Nathan Lynch <nathanl@linux.ibm.com>
Looks good to me. This locking scheme makes the code consistent with
dlpar_cpu() which also uses the high-level device APIs.
Reviewed-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
> ---
> arch/powerpc/kernel/rtas.c | 11 ++++++++---
> 1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c
> index 5faf0a64c92b..05824eb4323b 100644
> --- a/arch/powerpc/kernel/rtas.c
> +++ b/arch/powerpc/kernel/rtas.c
> @@ -871,15 +871,17 @@ static int rtas_cpu_state_change_mask(enum rtas_cpu_state state,
> return 0;
>
> for_each_cpu(cpu, cpus) {
> + struct device *dev = get_cpu_device(cpu);
> +
> switch (state) {
> case DOWN:
> - cpuret = cpu_down(cpu);
> + cpuret = device_offline(dev);
> break;
> case UP:
> - cpuret = cpu_up(cpu);
> + cpuret = device_online(dev);
> break;
> }
> - if (cpuret) {
> + if (cpuret < 0) {
> pr_debug("%s: cpu_%s for cpu#%d returned %d.\n",
> __func__,
> ((state == UP) ? "up" : "down"),
> @@ -968,6 +970,8 @@ int rtas_ibm_suspend_me(u64 handle)
> data.token = rtas_token("ibm,suspend-me");
> data.complete = &done;
>
> + lock_device_hotplug();
> +
> /* All present CPUs must be online */
> cpumask_andnot(offline_mask, cpu_present_mask, cpu_online_mask);
> cpuret = rtas_online_cpus_mask(offline_mask);
> @@ -1006,6 +1010,7 @@ int rtas_ibm_suspend_me(u64 handle)
> __func__);
>
> out:
> + unlock_device_hotplug();
> free_cpumask_var(offline_mask);
> return atomic_read(&data.error);
> }
> --
> 2.20.1
>
--
Thanks and Regards
gautham.
next prev parent reply other threads:[~2019-08-13 17:22 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-02 19:29 [PATCH v2 0/3] more migration vs CPU hotplug fixes etc Nathan Lynch
2019-08-02 19:29 ` [PATCH v2 1/3] powerpc/rtas: use device model APIs and serialization during LPM Nathan Lynch
2019-08-05 23:05 ` Tyrel Datwyler
2019-08-12 16:55 ` Nathan Lynch
2019-08-13 17:20 ` Gautham R Shenoy [this message]
2019-08-22 13:08 ` Michael Ellerman
2019-08-02 19:29 ` [PATCH v2 2/3] powerpc/rtas: allow rescheduling while changing cpu states Nathan Lynch
2019-08-13 17:17 ` Gautham R Shenoy
2019-08-13 18:14 ` Nathan Lynch
2019-08-02 19:29 ` [PATCH v2 3/3] powerpc/pseries/mobility: use cond_resched when updating device tree Nathan Lynch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAHZ_5WxJcgn4A0Tyq3K+ohKZo89pj4tnF3_L90akWkbQii1eyA@mail.gmail.com \
--to=ego.lkml@gmail.com \
--cc=ego@linux.vnet.ibm.com \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=nathanl@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).