* [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs @ 2018-07-29 13:18 Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 1/5] hotplug/cpu: Conditionally acquire/release DRC index Michael Bringmann ` (3 more replies) 0 siblings, 4 replies; 6+ messages in thread From: Michael Bringmann @ 2018-07-29 13:18 UTC (permalink / raw) To: linuxppc-dev Cc: Michael Bringmann, Nathan Fontenot, John Allen, Tyrel Datwyler, Thomas Falcon The migration of LPARs across Power systems affects many attributes including that of the associativity of CPUs. The patches in this set execute when a system is coming up fresh upon a migration target. They are intended to, * Recognize changes to the associativity of CPUs recorded in internal data structures when compared to the latest copies in the device tree. * Generate calls to other code layers to reset the data structures related to associativity of the CPUs. * Re-register the 'changed' entities into the target system. Re-registration of CPUs mostly entails acting as if they have been newly hot-added into the target system. Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com> Michael Bringmann (9): hotplug/cpu: Conditionally acquire/release DRC index hotplug/cpu: Add operation queuing function hotplug/cpu: Provide CPU readd operation mobility/numa: Ensure numa update does not overlap numa: Disable/enable arch_update_cpu_topology pmt/numa: Disable arch_update_cpu_topology during CPU readd powerpc/rtas: Allow disabling rtas_event_scan hotplug/rtas: No rtas_event_scan during PMT update hotplug/pmt: Update topology after PMT --- Changes in patch: -- Restructure and rearrange content of patches to co-locate similar or related modifications -- Rename pseries_update_drconf_cpu to pseries_update_processor -- Simplify code to update CPU nodes during mobility checks. Remove functions to generate extra HP_ELOG messages in favor of direct function calls to dlpar_cpu_readd_by_index. -- Revise code order in dlpar_cpu_readd_by_index() to present more appropriate error codes from underlying layers of the implementation. -- Add hotplug device lock around all property updates -- Add call to rebuild_sched_domains in case of changes -- Various code cleanups and compaction -- Rebase to 4.18 kernel -- Change operation to run CPU readd after end of migration store. -- Improve descriptive text -- Cleanup patch reference to outdated function -- Code cleanup a 'acquire_drc' check in dlpar_cpu_add. -- Code cleanup a 'release_drc' check in dlpar_cpu_remove. -- Add more information to patch descriptions. -- More code cleanup -- Rearrange call to rebuild_sched_domains to allow removal of some locking code. ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v08 1/5] hotplug/cpu: Conditionally acquire/release DRC index 2018-07-29 13:18 [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michael Bringmann @ 2018-07-29 13:19 ` Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 4/5] mobility/numa: Ensure numa update does not overlap Michael Bringmann ` (2 subsequent siblings) 3 siblings, 0 replies; 6+ messages in thread From: Michael Bringmann @ 2018-07-29 13:19 UTC (permalink / raw) To: linuxppc-dev Cc: Michael Bringmann, Nathan Fontenot, John Allen, Tyrel Datwyler, Thomas Falcon powerpc/cpu: Modify dlpar_cpu_add and dlpar_cpu_remove to allow the skipping of DRC index acquire or release operations during the CPU add or remove operations. This is intended to support subsequent changes to provide a 'CPU readd' operation. Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com> --- Changes in patch: -- Move new validity check added to pseries_smp_notifier to another patch -- Revise one of checks for 'acquire_drc' in dlpar_cpu_add. -- Revise one of checks for 'release_drc' in dlpar_cpu_remove. --- arch/powerpc/platforms/pseries/hotplug-cpu.c | 71 +++++++++++++++----------- 1 file changed, 40 insertions(+), 31 deletions(-) diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pseries/hotplug-cpu.c index 6ef77ca..7ede3b0 100644 --- a/arch/powerpc/platforms/pseries/hotplug-cpu.c +++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c @@ -432,7 +432,7 @@ static bool valid_cpu_drc_index(struct device_node *parent, u32 drc_index) return found; } -static ssize_t dlpar_cpu_add(u32 drc_index) +static ssize_t dlpar_cpu_add(u32 drc_index, bool acquire_drc) { struct device_node *dn, *parent; int rc, saved_rc; @@ -457,19 +457,22 @@ static ssize_t dlpar_cpu_add(u32 drc_index) return -EINVAL; } - rc = dlpar_acquire_drc(drc_index); - if (rc) { - pr_warn("Failed to acquire DRC, rc: %d, drc index: %x\n", - rc, drc_index); - of_node_put(parent); - return -EINVAL; + if (acquire_drc) { + rc = dlpar_acquire_drc(drc_index); + if (rc) { + pr_warn("Failed to acquire DRC, rc: %d, drc index: %x\n", + rc, drc_index); + of_node_put(parent); + return -EINVAL; + } } dn = dlpar_configure_connector(cpu_to_be32(drc_index), parent); if (!dn) { pr_warn("Failed call to configure-connector, drc index: %x\n", drc_index); - dlpar_release_drc(drc_index); + if (acquire_drc) + dlpar_release_drc(drc_index); of_node_put(parent); return -EINVAL; } @@ -484,9 +487,11 @@ static ssize_t dlpar_cpu_add(u32 drc_index) pr_warn("Failed to attach node %s, rc: %d, drc index: %x\n", dn->name, rc, drc_index); - rc = dlpar_release_drc(drc_index); - if (!rc) - dlpar_free_cc_nodes(dn); + if (acquire_drc) { + rc = dlpar_release_drc(drc_index); + if (!rc) + dlpar_free_cc_nodes(dn); + } return saved_rc; } @@ -498,7 +503,7 @@ static ssize_t dlpar_cpu_add(u32 drc_index) dn->name, rc, drc_index); rc = dlpar_detach_node(dn); - if (!rc) + if (!rc && acquire_drc) dlpar_release_drc(drc_index); return saved_rc; @@ -566,7 +571,8 @@ static int dlpar_offline_cpu(struct device_node *dn) } -static ssize_t dlpar_cpu_remove(struct device_node *dn, u32 drc_index) +static ssize_t dlpar_cpu_remove(struct device_node *dn, u32 drc_index, + bool release_drc) { int rc; @@ -579,12 +585,14 @@ static ssize_t dlpar_cpu_remove(struct device_node *dn, u32 drc_index) return -EINVAL; } - rc = dlpar_release_drc(drc_index); - if (rc) { - pr_warn("Failed to release drc (%x) for CPU %s, rc: %d\n", - drc_index, dn->name, rc); - dlpar_online_cpu(dn); - return rc; + if (release_drc) { + rc = dlpar_release_drc(drc_index); + if (rc) { + pr_warn("Failed to release drc (%x) for CPU %s, rc: %d\n", + drc_index, dn->name, rc); + dlpar_online_cpu(dn); + return rc; + } } rc = dlpar_detach_node(dn); @@ -593,8 +601,9 @@ static ssize_t dlpar_cpu_remove(struct device_node *dn, u32 drc_index) pr_warn("Failed to detach CPU %s, rc: %d", dn->name, rc); - rc = dlpar_acquire_drc(drc_index); - if (!rc) + if (release_drc) + rc = dlpar_acquire_drc(drc_index); + if (!release_drc || !rc) dlpar_online_cpu(dn); return saved_rc; @@ -622,7 +631,7 @@ static struct device_node *cpu_drc_index_to_dn(u32 drc_index) return dn; } -static int dlpar_cpu_remove_by_index(u32 drc_index) +static int dlpar_cpu_remove_by_index(u32 drc_index, bool release_drc) { struct device_node *dn; int rc; @@ -634,7 +643,7 @@ static int dlpar_cpu_remove_by_index(u32 drc_index) return -ENODEV; } - rc = dlpar_cpu_remove(dn, drc_index); + rc = dlpar_cpu_remove(dn, drc_index, release_drc); of_node_put(dn); return rc; } @@ -699,7 +708,7 @@ static int dlpar_cpu_remove_by_count(u32 cpus_to_remove) } for (i = 0; i < cpus_to_remove; i++) { - rc = dlpar_cpu_remove_by_index(cpu_drcs[i]); + rc = dlpar_cpu_remove_by_index(cpu_drcs[i], true); if (rc) break; @@ -710,7 +719,7 @@ static int dlpar_cpu_remove_by_count(u32 cpus_to_remove) pr_warn("CPU hot-remove failed, adding back removed CPUs\n"); for (i = 0; i < cpus_removed; i++) - dlpar_cpu_add(cpu_drcs[i]); + dlpar_cpu_add(cpu_drcs[i], true); rc = -EINVAL; } else { @@ -780,7 +789,7 @@ static int dlpar_cpu_add_by_count(u32 cpus_to_add) } for (i = 0; i < cpus_to_add; i++) { - rc = dlpar_cpu_add(cpu_drcs[i]); + rc = dlpar_cpu_add(cpu_drcs[i], true); if (rc) break; @@ -791,7 +800,7 @@ static int dlpar_cpu_add_by_count(u32 cpus_to_add) pr_warn("CPU hot-add failed, removing any added CPUs\n"); for (i = 0; i < cpus_added; i++) - dlpar_cpu_remove_by_index(cpu_drcs[i]); + dlpar_cpu_remove_by_index(cpu_drcs[i], true); rc = -EINVAL; } else { @@ -817,7 +826,7 @@ int dlpar_cpu(struct pseries_hp_errorlog *hp_elog) if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_COUNT) rc = dlpar_cpu_remove_by_count(count); else if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_INDEX) - rc = dlpar_cpu_remove_by_index(drc_index); + rc = dlpar_cpu_remove_by_index(drc_index, true); else rc = -EINVAL; break; @@ -825,7 +834,7 @@ int dlpar_cpu(struct pseries_hp_errorlog *hp_elog) if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_COUNT) rc = dlpar_cpu_add_by_count(count); else if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_INDEX) - rc = dlpar_cpu_add(drc_index); + rc = dlpar_cpu_add(drc_index, true); else rc = -EINVAL; break; @@ -850,7 +859,7 @@ static ssize_t dlpar_cpu_probe(const char *buf, size_t count) if (rc) return -EINVAL; - rc = dlpar_cpu_add(drc_index); + rc = dlpar_cpu_add(drc_index, true); return rc ? rc : count; } @@ -871,7 +880,7 @@ static ssize_t dlpar_cpu_release(const char *buf, size_t count) return -EINVAL; } - rc = dlpar_cpu_remove(dn, drc_index); + rc = dlpar_cpu_remove(dn, drc_index, true); of_node_put(dn); return rc ? rc : count; ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v08 4/5] mobility/numa: Ensure numa update does not overlap 2018-07-29 13:18 [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 1/5] hotplug/cpu: Conditionally acquire/release DRC index Michael Bringmann @ 2018-07-29 13:19 ` Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 5/5] hotplug/pmt: Update topology after PMT Michael Bringmann 2018-10-29 19:51 ` [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michal Suchánek 3 siblings, 0 replies; 6+ messages in thread From: Michael Bringmann @ 2018-07-29 13:19 UTC (permalink / raw) To: linuxppc-dev Cc: Michael Bringmann, Nathan Fontenot, John Allen, Tyrel Datwyler, Thomas Falcon mobility/numa: Ensure that numa_update_cpu_topology() can not be entered multiple times concurrently. It may be accessed through many different paths through the code. Without some protection against multiple entry, the code may acquire and update associativity information from the PHYP in multiple threads, and apply changes separately in each thread. Applying the changes concurrently may perform considerable work multiple times, but more importantly, may acquire locks in other modules concurrently and end up blocking each of the competing threads. This patch allows only the first entrant to the code to execute the operation and to recognize any CPU topology changes. Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com> --- Changes in patch: -- Added information to description. --- arch/powerpc/mm/numa.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c index a789d57..b22e27a 100644 --- a/arch/powerpc/mm/numa.c +++ b/arch/powerpc/mm/numa.c @@ -1079,6 +1079,7 @@ struct topology_update_data { static int topology_timer_secs = 1; static int topology_inited; static int topology_update_needed; +static struct mutex topology_update_lock; /* * Change polling interval for associativity changes. @@ -1320,6 +1321,11 @@ int numa_update_cpu_topology(bool cpus_locked) if (!updates) return 0; + if (!mutex_trylock(&topology_update_lock)) { + kfree(updates); + return 0; + } + cpumask_clear(&updated_cpus); for_each_cpu(cpu, &cpu_associativity_changes_mask) { @@ -1424,6 +1430,7 @@ int numa_update_cpu_topology(bool cpus_locked) out: kfree(updates); topology_update_needed = 0; + mutex_unlock(&topology_update_lock); return changed; } @@ -1598,6 +1605,8 @@ static ssize_t topology_write(struct file *file, const char __user *buf, static int topology_update_init(void) { + mutex_init(&topology_update_lock); + /* Do not poll for changes if disabled at boot */ if (topology_updates_enabled) start_topology_update(); ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v08 5/5] hotplug/pmt: Update topology after PMT 2018-07-29 13:18 [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 1/5] hotplug/cpu: Conditionally acquire/release DRC index Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 4/5] mobility/numa: Ensure numa update does not overlap Michael Bringmann @ 2018-07-29 13:19 ` Michael Bringmann 2018-10-29 19:51 ` [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michal Suchánek 3 siblings, 0 replies; 6+ messages in thread From: Michael Bringmann @ 2018-07-29 13:19 UTC (permalink / raw) To: linuxppc-dev Cc: Michael Bringmann, Nathan Fontenot, John Allen, Tyrel Datwyler, Thomas Falcon hotplug/pmt: Invoke rebuild_sched_domains before applying any CPU 'readd' modifications. The call leads to arch_update_cpu_topology() which will recognize and report any changes to the CPU topology are now applied, and that the relevant system data structures are updated. It may also initialize nodes that were unused in the topology of the source migration system. This will avoid some locking synchronization issues that were observed during development of this patch set. Also, the addition of the explicit call ensures that such changes are made even when the automatic topology update worker, triggered by a timer, has been disabled. Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com> --- Changes in patch: -- Add information to description. -- Remove long ssleep calls -- Rearrange code to perform rebuild before any other PMT mods --- arch/powerpc/platforms/pseries/dlpar.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/powerpc/platforms/pseries/dlpar.c b/arch/powerpc/platforms/pseries/dlpar.c index 13ac1cc..2de0f0d 100644 --- a/arch/powerpc/platforms/pseries/dlpar.c +++ b/arch/powerpc/platforms/pseries/dlpar.c @@ -16,6 +16,7 @@ #include <linux/notifier.h> #include <linux/spinlock.h> #include <linux/cpu.h> +#include <linux/cpuset.h> #include <linux/slab.h> #include <linux/of.h> @@ -437,6 +438,17 @@ static int dlpar_pmt(struct pseries_hp_errorlog *work) { struct list_head *pos, *q; + /* Rebuild the domains and init any memoryless nodes + * first to avoid later sync issues with CPU readd. + */ + rebuild_sched_domains(); + msleep(100); + /* Ensure that the worker for rebuild_sched_domains + * has the opportunity to actually begin work as we + * don't want it delayed by the CPU readd hotplug + * locking. + */ + list_for_each_safe(pos, q, &dlpar_delayed_list) { struct pseries_hp_errorlog *tmp; ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs 2018-07-29 13:18 [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michael Bringmann ` (2 preceding siblings ...) 2018-07-29 13:19 ` [PATCH v08 5/5] hotplug/pmt: Update topology after PMT Michael Bringmann @ 2018-10-29 19:51 ` Michal Suchánek 2018-11-05 17:50 ` Michael Bringmann 3 siblings, 1 reply; 6+ messages in thread From: Michal Suchánek @ 2018-10-29 19:51 UTC (permalink / raw) To: Michael Bringmann Cc: Nathan Fontenot, linuxppc-dev, Tyrel Datwyler, Thomas Falcon, John Allen On Sun, 29 Jul 2018 08:18:34 -0500 Michael Bringmann <mwb@linux.vnet.ibm.com> wrote: > The migration of LPARs across Power systems affects many attributes > including that of the associativity of CPUs. The patches in this > set execute when a system is coming up fresh upon a migration target. > They are intended to, > > * Recognize changes to the associativity of CPUs recorded in internal > data structures when compared to the latest copies in the device > tree. > * Generate calls to other code layers to reset the data structures > related to associativity of the CPUs. > * Re-register the 'changed' entities into the target system. > Re-registration of CPUs mostly entails acting as if they have been > newly hot-added into the target system. > > Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com> > Hello, what is the status of this patchset other than it no longer applies? Thanks Michal ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs 2018-10-29 19:51 ` [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michal Suchánek @ 2018-11-05 17:50 ` Michael Bringmann 0 siblings, 0 replies; 6+ messages in thread From: Michael Bringmann @ 2018-11-05 17:50 UTC (permalink / raw) To: Michal Suchánek Cc: Nathan Fontenot, linuxppc-dev, Tyrel Datwyler, Thomas Falcon, John Allen On 10/29/2018 02:51 PM, Michal Suchánek wrote: > On Sun, 29 Jul 2018 08:18:34 -0500 > Michael Bringmann <mwb@linux.vnet.ibm.com> wrote: > >> The migration of LPARs across Power systems affects many attributes >> including that of the associativity of CPUs. The patches in this >> set execute when a system is coming up fresh upon a migration target. >> They are intended to, >> >> * Recognize changes to the associativity of CPUs recorded in internal >> data structures when compared to the latest copies in the device >> tree. >> * Generate calls to other code layers to reset the data structures >> related to associativity of the CPUs. >> * Re-register the 'changed' entities into the target system. >> Re-registration of CPUs mostly entails acting as if they have been >> newly hot-added into the target system. >> >> Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com> >> > Hello, > > what is the status of this patchset other than it no longer applies? The current PowerPC systems already perform remove_processor / add_processor operations during migration events that subsume the necessary affinity changes. This patchset was overkill, so we pulled it. Memory affinity on the other hand does need intervention, and that patchset has been posted. I was just about to ping Michael Ellerman about the status from his end. > > Thanks > > Michal Thanks. Michael -- Michael W. Bringmann Linux Technology Center IBM Corporation Tie-Line 363-5196 External: (512) 286-5196 Cell: (512) 466-0650 mwb@linux.vnet.ibm.com ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-11-05 17:52 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-07-29 13:18 [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 1/5] hotplug/cpu: Conditionally acquire/release DRC index Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 4/5] mobility/numa: Ensure numa update does not overlap Michael Bringmann 2018-07-29 13:19 ` [PATCH v08 5/5] hotplug/pmt: Update topology after PMT Michael Bringmann 2018-10-29 19:51 ` [PATCH v08 0/5] powerpc/hotplug: Update affinity for migrated CPUs Michal Suchánek 2018-11-05 17:50 ` Michael Bringmann
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).