From: Yang Shi <shy828301@gmail.com>
To: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Yang Shi <yang.shi@linux.alibaba.com>,
David Rientjes <rientjes@google.com>,
Huang Ying <ying.huang@intel.com>,
Dan Williams <dan.j.williams@intel.com>
Subject: Re: [RFC][PATCH 3/9] mm/migrate: update migration order during on hotplug events
Date: Thu, 20 Aug 2020 15:07:48 -0700 [thread overview]
Message-ID: <CAHbLzkoKg-iPq1WZUJ5G-mzqTgiT2xrpff9NTFfTEmABGdmL0Q@mail.gmail.com> (raw)
In-Reply-To: <20200818184127.C3B49AB6@viggo.jf.intel.com>
On Tue, Aug 18, 2020 at 11:52 AM Dave Hansen
<dave.hansen@linux.intel.com> wrote:
>
>
> From: Dave Hansen <dave.hansen@linux.intel.com>
>
> Reclaim-based migration is attempting to optimize data placement in
> memory based on the system topology. If the system changes, so must
> the migration ordering.
>
> The implementation here is pretty simple and entirely unoptimized. On
> any memory or CPU hotplug events, assume that a node was added or
> removed and recalculate all migration targets. This ensures that the
> node_demotion[] array is always ready to be used in case the new
> reclaim mode is enabled.
>
> This recalculation is far from optimal, most glaringly that it does
> not even attempt to figure out if nodes are actually coming or going.
> But, given the expected paucity of hotplug events, this should be
> fine.
>
> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: Yang Shi <yang.shi@linux.alibaba.com>
> Cc: David Rientjes <rientjes@google.com>
> Cc: Huang Ying <ying.huang@intel.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> ---
>
> b/mm/migrate.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 93 insertions(+)
>
> diff -puN mm/migrate.c~enable-numa-demotion mm/migrate.c
> --- a/mm/migrate.c~enable-numa-demotion 2020-08-18 11:36:50.193583185 -0700
> +++ b/mm/migrate.c 2020-08-18 11:36:50.197583185 -0700
> @@ -49,6 +49,7 @@
> #include <linux/sched/mm.h>
> #include <linux/ptrace.h>
> #include <linux/oom.h>
> +#include <linux/memory.h>
>
> #include <asm/tlbflush.h>
>
> @@ -3130,9 +3131,101 @@ again:
> goto again;
> }
>
> +/*
> + * For callers that do not hold get_online_mems() already.
> + */
> void set_migration_target_nodes(void)
> {
> get_online_mems();
> __set_migration_target_nodes();
> put_online_mems();
> }
> +
> +/*
> + * React to hotplug events that might affect the migration targes
s/targes/targets
> + * like events that online or offline NUMA nodes.
> + *
> + * The ordering is also currently dependent on which nodes have
> + * CPUs. That means we need CPU on/offline notification too.
> + */
> +static int migration_online_cpu(unsigned int cpu)
> +{
> + set_migration_target_nodes();
> + return 0;
> +}
> +
> +static int migration_offline_cpu(unsigned int cpu)
> +{
> + set_migration_target_nodes();
> + return 0;
> +}
> +
> +/*
> + * This leaves migrate-on-reclaim transiently disabled
> + * between the MEM_GOING_OFFLINE and MEM_OFFLINE events.
> + * This runs reclaim-based micgration is enabled or not.
s/micgration/migration
> + * This ensures that the user can turn reclaim-based
> + * migration at any time without needing to recalcuate
s/recalcuate/recalculate
> + * migration targets.
> + *
> + * These callbacks already hold get_online_mems(). That
> + * is why __set_migration_target_nodes() can be used as
> + * opposed to set_migration_target_nodes().
> + */
> +#if defined(CONFIG_MEMORY_HOTPLUG)
> +static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
> + unsigned long action, void *arg)
> +{
> + switch (action) {
> + case MEM_GOING_OFFLINE:
> + /*
> + * Make sure there are not transient states where
> + * an offline node is a migration target. This
> + * will leave migration disabled until the offline
> + * completes and the MEM_OFFLINE case below runs.
> + */
> + disable_all_migrate_targets();
> + break;
> + case MEM_OFFLINE:
> + case MEM_ONLINE:
> + /*
> + * Recalculate the target nodes once the node
> + * reaches its final state (online or offline).
> + */
> + __set_migration_target_nodes();
> + break;
> + case MEM_CANCEL_OFFLINE:
> + /*
> + * MEM_GOING_OFFLINE disabled all the migration
> + * targets. Reenable them.
> + */
> + __set_migration_target_nodes();
> + break;
> + case MEM_GOING_ONLINE:
> + case MEM_CANCEL_ONLINE:
> + break;
> + }
> +
> + return notifier_from_errno(0);
> +}
> +
> +static int __init migrate_on_reclaim_init(void)
> +{
> + int ret;
> +
> + ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "migrate on reclaim",
> + migration_online_cpu,
> + migration_offline_cpu);
> + /*
> + * In the unlikely case that this fails, the automatic
> + * migration targets may become suboptimal for nodes
> + * where N_CPU changes. With such a small impact in a
> + * rare case, do not bother trying to do anything special.
> + */
> + WARN_ON(ret < 0);
> +
> + hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
> + return 0;
> +}
> +late_initcall(migrate_on_reclaim_init);
> +#endif /* CONFIG_MEMORY_HOTPLUG */
> _
next prev parent reply other threads:[~2020-08-20 22:08 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-18 18:41 [RFC][PATCH 0/9] [v3] Migrate Pages in lieu of discard Dave Hansen
2020-08-18 18:41 ` [RFC][PATCH 1/9] mm/numa: node demotion data structure and lookup Dave Hansen
2020-08-18 18:41 ` [RFC][PATCH 2/9] mm/numa: automatically generate node migration order Dave Hansen
2020-08-20 21:57 ` Yang Shi
2020-08-18 18:41 ` [RFC][PATCH 3/9] mm/migrate: update migration order during on hotplug events Dave Hansen
2020-08-20 22:07 ` Yang Shi [this message]
2020-08-18 18:41 ` [RFC][PATCH 4/9] mm/migrate: make migrate_pages() return nr_succeeded Dave Hansen
2020-09-17 1:25 ` Huang, Ying
2020-08-18 18:41 ` [RFC][PATCH 5/9] mm/migrate: demote pages during reclaim Dave Hansen
2020-08-20 8:06 ` Huang, Ying
2020-08-20 15:21 ` Dave Hansen
2020-08-20 16:26 ` Yang Shi
2020-08-21 0:57 ` Huang, Ying
2020-08-21 16:17 ` Yang Shi
2020-08-20 22:42 ` Yang Shi
2020-08-18 18:41 ` [RFC][PATCH 6/9] mm/vmscan: add page demotion counter Dave Hansen
2020-08-20 22:26 ` Yang Shi
2020-08-20 23:58 ` Yang Shi
2020-08-20 22:56 ` Yang Shi
2020-08-18 18:41 ` [RFC][PATCH 7/9] mm/vmscan: Consider anonymous pages without swap Dave Hansen
2020-08-18 18:41 ` [RFC][PATCH 8/9] mm/vmscan: never demote for memcg reclaim Dave Hansen
2020-08-20 22:50 ` Yang Shi
2020-08-18 18:41 ` [RFC][PATCH 9/9] mm/migrate: new zone_reclaim_mode to enable reclaim migration Dave Hansen
2020-08-20 0:47 ` [RFC][PATCH 0/9] [v3] Migrate Pages in lieu of discard Yang Shi
2020-08-24 22:36 ` Keith Busch
2020-10-07 16:17 [RFC][PATCH 0/9] [v4][RESEND] " Dave Hansen
2020-10-07 16:17 ` [RFC][PATCH 3/9] mm/migrate: update migration order during on hotplug events Dave Hansen
2020-10-07 18:08 ` osalvador
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAHbLzkoKg-iPq1WZUJ5G-mzqTgiT2xrpff9NTFfTEmABGdmL0Q@mail.gmail.com \
--to=shy828301@gmail.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=rientjes@google.com \
--cc=yang.shi@linux.alibaba.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).