From: David Hildenbrand <david@redhat.com> To: Vitaly Kuznetsov <vkuznets@redhat.com>, linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-hyperv@vger.kernel.org, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Andrew Morton <akpm@linux-foundation.org>, Michal Hocko <mhocko@kernel.org>, Oscar Salvador <osalvador@suse.de>, "Rafael J. Wysocki" <rafael@kernel.org>, Baoquan He <bhe@redhat.com>, Wei Yang <richard.weiyang@gmail.com> Subject: Re: [PATCH v1 5/5] mm/memory_hotplug: allow to specify a default online_type Date: Wed, 11 Mar 2020 18:05:58 +0100 [thread overview] Message-ID: <2586b3aa-42aa-c8e1-837d-5ba76f3de30c@redhat.com> (raw) In-Reply-To: <877dzqsuej.fsf@vitty.brq.redhat.com> On 11.03.20 17:55, Vitaly Kuznetsov wrote: > David Hildenbrand <david@redhat.com> writes: > >> For now, distributions implement advanced udev rules to essentially >> - Don't online any hotplugged memory (s390x) >> - Online all memory to ZONE_NORMAL (e.g., most virt environments like >> hyperv) >> - Online all memory to ZONE_MOVABLE in case the zone imbalance is taken >> care of (e.g., bare metal, special virt environments) >> >> In summary: All memory is usually onlined the same way, however, the >> kernel always has to ask userspace to come up with the same answer. >> E.g., HyperV always waits for a memory block to get onlined before >> continuing, otherwise it might end up adding memory faster than >> hotplugging it, which can result in strange OOM situations. >> >> Let's allow to specify a default online_type, not just "online" and >> "offline". This allows distributions to configure the default online_type >> when booting up and be done with it. >> >> We can now specify "offline", "online", "online_movable" and >> "online_kernel" via >> - "memhp_default_state=" on the kernel cmdline >> - /sys/devices/systemn/memory/auto_online_blocks >> just like we are able to specify for a single memory block via >> /sys/devices/systemn/memory/memoryX/state >> > > Thank you for picking this up! > > It's been awhile since I've added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE > but I vaguely recall one problem: memory hotplug may happen *very* early > (just because some memory is presented to a VM as hotplug memory, it is > not in e820). It happens way before we launch userspace (including > udev). The question is -- which ZONE will this memory be assigned too? If it's added via add_memory() ("hot/cold plugged memory") like ACPI DIMMs not part of e820, Hyper-V balloon added memory, XEN balloon added memory, s390x standby memory etc. the memory will be onlined as configured via memhp_default_online_type. Assume that one is set to "offline". *If* userspace changes memhp_default_online_type (as in my script in the cover letter), userspace has to online all memory that has been added before userspace was active itself (again, as done in my script). Memory not added via add_memory() is considered "initial memory" and not as hot/cold plugged memory. Same handling as for now using udev rules. (once userspace is up, udev rules for all early added memory is triggered as well) > > 'memhp_default_state=' resolves the issue but nobody likes additional > kernel parameters for anything but > debug. CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE was supposed to help, but it > is binary and distro-wide (so *all* deployments will get the same > default and as you validly stated we want it differently). > > We could've added something like your example onlining script to the > kernel itself but this is likely going to be hard to sell: "policies > belong to userspace!" will likely be the answer. Exactly my thought. -- Thanks, David / dhildenb
WARNING: multiple messages have this Message-ID (diff)
From: David Hildenbrand <david@redhat.com> To: Vitaly Kuznetsov <vkuznets@redhat.com>, linux-kernel@vger.kernel.org Cc: linux-hyperv@vger.kernel.org, Baoquan He <bhe@redhat.com>, "Rafael J. Wysocki" <rafael@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Michal Hocko <mhocko@kernel.org>, linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org>, Wei Yang <richard.weiyang@gmail.com>, linuxppc-dev@lists.ozlabs.org, Oscar Salvador <osalvador@suse.de> Subject: Re: [PATCH v1 5/5] mm/memory_hotplug: allow to specify a default online_type Date: Wed, 11 Mar 2020 18:05:58 +0100 [thread overview] Message-ID: <2586b3aa-42aa-c8e1-837d-5ba76f3de30c@redhat.com> (raw) In-Reply-To: <877dzqsuej.fsf@vitty.brq.redhat.com> On 11.03.20 17:55, Vitaly Kuznetsov wrote: > David Hildenbrand <david@redhat.com> writes: > >> For now, distributions implement advanced udev rules to essentially >> - Don't online any hotplugged memory (s390x) >> - Online all memory to ZONE_NORMAL (e.g., most virt environments like >> hyperv) >> - Online all memory to ZONE_MOVABLE in case the zone imbalance is taken >> care of (e.g., bare metal, special virt environments) >> >> In summary: All memory is usually onlined the same way, however, the >> kernel always has to ask userspace to come up with the same answer. >> E.g., HyperV always waits for a memory block to get onlined before >> continuing, otherwise it might end up adding memory faster than >> hotplugging it, which can result in strange OOM situations. >> >> Let's allow to specify a default online_type, not just "online" and >> "offline". This allows distributions to configure the default online_type >> when booting up and be done with it. >> >> We can now specify "offline", "online", "online_movable" and >> "online_kernel" via >> - "memhp_default_state=" on the kernel cmdline >> - /sys/devices/systemn/memory/auto_online_blocks >> just like we are able to specify for a single memory block via >> /sys/devices/systemn/memory/memoryX/state >> > > Thank you for picking this up! > > It's been awhile since I've added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE > but I vaguely recall one problem: memory hotplug may happen *very* early > (just because some memory is presented to a VM as hotplug memory, it is > not in e820). It happens way before we launch userspace (including > udev). The question is -- which ZONE will this memory be assigned too? If it's added via add_memory() ("hot/cold plugged memory") like ACPI DIMMs not part of e820, Hyper-V balloon added memory, XEN balloon added memory, s390x standby memory etc. the memory will be onlined as configured via memhp_default_online_type. Assume that one is set to "offline". *If* userspace changes memhp_default_online_type (as in my script in the cover letter), userspace has to online all memory that has been added before userspace was active itself (again, as done in my script). Memory not added via add_memory() is considered "initial memory" and not as hot/cold plugged memory. Same handling as for now using udev rules. (once userspace is up, udev rules for all early added memory is triggered as well) > > 'memhp_default_state=' resolves the issue but nobody likes additional > kernel parameters for anything but > debug. CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE was supposed to help, but it > is binary and distro-wide (so *all* deployments will get the same > default and as you validly stated we want it differently). > > We could've added something like your example onlining script to the > kernel itself but this is likely going to be hard to sell: "policies > belong to userspace!" will likely be the answer. Exactly my thought. -- Thanks, David / dhildenb
next prev parent reply other threads:[~2020-03-11 17:06 UTC|newest] Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-03-11 12:30 [PATCH v1 0/5] mm/memory_hotplug: allow to specify a default online_type David Hildenbrand 2020-03-11 12:30 ` David Hildenbrand 2020-03-11 12:30 ` [PATCH v1 1/5] drivers/base/memory: rename MMOP_ONLINE_KEEP to MMOP_ONLINE David Hildenbrand 2020-03-11 12:30 ` David Hildenbrand 2020-03-11 14:17 ` Wei Yang 2020-03-11 14:17 ` Wei Yang 2020-03-16 15:12 ` Michal Hocko 2020-03-16 15:12 ` Michal Hocko 2020-03-16 15:17 ` David Hildenbrand 2020-03-16 15:17 ` David Hildenbrand 2020-03-11 12:30 ` [PATCH v1 2/5] drivers/base/memory: map MMOP_OFFLINE to 0 David Hildenbrand 2020-03-11 12:30 ` David Hildenbrand 2020-03-11 14:18 ` Wei Yang 2020-03-11 14:18 ` Wei Yang 2020-03-16 15:19 ` Michal Hocko 2020-03-16 15:19 ` Michal Hocko 2020-03-11 12:30 ` [PATCH v1 3/5] drivers/base/memory: store mapping between MMOP_* and string in an array David Hildenbrand 2020-03-11 12:30 ` David Hildenbrand 2020-03-11 14:20 ` Wei Yang 2020-03-11 14:20 ` Wei Yang 2020-03-11 14:27 ` Wei Yang 2020-03-11 14:27 ` Wei Yang 2020-03-16 15:20 ` Michal Hocko 2020-03-16 15:20 ` Michal Hocko 2020-03-11 12:30 ` [PATCH v1 4/5] mm/memory_hotplug: convert memhp_auto_online to store an online_type David Hildenbrand 2020-03-11 12:30 ` David Hildenbrand 2020-03-11 14:25 ` Wei Yang 2020-03-11 14:25 ` Wei Yang 2020-03-16 15:24 ` Michal Hocko 2020-03-16 15:24 ` Michal Hocko 2020-03-16 15:34 ` David Hildenbrand 2020-03-16 15:34 ` David Hildenbrand 2020-03-16 15:46 ` Michal Hocko 2020-03-16 15:46 ` Michal Hocko 2020-03-11 12:30 ` [PATCH v1 5/5] mm/memory_hotplug: allow to specify a default online_type David Hildenbrand 2020-03-11 12:30 ` David Hildenbrand 2020-03-11 14:26 ` Wei Yang 2020-03-11 14:26 ` Wei Yang 2020-03-11 15:20 ` David Hildenbrand 2020-03-11 15:20 ` David Hildenbrand 2020-03-11 16:55 ` Vitaly Kuznetsov 2020-03-11 16:55 ` Vitaly Kuznetsov 2020-03-11 17:05 ` David Hildenbrand [this message] 2020-03-11 17:05 ` David Hildenbrand 2020-03-16 15:31 ` Michal Hocko 2020-03-16 15:31 ` Michal Hocko 2020-03-16 15:48 ` David Hildenbrand 2020-03-16 15:48 ` David Hildenbrand
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=2586b3aa-42aa-c8e1-837d-5ba76f3de30c@redhat.com \ --to=david@redhat.com \ --cc=akpm@linux-foundation.org \ --cc=bhe@redhat.com \ --cc=gregkh@linuxfoundation.org \ --cc=linux-hyperv@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=mhocko@kernel.org \ --cc=osalvador@suse.de \ --cc=rafael@kernel.org \ --cc=richard.weiyang@gmail.com \ --cc=vkuznets@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.