[PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[Vitaly Kuznetsov]

This patchset continues the work I started with:

commit 31bc3858ea3ebcc3157b3f5f0e624c5962f5a7a6
Author: Vitaly Kuznetsov <vkuznets@redhat.com>
Date:   Tue Mar 15 14:56:48 2016 -0700

    memory-hotplug: add automatic onlining policy for the newly added memory

Initially I was going to stop there and bring the policy setting logic to
userspace. I met two issues on this way:

1) It is possible to have memory hotplugged at boot (e.g. with QEMU). These
   blocks stay offlined if we turn the onlining policy on by userspace.

2) My attempt to bring this policy setting to systemd failed, systemd
   maintainers suggest to change the default in kernel or ... to use tmpfiles.d
   to alter the policy (which looks like a hack to me): 
   https://github.com/systemd/systemd/pull/2938

Here I suggest to add a config option to set the default value for the policy
and a kernel command line parameter to make the override.

Vitaly Kuznetsov (2):
  memory_hotplug: introduce CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE
  memory_hotplug: introduce memhp_default_state= command line parameter

 Documentation/kernel-parameters.txt |  8 ++++++++
 Documentation/memory-hotplug.txt    |  9 +++++----
 mm/Kconfig                          | 16 ++++++++++++++++
 mm/memory_hotplug.c                 | 15 +++++++++++++++
 4 files changed, 44 insertions(+), 4 deletions(-)

-- 
2.5.5

[PATCH 1/2] memory_hotplug: introduce CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE

[Vitaly Kuznetsov]

Introduce config option to set the default value for memory hotplug
onlining policy (/sys/devices/system/memory/auto_online_blocks). The
reason one would want to turn this option on are to have early onlining
for hotpluggable memory available at boot and to not require any userspace
actions to make memory hotplug work.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 Documentation/memory-hotplug.txt |  9 +++++----
 mm/Kconfig                       | 16 ++++++++++++++++
 mm/memory_hotplug.c              |  4 ++++
 3 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/Documentation/memory-hotplug.txt b/Documentation/memory-hotplug.txt
index 443f4b4..0d7cb95 100644
--- a/Documentation/memory-hotplug.txt
+++ b/Documentation/memory-hotplug.txt
@@ -261,10 +261,11 @@ it according to the policy which can be read from "auto_online_blocks" file:
 
 % cat /sys/devices/system/memory/auto_online_blocks
 
-The default is "offline" which means the newly added memory is not in a
-ready-to-use state and you have to "online" the newly added memory blocks
-manually. Automatic onlining can be requested by writing "online" to
-"auto_online_blocks" file:
+The default depends on the CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel config
+option. If it is disabled the default is "offline" which means the newly added
+memory is not in a ready-to-use state and you have to "online" the newly added
+memory blocks manually. Automatic onlining can be requested by writing "online"
+to "auto_online_blocks" file:
 
 % echo online > /sys/devices/system/memory/auto_online_blocks
 
diff --git a/mm/Kconfig b/mm/Kconfig
index 989f8f3..4c7dd6e 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -192,6 +192,22 @@ config MEMORY_HOTPLUG_SPARSE
 	def_bool y
 	depends on SPARSEMEM && MEMORY_HOTPLUG
 
+config MEMORY_HOTPLUG_DEFAULT_ONLINE
+        bool "Online the newly added memory blocks by default"
+        default n
+        depends on MEMORY_HOTPLUG
+        help
+	  This option changes the default policy setting for memory hotplug
+	  onlining policy (/sys/devices/system/memory/auto_online_blocks) which
+	  determines what happens to the newly added memory regions. Policy
+	  setting can always be changed in runtime.
+	  See Documentation/memory-hotplug.txt for more information.
+
+	  Say Y here if you want all hot-plugged memory blocks to appear in
+	  'online' state by default.
+	  Say N here if you want the default policy to keep all hot-plugged
+	  memory blocks in 'offline' state.
+
 config MEMORY_HOTREMOVE
 	bool "Allow for memory hot remove"
 	select MEMORY_ISOLATION
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index aa34431..072e0a1 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -78,7 +78,11 @@ static struct {
 #define memhp_lock_acquire()      lock_map_acquire(&mem_hotplug.dep_map)
 #define memhp_lock_release()      lock_map_release(&mem_hotplug.dep_map)
 
+#ifndef CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE
 bool memhp_auto_online;
+#else
+bool memhp_auto_online = true;
+#endif
 EXPORT_SYMBOL_GPL(memhp_auto_online);
 
 void get_online_mems(void)
-- 
2.5.5

[PATCH 2/2] memory_hotplug: introduce memhp_default_state= command line parameter

[Vitaly Kuznetsov]

CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE specifies the default value for the
memory hotplug onlining policy. Add a command line parameter to make it
possible to override the default. It may come handy for debug and testing
purposes.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 Documentation/kernel-parameters.txt |  8 ++++++++
 mm/memory_hotplug.c                 | 11 +++++++++++
 2 files changed, 19 insertions(+)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index ecc74fa..b05ee2f 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2141,6 +2141,14 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			[KNL,SH] Allow user to override the default size for
 			per-device physically contiguous DMA buffers.
 
+        memhp_default_state=online/offline
+			[KNL] Set the initial state for the memory hotplug
+			onlining policy. If not specified, the default value is
+			set according to the
+			CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel config
+			option.
+			See Documentation/memory-hotplug.txt.
+
 	memmap=exactmap	[KNL,X86] Enable setting of an exact
 			E820 memory map, as specified by the user.
 			Such memmap=exactmap lines can be constructed based on
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 072e0a1..179f3af 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -85,6 +85,17 @@ bool memhp_auto_online = true;
 #endif
 EXPORT_SYMBOL_GPL(memhp_auto_online);
 
+static int __init setup_memhp_default_state(char *str)
+{
+	if (!strcmp(str, "online"))
+		memhp_auto_online = true;
+	else if (!strcmp(str, "offline"))
+		memhp_auto_online = false;
+
+	return 1;
+}
+__setup("memhp_default_state=", setup_memhp_default_state);
+
 void get_online_mems(void)
 {
 	might_sleep();
-- 
2.5.5

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[Andrew Morton]

On Wed,  6 Apr 2016 15:45:10 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

> This patchset continues the work I started with:
> 
> commit 31bc3858ea3ebcc3157b3f5f0e624c5962f5a7a6
> Author: Vitaly Kuznetsov <vkuznets@redhat.com>
> Date:   Tue Mar 15 14:56:48 2016 -0700
> 
>     memory-hotplug: add automatic onlining policy for the newly added memory
> 
> Initially I was going to stop there and bring the policy setting logic to
> userspace. I met two issues on this way:
> 
> 1) It is possible to have memory hotplugged at boot (e.g. with QEMU). These
>    blocks stay offlined if we turn the onlining policy on by userspace.
> 
> 2) My attempt to bring this policy setting to systemd failed, systemd
>    maintainers suggest to change the default in kernel or ... to use tmpfiles.d
>    to alter the policy (which looks like a hack to me): 
>    https://github.com/systemd/systemd/pull/2938

That discussion really didn't come to a conclusion and I don't
understand why you consider Lennert's "recommended way" to be a hack?

> Here I suggest to add a config option to set the default value for the policy
> and a kernel command line parameter to make the override.

But the patchset looks pretty reasonable regardless of the above.

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[David Rientjes]

On Wed, 6 Apr 2016, Andrew Morton wrote:

> > This patchset continues the work I started with:
> > 
> > commit 31bc3858ea3ebcc3157b3f5f0e624c5962f5a7a6
> > Author: Vitaly Kuznetsov <vkuznets@redhat.com>
> > Date:   Tue Mar 15 14:56:48 2016 -0700
> > 
> >     memory-hotplug: add automatic onlining policy for the newly added memory
> > 
> > Initially I was going to stop there and bring the policy setting logic to
> > userspace. I met two issues on this way:
> > 
> > 1) It is possible to have memory hotplugged at boot (e.g. with QEMU). These
> >    blocks stay offlined if we turn the onlining policy on by userspace.
> > 
> > 2) My attempt to bring this policy setting to systemd failed, systemd
> >    maintainers suggest to change the default in kernel or ... to use tmpfiles.d
> >    to alter the policy (which looks like a hack to me): 
> >    https://github.com/systemd/systemd/pull/2938
> 
> That discussion really didn't come to a conclusion and I don't
> understand why you consider Lennert's "recommended way" to be a hack?
> 
> > Here I suggest to add a config option to set the default value for the policy
> > and a kernel command line parameter to make the override.
> 
> But the patchset looks pretty reasonable regardless of the above.
> 

I don't understand why initscripts simply cannot crawl sysfs memory blocks 
and online them for the same behavior.

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[Vitaly Kuznetsov]

Andrew Morton <akpm@linux-foundation.org> writes:

> On Wed,  6 Apr 2016 15:45:10 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
>> This patchset continues the work I started with:
>> 
>> commit 31bc3858ea3ebcc3157b3f5f0e624c5962f5a7a6
>> Author: Vitaly Kuznetsov <vkuznets@redhat.com>
>> Date:   Tue Mar 15 14:56:48 2016 -0700
>> 
>>     memory-hotplug: add automatic onlining policy for the newly added memory
>> 
>> Initially I was going to stop there and bring the policy setting logic to
>> userspace. I met two issues on this way:
>> 
>> 1) It is possible to have memory hotplugged at boot (e.g. with QEMU). These
>>    blocks stay offlined if we turn the onlining policy on by userspace.
>> 
>> 2) My attempt to bring this policy setting to systemd failed, systemd
>>    maintainers suggest to change the default in kernel or ... to use tmpfiles.d
>>    to alter the policy (which looks like a hack to me): 
>>    https://github.com/systemd/systemd/pull/2938
>
> That discussion really didn't come to a conclusion and I don't
> understand why you consider Lennert's "recommended way" to be a hack?

Just the name. To me 'tmpfiles.d' doesn't sound like an appropriate
place to search for kernel tunables settings. It would be much better in
case we had something like 'tunables.d' for that.

-- 
  Vitaly

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[Vitaly Kuznetsov]

David Rientjes <rientjes@google.com> writes:

> On Wed, 6 Apr 2016, Andrew Morton wrote:
>
>> > This patchset continues the work I started with:
>> > 
>> > commit 31bc3858ea3ebcc3157b3f5f0e624c5962f5a7a6
>> > Author: Vitaly Kuznetsov <vkuznets@redhat.com>
>> > Date:   Tue Mar 15 14:56:48 2016 -0700
>> > 
>> >     memory-hotplug: add automatic onlining policy for the newly added memory
>> > 
>> > Initially I was going to stop there and bring the policy setting logic to
>> > userspace. I met two issues on this way:
>> > 
>> > 1) It is possible to have memory hotplugged at boot (e.g. with QEMU). These
>> >    blocks stay offlined if we turn the onlining policy on by userspace.
>> > 
>> > 2) My attempt to bring this policy setting to systemd failed, systemd
>> >    maintainers suggest to change the default in kernel or ... to use tmpfiles.d
>> >    to alter the policy (which looks like a hack to me): 
>> >    https://github.com/systemd/systemd/pull/2938
>> 
>> That discussion really didn't come to a conclusion and I don't
>> understand why you consider Lennert's "recommended way" to be a hack?
>> 
>> > Here I suggest to add a config option to set the default value for the policy
>> > and a kernel command line parameter to make the override.
>> 
>> But the patchset looks pretty reasonable regardless of the above.
>> 
>
> I don't understand why initscripts simply cannot crawl sysfs memory blocks 
> and online them for the same behavior.

Yes, they can. With this patchset I don't bring any new features, it's
rather a convenience so linux distros can make memory hotplug work
'out of the box' without such distro-specific initscripts. Memory
hotplug is a standard feature of all major virt technologies so I think
it's pretty reasonable to have an option to make it work 'by default'
available.

-- 
  Vitaly

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[David Rientjes]

On Thu, 7 Apr 2016, Vitaly Kuznetsov wrote:

> >> > This patchset continues the work I started with:
> >> > 
> >> > commit 31bc3858ea3ebcc3157b3f5f0e624c5962f5a7a6
> >> > Author: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> > Date:   Tue Mar 15 14:56:48 2016 -0700
> >> > 
> >> >     memory-hotplug: add automatic onlining policy for the newly added memory
> >> > 
> >> > Initially I was going to stop there and bring the policy setting logic to
> >> > userspace. I met two issues on this way:
> >> > 
> >> > 1) It is possible to have memory hotplugged at boot (e.g. with QEMU). These
> >> >    blocks stay offlined if we turn the onlining policy on by userspace.
> >> > 
> >> > 2) My attempt to bring this policy setting to systemd failed, systemd
> >> >    maintainers suggest to change the default in kernel or ... to use tmpfiles.d
> >> >    to alter the policy (which looks like a hack to me): 
> >> >    https://github.com/systemd/systemd/pull/2938
> >> 
> >> That discussion really didn't come to a conclusion and I don't
> >> understand why you consider Lennert's "recommended way" to be a hack?
> >> 
> >> > Here I suggest to add a config option to set the default value for the policy
> >> > and a kernel command line parameter to make the override.
> >> 
> >> But the patchset looks pretty reasonable regardless of the above.
> >> 
> >
> > I don't understand why initscripts simply cannot crawl sysfs memory blocks 
> > and online them for the same behavior.
> 
> Yes, they can. With this patchset I don't bring any new features, it's
> rather a convenience so linux distros can make memory hotplug work
> 'out of the box' without such distro-specific initscripts. Memory
> hotplug is a standard feature of all major virt technologies so I think
> it's pretty reasonable to have an option to make it work 'by default'
> available.
> 

I'd personally disagree that we need more and more config options to take 
care of something that an initscript can easily do and most distros 
already have their own initscripts that this can be added to.  I don't see 
anything that the config option adds.

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[Vitaly Kuznetsov]

David Rientjes <rientjes@google.com> writes:

> On Thu, 7 Apr 2016, Vitaly Kuznetsov wrote:
>
>> >> > This patchset continues the work I started with:
>> >> > 
>> >> > commit 31bc3858ea3ebcc3157b3f5f0e624c5962f5a7a6
>> >> > Author: Vitaly Kuznetsov <vkuznets@redhat.com>
>> >> > Date:   Tue Mar 15 14:56:48 2016 -0700
>> >> > 
>> >> >     memory-hotplug: add automatic onlining policy for the newly added memory
>> >> > 
>> >> > Initially I was going to stop there and bring the policy setting logic to
>> >> > userspace. I met two issues on this way:
>> >> > 
>> >> > 1) It is possible to have memory hotplugged at boot (e.g. with QEMU). These
>> >> >    blocks stay offlined if we turn the onlining policy on by userspace.
>> >> > 
>> >> > 2) My attempt to bring this policy setting to systemd failed, systemd
>> >> >    maintainers suggest to change the default in kernel or ... to use tmpfiles.d
>> >> >    to alter the policy (which looks like a hack to me): 
>> >> >    https://github.com/systemd/systemd/pull/2938
>> >> 
>> >> That discussion really didn't come to a conclusion and I don't
>> >> understand why you consider Lennert's "recommended way" to be a hack?
>> >> 
>> >> > Here I suggest to add a config option to set the default value for the policy
>> >> > and a kernel command line parameter to make the override.
>> >> 
>> >> But the patchset looks pretty reasonable regardless of the above.
>> >> 
>> >
>> > I don't understand why initscripts simply cannot crawl sysfs memory blocks 
>> > and online them for the same behavior.
>> 
>> Yes, they can. With this patchset I don't bring any new features, it's
>> rather a convenience so linux distros can make memory hotplug work
>> 'out of the box' without such distro-specific initscripts. Memory
>> hotplug is a standard feature of all major virt technologies so I think
>> it's pretty reasonable to have an option to make it work 'by default'
>> available.
>> 
>
> I'd personally disagree that we need more and more config options to take 
> care of something that an initscript can easily do and most distros 
> already have their own initscripts that this can be added to.  I don't see 
> anything that the config option adds.

Yes, but why does every distro need to solve the exact same issue by 
a distro-specific init script when we can allow setting reasonable
default in kernel?

If the config option itself is a problem (though I don't understand why)
we can get rid of it making the default 'online' and keeping the command
line parameter to disable it for cases when something goes wrong but why
not leave an option for those who want it the other way around?

Other than the above, let's imagine a 'unikernel' scenario when there
are no initscripts and we're in a virtualized environment. We may want to
have memory hotplug there too, but where would we put the 'onlining'
logic? In every userspace we want to run? This doesn't sound right.

-- 
  Vitaly

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[David Rientjes]

On Tue, 19 Apr 2016, Vitaly Kuznetsov wrote:

> > I'd personally disagree that we need more and more config options to take 
> > care of something that an initscript can easily do and most distros 
> > already have their own initscripts that this can be added to.  I don't see 
> > anything that the config option adds.
> 
> Yes, but why does every distro need to solve the exact same issue by 
> a distro-specific init script when we can allow setting reasonable
> default in kernel?
> 

No, only distros that want to change the long-standing default which is 
"offline" since they apparently aren't worried about breaking existing 
userspace.

Changing defaults is always risky business in the kernel, especially when 
it's long standing.  If the default behavior is changeable, userspace 
needs to start testing for that and acting accordingly if it actually 
wants to default to offline (and there are existing tools that suppose the 
long-standing default).  The end result is that the kernel default doesn't 
matter anymore, we've just pushed it to userspace to either online or 
offline at the time of hotplug.

> If the config option itself is a problem (though I don't understand why)
> we can get rid of it making the default 'online' and keeping the command
> line parameter to disable it for cases when something goes wrong but why
> not leave an option for those who want it the other way around?
> 

That could break existing userspace that assumes the default is offline; 
if users are currently hotadding memory and then onlining it when needed 
rather than immediately, they break.  So that's not a possibility.

> Other than the above, let's imagine a 'unikernel' scenario when there
> are no initscripts and we're in a virtualized environment. We may want to
> have memory hotplug there too, but where would we put the 'onlining'
> logic? In every userspace we want to run? This doesn't sound right.
> 

Nobody is resisting hotplug notifiers.

Re: [PATCH 0/2] memory_hotplug: introduce config and command line options to set the default onlining policy

[Vitaly Kuznetsov]

David Rientjes <rientjes@google.com> writes:

> On Tue, 19 Apr 2016, Vitaly Kuznetsov wrote:
>
>> > I'd personally disagree that we need more and more config options to take 
>> > care of something that an initscript can easily do and most distros 
>> > already have their own initscripts that this can be added to.  I don't see 
>> > anything that the config option adds.
>> 
>> Yes, but why does every distro need to solve the exact same issue by 
>> a distro-specific init script when we can allow setting reasonable
>> default in kernel?
>> 
>
> No, only distros that want to change the long-standing default which is 
> "offline" since they apparently aren't worried about breaking existing 
> userspace.
>
> Changing defaults is always risky business in the kernel, especially when 
> it's long standing.  If the default behavior is changeable, userspace 
> needs to start testing for that and acting accordingly if it actually 
> wants to default to offline (and there are existing tools that suppose the 
> long-standing default).  The end result is that the kernel default doesn't 
> matter anymore, we've just pushed it to userspace to either online or 
> offline at the time of hotplug.
>

"We don't break userspace". Yes, I know, but is there an example of such
userspace which is going to break?

E.g. RHEL7 ships the following udev rule by default:
# Memory hotadd request
SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"

which is not very smart but it does the job (with issues I'm trying to
solve). I'm not aware of any breakages reported after it was introduced.

My understanding is that the legacy default 'offline' was introduced
before memory hotplug became a frequently used feature in virtual
machines. When you hotplug physical memory you go to your server room,
open your server, insert memory dimm, ... - in this scenario 'offline'
is a reasonable default. But in VMs mempory hotplug is usually an
automatic from host side -- we address high memory pressure/tenant
requests.

>> If the config option itself is a problem (though I don't understand why)
>> we can get rid of it making the default 'online' and keeping the command
>> line parameter to disable it for cases when something goes wrong but why
>> not leave an option for those who want it the other way around?
>> 
>
> That could break existing userspace that assumes the default is offline; 
> if users are currently hotadding memory and then onlining it when needed 
> rather than immediately, they break.  So that's not a possibility.
>

Yes, so I introduce a config option. Next thing we do we enable it in
'bleeding edge' distros, e.g. Fedora and see who complains. My guess is
that nobody is going to complain.

>> Other than the above, let's imagine a 'unikernel' scenario when there
>> are no initscripts and we're in a virtualized environment. We may want to
>> have memory hotplug there too, but where would we put the 'onlining'
>> logic? In every userspace we want to run? This doesn't sound right.
>> 
>
> Nobody is resisting hotplug notifiers.

Yes, but we need to teach memory hotplug to every userspace instead.

-- 
  Vitaly