[RFC PATCH]: ACPI: Automatically online hot-added memory

[RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

New sockets have on-die memory controllers.  This means that in certain
HW configurations the memory behind the socket comes and goes as the socket
is physically enabled and disabled.

Since the cpu bringup code does on node memory allocations, the memory on the
added socket must be onlined first.

Add a .config option to automatically online hot added memory, and enable it
in the acpi memory add path.

Signed-off-by: Prarit Bhargava <prarit@redhat.com>

diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
index 93d2c79..dece6bd 100644
--- a/drivers/acpi/Kconfig
+++ b/drivers/acpi/Kconfig
@@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
 	  To compile this driver as a module, choose M here:
 	  the module will be called acpi_memhotplug.
 
+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+	bool "Automatically online hotplugged memory"
+	depends on ACPI_HOTPLUG_MEMORY
+	default n
+	help
+	  This forces memory that is brought into service by ACPI
+	  to be automatically onlined.
+
 config ACPI_SBS
 	tristate "Smart Battery System"
 	depends on X86
diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
index 3597d73..9814c50 100644
--- a/drivers/acpi/acpi_memhotplug.c
+++ b/drivers/acpi/acpi_memhotplug.c
@@ -30,6 +30,7 @@
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/memory_hotplug.h>
+#include <linux/memory.h>
 #include <acpi/acpi_drivers.h>
 
 #define ACPI_MEMORY_DEVICE_CLASS		"memory"
@@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
 		result = add_memory(node, info->start_addr, info->length);
 		if (result)
 			continue;
+#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+		/*
+		 * New processors require memory to be online before cpus.
+		 * No notifications are required here as "we" are the only
+		 * ones who know about the new memory right now.
+		 */
+		result = online_pages(info->start_addr >> PAGE_SHIFT,
+				      info->length >> PAGE_SHIFT);
+		if (!result)
+			set_memory_state(info->start_addr, MEM_ONLINE);
+		else
+			printk("Memory online failed.\n");
+#endif
 		info->enabled = 1;
 		num_enabled++;
 	}
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 2f86915..fb465d5 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
 }
 
 /*
+ * need an interface for the VM to mark sections on and offline when hot-adding
+ * memory.
+ */
+void set_memory_state(unsigned long addr, unsigned long state)
+{
+	struct mem_section *section;
+	struct memory_block *mem;
+
+	section = __pfn_to_section(addr >> PAGE_SHIFT);
+	mem = find_memory_block(section);
+	mem->state = state;
+}
+
+/*
  * need an interface for the VM to add new memory regions,
  * but without onlining it.
  */
diff --git a/include/linux/memory.h b/include/linux/memory.h
index 1adfe77..5d8d78c 100644
--- a/include/linux/memory.h
+++ b/include/linux/memory.h
@@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
 extern int memory_notify(unsigned long val, void *v);
 extern int memory_isolate_notify(unsigned long val, void *v);
 extern struct memory_block *find_memory_block(struct mem_section *);
+extern void set_memory_state(unsigned long, unsigned long);
 #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
 enum mem_add_context { BOOT, HOTPLUG };
 #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Matthew Garrett]

On Tue, Mar 09, 2010 at 09:12:03AM -0500, Prarit Bhargava wrote:
> New sockets have on-die memory controllers.  This means that in certain
> HW configurations the memory behind the socket comes and goes as the socket
> is physically enabled and disabled.
> 
> Since the cpu bringup code does on node memory allocations, the memory on the
> added socket must be onlined first.
> 
> Add a .config option to automatically online hot added memory, and enable it
> in the acpi memory add path.

This seems like the right thing to do.

> +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
> +	bool "Automatically online hotplugged memory"
> +	depends on ACPI_HOTPLUG_MEMORY
> +	default n

default !S390? default x86?

> +		result = online_pages(info->start_addr >> PAGE_SHIFT,
> +				      info->length >> PAGE_SHIFT);
> +		if (!result)
> +			set_memory_state(info->start_addr, MEM_ONLINE);
> +		else
> +			printk("Memory online failed.\n");

That probably wants to be more descriptive and have a loglevel. What 
happens if this fails? The CPU presumably can't be brought up?

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

Matthew Garrett wrote:
> On Tue, Mar 09, 2010 at 09:12:03AM -0500, Prarit Bhargava wrote:
>   
>> New sockets have on-die memory controllers.  This means that in certain
>> HW configurations the memory behind the socket comes and goes as the socket
>> is physically enabled and disabled.
>>
>> Since the cpu bringup code does on node memory allocations, the memory on the
>> added socket must be onlined first.
>>
>> Add a .config option to automatically online hot added memory, and enable it
>> in the acpi memory add path.
>>     
>
> This seems like the right thing to do.
>
>   
>> +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
>> +	bool "Automatically online hotplugged memory"
>> +	depends on ACPI_HOTPLUG_MEMORY
>> +	default n
>>     
>
> default !S390? default x86?
>   

I've been inquiring about this.  It seems that some vendors want the 
capability to manually bring memory online.  I'm trying to get more 
details as this doesn't seem intuitive.  It seems to me that if you add 
memory you want to online it right now.

>   
>> +		result = online_pages(info->start_addr >> PAGE_SHIFT,
>> +				      info->length >> PAGE_SHIFT);
>> +		if (!result)
>> +			set_memory_state(info->start_addr, MEM_ONLINE);
>> +		else
>> +			printk("Memory online failed.\n");
>>     
>
> That probably wants to be more descriptive and have a loglevel. 

Oops.  I'm very bad about setting the loglevel's on my printk's ;) ... 
I'll fix that in the next patch.

> What 
> happens if this fails? The CPU presumably can't be brought up?
>   

That's a good question...  After thinking about it I think that 
acpi_processor_add() must check the node to see if it has memory before 
bringing the cpu online.

I'll code that up, test, and repost.

P.

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Alex Chiang]

I think if you're going to poke into drivers/base/memory.c we
should let the -mm guys know that you're creating a new
interface.

The ACPI part looks fine to me.

cc added.

/ac

* Prarit Bhargava <prarit@redhat.com>:
> New sockets have on-die memory controllers.  This means that in certain
> HW configurations the memory behind the socket comes and goes as the socket
> is physically enabled and disabled.
> 
> Since the cpu bringup code does on node memory allocations, the memory on the
> added socket must be onlined first.
> 
> Add a .config option to automatically online hot added memory, and enable it
> in the acpi memory add path.
> 
> Signed-off-by: Prarit Bhargava <prarit@redhat.com>
> 
> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
> index 93d2c79..dece6bd 100644
> --- a/drivers/acpi/Kconfig
> +++ b/drivers/acpi/Kconfig
> @@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
>  	  To compile this driver as a module, choose M here:
>  	  the module will be called acpi_memhotplug.
>  
> +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
> +	bool "Automatically online hotplugged memory"
> +	depends on ACPI_HOTPLUG_MEMORY
> +	default n
> +	help
> +	  This forces memory that is brought into service by ACPI
> +	  to be automatically onlined.
> +
>  config ACPI_SBS
>  	tristate "Smart Battery System"
>  	depends on X86
> diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
> index 3597d73..9814c50 100644
> --- a/drivers/acpi/acpi_memhotplug.c
> +++ b/drivers/acpi/acpi_memhotplug.c
> @@ -30,6 +30,7 @@
>  #include <linux/init.h>
>  #include <linux/types.h>
>  #include <linux/memory_hotplug.h>
> +#include <linux/memory.h>
>  #include <acpi/acpi_drivers.h>
>  
>  #define ACPI_MEMORY_DEVICE_CLASS		"memory"
> @@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
>  		result = add_memory(node, info->start_addr, info->length);
>  		if (result)
>  			continue;
> +#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
> +		/*
> +		 * New processors require memory to be online before cpus.
> +		 * No notifications are required here as "we" are the only
> +		 * ones who know about the new memory right now.
> +		 */
> +		result = online_pages(info->start_addr >> PAGE_SHIFT,
> +				      info->length >> PAGE_SHIFT);
> +		if (!result)
> +			set_memory_state(info->start_addr, MEM_ONLINE);
> +		else
> +			printk("Memory online failed.\n");
> +#endif
>  		info->enabled = 1;
>  		num_enabled++;
>  	}
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index 2f86915..fb465d5 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
>  }
>  
>  /*
> + * need an interface for the VM to mark sections on and offline when hot-adding
> + * memory.
> + */
> +void set_memory_state(unsigned long addr, unsigned long state)
> +{
> +	struct mem_section *section;
> +	struct memory_block *mem;
> +
> +	section = __pfn_to_section(addr >> PAGE_SHIFT);
> +	mem = find_memory_block(section);
> +	mem->state = state;
> +}
> +
> +/*
>   * need an interface for the VM to add new memory regions,
>   * but without onlining it.
>   */
> diff --git a/include/linux/memory.h b/include/linux/memory.h
> index 1adfe77..5d8d78c 100644
> --- a/include/linux/memory.h
> +++ b/include/linux/memory.h
> @@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
>  extern int memory_notify(unsigned long val, void *v);
>  extern int memory_isolate_notify(unsigned long val, void *v);
>  extern struct memory_block *find_memory_block(struct mem_section *);
> +extern void set_memory_state(unsigned long, unsigned long);
>  #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
>  enum mem_add_context { BOOT, HOTPLUG };
>  #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Alex Chiang]

I think if you're going to poke into drivers/base/memory.c we
should let the -mm guys know that you're creating a new
interface.

The ACPI part looks fine to me.

cc added.

/ac

* Prarit Bhargava <prarit@redhat.com>:
> New sockets have on-die memory controllers.  This means that in certain
> HW configurations the memory behind the socket comes and goes as the socket
> is physically enabled and disabled.
> 
> Since the cpu bringup code does on node memory allocations, the memory on the
> added socket must be onlined first.
> 
> Add a .config option to automatically online hot added memory, and enable it
> in the acpi memory add path.
> 
> Signed-off-by: Prarit Bhargava <prarit@redhat.com>
> 
> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
> index 93d2c79..dece6bd 100644
> --- a/drivers/acpi/Kconfig
> +++ b/drivers/acpi/Kconfig
> @@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
>  	  To compile this driver as a module, choose M here:
>  	  the module will be called acpi_memhotplug.
>  
> +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
> +	bool "Automatically online hotplugged memory"
> +	depends on ACPI_HOTPLUG_MEMORY
> +	default n
> +	help
> +	  This forces memory that is brought into service by ACPI
> +	  to be automatically onlined.
> +
>  config ACPI_SBS
>  	tristate "Smart Battery System"
>  	depends on X86
> diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
> index 3597d73..9814c50 100644
> --- a/drivers/acpi/acpi_memhotplug.c
> +++ b/drivers/acpi/acpi_memhotplug.c
> @@ -30,6 +30,7 @@
>  #include <linux/init.h>
>  #include <linux/types.h>
>  #include <linux/memory_hotplug.h>
> +#include <linux/memory.h>
>  #include <acpi/acpi_drivers.h>
>  
>  #define ACPI_MEMORY_DEVICE_CLASS		"memory"
> @@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
>  		result = add_memory(node, info->start_addr, info->length);
>  		if (result)
>  			continue;
> +#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
> +		/*
> +		 * New processors require memory to be online before cpus.
> +		 * No notifications are required here as "we" are the only
> +		 * ones who know about the new memory right now.
> +		 */
> +		result = online_pages(info->start_addr >> PAGE_SHIFT,
> +				      info->length >> PAGE_SHIFT);
> +		if (!result)
> +			set_memory_state(info->start_addr, MEM_ONLINE);
> +		else
> +			printk("Memory online failed.\n");
> +#endif
>  		info->enabled = 1;
>  		num_enabled++;
>  	}
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index 2f86915..fb465d5 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
>  }
>  
>  /*
> + * need an interface for the VM to mark sections on and offline when hot-adding
> + * memory.
> + */
> +void set_memory_state(unsigned long addr, unsigned long state)
> +{
> +	struct mem_section *section;
> +	struct memory_block *mem;
> +
> +	section = __pfn_to_section(addr >> PAGE_SHIFT);
> +	mem = find_memory_block(section);
> +	mem->state = state;
> +}
> +
> +/*
>   * need an interface for the VM to add new memory regions,
>   * but without onlining it.
>   */
> diff --git a/include/linux/memory.h b/include/linux/memory.h
> index 1adfe77..5d8d78c 100644
> --- a/include/linux/memory.h
> +++ b/include/linux/memory.h
> @@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
>  extern int memory_notify(unsigned long val, void *v);
>  extern int memory_isolate_notify(unsigned long val, void *v);
>  extern struct memory_block *find_memory_block(struct mem_section *);
> +extern void set_memory_state(unsigned long, unsigned long);
>  #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
>  enum mem_add_context { BOOT, HOTPLUG };
>  #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

Alex Chiang wrote:
> I think if you're going to poke into drivers/base/memory.c we
> should let the -mm guys know that you're creating a new
> interface.
>   

Absolutely -- I was just RFC'ing for now ;)

> The ACPI part looks fine to me.
>
> cc added.
>
>   

Thanks :)

P.

> /ac
>
> * Prarit Bhargava <prarit@redhat.com>:
>   
>> New sockets have on-die memory controllers.  This means that in certain
>> HW configurations the memory behind the socket comes and goes as the socket
>> is physically enabled and disabled.
>>
>> Since the cpu bringup code does on node memory allocations, the memory on the
>> added socket must be onlined first.
>>
>> Add a .config option to automatically online hot added memory, and enable it
>> in the acpi memory add path.
>>
>> Signed-off-by: Prarit Bhargava <prarit@redhat.com>
>>
>> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
>> index 93d2c79..dece6bd 100644
>> --- a/drivers/acpi/Kconfig
>> +++ b/drivers/acpi/Kconfig
>> @@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
>>  	  To compile this driver as a module, choose M here:
>>  	  the module will be called acpi_memhotplug.
>>  
>> +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
>> +	bool "Automatically online hotplugged memory"
>> +	depends on ACPI_HOTPLUG_MEMORY
>> +	default n
>> +	help
>> +	  This forces memory that is brought into service by ACPI
>> +	  to be automatically onlined.
>> +
>>  config ACPI_SBS
>>  	tristate "Smart Battery System"
>>  	depends on X86
>> diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
>> index 3597d73..9814c50 100644
>> --- a/drivers/acpi/acpi_memhotplug.c
>> +++ b/drivers/acpi/acpi_memhotplug.c
>> @@ -30,6 +30,7 @@
>>  #include <linux/init.h>
>>  #include <linux/types.h>
>>  #include <linux/memory_hotplug.h>
>> +#include <linux/memory.h>
>>  #include <acpi/acpi_drivers.h>
>>  
>>  #define ACPI_MEMORY_DEVICE_CLASS		"memory"
>> @@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
>>  		result = add_memory(node, info->start_addr, info->length);
>>  		if (result)
>>  			continue;
>> +#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
>> +		/*
>> +		 * New processors require memory to be online before cpus.
>> +		 * No notifications are required here as "we" are the only
>> +		 * ones who know about the new memory right now.
>> +		 */
>> +		result = online_pages(info->start_addr >> PAGE_SHIFT,
>> +				      info->length >> PAGE_SHIFT);
>> +		if (!result)
>> +			set_memory_state(info->start_addr, MEM_ONLINE);
>> +		else
>> +			printk("Memory online failed.\n");
>> +#endif
>>  		info->enabled = 1;
>>  		num_enabled++;
>>  	}
>> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
>> index 2f86915..fb465d5 100644
>> --- a/drivers/base/memory.c
>> +++ b/drivers/base/memory.c
>> @@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
>>  }
>>  
>>  /*
>> + * need an interface for the VM to mark sections on and offline when hot-adding
>> + * memory.
>> + */
>> +void set_memory_state(unsigned long addr, unsigned long state)
>> +{
>> +	struct mem_section *section;
>> +	struct memory_block *mem;
>> +
>> +	section = __pfn_to_section(addr >> PAGE_SHIFT);
>> +	mem = find_memory_block(section);
>> +	mem->state = state;
>> +}
>> +
>> +/*
>>   * need an interface for the VM to add new memory regions,
>>   * but without onlining it.
>>   */
>> diff --git a/include/linux/memory.h b/include/linux/memory.h
>> index 1adfe77..5d8d78c 100644
>> --- a/include/linux/memory.h
>> +++ b/include/linux/memory.h
>> @@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
>>  extern int memory_notify(unsigned long val, void *v);
>>  extern int memory_isolate_notify(unsigned long val, void *v);
>>  extern struct memory_block *find_memory_block(struct mem_section *);
>> +extern void set_memory_state(unsigned long, unsigned long);
>>  #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
>>  enum mem_add_context { BOOT, HOTPLUG };
>>  #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>     

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

Alex Chiang wrote:
> I think if you're going to poke into drivers/base/memory.c we
> should let the -mm guys know that you're creating a new
> interface.
>   

Absolutely -- I was just RFC'ing for now ;)

> The ACPI part looks fine to me.
>
> cc added.
>
>   

Thanks :)

P.

> /ac
>
> * Prarit Bhargava <prarit@redhat.com>:
>   
>> New sockets have on-die memory controllers.  This means that in certain
>> HW configurations the memory behind the socket comes and goes as the socket
>> is physically enabled and disabled.
>>
>> Since the cpu bringup code does on node memory allocations, the memory on the
>> added socket must be onlined first.
>>
>> Add a .config option to automatically online hot added memory, and enable it
>> in the acpi memory add path.
>>
>> Signed-off-by: Prarit Bhargava <prarit@redhat.com>
>>
>> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
>> index 93d2c79..dece6bd 100644
>> --- a/drivers/acpi/Kconfig
>> +++ b/drivers/acpi/Kconfig
>> @@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
>>  	  To compile this driver as a module, choose M here:
>>  	  the module will be called acpi_memhotplug.
>>  
>> +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
>> +	bool "Automatically online hotplugged memory"
>> +	depends on ACPI_HOTPLUG_MEMORY
>> +	default n
>> +	help
>> +	  This forces memory that is brought into service by ACPI
>> +	  to be automatically onlined.
>> +
>>  config ACPI_SBS
>>  	tristate "Smart Battery System"
>>  	depends on X86
>> diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
>> index 3597d73..9814c50 100644
>> --- a/drivers/acpi/acpi_memhotplug.c
>> +++ b/drivers/acpi/acpi_memhotplug.c
>> @@ -30,6 +30,7 @@
>>  #include <linux/init.h>
>>  #include <linux/types.h>
>>  #include <linux/memory_hotplug.h>
>> +#include <linux/memory.h>
>>  #include <acpi/acpi_drivers.h>
>>  
>>  #define ACPI_MEMORY_DEVICE_CLASS		"memory"
>> @@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
>>  		result = add_memory(node, info->start_addr, info->length);
>>  		if (result)
>>  			continue;
>> +#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
>> +		/*
>> +		 * New processors require memory to be online before cpus.
>> +		 * No notifications are required here as "we" are the only
>> +		 * ones who know about the new memory right now.
>> +		 */
>> +		result = online_pages(info->start_addr >> PAGE_SHIFT,
>> +				      info->length >> PAGE_SHIFT);
>> +		if (!result)
>> +			set_memory_state(info->start_addr, MEM_ONLINE);
>> +		else
>> +			printk("Memory online failed.\n");
>> +#endif
>>  		info->enabled = 1;
>>  		num_enabled++;
>>  	}
>> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
>> index 2f86915..fb465d5 100644
>> --- a/drivers/base/memory.c
>> +++ b/drivers/base/memory.c
>> @@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
>>  }
>>  
>>  /*
>> + * need an interface for the VM to mark sections on and offline when hot-adding
>> + * memory.
>> + */
>> +void set_memory_state(unsigned long addr, unsigned long state)
>> +{
>> +	struct mem_section *section;
>> +	struct memory_block *mem;
>> +
>> +	section = __pfn_to_section(addr >> PAGE_SHIFT);
>> +	mem = find_memory_block(section);
>> +	mem->state = state;
>> +}
>> +
>> +/*
>>   * need an interface for the VM to add new memory regions,
>>   * but without onlining it.
>>   */
>> diff --git a/include/linux/memory.h b/include/linux/memory.h
>> index 1adfe77..5d8d78c 100644
>> --- a/include/linux/memory.h
>> +++ b/include/linux/memory.h
>> @@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
>>  extern int memory_notify(unsigned long val, void *v);
>>  extern int memory_isolate_notify(unsigned long val, void *v);
>>  extern struct memory_block *find_memory_block(struct mem_section *);
>> +extern void set_memory_state(unsigned long, unsigned long);
>>  #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
>>  enum mem_add_context { BOOT, HOTPLUG };
>>  #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>     

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[ykzhao]

On Wed, 2010-03-10 at 02:27 +0800, Prarit Bhargava wrote:
> 
> Matthew Garrett wrote:
> > On Tue, Mar 09, 2010 at 09:12:03AM -0500, Prarit Bhargava wrote:
> >   
> >> New sockets have on-die memory controllers.  This means that in certain
> >> HW configurations the memory behind the socket comes and goes as the socket
> >> is physically enabled and disabled.
> >>
> >> Since the cpu bringup code does on node memory allocations, the memory on the
> >> added socket must be onlined first.
> >>
> >> Add a .config option to automatically online hot added memory, and enable it
> >> in the acpi memory add path.
> >>     
> >
> > This seems like the right thing to do.
> >
> >   
> >> +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
> >> +	bool "Automatically online hotplugged memory"
> >> +	depends on ACPI_HOTPLUG_MEMORY
> >> +	default n
> >>     
> >
> > default !S390? default x86?
> >   
> 
> I've been inquiring about this.  It seems that some vendors want the 
> capability to manually bring memory online.  I'm trying to get more 
> details as this doesn't seem intuitive.  It seems to me that if you add 
> memory you want to online it right now.
> 
> >   
> >> +		result = online_pages(info->start_addr >> PAGE_SHIFT,
> >> +				      info->length >> PAGE_SHIFT);
> >> +		if (!result)
> >> +			set_memory_state(info->start_addr, MEM_ONLINE);
> >> +		else
> >> +			printk("Memory online failed.\n");
> >>     
> >
> > That probably wants to be more descriptive and have a loglevel. 
> 
> Oops.  I'm very bad about setting the loglevel's on my printk's ;) ... 
> I'll fix that in the next patch.

One note is that the hot-added memory will be automatically onlined. And
we had better add an interface so that the user-space can quiry whether
the hot-added memory is automatically onlined. Otherwise the user-space
will receive the corresponding event and online hot-added memory again.
In such case it will get some error messages.

> 
> > What 
> > happens if this fails? The CPU presumably can't be brought up?
> >   
> 
> That's a good question...  After thinking about it I think that 
> acpi_processor_add() must check the node to see if it has memory before 
> bringing the cpu online.

Why do we need to see whether the memory is onlined before bringing cpu
to online state? It seems that there is no dependency between cpu online
and memory online.

Thanks.
> 
> I'll code that up, test, and repost.
> 
> P.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

>
> Why do we need to see whether the memory is onlined before bringing cpu
> to online state? It seems that there is no dependency between cpu online
> and memory online.
>
>   

Yakui,

Here's a deeper look into the issue.  New Intel processors have an 
on-die memory controller and this means that as the socket comes and 
goes, so does the memory "behind" the socket.

ie) with new processors it is possible that an entire node which 
consists of memory and cpus comes and goes with the socket enable and 
disable.

The cpu bringup code does local node allocations for the cpu.  If the 
memory connected to the node (which is "behind" the socket) isn't 
online, then these allocations fail, and then the cpu bringup fails.

P.

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[ykzhao]

On Wed, 2010-03-10 at 21:28 +0800, Prarit Bhargava wrote:
> >
> > Why do we need to see whether the memory is onlined before bringing cpu
> > to online state? It seems that there is no dependency between cpu online
> > and memory online.
> >
> >   
> 
> Yakui,
> 

Thanks for the explanation.

> Here's a deeper look into the issue.  New Intel processors have an 
> on-die memory controller and this means that as the socket comes and 
> goes, so does the memory "behind" the socket.

Yes. The nehalem processor has the integrated memory controller. But it
is not required that the hot-added memory should be onlined before
bringing up CPU.
    I do the following memory-hotplug test on one Machine.
    a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
    b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

In the above testing case the CPU is brought up before onlining the
hot-added memory. And the test shows that it can work well.

> 
> ie) with new processors it is possible that an entire node which 
> consists of memory and cpus comes and goes with the socket enable and 
> disable.
> 
> The cpu bringup code does local node allocations for the cpu.  If the 
> memory connected to the node (which is "behind" the socket) isn't 
> online, then these allocations fail, and then the cpu bringup fails.

If the CPU can't allocate the memory from its own node, it can turn to
other node and see whether the memory can be allocated. And this depends
on the NUMA allocation policy.

   
> 
> P.
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

>
> Yes. The nehalem processor has the integrated memory controller. But it
> is not required that the hot-added memory should be onlined before
> bringing up CPU.
>     I do the following memory-hotplug test on one Machine.
>     a. Before hot plugging memory, four CPUs socket are installed and
> all the logical CPU are brought up. (Only one node has the memory)
>     b. The memory is hot-plugged and then the memory is onlined so that
> it can be accessed by the system.
>
> In the above testing case the CPU is brought up before onlining the
> hot-added memory. And the test shows that it can work well.
>   

That doesn't work when you have multiple nodes AFAICT.  The cpus do not 
come into service because of a lack of memory on the node....  per node 
allocations will fail.

Just curious, exactly what did you test with?  2.6.33 (or newer)?

>   
>> ie) with new processors it is possible that an entire node which 
>> consists of memory and cpus comes and goes with the socket enable and 
>> disable.
>>
>> The cpu bringup code does local node allocations for the cpu.  If the 
>> memory connected to the node (which is "behind" the socket) isn't 
>> online, then these allocations fail, and then the cpu bringup fails.
>>     
>
> If the CPU can't allocate the memory from its own node, it can turn to
> other node and see whether the memory can be allocated. And this depends
> on the NUMA allocation policy.
>   

Maybe that could work, but I haven't gotten that to work.  Even if it 
does work, it's a HUGE performance hit :(.  I can't imagine incurring an 
extra hop just to get to per_cpu memory.  I'd rather bring the memory on 
the local node up first.

I think it is much better to bring the memory up first.

Either way, it's a nice feature to have.

P.

>    
>   
>> P.
>>
>>     
>
>   

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[ykzhao]

On Thu, 2010-03-11 at 10:18 +0800, Prarit Bhargava wrote:
> >
> > Yes. The nehalem processor has the integrated memory controller. But it
> > is not required that the hot-added memory should be onlined before
> > bringing up CPU.
> >     I do the following memory-hotplug test on one Machine.
> >     a. Before hot plugging memory, four CPUs socket are installed and
> > all the logical CPU are brought up. (Only one node has the memory)
> >     b. The memory is hot-plugged and then the memory is onlined so that
> > it can be accessed by the system.
> >
> > In the above testing case the CPU is brought up before onlining the
> > hot-added memory. And the test shows that it can work well.
> >   
> 
> That doesn't work when you have multiple nodes AFAICT.  The cpus do not 
> come into service because of a lack of memory on the node....  per node 
> allocations will fail.

In the test the system has multiple nodes. The reason is that the cpu
without memory can turn to other node and allocate the memory.

> 
> Just curious, exactly what did you test with?  2.6.33 (or newer)?

I test it on 2.6.32 kernel.

> 
> >   
> >> ie) with new processors it is possible that an entire node which 
> >> consists of memory and cpus comes and goes with the socket enable and 
> >> disable.
> >>
> >> The cpu bringup code does local node allocations for the cpu.  If the 
> >> memory connected to the node (which is "behind" the socket) isn't 
> >> online, then these allocations fail, and then the cpu bringup fails.
> >>     
> >
> > If the CPU can't allocate the memory from its own node, it can turn to
> > other node and see whether the memory can be allocated. And this depends
> > on the NUMA allocation policy.
> >   
> 
> Maybe that could work, but I haven't gotten that to work.  Even if it 
> does work, it's a HUGE performance hit :(.  I can't imagine incurring an 
> extra hop just to get to per_cpu memory.  I'd rather bring the memory on 
> the local node up first.
> 
> I think it is much better to bring the memory up first.
> 
> Either way, it's a nice feature to have.
> 
> P.
> 
> >    
> >   
> >> P.
> >>
> >>     
> >
> >   

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[chen gong]

On 2010-3-11 16:07, ykzhao wrote:
> On Thu, 2010-03-11 at 10:18 +0800, Prarit Bhargava wrote:
>>>
>>> Yes. The nehalem processor has the integrated memory controller. But it
>>> is not required that the hot-added memory should be onlined before
>>> bringing up CPU.
>>>     I do the following memory-hotplug test on one Machine.
>>>     a. Before hot plugging memory, four CPUs socket are installed and
>>> all the logical CPU are brought up. (Only one node has the memory)
>>>     b. The memory is hot-plugged and then the memory is onlined so that
>>> it can be accessed by the system.
>>>
>>> In the above testing case the CPU is brought up before onlining the
>>> hot-added memory. And the test shows that it can work well.
>>>   
>>
>> That doesn't work when you have multiple nodes AFAICT.  The cpus do not 
>> come into service because of a lack of memory on the node....  per node 
>> allocations will fail.
> 
> In the test the system has multiple nodes. The reason is that the cpu
> without memory can turn to other node and allocate the memory.

I agree with Yakui. The memory and CPU are irrelevant in some way. CPU can
get memory from other nodes if it hasn't local memory, though for now it has some issues.
Andi is working on it now (http://lkml.org/lkml/2010/2/3/343)

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some
Novell guy is working on it.

> 
>>
>> Just curious, exactly what did you test with?  2.6.33 (or newer)?
> 
> I test it on 2.6.32 kernel.
> 
>>
>>>   
>>>> ie) with new processors it is possible that an entire node which 
>>>> consists of memory and cpus comes and goes with the socket enable and 
>>>> disable.
>>>>
>>>> The cpu bringup code does local node allocations for the cpu.  If the 
>>>> memory connected to the node (which is "behind" the socket) isn't 
>>>> online, then these allocations fail, and then the cpu bringup fails.
>>>>     
>>>
>>> If the CPU can't allocate the memory from its own node, it can turn to
>>> other node and see whether the memory can be allocated. And this depends
>>> on the NUMA allocation policy.
>>>   
>>
>> Maybe that could work, but I haven't gotten that to work.  Even if it 
>> does work, it's a HUGE performance hit :(.  I can't imagine incurring an 
>> extra hop just to get to per_cpu memory.  I'd rather bring the memory on 
>> the local node up first.
>>
>> I think it is much better to bring the memory up first.
>>
>> Either way, it's a nice feature to have.

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

ykzhao wrote:
> On Thu, 2010-03-11 at 10:18 +0800, Prarit Bhargava wrote:
>   
>>> Yes. The nehalem processor has the integrated memory controller. But it
>>> is not required that the hot-added memory should be onlined before
>>> bringing up CPU.
>>>     I do the following memory-hotplug test on one Machine.
>>>     a. Before hot plugging memory, four CPUs socket are installed and
>>> all the logical CPU are brought up. (Only one node has the memory)
>>>     b. The memory is hot-plugged and then the memory is onlined so that
>>> it can be accessed by the system.
>>>
>>> In the above testing case the CPU is brought up before onlining the
>>> hot-added memory. And the test shows that it can work well.
>>>   
>>>       
>> That doesn't work when you have multiple nodes AFAICT.  The cpus do not 
>> come into service because of a lack of memory on the node....  per node 
>> allocations will fail.
>>     
>
> In the test the system has multiple nodes. The reason is that the cpu
> without memory can turn to other node and allocate the memory.
>
>   
>> Just curious, exactly what did you test with?  2.6.33 (or newer)?
>>     
>
> I test it on 2.6.32 kernel.
>   

2.6.32 doesn't work with memory and cpu.
2.6.33 doesn't work with memory and cpu.

Other patches (which have been posted upstream) are required.


I'll try to dig up the patches and you can retest.

I will try a memory-free cpu add as well to see if that works... but the 
performance hit due to the off-node allocations probably doesn't make it 
worthwhile.


P.

>   

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

chen gong wrote:
> On 2010-3-11 16:07, ykzhao wrote:
>   
>> On Thu, 2010-03-11 at 10:18 +0800, Prarit Bhargava wrote:
>>     
>>>> Yes. The nehalem processor has the integrated memory controller. But it
>>>> is not required that the hot-added memory should be onlined before
>>>> bringing up CPU.
>>>>     I do the following memory-hotplug test on one Machine.
>>>>     a. Before hot plugging memory, four CPUs socket are installed and
>>>> all the logical CPU are brought up. (Only one node has the memory)
>>>>     b. The memory is hot-plugged and then the memory is onlined so that
>>>> it can be accessed by the system.
>>>>
>>>> In the above testing case the CPU is brought up before onlining the
>>>> hot-added memory. And the test shows that it can work well.
>>>>   
>>>>         
>>> That doesn't work when you have multiple nodes AFAICT.  The cpus do not 
>>> come into service because of a lack of memory on the node....  per node 
>>> allocations will fail.
>>>       
>> In the test the system has multiple nodes. The reason is that the cpu
>> without memory can turn to other node and allocate the memory.
>>     
>
> I agree with Yakui. The memory and CPU are irrelevant in some way. CPU can
> get memory from other nodes if it hasn't local memory, though for now it has some issues.
> Andi is working on it now (http://lkml.org/lkml/2010/2/3/343)
>   

Right -- I'm working with Andi on this to get RHEL working.  So I have 
his slab patches, etc..  That's one of the reasons it isn't working 
right now.

> BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some
> Novell guy is working on it.
>
>   

I was thinking about that, but the CPU comes online automatically, so 
why not memory?  It seems to be a discrepancy between the two.   And ... 
the two of you still aren't considering the obvious performance hit 
which I would think makes this a must have.

P.


>   

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[ykzhao]

On Thu, 2010-03-11 at 19:18 +0800, Prarit Bhargava wrote:
> 
> ykzhao wrote:
> > On Thu, 2010-03-11 at 10:18 +0800, Prarit Bhargava wrote:
> >   
> >>> Yes. The nehalem processor has the integrated memory controller. But it
> >>> is not required that the hot-added memory should be onlined before
> >>> bringing up CPU.
> >>>     I do the following memory-hotplug test on one Machine.
> >>>     a. Before hot plugging memory, four CPUs socket are installed and
> >>> all the logical CPU are brought up. (Only one node has the memory)
> >>>     b. The memory is hot-plugged and then the memory is onlined so that
> >>> it can be accessed by the system.
> >>>
> >>> In the above testing case the CPU is brought up before onlining the
> >>> hot-added memory. And the test shows that it can work well.
> >>>   
> >>>       
> >> That doesn't work when you have multiple nodes AFAICT.  The cpus do not 
> >> come into service because of a lack of memory on the node....  per node 
> >> allocations will fail.
> >>     
> >
> > In the test the system has multiple nodes. The reason is that the cpu
> > without memory can turn to other node and allocate the memory.
> >
> >   
> >> Just curious, exactly what did you test with?  2.6.33 (or newer)?
> >>     
> >
> > I test it on 2.6.32 kernel.
> >   
> 
> 2.6.32 doesn't work with memory and cpu.
> 2.6.33 doesn't work with memory and cpu.

I don't test the scenario of both cpu and memory hot-plugging. 

What I mean is that the 2.6.32 kernel can work if only memory is
hot-plugged (The cpu is already brought-up in the boot phase).

> 
> Other patches (which have been posted upstream) are required.
> 
> 
> I'll try to dig up the patches and you can retest.
> 
> I will try a memory-free cpu add as well to see if that works... but the 
> performance hit due to the off-node allocations probably doesn't make it 
> worthwhile.
> 
> 
> P.
> 
> >   

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Thomas Renninger]

On Thursday 11 March 2010 01:55:15 ykzhao wrote:
> On Wed, 2010-03-10 at 21:28 +0800, Prarit Bhargava wrote:
> > >
> > > Why do we need to see whether the memory is onlined before bringing cpu
> > > to online state? It seems that there is no dependency between cpu online
> > > and memory online.
> > >
> > >   
> > 
> > Yakui,
> > 
> 
> Thanks for the explanation.
> 
> > Here's a deeper look into the issue.  New Intel processors have an 
> > on-die memory controller and this means that as the socket comes and 
> > goes, so does the memory "behind" the socket.
> 
> Yes. The nehalem processor has the integrated memory controller. But it
> is not required that the hot-added memory should be onlined before
> bringing up CPU.
>     I do the following memory-hotplug test on one Machine.
>     a. Before hot plugging memory, four CPUs socket are installed and
> all the logical CPU are brought up. (Only one node has the memory)
>     b. The memory is hot-plugged and then the memory is onlined so that
> it can be accessed by the system.
> 
> In the above testing case the CPU is brought up before onlining the
> hot-added memory. And the test shows that it can work well.
> 
> > 
> > ie) with new processors it is possible that an entire node which 
> > consists of memory and cpus comes and goes with the socket enable and 
> > disable.
> > 
> > The cpu bringup code does local node allocations for the cpu.  If the 
> > memory connected to the node (which is "behind" the socket) isn't 
> > online, then these allocations fail, and then the cpu bringup fails.
> 
> If the CPU can't allocate the memory from its own node, it can turn to
> other node and see whether the memory can be allocated. And this depends
> on the NUMA allocation policy.
Yes and this is broken and needs fixing.
Yakui, I expect you miss this patch and wrongly online the cpus to existing
nodes, therefore you do not run into "out of memory" conditions:
0271f91003d3703675be13b8865618359a6caa1f

I know for sure that slab is broken.
slub behaves different, but I am not sure whether this is due to wrong CPU
hotadd code (processor_core.c is also broken and you get wrong C-state info
from BIOS tables on hotadded CPUs)

Prarit: Can you retest with slub and processor.max_cstate=1, this could/should
work.

AFAIK vmware injects memory in the same way into clients, so you may have
different behavior of virtualized Linux clients.

This is a work around for current memory management not being able
to allocate from foreign nodes. That should not mean that I generally
vote against this to get added. If it works reliably why not add a work
around until the more complicated stuff works.

One question: You also want to automatically add the CPUs, once a CPU hotplug
event got fired, right?
The fact that the memory hotplug driver adds the memory immediately once notified,
does not ensure that the HW/BIOS fires this event first.
Theoretically you need a logic to not add CPUs to memoryless nodes, poll/wait
until memory got added, etc.

   Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Thomas Renninger]

On Thursday 11 March 2010 09:32:09 chen gong wrote:
> On 2010-3-11 16:07, ykzhao wrote:
... 
> BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some
> Novell guy is working on it.
I also know this guy :)

These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}"
SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"

But this should be the same as you suggest (at least the memory rule)
to do in the kernel:
automatically online the memory, once hotadded.

I would not add any udev rules before this does not work
reliably and currently it is totally broken, mainly because:
  - not being able to alloc memory on foreign nodes (at least with slab)
  - C-state, throttling and cpufreq set up is done without valid
    cpu_data(new_cpu) resulting in wrong C-state (and other) info

   Thomas

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

Thomas Renninger wrote:
> On Thursday 11 March 2010 01:55:15 ykzhao wrote:
>   
>> On Wed, 2010-03-10 at 21:28 +0800, Prarit Bhargava wrote:
>>     
>>>> Why do we need to see whether the memory is onlined before bringing cpu
>>>> to online state? It seems that there is no dependency between cpu online
>>>> and memory online.
>>>>
>>>>   
>>>>         
>>> Yakui,
>>>
>>>       
>> Thanks for the explanation.
>>
>>     
>>> Here's a deeper look into the issue.  New Intel processors have an 
>>> on-die memory controller and this means that as the socket comes and 
>>> goes, so does the memory "behind" the socket.
>>>       
>> Yes. The nehalem processor has the integrated memory controller. But it
>> is not required that the hot-added memory should be onlined before
>> bringing up CPU.
>>     I do the following memory-hotplug test on one Machine.
>>     a. Before hot plugging memory, four CPUs socket are installed and
>> all the logical CPU are brought up. (Only one node has the memory)
>>     b. The memory is hot-plugged and then the memory is onlined so that
>> it can be accessed by the system.
>>
>> In the above testing case the CPU is brought up before onlining the
>> hot-added memory. And the test shows that it can work well.
>>
>>     
>>> ie) with new processors it is possible that an entire node which 
>>> consists of memory and cpus comes and goes with the socket enable and 
>>> disable.
>>>
>>> The cpu bringup code does local node allocations for the cpu.  If the 
>>> memory connected to the node (which is "behind" the socket) isn't 
>>> online, then these allocations fail, and then the cpu bringup fails.
>>>       
>> If the CPU can't allocate the memory from its own node, it can turn to
>> other node and see whether the memory can be allocated. And this depends
>> on the NUMA allocation policy.
>>     
> Yes and this is broken and needs fixing.
> Yakui, I expect you miss this patch and wrongly online the cpus to existing
> nodes, therefore you do not run into "out of memory" conditions:
> 0271f91003d3703675be13b8865618359a6caa1f
>   

FWIW, I'm working on a 2.6.32 based tree, but I have that patch in (as 
well as several others).  I'm also running the latest upstream (tip as 
of this morning).  The issues I see in my 2.6.32 based tree are the same 
AFAICT that I see upstream: a cpu comes online and attempts to make a 
per_node allocation which fails, so the cpu bringup fails.

> I know for sure that slab is broken.
>   

Yes, but I believe Andi Kleen has added some patches that resolve (at 
least some of) the issues.  I've been using slab (and occasionally 
testing slub).

> slub behaves different, but I am not sure whether this is due to wrong CPU
> hotadd code (processor_core.c is also broken and you get wrong C-state info
> from BIOS tables on hotadded CPUs)
>
> Prarit: Can you retest with slub and processor.max_cstate=1, this could/should
> work.
>
>   

Two tests:

1.  WITHOUT my auto online patch, the cpus fail to come into service 
because of a per_node allocation failure.
2.  WITH my auto online patch, the cpus come into service

... I have NOT done any sort of testing to see if the cpus are really 
live ;)

> AFAIK vmware injects memory in the same way into clients, so you may have
> different behavior of virtualized Linux clients.
>
>   

Are you referring the vmware ballooning driver (or whatever they call 
it).  IIRC (and I'm not saying I do ;) ), vmware adds memory and 
automatically onlines it in a guest.  I'm not sure how that's done -- it 
could be via udev.

I'll see if anyone here knows.

> One question: You also want to automatically add the CPUs, once a CPU hotplug
> event got fired, right?
>   

Yes,  That's correct.

> The fact that the memory hotplug driver adds the memory immediately once notified,
> does not ensure that the HW/BIOS fires this event first.
>   

Is that right?  FWIW I always see this sequence of events:

ACPI memory added
ACPI cpus added

I never see them out-of-order.  OTOH, I'm only testing on Intel's latest 
platform so maybe there are some older systems that don't do this in 
that order. 

> Theoretically you need a logic to not add CPUs to memoryless nodes, poll/wait
> until memory got added, etc.
>   
Theoretically yes -- but are there any systems that generate cpu add 
events before memory add events?

Thanks for the input Thomas :)

P.
>    Thomas
>   
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

Thomas Renninger wrote:
> On Thursday 11 March 2010 09:32:09 chen gong wrote:
>   
>> On 2010-3-11 16:07, ykzhao wrote:
>>     
> ... 
>   
>> BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some
>> Novell guy is working on it.
>>     
> I also know this guy :)
>
> These would be the udev rules to automatically add memory/cpus:
> SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}"
> SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"
>
> But this should be the same as you suggest (at least the memory rule)
> to do in the kernel:
> automatically online the memory, once hotadded.
>
> I would not add any udev rules before this does not work
> reliably and currently it is totally broken, mainly because:
>   - not being able to alloc memory on foreign nodes (at least with slab)
>   - C-state, throttling and cpufreq set up is done without valid
>     cpu_data(new_cpu) resulting in wrong C-state (and other) info
>
>   

Thomas, forgive my ignorance of udev rules ... but can one udev rule 
block another?  ... That's not the best way to explain it ... here's an 
example:

CPU hot add on Nehalem-EX.  Memory is brought online.  Memory udev 
events are generated.  CPU udev events are generated.

Can the memory udev events block the cpu udev events?  ie) can I be 
assured that memory will come online before the cpus?

P.

>    Thomas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Thomas Renninger]

On Wednesday 17 March 2010 07:47:55 pm Prarit Bhargava wrote:
> Thomas Renninger wrote:
> > On Thursday 11 March 2010 09:32:09 chen gong wrote:
> >> On 2010-3-11 16:07, ykzhao wrote:
> >
> > ...
> >
> >> BTW, how about using UDEV rules to do this operation. It looks more
> >> smooth. I know some Novell guy is working on it.
> >
> > I also know this guy :)
> >
> > These would be the udev rules to automatically add memory/cpus:
> > SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0",
> > ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}"
> > SUBSYSTEM=="memory", ACTION=="add", TEST=="state",
> > ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining
> > memory: $env{DEVPATH}"
> >
> > But this should be the same as you suggest (at least the memory rule)
> > to do in the kernel:
> > automatically online the memory, once hotadded.
> >
> > I would not add any udev rules before this does not work
> > reliably and currently it is totally broken, mainly because:
> >   - not being able to alloc memory on foreign nodes (at least with slab)
> >   - C-state, throttling and cpufreq set up is done without valid
> >     cpu_data(new_cpu) resulting in wrong C-state (and other) info
>
> Thomas, forgive my ignorance of udev rules ... but can one udev rule
> block another?  ... That's not the best way to explain it ... here's an
> example:
>
> CPU hot add on Nehalem-EX.  Memory is brought online.  Memory udev
> events are generated.  CPU udev events are generated.
>
> Can the memory udev events block the cpu udev events?  ie) can I be
> assured that memory will come online before the cpus?

Blocking does not work.
But I have an idea, whatabout:
  - CPU add rule which only adds a CPU if the corresponding Numa node already
    has onlined memory
  - Memory add rule which adds the memory and also onlines CPUs if the Numa
    node still has offlined CPUs

This has the side effect that you online a CPU which may have been offlined on
purpose if you hotadd memory on the same node..., a rather uncommon case.

A quick try showed that it would be convenient to add the Numa node to the 
uevent which is a bit tricky...
Ok, I got this working, but messed up the cpu hotplugging with my patches:
...
Unable to map lapic 32 to logical cpu number
...
This already worked...

I wanted to post something today, but I couldn't made it.
Still I more or less could prove that above works and I hopefully can show
some results on Monday.

Comments?

   Thomas

BTW: I was wrong with slub. It also has bad side effects. What is strange is 
that with deep C-states one CPU (not the onlined) gets stuck (softlock up), 
but with processor.max_cstate=1 it does not. It reproducable hangs (loops?) 
in mm code, looks like it tries to alloc foreign memory. What I do not 
understand is why this happens with C3, but not with C1, it shouldn't have 
anything to do with memory setup. Anyway, I go back to slab again, lets 
ensure somehow that memory hot add comes first.

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

>
> Blocking does not work.
> But I have an idea, whatabout:
>   - CPU add rule which only adds a CPU if the corresponding Numa node already
>     has onlined memory
>   - Memory add rule which adds the memory and also onlines CPUs if the Numa
>     node still has offlined CPUs
>
> This has the side effect that you online a CPU which may have been offlined on
> purpose if you hotadd memory on the same node..., a rather uncommon case.
>
>   
That actually might be more of a problem than you think.  It's not 
atypical that in the evening a system's components are offlined in order 
to save power.  Maintenance also is scheduled for downtime so when 
memory is added to the system we may bring cpus into service 
erroneously.  I think the "end-user" may not be too happy with this result.

[OTOH, a little documentation could fix that issue]

> A quick try showed that it would be convenient to add the Numa node to the 
> uevent which is a bit tricky...
> Ok, I got this working, but messed up the cpu hotplugging with my patches:
> ...
> Unable to map lapic 32 to logical cpu number
> ...
> This already worked...
>
> I wanted to post something today, but I couldn't made it.
> Still I more or less could prove that above works and I hopefully can show
> some results on Monday.
>
>   

If it works then I'm all for it :)

> Comments?
>
>    Thomas
>
> BTW: I was wrong with slub. It also has bad side effects. What is strange is 
>   

FWIW, I'm testing with slab.

P.

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Thomas Renninger]

On Friday 19 March 2010 06:23:33 pm Prarit Bhargava wrote:
> > Blocking does not work.
> > But I have an idea, whatabout:
> >   - CPU add rule which only adds a CPU if the corresponding Numa node
> > already has onlined memory
> >   - Memory add rule which adds the memory and also onlines CPUs if the
> > Numa node still has offlined CPUs
> >
> > This has the side effect that you online a CPU which may have been
> > offlined on purpose if you hotadd memory on the same node..., a rather
> > uncommon case.
>
> That actually might be more of a problem than you think.  It's not
> atypical that in the evening a system's components are offlined in order
> to save power.
This shouldn't be a problem currently, because offlined CPUs drain more
power than online ones:
http://bugzilla.kernel.org/show_bug.cgi?id=5471

> Maintenance also is scheduled for downtime
I don't understand this part.
> so when  
> memory is added to the system we may bring cpus into service
> erroneously.  I think the "end-user" may not be too happy with this result.
...
> > I wanted to post something today, but I couldn't made it.
> > Still I more or less could prove that above works and I hopefully can
> > show some results on Monday.
>
> If it works then I'm all for it :)
Great. I'll hopefully post some patches in a new thread soon and try to 
collect all interested people in CC.

     Thomas

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Thomas Renninger]

On Wednesday 17 March 2010 19:47:55 Prarit Bhargava wrote:
> 
> Thomas Renninger wrote:
> > On Thursday 11 March 2010 09:32:09 chen gong wrote:
> >   
> >> On 2010-3-11 16:07, ykzhao wrote:
> >>     
> > ... 
> >   
> >> BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some
> >> Novell guy is working on it.
> >>     
> > I also know this guy :)
> >
> > These would be the udev rules to automatically add memory/cpus:
> > SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}"
> > SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"
> >
> > But this should be the same as you suggest (at least the memory rule)
> > to do in the kernel:
> > automatically online the memory, once hotadded.
> >
> > I would not add any udev rules before this does not work
> > reliably and currently it is totally broken, mainly because:
> >   - not being able to alloc memory on foreign nodes (at least with slab)
> >   - C-state, throttling and cpufreq set up is done without valid
> >     cpu_data(new_cpu) resulting in wrong C-state (and other) info
> >
> >   
> 
> Thomas, forgive my ignorance of udev rules ... but can one udev rule 
> block another?  ... That's not the best way to explain it ... here's an 
> example:
> 
> CPU hot add on Nehalem-EX.  Memory is brought online.  Memory udev 
> events are generated.  CPU udev events are generated.
> 
> Can the memory udev events block the cpu udev events?  ie) can I be 
> assured that memory will come online before the cpus?
I give up, there are too many open issues.
Onlining cpus and memory in parallel currently is not a good idea.

If the cpu event comes first, you run into other issues. I accidently had
the acpi_memhotplug driver not loaded, in this case a new node was not
created at all and the cpus ended up on the wrong node.

Some comments on your patch:
Your patch misses EXPORT_SYMBOL(_GPL?)(set_memory_state); (I wonder why
drivers/base/memory.c does not export symbols under gpl...) and
EXPORT_SYMBOL_GPL(online_pages);

You also need to set each memory block MEM_ONLINE somehow,
while all memory was active, only one memory block showed "online" here:
/sys/devices/system/node/node3/memory*/state
which has side-effects.

Would be great if you add me to CC if you repost or on related topics.

Thanks,

     Thomas

Re: [RFC PATCH]: ACPI: Automatically online hot-added memory

[Prarit Bhargava]

Thomas Renninger wrote:
> On Wednesday 17 March 2010 19:47:55 Prarit Bhargava wrote:
>   
>> Thomas Renninger wrote:
>>     
>>> On Thursday 11 March 2010 09:32:09 chen gong wrote:
>>>   
>>>       
>>>> On 2010-3-11 16:07, ykzhao wrote:
>>>>     
>>>>         
>>> ... 
>>>   
>>>       
>>>> BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some
>>>> Novell guy is working on it.
>>>>     
>>>>         
>>> I also know this guy :)
>>>
>>> These would be the udev rules to automatically add memory/cpus:
>>> SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}"
>>> SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"
>>>
>>> But this should be the same as you suggest (at least the memory rule)
>>> to do in the kernel:
>>> automatically online the memory, once hotadded.
>>>
>>> I would not add any udev rules before this does not work
>>> reliably and currently it is totally broken, mainly because:
>>>   - not being able to alloc memory on foreign nodes (at least with slab)
>>>   - C-state, throttling and cpufreq set up is done without valid
>>>     cpu_data(new_cpu) resulting in wrong C-state (and other) info
>>>
>>>   
>>>       
>> Thomas, forgive my ignorance of udev rules ... but can one udev rule 
>> block another?  ... That's not the best way to explain it ... here's an 
>> example:
>>
>> CPU hot add on Nehalem-EX.  Memory is brought online.  Memory udev 
>> events are generated.  CPU udev events are generated.
>>
>> Can the memory udev events block the cpu udev events?  ie) can I be 
>> assured that memory will come online before the cpus?
>>     
> I give up, there are too many open issues.
> Onlining cpus and memory in parallel currently is not a good idea.
>
>   

Thomas,

> If the cpu event comes first, you run into other issues. I accidently had
> the acpi_memhotplug driver not loaded, in this case a new node was not
> created at all and the cpus ended up on the wrong node.
>
>   

Right :( 
> Some comments on your patch:
> Your patch misses EXPORT_SYMBOL(_GPL?)(set_memory_state); (I wonder why
> drivers/base/memory.c does not export symbols under gpl...) and
> EXPORT_SYMBOL_GPL(online_pages);
>
>   

Okay, I'll fix that up. 

> You also need to set each memory block MEM_ONLINE somehow,
> while all memory was active, only one memory block showed "online" here:
> /sys/devices/system/node/node3/memory*/state
> which has side-effects.
>
>   

Huh -- I wonder if that has to do with they way I'm calling 
online_pages.  I wonder if I have to make that clal "deeper" in the call 
path and call it on EACH segment that I'm bringing into service...

> Would be great if you add me to CC if you repost or on related topics.
>
>   
Absolutely no problem :)  I didn't mean to leave you out of the 
conversation :)

P.

> Thanks,
>
>      Thomas
>