[PATCH 0/6] Add memory add support to Xen

[PATCH 0/6] Add memory add support to Xen

[Jiang, Yunhong]

Hi, all, this patchset add support so that we can add memory dynamically to the system. This can be used for memory hot add.

The change includes 
a) changes to boot allocator, frametable setup, m2p table setup, so that they will be called not-only when system boot up, but also when hot add happens
b) Changes to page fault handler, to propgate changes to frametable/m2p table to all guest.

Please review  it and gives feedback.

Thanks
Yunhong Jiang

BTW, we also want to have some discussion on how to support memory hot add. The main challenges comes from ACPI. 
When memory hot add happen, it will trigger an ACPI event. The ACPI driver will get memory device's resource information, including start/end address, NUMA informatin (proximity domain information) etc and in the end add the memory to OSPM.
In Xen, the ACPI event will be handled firstly by Dom0 , but dom0 have no idea of physical memory. The real memory adding should happen in Xen HV.

There are several option to support it. 
a) Move the ACPI event to Xen HV, that is sure to be a big changes.
b) Add some hook in Dom0, so that instead of add the new memory to dom0, it will use hypercall to notify Xen HV.
c) Change current dom0 implementation. When dom0 get the ACPI event and get the memory device's resource, it will not directly add the memry to system, instead, it will notify user space with uevent. The uevent script will ask kernel to add the new memory to system. In Xen environment, the uevent script will notify Xen HV. We are still investigating on this option.

We are in early stage . any suggestion or new idea will be helpful to us and welcome.

Of course, no matter how to support memory hotadd, this patchset is not impacted . For example, following usage model will require this function: an "mem=" optoin .is used to limit xen's memory, and later, we want to bring all memory to xen.

Re: [PATCH 0/6] Add memory add support to Xen

[Keir Fraser]

On 28/06/2009 10:26, "Jiang, Yunhong" <yunhong.jiang@intel.com> wrote:

> BTW, we also want to have some discussion on how to support memory hot add.
> The main challenges comes from ACPI.
> When memory hot add happen, it will trigger an ACPI event. The ACPI driver
> will get memory device's resource information, including start/end address,
> NUMA informatin (proximity domain information) etc and in the end add the
> memory to OSPM.
> In Xen, the ACPI event will be handled firstly by Dom0 , but dom0 have no idea
> of physical memory. The real memory adding should happen in Xen HV.
> 
> There are several option to support it.
> a) Move the ACPI event to Xen HV, that is sure to be a big changes.

There's been some discussion of making Xen the OSPM instead of dom0 (and
hence giving Xen an AML interpreter, etc). Could be an interesting thing to
think about. I'm not sure how well it would work since there will still be
aspects of platform management that dom0 is in control of and would
logically be the OSPM for.

 -- Keir

RE: [PATCH 0/6] Add memory add support to Xen

[Jiang, Yunhong]

[-- Attachment #1: Type: text/plain, Size: 2108 bytes --]

 

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com 
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
>Sent: 2009年6月28日 17:48
>To: Jiang, Yunhong; Jan Beulich; Tim Deegan; 
>xen-devel@lists.xensource.com
>Subject: [Xen-devel] Re: [PATCH 0/6] Add memory add support to Xen
>
>On 28/06/2009 10:26, "Jiang, Yunhong" <yunhong.jiang@intel.com> wrote:
>
>> BTW, we also want to have some discussion on how to support 
>memory hot add.
>> The main challenges comes from ACPI.
>> When memory hot add happen, it will trigger an ACPI event. 
>The ACPI driver
>> will get memory device's resource information, including 
>start/end address,
>> NUMA informatin (proximity domain information) etc and in 
>the end add the
>> memory to OSPM.
>> In Xen, the ACPI event will be handled firstly by Dom0 , but 
>dom0 have no idea
>> of physical memory. The real memory adding should happen in Xen HV.
>> 
>> There are several option to support it.
>> a) Move the ACPI event to Xen HV, that is sure to be a big changes.
>
>There's been some discussion of making Xen the OSPM instead of 
>dom0 (and
>hence giving Xen an AML interpreter, etc). Could be an 
>interesting thing to
>think about. I'm not sure how well it would work since there 
>will still be
>aspects of platform management that dom0 is in control of and would
>logically be the OSPM for.

Yes, the ACPI is not virtualization friendly, at least not Xen arch friendly. And it is difficult for both PM and hotplug (both CPU and memory hotplug). Or we can split dom0 into two domain, one is as a generic driver domain, and one is for platform domain. The platform domain will act for ACPI, IOAPIC parse etc, while the driver domain will own all PCI devices and act as pciback (I didn't think in deep on it still).

Maybe option 3 will be cleaner, we are still investigate it.

Thanks
Yunhong Jiang


>
> -- Keir
>
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Re: [PATCH 0/6] Add memory add support to Xen

[Dan Magenheimer]

I seem to recall that an AML interpreter is larger than
an entire hypervisor.  This was a few years ago so may
not still be valid, but if it is anywhere near true,
it seems like a potential issue. :-)

> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Sunday, June 28, 2009 3:48 AM
> To: Jiang, Yunhong; Jan Beulich; Tim Deegan;
> xen-devel@lists.xensource.com
> Subject: [Xen-devel] Re: [PATCH 0/6] Add memory add support to Xen
> 
> 
> On 28/06/2009 10:26, "Jiang, Yunhong" <yunhong.jiang@intel.com> wrote:
> 
> > BTW, we also want to have some discussion on how to support 
> memory hot add.
> > The main challenges comes from ACPI.
> > When memory hot add happen, it will trigger an ACPI event. 
> The ACPI driver
> > will get memory device's resource information, including 
> start/end address,
> > NUMA informatin (proximity domain information) etc and in 
> the end add the
> > memory to OSPM.
> > In Xen, the ACPI event will be handled firstly by Dom0 , 
> but dom0 have no idea
> > of physical memory. The real memory adding should happen in Xen HV.
> > 
> > There are several option to support it.
> > a) Move the ACPI event to Xen HV, that is sure to be a big changes.
> 
> There's been some discussion of making Xen the OSPM instead 
> of dom0 (and
> hence giving Xen an AML interpreter, etc). Could be an 
> interesting thing to
> think about. I'm not sure how well it would work since there 
> will still be
> aspects of platform management that dom0 is in control of and would
> logically be the OSPM for.
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: [PATCH 0/6] Add memory add support to Xen

[Nakajima, Jun]

Jiang, Yunhong wrote on Sun, 28 Jun 2009 at 02:26:51:

> 
> There are several option to support it.
> a) Move the ACPI event to Xen HV, that is sure to be a big changes.
> b) Add some hook in Dom0, so that instead of add the new memory to dom0,
> it will use hypercall to notify Xen HV.
> c) Change current dom0 implementation. When dom0 get the ACPI event and
> get the memory device's resource, it will not directly add the memry to
> system, instead, it will notify user space with uevent. The uevent
> script will ask kernel to add the new memory to system. In Xen
> environment, the uevent script will notify Xen HV. We are still
> investigating on this option.
> 
> We are in early stage . any suggestion or new idea will be helpful to
> us and welcome.
> 

I think the option b is the practical choice because the kernel typically has a well-defined routine for that, and the maintenance cost of the diff for dom0 should be very low. The option c may not work for other OS like Solaris...

Jun
___
Intel Open Source Technology Center

RE: [PATCH 0/6] Add memory add support to Xen

[Jiang, Yunhong]

Nakajima, Jun wrote:
> Jiang, Yunhong wrote on Sun, 28 Jun 2009 at 02:26:51:
> 
>> 
>> There are several option to support it.
>> a) Move the ACPI event to Xen HV, that is sure to be a big changes.
>> b) Add some hook in Dom0, so that instead of add the new memory to
>> dom0, it will use hypercall to notify Xen HV.
>> c) Change current dom0 implementation. When dom0 get the ACPI event
>> and get the memory device's resource, it will not directly add the
>> memry to system, instead, it will notify user space with uevent. The
>> uevent script will ask kernel to add the new memory to system. In Xen
>> environment, the uevent script will notify Xen HV. We are still
>> investigating on this option. 
>> 
>> We are in early stage . any suggestion or new idea will be helpful
>> to us and welcome. 
>> 
> 
> I think the option b is the practical choice because the
> kernel typically has a well-defined routine for that, and the
> maintenance cost of the diff for dom0 should be very low. The
> option c may not work for other OS like Solaris...

Yes, the diff will be low. We will work on this way.

--jyh

> 
> Jun
> ___
> Intel Open Source Technology Center