RE: [Lhms-devel] RE: memory hotremove prototype, take 3

RE: [Lhms-devel] RE: memory hotremove prototype, take 3

[Perez-Gonzalez, Inaky]

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Yasunori Goto
> Sent: Wednesday, December 03, 2003 1:23 PM
> To: Perez-Gonzalez, Inaky
> Cc: Pavel Machek; linux-kernel@vger.kernel.org; Luck, Tony; IWAMOTO Toshihiro; Hirokazu Takahashi; Linux Hotplug Memory Support
> Subject: Re: [Lhms-devel] RE: memory hotremove prototype, take 3
> 
> 
> > > IMHO, To hot-remove memory, memory attribute should be divided
> > > into Hotpluggable and no-Hotpluggable, and each attribute memory
> > > should be allocated each unit(ex. node).
> >
> > Why? I still don't get that -- we should be able to use the virtual
> > addressing mechanism of any CPU to swap under the rug any virtual
> > address without needing to do anything more than allocate a page frame
> > for the new physical location
> 
> My understanding is that the kernel and device drivers sometimes
> assume physical memory address is not changed.
> For example, IA32 kernel often uses __PAGE_OFFSET.
> I suppose that there are many case which the kernel and device drivers
> assume physical address is not changed like this.
> Even if we use Mr. Iwamoto's method use, some memories will remain.

Grrrr ... my concern is that Murphy's Law says that we'll need to hotplug
the memory that we cannot in most of the cases (pray not). I guess I will
need to research some more to think how to do it.

I still think we could use the CPU's virtualization mechanism--of course,
and as you and Tony Luck mention, we'd had to track down and modify the
parts that assume physical memory et al. That they use large pages or
not doesn't seem to be such a big problem (other than allocating a bigger
chunk for transfer) and that performance wise, it'd take longer to 
do _that_ part, but still I prefer it to taking the machine down.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

Re: [Lhms-devel] RE: memory hotremove prototype, take 3

[Pavel Machek]

Hi!

> > > > IMHO, To hot-remove memory, memory attribute should be divided
> > > > into Hotpluggable and no-Hotpluggable, and each attribute memory
> > > > should be allocated each unit(ex. node).
> > >
> > > Why? I still don't get that -- we should be able to use the virtual
> > > addressing mechanism of any CPU to swap under the rug any virtual
> > > address without needing to do anything more than allocate a page frame
> > > for the new physical location
> > 
> > My understanding is that the kernel and device drivers sometimes
> > assume physical memory address is not changed.
> > For example, IA32 kernel often uses __PAGE_OFFSET.
> > I suppose that there are many case which the kernel and device drivers
> > assume physical address is not changed like this.
> > Even if we use Mr. Iwamoto's method use, some memories will remain.
> 
> Grrrr ... my concern is that Murphy's Law says that we'll need to hotplug
> the memory that we cannot in most of the cases (pray not). I guess I will
> need to research some more to think how to do it.
> 
> I still think we could use the CPU's virtualization mechanism--of course,
> and as you and Tony Luck mention, we'd had to track down and modify the
> parts that assume physical memory et al. That they use large pages
> or

...which means basically auditing whole kernel, and rewriting half of
drivers. Good luck with _that_.
								Pavel
-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

RE: [Lhms-devel] RE: memory hotremove prototype, take 3

[Luck, Tony]

> > Bingo...just the perfect excuse I need to give to my manager to keep
> > a low profile while tinkering around for a long time :)
> > 
> > Okay, so I will play a wee bit more the devil's advocate as an 
> > exercise of futility, if you don't mind. Just trying to compile a 
> > (possibly incomplete) quick list of what would be needed, can you 
> > guys help me? you know way more than I do:
> > 
> > 1) the core kernel needs to be independent of physical 
> memory position
> > 1.1) same with drivers/subsystems
> > 1.2) filesystems
> > [it cannot be really incomplete because I have added all the code
> > :/]
> 
> ...and you have bad problem at any place where physical address is
> passed to the hardware. UHCI is going to be "interesting".

Which is why this hot remove project is side-stepping all
those hard problems, and just dealling with the "easy" cases
of only allocating user memory to removeable physical memory.
This is useful for large systems made up of removeable nodes
(either to physically remove a node, or to logically re-assign
a node to a different "partition" in the machine).

Anyone who wants more than this is welcome to virtualize the
base kernel, filesystems, driver code etc. to allow hot remove
of the remainder of memory not dealt with by this patch :-)

-Tony

RE: [Lhms-devel] RE: memory hotremove prototype, take 3

[Perez-Gonzalez, Inaky]

> From: Pavel Machek [mailto:pavel@suse.cz]


> > 1) the core kernel needs to be independent of physical memory position
> > 1.1) same with drivers/subsystems
> > 1.2) filesystems
> > [it cannot be really incomplete because I have added all the code
> > :/]
> 
> ...and you have bad problem at any place where physical address is
> passed to the hardware. UHCI is going to be "interesting".

That one falls under the category of "every device driver that talks
in physical with its device" needs a callback to reallocate buffers
or cancel and reissue transactions.

Still, when I started to hate UHCI's guts in 1997 I had a reason...it
still holds true.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

Re: [Lhms-devel] RE: memory hotremove prototype, take 3

[Pavel Machek]

Hi!

> > > I still think we could use the CPU's virtualization mechanism--of course,
> > > and as you and Tony Luck mention, we'd had to track down and modify the
> > > parts that assume physical memory et al. That they use large pages
> > > or
> > 
> > ...which means basically auditing whole kernel, and rewriting half of
> > drivers. Good luck with _that_.
> 
> Bingo...just the perfect excuse I need to give to my manager to keep
> a low profile while tinkering around for a long time :)
> 
> Okay, so I will play a wee bit more the devil's advocate as an 
> exercise of futility, if you don't mind. Just trying to compile a 
> (possibly incomplete) quick list of what would be needed, can you 
> guys help me? you know way more than I do:
> 
> 1) the core kernel needs to be independent of physical memory position
> 1.1) same with drivers/subsystems
> 1.2) filesystems
> [it cannot be really incomplete because I have added all the code
> :/]

...and you have bad problem at any place where physical address is
passed to the hardware. UHCI is going to be "interesting".
								Pavel
-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

RE: [Lhms-devel] RE: memory hotremove prototype, take 3

[Perez-Gonzalez, Inaky]

> From: Pavel Machek [mailto:pavel@suse.cz]


> > I still think we could use the CPU's virtualization mechanism--of course,
> > and as you and Tony Luck mention, we'd had to track down and modify the
> > parts that assume physical memory et al. That they use large pages
> > or
> 
> ...which means basically auditing whole kernel, and rewriting half of
> drivers. Good luck with _that_.

Bingo...just the perfect excuse I need to give to my manager to keep
a low profile while tinkering around for a long time :)

Okay, so I will play a wee bit more the devil's advocate as an 
exercise of futility, if you don't mind. Just trying to compile a 
(possibly incomplete) quick list of what would be needed, can you 
guys help me? you know way more than I do:

1) the core kernel needs to be independent of physical memory position
1.1) same with drivers/subsystems
1.2) filesystems
[it cannot be really incomplete because I have added all the code :/]

Oh well, forget it, that's more than enough. Another project for the
stack of interesting things to work on.

Thanks to all

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

Re: [Lhms-devel] RE: memory hotremove prototype, take 3

[Yasunori Goto]

> > IMHO, To hot-remove memory, memory attribute should be divided
> > into Hotpluggable and no-Hotpluggable, and each attribute memory
> > should be allocated each unit(ex. node).
> 
> Why? I still don't get that -- we should be able to use the virtual
> addressing mechanism of any CPU to swap under the rug any virtual
> address without needing to do anything more than allocate a page frame
> for the new physical location 

My understanding is that the kernel and device drivers sometimes
assume physical memory address is not changed.
For example, IA32 kernel often uses __PAGE_OFFSET. 
I suppose that there are many case which the kernel and device drivers
assume physical address is not changed like this.
Even if we use Mr. Iwamoto's method use, some memories will remain.

But, if remaining memories will be on only one (or a few) node,
other nodes will be able to be hot-removed.
I think that dividing attribute hotpluggable or not is realistic way.


> (I am ignoring here devices that are 
> directly accessing physical memory--a callback in the device model could
> be added to require them to reallocate their buffers).

Mr. Iwamoto and Mr. Takahashi told me that some memories which is used
by NFS are especially difficult to be hotplugged....

Thanks.

-- 
Yasunori Goto <ygoto at fsw.fujitsu.com>