RE: 2.7 thoughts

RE: 2.7 thoughts

[Frederick, Fabian]

Gabor, 
	Here's a first "2.7 thoughts" listing with some questions...

2.7 features

*	slab allocation quota
*	ntfs full support
*	kernel web server (Interfaced to Roman config tool)
*	ipc to sysfs
*	complete user quota centralization 
*	Add _responsibilities_ for virtual process tree and possible
relation in oops cases 
*	Does the whole proc vm stuff root/box relevant ?I don't think
so....Hence, those proc entries deserve security relevant attributes 
*	Devices should be limited as well against bad usage(floppy defect),
viral activity(netcard rush)... 
*	All this guides me to a more global conclusion in which all that
stuff should be kobject registration relevant 
*	Meanwhile, we don't have a kobject <-> security bridge :( 

* bind mount support for all general mount options (nodev,ro,noexec etc)
  with SECURE implementation with any (maybe even future) filesystems?

* union mount (possible with option to declare on what fs a new file
  should be created: on fixed ones, random algorithm, on fs with the
  largest free space available etc ...)
<LVM ?

* guaranteed i/o bandwidth allocation?
* netfilter's ability to do tricks which OpenBSD can do now with its
  packet filter

* ENBD support in official kernel with enterprise-class 'through the
  network' volume management
< NFS ?

* more and more tunable kernel parameters to be able to have some user
  space program which can 'tune' the system for the current load,usage,etc
  of the server ("selftune")
<What parameters would you add in procfs ?

* more configuration options to be able to use Linux at the low end as well
  (current kernels are too complex, too huge and sometimes contains too
  many unwanted features for a simple system, though for most times it is
  enough but can be even better)

* Virtual machine support
<Maybe more 3.0 relevant ?

Re: 2.7 thoughts

[Jens Axboe]

On Thu, Oct 09 2003, Frederick, Fabian wrote:
> Gabor, 
> 	Here's a first "2.7 thoughts" listing with some questions...

Looks more like a wish list.

-- 
Jens Axboe

Re: 2.7 thoughts

[Gábor Lénárt]

On Thu, Oct 09, 2003 at 03:17:05PM +0200, Frederick, Fabian wrote:
> * bind mount support for all general mount options (nodev,ro,noexec etc)
>   with SECURE implementation with any (maybe even future) filesystems?
> 
> * union mount (possible with option to declare on what fs a new file
>   should be created: on fixed ones, random algorithm, on fs with the
>   largest free space available etc ...)
> <LVM ?

Dunno about LVM, I've not plaied a lot with it ...

> 
> * guaranteed i/o bandwidth allocation?
> * netfilter's ability to do tricks which OpenBSD can do now with its
>   packet filter
> 
> * ENBD support in official kernel with enterprise-class 'through the
>   network' volume management
> < NFS ?

NFS is a filesystem. However I meant filesystem independent access, ie
raw device access through the network and creating any filesystem on it.
NBD can do this, ENBD is even better, but I think it can be further developed
sure, and at least AFAIK ENBD is not inside the official kernel. Or maybe
it is? :)

> * more and more tunable kernel parameters to be able to have some user
>   space program which can 'tune' the system for the current load,usage,etc
>   of the server ("selftune")
> <What parameters would you add in procfs ?

In general :) Maybe this is not the question of the kernel nowdays, but that
userspace program. I mean now we can hear news about chaning scheduler
behaviour on-the-fly and other interesing stuffs. I meant these class of
parameters for example. Like the case of the OOM killer and others I know
that implementing a quite complex algorithm inside the kernel is not a good
point, probably. That's why I thought that this kind of on-line "selftuning"
can be possible with a user space process, the only need is to provide
enough statistical parameters and tuning possibilites by the kernel, so the
tuner can do its job. However I don't know that this theory is useful for
reality it was only a wild idea (but sounds good IMHO :)

> * Virtual machine support
> <Maybe more 3.0 relevant ?

:)

-- 
- Gábor (larta'H)

Re: 2.7 thoughts

[Bartlomiej Zolnierkiewicz]

Is this your TODO list?
You are very ambitious person.

;-)

--bartlomiej

On Thursday 09 of October 2003 15:17, Frederick, Fabian wrote:
> Gabor,
> 	Here's a first "2.7 thoughts" listing with some questions...
>
> 2.7 features
>
> *	slab allocation quota
> *	ntfs full support
> *	kernel web server (Interfaced to Roman config tool)
> *	ipc to sysfs
> *	complete user quota centralization
> *	Add _responsibilities_ for virtual process tree and possible
> relation in oops cases
> *	Does the whole proc vm stuff root/box relevant ?I don't think
> so....Hence, those proc entries deserve security relevant attributes
> *	Devices should be limited as well against bad usage(floppy defect),
> viral activity(netcard rush)...
> *	All this guides me to a more global conclusion in which all that
> stuff should be kobject registration relevant
> *	Meanwhile, we don't have a kobject <-> security bridge :(
>
> * bind mount support for all general mount options (nodev,ro,noexec etc)
>   with SECURE implementation with any (maybe even future) filesystems?
>
> * union mount (possible with option to declare on what fs a new file
>   should be created: on fixed ones, random algorithm, on fs with the
>   largest free space available etc ...)
> <LVM ?
>
> * guaranteed i/o bandwidth allocation?
> * netfilter's ability to do tricks which OpenBSD can do now with its
>   packet filter
>
> * ENBD support in official kernel with enterprise-class 'through the
>   network' volume management
> < NFS ?
>
> * more and more tunable kernel parameters to be able to have some user
>   space program which can 'tune' the system for the current load,usage,etc
>   of the server ("selftune")
> <What parameters would you add in procfs ?
>
> * more configuration options to be able to use Linux at the low end as well
>   (current kernels are too complex, too huge and sometimes contains too
>   many unwanted features for a simple system, though for most times it is
>   enough but can be even better)
>
> * Virtual machine support
> <Maybe more 3.0 relevant ?

Re: 2.7 thoughts

[Zwane Mwaikambo]

On Fri, 10 Oct 2003, William Lee Irwin III wrote:

> On Fri, Oct 10, 2003 at 07:47:23AM -0700, William Lee Irwin III wrote:
> >> You need at least enough warning to get out of critical sections (e.g.
> >> holding a spinlock) and dump registers out to memory. i.e. as long as it
> >> takes to schedule out whatever's currently running on the thing.
> 
> On Fri, Oct 10, 2003 at 11:03:20AM -0700, Tim Hockin wrote:
> > I've got a patch against RedHat's 2.4.x kernel with some version of O(1)
> > scheduler that migrates all processes off a CPU and makes it ineleigible to
> > run new tasks.  When the syscall returns, it is safe to remove the CPU.
> > Processes that are running or sleeping and can be migrated elsewhere are
> > migrated.  Running processes that have set_cpus_allowed to ONLY the
> > processor in question are marked TASK_UNRUNNABLE and moved off the runqueue.
> > Sleeping processes that have set_cpus_allowed to ONLY the processor in
> > question are left unmolested until they wake up, at which point they will be
> > marked UNRUNNABLE.
> > It was done to allow software to bring CPUs on/offline, but it should work
> > for this.  IRQs not done yet, either.
> 
> I think there is some generalized cpu hotplug stuff that's gone in that
> direction already, though I don't know any details. The bits about non-
> cooperative offlining were very interesting to hear, though.

The task migration is one of the easier bits, there is a hive of other 
problems which crop up in the various subsystems, the current patchset 
does very well however, Tim perhaps you'd like to help out? 

For interrupts we simple set interrupt affinity mask on the interrupt 
controller whilst the processor is still online, quiesce, 
local_irq_disable. The wait period is to make sure that interrupts which 
are already queued get handled before we completely disable interrupt 
processing, although it could do with some verification to make sure 
there aren't any lingering vectors.

Re: 2.7 thoughts

[Xose Vazquez Perez]

 Frederick, Fabian wrote:

> Some thoughts for 2.7.Someone has other ideas, comments ?

Here http://kernelnewbies.org/wiki/moin.cgi/KernelFuture there are
enough ideas for 3.0  :-)

2.7 thoughts

[Svetoslav Slavtchev]

my missed in 2.6 & must for 2.7:

finaly merge linuxconsole's ruby tree,
or with other words
real local multi user 
(
for now only usable under XFree,
but soon also with framebuffer console
)

svetljo

-- 
NEU FÜR ALLE - GMX MediaCenter - für Fotos, Musik, Dateien...
Fotoalbum, File Sharing, MMS, Multimedia-Gruß, GMX FotoService

Jetzt kostenlos anmelden unter http://www.gmx.net

+++ GMX - die erste Adresse für Mail, Message, More! +++

2.7 thoughts

[Frederick, Fabian]

Hi,

	Some thoughts for 2.7.Someone has other ideas, comments ?
	
*	slab allocation limit
*	complete user quota centralization 
*	Add _responsibilities_ for virtual process tree and possible
relation in oops cases 
*	Does the whole proc vm stuff root/box relevant ?I don't think
so....Hence, those proc entries deserve security relevant attributes 
*	Devices should be limited as well against bad usage(floppy defect),
viral activity(netcard rush)... 
*	All this guides me to a more global conclusion in which all that
stuff should be kobject registration relevant 
*	Meanwhile, we don't have a kobject <-> security bridge :( 

Regards,
Fabian

Re: 2.7 thoughts

[John Bradford]

> 	Some thoughts for 2.7.Someone has other ideas, comments ?

Better support for huge numbers of IDE disks.

John.

Re: 2.7 thoughts

[mario noioso]

JJohn Bradford wrote:

>>     Some thoughts for 2.7.Someone has other ideas, comments ?
>>   
>
>
> Better support for huge numbers of IDE disks.
>
>  
>
Is there somethings to do for better NTPL and Java VM support  in the 
kernel ?

Re: 2.7 thoughts

[Gábor Lénárt]

On Thu, Oct 09, 2003 at 10:08:00AM +0200, Frederick, Fabian wrote:
> Hi,
> 	Some thoughts for 2.7.Someone has other ideas, comments ?
[...] 	
> *	All this guides me to a more global conclusion in which all that
> stuff should be kobject registration relevant 
> *	Meanwhile, we don't have a kobject <-> security bridge :( 

well, maybe stupid ideas, but they're which supported on other unix like
system(s) or it would be very nice according to my experiences:

* bind mount support for all general mount options (nodev,ro,noexec etc)
  with SECURE implementation with any (maybe even future) filesystems?
* union mount (possible with option to declare on what fs a new file
  should be created: on fixed ones, random algorithm, on fs with the
  largest free space available etc ...)
* guaranteed i/o bandwidth allocation?
* netfilter's ability to do tricks which OpenBSD can do now with its
  packet filter
* ENBD support in official kernel with enterprise-class 'through the
  network' volume management
* more and more tunable kernel parameters to be able to have some user
  space program which can 'tune' the system for the current load,usage,etc
  of the server ("selftune")
* more configuration options to be able to use Linux at the low end as well
  (current kernels are too complex, too huge and sometimes contains too
  many unwanted features for a simple system, though for most times it is
  enough but can be even better)
* maybe some 'official in the kernel' general framework to implement
  virtual machines without the need to load third party kernel modeles
  from vmware, plex86 etc ...


-- 
- Gábor (larta'H)

Re: 2.7 thoughts

[Stephan von Krawczynski]

* hotplug CPU
* hotplug RAM

Regards,
Stephan

Re: 2.7 thoughts

[Stuart Longland]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stephan von Krawczynski wrote:
> * hotplug CPU
> * hotplug RAM

* hotplug motherboard & entire computer too I spose ;-)

Although sarcasm aside, a couple of ideas that have been bantered around 
on this list (and a few of my own ideas):

	- /proc interface alternative to modutils/module-init-tools.

		That is, to have a directory of virtual nodes in /proc
		to provide the functionality of insmod, rmmod, lsmod &
		modprobe would be great -- especially from the viewpoint
		of recue disk images, etc.

	- Software RAID 0+1 perhaps?

		A lot of hardware RAID cards support it, why not the
		kernel?  By RAID 0+1 I mean mirror-RAIDing two (or more) 		stripe-RAID 
arrays.  (Or can this be done already?)

	- Transparent Software-RAID for IDE RAID cards...
		This could be done by using the Software RAID
		functionality of the kernel, but making the RAID
		interface transparent, so you only see a /dev/md?
		device, rather than multiple /dev/?da* entries.

	- Support for the Casio DV-01 USB digital notetaker.

Just some thoughts...
And I spose, a few people have mentioned this already, what is planned 
for Linux 3.0?
- -- 
+-------------------------------------------------------------+
| Stuart Longland           stuartl at longlandclan.hopto.org |
| Brisbane Mesh Node: 719             http://stuartl.cjb.net/ |
| I haven't lost my mind - it's backed up on a tape somewhere |
| Griffith Student No:           Course: Bachelor/IT (Nathan) |
+-------------------------------------------------------------+
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE/hk+GIGJk7gLSDPcRArgVAJ0ZPJMRkMiFrGBBVVmL180fuZq2AACdFpCh
XjqIDsrAfvIYZgxipX3THdY=
=mRNi
-----END PGP SIGNATURE-----

Re: 2.7 thoughts

[William Lee Irwin III]

On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
Stephan von Krawczynski wrote:
>>* hotplug CPU
>>* hotplug RAM

Generally thought of as desirable, but hotplug RAM needs a *LOT* of
qualification.


On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
> * hotplug motherboard & entire computer too I spose ;-)

Um, this is worse than the above wrt. being too vague.


On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
> Although sarcasm aside, a couple of ideas that have been bantered around 
> on this list (and a few of my own ideas):
> 	- /proc interface alternative to modutils/module-init-tools.
> 		That is, to have a directory of virtual nodes in /proc
> 		to provide the functionality of insmod, rmmod, lsmod &
> 		modprobe would be great -- especially from the viewpoint
> 		of recue disk images, etc.

No way in Hell.


-- wli

Re: 2.7 thoughts

[YoshiyaETO]

> On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
> > * hotplug motherboard & entire computer too I spose ;-)
>
> Um, this is worse than the above wrt. being too vague.
----
"Hotplug node" is a better explanation, I think.
"Node" includes CPUs and/or Memory and/or some kind of IOs.
And "Node" should be flexibly configurable also.

----- Original Message ----- 
From: "William Lee Irwin III" <wli@holomorphy.com>
To: "Stuart Longland" <stuartl@longlandclan.hopto.org>
Cc: "Stephan von Krawczynski" <skraw@ithnet.com>; <lgb@lgb.hu>;
<Fabian.Frederick@prov-liege.be>; <linux-kernel@vger.kernel.org>
Sent: Friday, October 10, 2003 3:30 PM
Subject: Re: 2.7 thoughts


> On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
> Stephan von Krawczynski wrote:
> >>* hotplug CPU
> >>* hotplug RAM
>
> Generally thought of as desirable, but hotplug RAM needs a *LOT* of
> qualification.
>
>
> On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
> > * hotplug motherboard & entire computer too I spose ;-)
>
> Um, this is worse than the above wrt. being too vague.
>
>
> On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
> > Although sarcasm aside, a couple of ideas that have been bantered around
> > on this list (and a few of my own ideas):
> > - /proc interface alternative to modutils/module-init-tools.
> > That is, to have a directory of virtual nodes in /proc
> > to provide the functionality of insmod, rmmod, lsmod &
> > modprobe would be great -- especially from the viewpoint
> > of recue disk images, etc.
>
> No way in Hell.
>
>
> -- wli
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.7 thoughts

[William Lee Irwin III]

On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
>>> * hotplug motherboard & entire computer too I spose ;-)

> From: "William Lee Irwin III" <wli@holomorphy.com>
>> Um, this is worse than the above wrt. being too vague.

On Fri, Oct 10, 2003 at 04:30:27PM +0900, YoshiyaETO wrote:
> "Hotplug node" is a better explanation, I think.
> "Node" includes CPUs and/or Memory and/or some kind of IOs.
> And "Node" should be flexibly configurable also.

I don't see any reason to connect it with the notion of a node.

The main points of contention would appear to be cooperative vs.
forcible (where I believe cooperative is acknowledged as the only
feasible problem), and the potential connections with ZONE_HIGHMEM
wrt. constraints that would artificially introduce to 64-bit kernels.

The fact some systems would want to do whole nodes at a time with
some cpus and io buses in tandem is largely immaterial and doesn't
simplify, complicate, or otherwise affect the VM mechanics.

-- wli

Re: 2.7 thoughts

[YoshiyaETO]

> I don't see any reason to connect it with the notion of a node.
    If the word "Node" is not so appropriate, I will use "Unit".
And I also make it simple, "Unit" will have CPUs and/or Memory.
On the other hand IO-Unit will have IOs.

> The main points of contention would appear to be cooperative vs.
> forcible (where I believe cooperative is acknowledged as the only
    I could not understand what is forcible.
Everything should be cooperative, I think.

----- Original Message ----- 
From: "William Lee Irwin III" <wli@holomorphy.com>
To: "YoshiyaETO" <eto@soft.fujitsu.com>
Cc: "Stuart Longland" <stuartl@longlandclan.hopto.org>;
<linux-kernel@vger.kernel.org>; "Stephan von Krawczynski"
<skraw@ithnet.com>; <lgb@lgb.hu>; <Fabian.Frederick@prov-liege.be>
Sent: Friday, October 10, 2003 4:40 PM
Subject: Re: 2.7 thoughts


> On Fri, Oct 10, 2003 at 04:19:46PM +1000, Stuart Longland wrote:
> >>> * hotplug motherboard & entire computer too I spose ;-)
>
> > From: "William Lee Irwin III" <wli@holomorphy.com>
> >> Um, this is worse than the above wrt. being too vague.
>
> On Fri, Oct 10, 2003 at 04:30:27PM +0900, YoshiyaETO wrote:
> > "Hotplug node" is a better explanation, I think.
> > "Node" includes CPUs and/or Memory and/or some kind of IOs.
> > And "Node" should be flexibly configurable also.
>
> I don't see any reason to connect it with the notion of a node.
>
> The main points of contention would appear to be cooperative vs.
> forcible (where I believe cooperative is acknowledged as the only
> feasible problem), and the potential connections with ZONE_HIGHMEM
> wrt. constraints that would artificially introduce to 64-bit kernels.
>
> The fact some systems would want to do whole nodes at a time with
> some cpus and io buses in tandem is largely immaterial and doesn't
> simplify, complicate, or otherwise affect the VM mechanics.
>
> -- wli
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.7 thoughts

[William Lee Irwin III]

At some point in the past, I wrote:
>> I don't see any reason to connect it with the notion of a node.

On Fri, Oct 10, 2003 at 05:47:37PM +0900, YoshiyaETO wrote:
>     If the word "Node" is not so appropriate, I will use "Unit".
> And I also make it simple, "Unit" will have CPUs and/or Memory.
> On the other hand IO-Unit will have IOs.

Well, that's precisely what I was saying was unnecessary. The VM
mechanics are orthogonal to the rest, so there's no reason to tie
their handling together. The coincidence that they appear bundled
on one system or another is irrelevant.


At some point in the past, I wrote:
> > The main points of contention would appear to be cooperative vs.
> > forcible (where I believe cooperative is acknowledged as the only

On Fri, Oct 10, 2003 at 05:47:37PM +0900, YoshiyaETO wrote:
>     I could not understand what is forcible.
> Everything should be cooperative, I think.

"Forcible" would be "the kernel receives a magic interrupt, and in
some mailbox the interrupt handler discovers the memory has either
already disappeared or will disappear in some amount of time regardless
of whether the kernel is prepared to handle its removal." The
distinction is meaningless for the case of onlining. The case of
offlining (perhaps by some deadline) is widely considered infeasible,
but there are some environments that could consider it desirable.

The bit that was actually expected to spark debate was the ZONE_HIGHMEM
notion purported to be a desirable method for resolving the conflict
between pinned/wired kernel allocations and cooperative offlining by
restricting pinned/wired kernel allocations to some fixed physical
arena. The two issues mentioned above are in reality non-issues.


-- wli

Re: 2.7 thoughts

[YoshiyaETO]

> Well, that's precisely what I was saying was unnecessary. The VM
> mechanics are orthogonal to the rest, so there's no reason to tie
> their handling together. The coincidence that they appear bundled
> on one system or another is irrelevant.
----
     In the kernel logic, VM mechanics are really orthogonal to the CPU.
But, in the real world, someone, like hardware maintenance person,
should treat CPUs and memory at a time to maintain Unit. So, notion
of Unit should be somewhere, might be sysfs + userland.

> "Forcible" would be "the kernel receives a magic interrupt, and in
> some mailbox the interrupt handler discovers the memory has either
> already disappeared or will disappear in some amount of time regardless
> of whether the kernel is prepared to handle its removal." The
----
    "Forcible" way is not a good idea.
In that mean, it should be a "cooperative" way with kernel, userland,
and sometimes with system operator who can make an appropriate
environment.

> The bit that was actually expected to spark debate was the ZONE_HIGHMEM
> notion purported to be a desirable method for resolving the conflict
> between pinned/wired kernel allocations and cooperative offlining by
> restricting pinned/wired kernel allocations to some fixed physical
      I also expect debate, but not now.
I think the way using "ZONE_HIGHTMEM" simplify the issues to
be able to remove physical arena other than kernel memory
allocated one that would be fixed.

> arena. The two issues mentioned above are in reality non-issues.
    Could tell me what is the real issue you think?

----- Original Message ----- 
From: "William Lee Irwin III" <wli@holomorphy.com>
To: "YoshiyaETO" <eto@soft.fujitsu.com>
Cc: "Stuart Longland" <stuartl@longlandclan.hopto.org>;
<linux-kernel@vger.kernel.org>; "Stephan von Krawczynski"
<skraw@ithnet.com>; <lgb@lgb.hu>; <Fabian.Frederick@prov-liege.be>
Sent: Friday, October 10, 2003 6:09 PM
Subject: Re: 2.7 thoughts


> At some point in the past, I wrote:
> >> I don't see any reason to connect it with the notion of a node.
>
> On Fri, Oct 10, 2003 at 05:47:37PM +0900, YoshiyaETO wrote:
> >     If the word "Node" is not so appropriate, I will use "Unit".
> > And I also make it simple, "Unit" will have CPUs and/or Memory.
> > On the other hand IO-Unit will have IOs.
>
> Well, that's precisely what I was saying was unnecessary. The VM
> mechanics are orthogonal to the rest, so there's no reason to tie
> their handling together. The coincidence that they appear bundled
> on one system or another is irrelevant.
>
>
> At some point in the past, I wrote:
> > > The main points of contention would appear to be cooperative vs.
> > > forcible (where I believe cooperative is acknowledged as the only
>
> On Fri, Oct 10, 2003 at 05:47:37PM +0900, YoshiyaETO wrote:
> >     I could not understand what is forcible.
> > Everything should be cooperative, I think.
>
> "Forcible" would be "the kernel receives a magic interrupt, and in
> some mailbox the interrupt handler discovers the memory has either
> already disappeared or will disappear in some amount of time regardless
> of whether the kernel is prepared to handle its removal." The
> distinction is meaningless for the case of onlining. The case of
> offlining (perhaps by some deadline) is widely considered infeasible,
> but there are some environments that could consider it desirable.
>
> The bit that was actually expected to spark debate was the ZONE_HIGHMEM
> notion purported to be a desirable method for resolving the conflict
> between pinned/wired kernel allocations and cooperative offlining by
> restricting pinned/wired kernel allocations to some fixed physical
> arena. The two issues mentioned above are in reality non-issues.
>
>
> -- wli
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.7 thoughts

[William Lee Irwin III]

At some point in the past, I wrote:
>> arena. The two issues mentioned above are in reality non-issues.

On Fri, Oct 10, 2003 at 07:26:26PM +0900, YoshiyaETO wrote:
>     Could tell me what is the real issue you think?

Using reserved areas for kernel metadata marked as ZONE_HIGHMEM creates
two serious problems. First memory placement of kernel metadata is
disturbed, just like the "all kernel memory references are on node 0"
problem on ia32 NUMA. Second, it also inherits all of the workload
feasibility problems also seen on ia32 PAE. A third, less serious
problem is that the confusions between notions such as the capability
to do io to memory vs. placement in ZONE_HIGHMEM, the likely newly
introduced confusion between being able to directly access the data
in ZONE_HIGHMEM without establishing a new mapping (TLB entry) vs.
copying overheads etc., and the placement of some pinned/wired kernel
metadata like leaf pagetable nodes in ZONE_HIGHMEM, are all nasty
details conveniently left out of the quickly rattled-off zoning schemes
for inhibiting pinned/wired allocations from offlineable memory.

The first point explodes the entire theory of nodes coming on and off
line; the moment node-local kernel data is allowed, the entire zoning
scheme falls apart and the node can't be fully offlined at will as
envisioned at all.

The second point raises the further question of whether allowing the
zoning to get anywhere near 64-bit is even desirable; Linux tends to
want to consume all physical memory for kernel metadata as it pleases,
and confining it to small subsets of it has proved incompatible with
its design and/or the wishes of the kernel community in the past. The
general notions of process-local paged kernel metadata and the like
which would create some allowance for relocatable kernel metadata (or
in the PAE case, reduce pressure on lowmem) have been proposed before
and met heavy resistance. OTOH, certain things, like mem_map[] and
pgdats, could easily be made node-local as they would be torn down
when the node is offlined anyway.


-- wli

(The answers to the issues raised in the third point are all basically
API cleanups and the implementation of a thing that should have been in
the kernel to begin with.)

Re: 2.7 thoughts

[William Lee Irwin III]

On Fri, Oct 10, 2003 at 07:26:26PM +0900, YoshiyaETO wrote:
>>     Could tell me what is the real issue you think?

On Fri, Oct 10, 2003 at 03:50:21AM -0700, William Lee Irwin III wrote:
> Using reserved areas for kernel metadata marked as ZONE_HIGHMEM creates

Ergh, s/HIGHMEM/NORMAL/ here.


-- wli

Re: 2.7 thoughts

[Stephan von Krawczynski]

On Fri, 10 Oct 2003 16:19:46 +1000
Stuart Longland <stuartl@longlandclan.hopto.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Stephan von Krawczynski wrote:
> > * hotplug CPU
> > * hotplug RAM
> 
> * hotplug motherboard & entire computer too I spose ;-)
> 
> Although sarcasm aside, a couple of ideas that have been bantered around 
> on this list (and a few of my own ideas):

You are obviously not quite familiar with industrial boxes where this is
state-of-the-art. 

My thinking is:
* CPU hotplug: should not be too hard
* RAM hotplug: has already been discussed in several threads, sure needs brain,
but must be doable.

Generally spoken every part of a computer should be thought of as a "resource"
that can be added or removed at any time during runtime. CPU or RAM is in no
way different.

Regards,
Stephan

Re: 2.7 thoughts

[Stuart Longland]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stephan von Krawczynski wrote:

> You are obviously not quite familiar with industrial boxes where this is
> state-of-the-art. 
 >
 > [...]
 >
> Generally spoken every part of a computer should be thought of as a "resource"
> that can be added or removed at any time during runtime. CPU or RAM is in no
> way different.

Oh, okay, this sort of thing is supported by industrial boxes? 
Interesting... Live and learn I spose ;-) (You're right, I'm not 
familiar with industrial boxes at all.  My experience is with mostly 
desktop computers, laptops, and some entry-level servers)

Hotplug RAM I could see would be possible, but hotplug CPUs?  I spose if 
you've got a multiprocessor box, you could swap them one at a time, but 
my thinking is that this would cause issues with the OS as it wouldn't 
be expecting the CPU to suddenly disappear.  Problems would be even 
worse if the old and new CPUs were of different types too.

Hotplug RAM would also be interesting, but then again, I spose the 
procedure would be to alert the kernel that the memory area from byte X 
to byte Y would disappear, so it could page that out to swapspace.

Anyways, I don't profess to be any hardware/software/kernel guru, I had 
never heard of this level of hotplug, and it struck me as unusual.
- -- 
+-------------------------------------------------------------+
| Stuart Longland           stuartl at longlandclan.hopto.org |
| Brisbane Mesh Node: 719             http://stuartl.cjb.net/ |
| I haven't lost my mind - it's backed up on a tape somewhere |
| Griffith Student No:           Course: Bachelor/IT (Nathan) |
+-------------------------------------------------------------+
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE/hr0OIGJk7gLSDPcRAl9CAJ0c+iU//ELVoO8czbezWgvd7UBzjwCeNApa
ftZkG88NrefELUoGaEop+NI=
=tTfj
-----END PGP SIGNATURE-----

Re: 2.7 thoughts

[Stephan von Krawczynski]

On Sat, 11 Oct 2003 00:07:10 +1000
Stuart Longland <stuartl@longlandclan.hopto.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Stephan von Krawczynski wrote:
> 
> > You are obviously not quite familiar with industrial boxes where this is
> > state-of-the-art. 
>  >
>  > [...]
>  >
> > Generally spoken every part of a computer should be thought of as a
> > "resource" that can be added or removed at any time during runtime. CPU or
> > RAM is in no way different.
>
> Hotplug RAM I could see would be possible, but hotplug CPUs?  I spose if 
> you've got a multiprocessor box, you could swap them one at a time, but 
> my thinking is that this would cause issues with the OS as it wouldn't 
> be expecting the CPU to suddenly disappear.  Problems would be even 
> worse if the old and new CPUs were of different types too.

I wouldn't expect a hotplug feature for differing CPU types actually. But in
fact there are boxes out there that complain via phonecall that a processor
board just died and request replacement. Same goes for RAM. It is obvious that
no feature like this exists on desktop nowadays, but 2.7 and release of 2.8 is
far ahead, so I see no good reason why people should not be enabled to do this
- _then_ .
In fact it is sounds really simple to plug in additional boards to a numa-type
box. Boards meaning RAM+CPU.
HP builds hotplug PCI stuff in comparably cheap boxes, so this is not
completely off-reality.
 
> Hotplug RAM would also be interesting, but then again, I spose the 
> procedure would be to alert the kernel that the memory area from byte X 
> to byte Y would disappear, so it could page that out to swapspace.

This direction sounds likely.

Regards,
Stephan

Re: 2.7 thoughts

[Gábor Lénárt]

On Sat, Oct 11, 2003 at 12:07:10AM +1000, Stuart Longland wrote:
> Oh, okay, this sort of thing is supported by industrial boxes? 
> Interesting... Live and learn I spose ;-) (You're right, I'm not 
> familiar with industrial boxes at all.  My experience is with mostly 
> desktop computers, laptops, and some entry-level servers)
> 
> Hotplug RAM I could see would be possible, but hotplug CPUs?  I spose if 
> you've got a multiprocessor box, you could swap them one at a time, but 
> my thinking is that this would cause issues with the OS as it wouldn't 
> be expecting the CPU to suddenly disappear.  Problems would be even 

Why? Sure, you should note OS before unplug a CPU ;-) but in this case OS
should be noticed that it could not use that CPU any more (like scheduler,
interrupt delivering etc). And then it's not an issue if you unplug THAT
cpu. Sure, some hw supporting is needed, a "normal" motherboard does not
like to unplug a CPU when powered of course by hardware issues, but "CPU
hotplug capable" motherboards can support this. Maybe of course some notice
mechanism is needed for the motherboard as well not only for the OS, but it
only mean that notfication before unplugging the CPU should be delivered to
the OS _and_ to the hardware. Sure, I have no experience at all in this
area, so this is only a theory (I've never see a hardware support this yet).

> worse if the old and new CPUs were of different types too.

Yes this IS an issue. I think this is no good workaround for this problem,
but I also don't think that this situation is well supported by other
OSes as well ... If you insert a new CPU which is fully compatible with
the old one this should not be an issue however.

- Gábor (larta'H)

Re: 2.7 thoughts

[William Lee Irwin III]

On Fri, Oct 10, 2003 at 04:35:29PM +0200, G?bor L?n?rt wrote:
> Why? Sure, you should note OS before unplug a CPU ;-) but in this case OS
> should be noticed that it could not use that CPU any more (like scheduler,
> interrupt delivering etc). And then it's not an issue if you unplug THAT
> cpu. Sure, some hw supporting is needed, a "normal" motherboard does not
> like to unplug a CPU when powered of course by hardware issues, but "CPU
> hotplug capable" motherboards can support this. Maybe of course some notice
> mechanism is needed for the motherboard as well not only for the OS, but it
> only mean that notfication before unplugging the CPU should be delivered to
> the OS _and_ to the hardware. Sure, I have no experience at all in this
> area, so this is only a theory (I've never see a hardware support this yet).

You need at least enough warning to get out of critical sections (e.g.
holding a spinlock) and dump registers out to memory. i.e. as long as it
takes to schedule out whatever's currently running on the thing.

... and unless you want to start enforcing realtime bounds, the answer
to "how long do you have to give the kernel to do it?" is "forever".
In practice, it won't take forever, but no finite time is enforcible.


-- wli

Re: 2.7 thoughts

[Mark Mielke]

On Fri, Oct 10, 2003 at 07:47:23AM -0700, William Lee Irwin III wrote:
> On Fri, Oct 10, 2003 at 04:35:29PM +0200, G?bor L?n?rt wrote:
> > Why? Sure, you should note OS before unplug a CPU ;-) but in this case OS
> > should be noticed that it could not use that CPU any more (like scheduler,
> > interrupt delivering etc). And then it's not an issue if you unplug THAT
> > cpu. Sure, some hw supporting is needed, a "normal" motherboard does not
> > like to unplug a CPU when powered of course by hardware issues, but "CPU
> > hotplug capable" motherboards can support this. Maybe of course some notice
> > mechanism is needed for the motherboard as well not only for the OS, but it
> > only mean that notfication before unplugging the CPU should be delivered to
> > the OS _and_ to the hardware. Sure, I have no experience at all in this

Perhaps I've naive here, but - with hot-pluggable CPU machines, do you not
de-activate the CPU through software first, before pulling the CPU out, at
which point it is not in use?

mark

-- 
mark@mielke.cc/markm@ncf.ca/markm@nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

Re: 2.7 thoughts

[William Lee Irwin III]

On Fri, Oct 10, 2003 at 10:48:37AM -0400, Mark Mielke wrote:
> Perhaps I've naive here, but - with hot-pluggable CPU machines, do you not
> de-activate the CPU through software first, before pulling the CPU out, at
> which point it is not in use?

Well, you deleted my reply, but never mind that.

This obviously can't work unless the kernel gets some kind of warning.
Userspace and kernel register state, once lost that way, can't be
recovered, and if tasks are automatically suspended (e.g. cpu dumps to
somewhere and a miracle occurs), you'll deadlock if the kernel was in
a non-preemptible critical section at the time.

What I suspect to be the case is some kind of warning is given to the
kernel, and it has to respond within a certain time. Apparently it
succeeds most of the time despite my naysaying if this actually works.


-- wli

Re: 2.7 thoughts

[Mark Mielke]

On Fri, Oct 10, 2003 at 08:01:22AM -0700, William Lee Irwin III wrote:
> On Fri, Oct 10, 2003 at 10:48:37AM -0400, Mark Mielke wrote:
> > Perhaps I've naive here, but - with hot-pluggable CPU machines, do you not
> > de-activate the CPU through software first, before pulling the CPU out, at
> > which point it is not in use?
> Well, you deleted my reply, but never mind that.

I wasn't responding to you. You're article just happened to be the one
that I pushed 'g' to... :-)

> This obviously can't work unless the kernel gets some kind of warning.
> Userspace and kernel register state, once lost that way, can't be
> recovered, and if tasks are automatically suspended (e.g. cpu dumps to
> somewhere and a miracle occurs), you'll deadlock if the kernel was in
> a non-preemptible critical section at the time.

Again, I'm perhaps naive here - I've never been able to afford such a
machine with hot-pluggable CPU's, however, I have heard from people who
have used them at work, that you use software (i.e. system calls to the
kernel) to instruct the kernel to no longer schedule tasks for the CPU.
Once the CPU is no longer scheduled for use, you can feel free to unplug
the CPU from the motherboard. Note that I didn't say that the software
approach could *guarantee* immediate success. You wouldn't unplug the
CPU until your had successfully deregistered the CPU from having anything
scheduled for it.

Is this not the way things (should) work?

mark

-- 
mark@mielke.cc/markm@ncf.ca/markm@nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

Re: 2.7 thoughts

[Chris Friesen]

Mark Mielke wrote:

> Note that I didn't say that the software
> approach could *guarantee* immediate success. You wouldn't unplug the
> CPU until your had successfully deregistered the CPU from having anything
> scheduled for it.
> 
> Is this not the way things (should) work?

Note that if you're doing this for high availability purposes, you 
already need to have some way of handling a cpu that just dies in the 
middle of processing.  Once you've done that, you can just re-use that 
to handle hot removal--it just gets treated like a fault.  This is not 
to say that you can't try and shut it down nicely first, but its not a 
hard requirement.

Chris


-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: 2.7 thoughts

[William Lee Irwin III]

Mark Mielke wrote:
>> Note that I didn't say that the software
>> approach could *guarantee* immediate success. You wouldn't unplug the
>> CPU until your had successfully deregistered the CPU from having anything
>> scheduled for it.
>> Is this not the way things (should) work?

On Fri, Oct 10, 2003 at 12:35:57PM -0400, Chris Friesen wrote:
> Note that if you're doing this for high availability purposes, you 
> already need to have some way of handling a cpu that just dies in the 
> middle of processing.  Once you've done that, you can just re-use that 
> to handle hot removal--it just gets treated like a fault.  This is not 
> to say that you can't try and shut it down nicely first, but its not a 
> hard requirement.

If you've lost the registers in-kernel, you can't even get away with
shooting the process. I'm not sure what extreme HA wants to do with
that, but at a such a point it's not really even possible to figure
out how to extract the thing from whatever critical section it may
have been in.


-- wli

Re: 2.7 thoughts

[Pavel Machek]

Hi!

> > Perhaps I've naive here, but - with hot-pluggable CPU machines, do you not
> > de-activate the CPU through software first, before pulling the CPU out, at
> > which point it is not in use?
> 
> Well, you deleted my reply, but never mind that.
> 
> This obviously can't work unless the kernel gets some kind of warning.
       ~~~~~~~~~~~~~~~~~~~~
> Userspace and kernel register state, once lost that way, can't be
> recovered, and if tasks are automatically suspended (e.g. cpu dumps to
> somewhere and a miracle occurs), you'll deadlock if the kernel was in
> a non-preemptible critical section at the time.

Of course it _can_ work without warning, it would just be *extremely*
hard to do within linux. [When you have big enough cluster, you have
to deal with nodes failing randomly; when you get big enough SMP,
you'll have same issue].

One [stupid] method to handle this would be to do periodic system
snapshots, and if cpu fails rollback and try again without that
cpu. You'll get nasty issues with network (duplicated packets on tx),
and funny stuff on console, but certainly way to make this work
exists.

								Pavel
-- 
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]

Re: 2.7 thoughts

[Jamie Lokier]

William Lee Irwin III wrote:
> You need at least enough warning to get out of critical sections (e.g.
> holding a spinlock) and dump registers out to memory. i.e. as long as it
> takes to schedule out whatever's currently running on the thing.
> 
> ... and unless you want to start enforcing realtime bounds, the answer
> to "how long do you have to give the kernel to do it?" is "forever".
> In practice, it won't take forever, but no finite time is enforcible.

You can create a very peculiar scheduling state to make even
spinlocked sections multi-task, so the CPU can be released in a finite
time and quickly - about as quickly as taking an NMI and broadcasting
the critical IPIs to tell other CPUs to take over.

The peculiar state is restored to normal as soon as the number
of concurrent critical sections no longer exceeds the number of real CPUs.

Come to think of it, this would be an excellent HA mechanism: CPU or
node detects hardware fault, raises NMI which saves the CPU registers,
does the IPIs and then immediately disables the CPU.  That's _it_.

The peculiar scheduling state obviously slows down the code which is
unlucky enough to be caught in a critical section when a CPU is
removed, so if there is _timing_ critical code it will be affected,
but that is a similar problem to speedstep etc.

-- Jamie

Re: 2.7 thoughts

[William Lee Irwin III]

William Lee Irwin III wrote:
>> You need at least enough warning to get out of critical sections (e.g.
>> holding a spinlock) and dump registers out to memory. i.e. as long as it
>> takes to schedule out whatever's currently running on the thing.
>> ... and unless you want to start enforcing realtime bounds, the answer
>> to "how long do you have to give the kernel to do it?" is "forever".
>> In practice, it won't take forever, but no finite time is enforcible.

On Fri, Oct 10, 2003 at 04:12:31PM +0100, Jamie Lokier wrote:
> You can create a very peculiar scheduling state to make even
> spinlocked sections multi-task, so the CPU can be released in a finite
> time and quickly - about as quickly as taking an NMI and broadcasting
> the critical IPIs to tell other CPUs to take over.
> The peculiar state is restored to normal as soon as the number
> of concurrent critical sections no longer exceeds the number of real CPUs.

DOH. Okay, that would do it.


-- wli

Re: 2.7 thoughts

[Tim Hockin]

On Fri, Oct 10, 2003 at 07:47:23AM -0700, William Lee Irwin III wrote:
> You need at least enough warning to get out of critical sections (e.g.
> holding a spinlock) and dump registers out to memory. i.e. as long as it
> takes to schedule out whatever's currently running on the thing.

I've got a patch against RedHat's 2.4.x kernel with some version of O(1)
scheduler that migrates all processes off a CPU and makes it ineleigible to
run new tasks.  When the syscall returns, it is safe to remove the CPU.
Processes that are running or sleeping and can be migrated elsewhere are
migrated.  Running processes that have set_cpus_allowed to ONLY the
processor in question are marked TASK_UNRUNNABLE and moved off the runqueue.
Sleeping processes that have set_cpus_allowed to ONLY the processor in
question are left unmolested until they wake up, at which point they will be
marked UNRUNNABLE.

It was done to allow software to bring CPUs on/offline, but it should work
for this.  IRQs not done yet, either.

-- 
Notice that as computers are becoming easier and easier to use,
suddenly there's a big market for "Dummies" books.  Cause and effect,
or merely an ironic juxtaposition of unrelated facts?

Re: 2.7 thoughts

[William Lee Irwin III]

On Fri, Oct 10, 2003 at 07:47:23AM -0700, William Lee Irwin III wrote:
>> You need at least enough warning to get out of critical sections (e.g.
>> holding a spinlock) and dump registers out to memory. i.e. as long as it
>> takes to schedule out whatever's currently running on the thing.

On Fri, Oct 10, 2003 at 11:03:20AM -0700, Tim Hockin wrote:
> I've got a patch against RedHat's 2.4.x kernel with some version of O(1)
> scheduler that migrates all processes off a CPU and makes it ineleigible to
> run new tasks.  When the syscall returns, it is safe to remove the CPU.
> Processes that are running or sleeping and can be migrated elsewhere are
> migrated.  Running processes that have set_cpus_allowed to ONLY the
> processor in question are marked TASK_UNRUNNABLE and moved off the runqueue.
> Sleeping processes that have set_cpus_allowed to ONLY the processor in
> question are left unmolested until they wake up, at which point they will be
> marked UNRUNNABLE.
> It was done to allow software to bring CPUs on/offline, but it should work
> for this.  IRQs not done yet, either.

I think there is some generalized cpu hotplug stuff that's gone in that
direction already, though I don't know any details. The bits about non-
cooperative offlining were very interesting to hear, though.

-- wli

Re: 2.7 thoughts

[Tim Hockin]

On Fri, Oct 10, 2003 at 11:29:09AM -0700, William Lee Irwin III wrote:
> I think there is some generalized cpu hotplug stuff that's gone in that
> direction already, though I don't know any details. The bits about non-
> cooperative offlining were very interesting to hear, though.

I spoke with Rusty about it at OLS.  I haven't tracked the hotplug CPU
projects.  I think that TASK_UNRUNNABLE is the sane way to handle said edge
case, but at the time Rusty was leaning towards SIGPOWER.


-- 
Notice that as computers are becoming easier and easier to use,
suddenly there's a big market for "Dummies" books.  Cause and effect,
or merely an ironic juxtaposition of unrelated facts?

Re: 2.7 thoughts

[Rusty Russell]

On Fri, 10 Oct 2003 11:56:09 -0700
Tim Hockin <thockin@hockin.org> wrote:

> On Fri, Oct 10, 2003 at 11:29:09AM -0700, William Lee Irwin III wrote:
> > I think there is some generalized cpu hotplug stuff that's gone in that
> > direction already, though I don't know any details. The bits about non-
> > cooperative offlining were very interesting to hear, though.
> 
> I spoke with Rusty about it at OLS.  I haven't tracked the hotplug CPU
> projects.  I think that TASK_UNRUNNABLE is the sane way to handle said edge
> case, but at the time Rusty was leaning towards SIGPOWER.

Yeah, if a task has become unrunnable, I do a SIGPOWER (with a new cpu siginfo
field).  If hotplug CPUs had been introduced at the same time as affinity,
this would IMHO have been a valid approach, but may not be so now.

See my kernel.org page, and the sourceforge mailing list for details.

I haven't done anything fancy with cpu offlining, but the most painful bit
is all the callbacks for workqueue threads, migration threads etc.  Once
that's in place, doing the atomic-style switch should be quite possible.
There's a theoretical case where code would do:

	spin_lock
	foo[smp_processor_id()]++;
	...
	foo[smp_processor_id()]--;
	spin_unlock

So you might want to fake up the answer to smp_processor_id() for that
task/interrupt.  Other real per-cpu things might have problems (if you
were halfway through fiddling with the interrupt state on that CPU,
maybe MTRR), but that's what makes it fun.

Cheers,
Rusty.
-- 
   there are those who do and those who hang on and you don't see too
   many doers quoting their contemporaries.  -- Larry McVoy

Re: 2.7 thoughts

[Anton Blanchard]

 
> Hotplug RAM I could see would be possible, but hotplug CPUs?  I spose if 
> you've got a multiprocessor box, you could swap them one at a time, but 
> my thinking is that this would cause issues with the OS as it wouldn't 
> be expecting the CPU to suddenly disappear.  Problems would be even 
> worse if the old and new CPUs were of different types too.

Our ppc64 hardware supports hotswap cpu - not physically adding a cpu but
rather moving the pool of cpus into and out of partitions.

There are many uses for cpu hotplug. If you hit a threshold of
correctible errors on a cpu, we can remove the failing cpu and swap in a
spare if available. If we do this before we get a UE then all is good.
One day we can get even more intelligent and recover from certain UEs, I
was talking to Andi and the ia64 guys at OLS about this (eg UE in a cpu
array mapped to a userspace address, we can kill the task and continue
on).

You can also move cpus between partitions on the fly, moving cpus out of
idle partitions and into the busy ones.

Anton

Re: 2.7 thoughts

[Kevin Corry]

On Friday 10 October 2003 01:19, Stuart Longland wrote:
> 	- Software RAID 0+1 perhaps?
>
> 		A lot of hardware RAID cards support it, why not the
> 		kernel?  By RAID 0+1 I mean mirror-RAIDing two (or more)
>		stripe-RAID arrays.  (Or can this be done already?)

This can be done already. For example, you have six drives, sd[a-f]. Create 
md0 (raid-0) using sda, sdb, and sdc. Create md1 (raid-0) using sdd, sde, and 
sdf. Then create md2 (raid-1) using md0 and md1. This setup can easily be 
accomplished using evms, mdadm, or raidtools.

-- 
Kevin Corry
kevcorry@us.ibm.com
http://evms.sourceforge.net/

Re: 2.7 thoughts

[Lars Marowsky-Bree]

On 2003-10-10T08:30:03,
   Kevin Corry <kevcorry@us.ibm.com> said:

> On Friday 10 October 2003 01:19, Stuart Longland wrote:
> > 	- Software RAID 0+1 perhaps?

Because RAID 0+1 is a rather bad idea. You want RAID 1+0. Make up the
fault matrix and simulate what happens if drives fail.

We can do both though, as Kevin pointed out. So if you want to shot
yourself in the foot, in the best Unix tradition, we allow you to ;)

I'd suggest moving this to the linux-raid list though.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering		ever tried. ever failed. no matter.
SuSE Labs				try again. fail again. fail better.
Research & Development, SUSE LINUX AG		-- Samuel Beckett

Re: 2.7 thoughts

[jw schultz]

On Fri, Oct 10, 2003 at 08:29:18PM +0200, Lars Marowsky-Bree wrote:
> On 2003-10-10T08:30:03,
>    Kevin Corry <kevcorry@us.ibm.com> said:
> 
> > On Friday 10 October 2003 01:19, Stuart Longland wrote:
> > > 	- Software RAID 0+1 perhaps?
> 
> Because RAID 0+1 is a rather bad idea. You want RAID 1+0. Make up the
> fault matrix and simulate what happens if drives fail.
> 
> We can do both though, as Kevin pointed out. So if you want to shot
> yourself in the foot, in the best Unix tradition, we allow you to ;)

I concur with one caviat.  0+1 has the advantage of
extendability that doesn't exist with 1+0.

	1. break mirror downing side A
	2. break stripe A
	3. build new stripe A with added disk(s)
	4. copying stripe B to stripe A
	5. break stripe B
	6. build new stripe B with added disk(s)
	7. build mirror (A->B)

It may even be possible to do this live.  So if gradual
extendability is more important than surviving multiple
failures 0+1 has the advantage.  Normally i prefer the
reliability or to do striping at the logical volume level.


	
	


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Re: 2.7 thoughts

[Lars Marowsky-Bree]

On 2003-10-10T20:49:51,
   jw schultz <jw@pegasys.ws> said:

> I concur with one caviat.  0+1 has the advantage of
> extendability that doesn't exist with 1+0.

Right, this annoying complicated approach you describe can be done much
easier with 1+0. With [EL]VMS?[12] you can simply create a new raid1 set
and add it as a physical volume to the volume group and then extend the
LVs accordingly. (I am unsure whether you can add a new disk to a raid0
set if you don't want to use a volume manager, but if it's not
currently, it sounds fairly straightforward to add.)

Your approach with breaking the mirrors etc includes prolonged periods
of no redundancy and makes me shiver.

There's some *buy it* good book about *buy it* all this, but if I go
hype *buy it* "Blueprints for High Availability" *buy it* one more time,
people are going to accuse me of *buy this book already!* of flogging a
dead horse *buy it*, so for this time, I am going to recommend reading
some linux LVM and RAID howtos ;-)

So, I think, as far as RAID and Volume Management is concerned, Linux
does pretty well. There's some advanced and fancy stuff missing (>2
mirrors, online consistency check, etc), but the basics are pretty well
done.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering		ever tried. ever failed. no matter.
SuSE Labs				try again. fail again. fail better.
Research & Development, SUSE LINUX AG		-- Samuel Beckett

Re: 2.7 thoughts

[jw schultz]

On Sat, Oct 11, 2003 at 03:24:22PM +0200, Lars Marowsky-Bree wrote:
> On 2003-10-10T20:49:51,
>    jw schultz <jw@pegasys.ws> said:
> 
> > I concur with one caviat.  0+1 has the advantage of
> > extendability that doesn't exist with 1+0.
> 
> Right, this annoying complicated approach you describe can be done much
> easier with 1+0. With [EL]VMS?[12] you can simply create a new raid1 set
> and add it as a physical volume to the volume group and then extend the
> LVs accordingly. (I am unsure whether you can add a new disk to a raid0
> set if you don't want to use a volume manager, but if it's not
> currently, it sounds fairly straightforward to add.)
> 
> Your approach with breaking the mirrors etc includes prolonged periods
> of no redundancy and makes me shiver.

I shivered as i wrote it.  I don't consider the periods of
non-redundancy to be prolonged but they are periods of
highest stress on the drives so they are more likely to fail
(even sans Murphy) while doing that than they would
otherwise.  And if you look back at my closing statement
you'll see that i don't recommend it.  I only site it as a
possibility.

Extendibility is i think the only rationalised excuse for
0+1 when 1+0 is available.  I consider it a rationalisation
because by the time you need to extend an array the value of
same-size disks will be questionable.

> [book flogging]
> I am going to recommend reading
> some linux LVM and RAID howtos ;-)
> 
> So, I think, as far as RAID and Volume Management is concerned, Linux
> does pretty well. There's some advanced and fancy stuff missing (>2
> mirrors, online consistency check, etc), but the basics are pretty well
> done.

Absolutely.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Re: 2.7 thoughts

[Matt Simonsen]

On Thu, 2003-10-09 at 23:19, Stuart Longland wrote:
> 	- Software RAID 0+1 perhaps?
> 
> 		A lot of hardware RAID cards support it, why not the
> 		kernel?  By RAID 0+1 I mean mirror-RAIDing two (or more) 		stripe-RAID 
> arrays.  (Or can this be done already?)


You can do this now (as you rightly suspected). 

Just create 2 raid 0 arrays (md0, md1). From there create md2 as a RAID1
array using /dev/md0 and /dev/md1. Create the filesystem and you've got
yourself RAID 0+1. 

You can do RAID 5+0 or any other wild combo you could think of.

Matt

Re: 2.7 thoughts

[Pedro Larroy]

On Thu, Oct 09, 2003 at 01:58:09PM +0200, Gábor Lénárt wrote:
> On Thu, Oct 09, 2003 at 10:08:00AM +0200, Frederick, Fabian wrote:
> > Hi,
> > 	Some thoughts for 2.7.Someone has other ideas, comments ?
> [...] 	
> > *	All this guides me to a more global conclusion in which all that
> > stuff should be kobject registration relevant 
> > *	Meanwhile, we don't have a kobject <-> security bridge :( 
> 
> well, maybe stupid ideas, but they're which supported on other unix like
> system(s) or it would be very nice according to my experiences:
> 
> * bind mount support for all general mount options (nodev,ro,noexec etc)
>   with SECURE implementation with any (maybe even future) filesystems?
> * union mount (possible with option to declare on what fs a new file
>   should be created: on fixed ones, random algorithm, on fs with the
>   largest free space available etc ...)
> * guaranteed i/o bandwidth allocation?
> * netfilter's ability to do tricks which OpenBSD can do now with its
>   packet filter

Can you describe those please?

> * ENBD support in official kernel with enterprise-class 'through the
>   network' volume management
> * more and more tunable kernel parameters to be able to have some user
>   space program which can 'tune' the system for the current load,usage,etc
>   of the server ("selftune")
> * more configuration options to be able to use Linux at the low end as well
>   (current kernels are too complex, too huge and sometimes contains too
>   many unwanted features for a simple system, though for most times it is
>   enough but can be even better)

Maybe hardware detection -> automatic kernel configuration maker


> * maybe some 'official in the kernel' general framework to implement
>   virtual machines without the need to load third party kernel modeles
>   from vmware, plex86 etc ...
> 


Regards.
-- 
  Pedro Larroy Tovar  |  piotr%member.fsf.org 

Software patents are a threat to innovation in Europe please check: 
	http://www.eurolinux.org/     

Re: 2.7 thoughts

[Greg KH]

On Thu, Oct 09, 2003 at 10:08:00AM +0200, Frederick, Fabian wrote:
> *	Meanwhile, we don't have a kobject <-> security bridge :( 

What do you mean by this?

greg k-h

Re: 2.7 thoughts

[Pavel Machek]

Hi!

> 	Some thoughts for 2.7.Someone has other ideas, comments ?
> 	

* mosix-like clustering support
* network char device for using remote soundcards etc.
* whole-system snapshot / rollback
* ACLs

				Pavel
-- 
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...