What are the VM motivations??

What are the VM motivations??

[Jason McMullan]

<rant>

	I've been reading the VM thread off-and-on for, oh, the last
8 _years_ on linux-kernel. It doesn't seem that much progress gets
made in any one direction. For every throughput optimination for servers,
the desktop people yell 'interactivity'. For every 'long-disk-idle'
desire the laptop guys have, others want large buffer caches.

	It goes back and forth. Everybody pulling from all sides, and
the VM performance has stayed (mostly) in the center.

	Well, I have an idea. Let's take a page from the Neural
Network guys (not the code, just the ideas), and look at VM from a
motivational perspective.

	What if the VM were your little Tuxigachi. A little critter
that lived in your computer, handling all the memory, swap, and
cache management. What would be the positive and negative feedback
you'd give him to tell him how well he's doing VM?

	Here's a short, off-the-cuff list that hopefully most
everyone can agree on.

	Positive
	--------
		* Low system CPU load for the VM timeslice
		* Process IO requests / Disk IO is less than 1.0
		* Large idle times between disk activity
		* Process don't have to wait long for pages from VM.
		* etc.

	Negative
	--------
		* High CPU usage for VM
		* High disk IO for low number of process IO requests.
		* Disk is rarely idle
		* Processes stall for a long time waiting for VM.
		* Deadlocks (fatal!)
		* etc.

	One we know how we would 'train' our little VM critter, we 
will know how to measure its performance. Once we have measures, we
can have good benchmarks. Once we have good benchmarks - we can pick
a good VM alg. 

	Or heck, let's just make the VM a _real_ Neural Network, that
self trains itself to the load you put on the system. Hideously
complex and evil? Well, why not wire up that roach on the floor, eating
that stale cheese doodle. It can't do any worse job on VM that some of the
VM patches I've seen...

</rant>

-- 
Jason McMullan, Senior Linux Consultant
Linuxcare, Inc. 412.432.6457 tel, 412.656.3519 cell
jmcmullan@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Putting open source to work.

Re: [OT] What are the VM motivations??

[Justin Guyett]

On Thu, 21 Jun 2001, Jason McMullan wrote:

> 	One we know how we would 'train' our little VM critter, we
> will know how to measure its performance. Once we have measures, we
> can have good benchmarks. Once we have good benchmarks - we can pick
> a good VM alg.
>
> 	Or heck, let's just make the VM a _real_ Neural Network, that
> self trains itself to the load you put on the system. Hideously
> complex and evil? Well, why not wire up that roach on the floor, eating
> that stale cheese doodle. It can't do any worse job on VM that some of the
> VM patches I've seen...

I can see it now... "Care and Feeding of your linux kernel VM for Dummies"


justin

Re: What are the VM motivations??

[Rik van Riel]

On Thu, 21 Jun 2001, Jason McMullan wrote:

> 	Or heck, let's just make the VM a _real_ Neural Network,
> that self trains itself to the load you put on the system.
> Hideously complex and evil?

Considering the amount of parameters the neural network
would have to tune, and the fact that there are no easy
parameters to tune, good luck...

> the floor, eating that stale cheese doodle. It can't do any
> worse job on VM that some of the VM patches I've seen...

You'd be surprised.

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: [OT] What are the VM motivations??

[Rik van Riel]

On Thu, 21 Jun 2001, Justin Guyett wrote:

> > 	One we know how we would 'train' our little VM critter, we
> > will know how to measure its performance. Once we have measures, we
> > can have good benchmarks. Once we have good benchmarks - we can pick
> > a good VM alg.
> 
> I can see it now... "Care and Feeding of your linux kernel VM for Dummies"

Shouldn't that be "by Dummies(tm)" ?  ;)

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: What are the VM motivations??

[Marcelo Tosatti]

On Thu, 21 Jun 2001, Jason McMullan wrote:

> <rant>
> 
> 	I've been reading the VM thread off-and-on for, oh, the last
> 8 _years_ on linux-kernel. It doesn't seem that much progress gets
> made in any one direction. For every throughput optimination for servers,
> the desktop people yell 'interactivity'. For every 'long-disk-idle'
> desire the laptop guys have, others want large buffer caches.
> 
> 	It goes back and forth. Everybody pulling from all sides, and
> the VM performance has stayed (mostly) in the center.
> 
> 	Well, I have an idea. Let's take a page from the Neural
> Network guys (not the code, just the ideas), and look at VM from a
> motivational perspective.
> 
> 	What if the VM were your little Tuxigachi. A little critter
> that lived in your computer, handling all the memory, swap, and
> cache management. What would be the positive and negative feedback
> you'd give him to tell him how well he's doing VM?
> 
> 	Here's a short, off-the-cuff list that hopefully most
> everyone can agree on.
> 
> 	Positive
> 	--------
> 		* Low system CPU load for the VM timeslice
> 		* Process IO requests / Disk IO is less than 1.0
> 		* Large idle times between disk activity
> 		* Process don't have to wait long for pages from VM.
> 		* etc.
> 
> 	Negative
> 	--------
> 		* High CPU usage for VM
> 		* High disk IO for low number of process IO requests.
> 		* Disk is rarely idle
> 		* Processes stall for a long time waiting for VM.
> 		* Deadlocks (fatal!)
> 		* etc.
> 
> 	One we know how we would 'train' our little VM critter, we 
> will know how to measure its performance. Once we have measures, we
> can have good benchmarks. Once we have good benchmarks - we can pick
> a good VM alg. 

We are talking with the OSDL people (http://www.osdlab.org) to setup an
automatic testing system (differents benchmarks, different configurations,
etc) which will give us a wider notion of VM changes wrt performance.

Re: What are the VM motivations??

[watermodem]

Rik van Riel wrote:
> 
> On Thu, 21 Jun 2001, Jason McMullan wrote:
> 
> >       Or heck, let's just make the VM a _real_ Neural Network,
> > that self trains itself to the load you put on the system.
> > Hideously complex and evil?
> 
> Considering the amount of parameters the neural network
> would have to tune, and the fact that there are no easy
> parameters to tune, good luck...
> 

Would never work with the ac-series.  Not enough time
to form a neural pattern between builds.  There is
a semi-prior art here.  Unix on the Tandem (now Compaq) 
Helix shipped (and maybe still does) with a Neural Net
for system sanity and tuning.  Only problem is that 
the learning period usually exceeds the average time
between installing releases (communications customers).  
8^)

Now if it was a FAST LEARNER...


> > the floor, eating that stale cheese doodle. It can't do any
> > worse job on VM that some of the VM patches I've seen...
> 
> You'd be surprised.
> 
> Rik
> --
> Executive summary of a recent Microsoft press release:
>    "we are concerned about the GNU General Public License (GPL)"
> 
>                 http://www.surriel.com/
> http://www.conectiva.com/       http://distro.conectiva.com/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: What are the VM motivations??

[Rik van Riel]

On Thu, 21 Jun 2001, Jason McMullan wrote:

> 	One we know how we would 'train' our little VM critter, we 
> will know how to measure its performance. Once we have measures, we
> can have good benchmarks. Once we have good benchmarks - we can pick
> a good VM alg. 
> 
> 	Or heck, let's just make the VM a _real_ Neural Network, that

OK.  I challenge you to come up with:

1) the set of inputs for the neural network
2) the set of outputs
3) the goal for training the thing

I'm pretty fed up with people who want to "change the VM"
but never give any details of their ideas.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: What are the VM motivations??

[Colonel]

In clouddancer.list.kernel, you wrote:
>
>On Thu, 21 Jun 2001, Jason McMullan wrote:
>
>> 	One we know how we would 'train' our little VM critter, we 
>> will know how to measure its performance. Once we have measures, we
>> can have good benchmarks. Once we have good benchmarks - we can pick
>> a good VM alg. 
>> 
>> 	Or heck, let's just make the VM a _real_ Neural Network, that
>
>OK.  I challenge you to come up with:
>
>1) the set of inputs for the neural network
>2) the set of outputs
>3) the goal for training the thing
>
>I'm pretty fed up with people who want to "change the VM"
>but never give any details of their ideas.


It's simple.  I want the old reliable behavior back, the one I found
in kernels from 1.1.41 thru 2.2.14.  The behavior that I have enjoyed
for 7 years, the one that allows me to chuckle when listening to
windoze user's expressing their woes.  Not the one that kills
important processes because of some fantasy AI, not the one that locks
up the machine.  The one that lets me run an unattended machine for
months rather than barely make it thru a week.

What you hear now is nothing compared to the roar that will occur
shortly (as the distributions start releasing 2.4 versions) when the
masses encounter the current surprises.  They will not be able to say
in detail how to change the VM, other than the above.  We want to USE
linux and depend upon it, not become VM gurus to fix our kernels.



-- 
"Or heck, let's just make the VM a _real_ Neural Network, that self
trains itself to the load you put on the system. Hideously complex and
evil?  Well, why not wire up that roach on the floor, eating that stale
cheese doodle. It can't do any worse job on VM that some of the VM
patches I've seen..."  -- Jason McMullan

ditto

Re: What are the VM motivations??

[Rik van Riel]

On Sun, 24 Jun 2001, Colonel wrote:

> It's simple.  I want the old reliable behavior back, the one I found
> in kernels from 1.1.41 thru 2.2.14.

And which one would that be ?   Note that there have been
4 different VM subsystems in that time and the kernel has
made the transition from the buffer cache to the page cache
in that period.

Please make up your mind before making feature requests ;)

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: What are the VM motivations??

[Jason McMullan]

On Sun, Jun 24, 2001 at 12:04:43PM -0300, Rik van Riel wrote:
> OK.  I challenge you to come up with:
> 
> 1) the set of inputs for the neural network
> 2) the set of outputs
> 3) the goal for training the thing
> 
> I'm pretty fed up with people who want to "change the VM"
> but never give any details of their ideas.


	Uhh. That's not what I was ranting about. What I was
ranting about is that we have never 'put to paper' the
requirements ('motiviations') for a good VM, nor have we
looked at said nonexistent list and figured out what instrumentation
would be needed.

	I don't now that much about VM, but I do know a bunch of
people each scratching their own itch, and most of them not looking
at the bigger picture. Linus, RvR, etc. excepted. Mostly. ;^)

	The whole 'neural network' bit was mostly troll. Sorry,
I got a little carried away at that point.

-- 
Jason McMullan, Senior Linux Consultant
Linuxcare, Inc. 412.432.6457 tel, 412.656.3519 cell
jmcmullan@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Putting open source to work.

Re: What are the VM motivations??

[Rik van Riel]

On Sun, 24 Jun 2001, Jason McMullan wrote:

> 	Uhh. That's not what I was ranting about. What I was
> ranting about is that we have never 'put to paper' the
> requirements ('motiviations') for a good VM, nor have we
> looked at said nonexistent list and figured out what instrumentation
> would be needed.

But we have.  The fact that you missed the event doesn't
make it any less true.

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: What are the VM motivations??

[Jason McMullan]

On Sun, Jun 24, 2001 at 03:26:13PM -0300, Rik van Riel wrote:
> On Sun, 24 Jun 2001, Jason McMullan wrote:
> > 	Uhh. That's not what I was ranting about. What I was
> > ranting about is that we have never 'put to paper' the
> > requirements ('motiviations') for a good VM, nor have we
> > looked at said nonexistent list and figured out what instrumentation
> > would be needed.
> 
> But we have.  The fact that you missed the event doesn't
> make it any less true.

	Excellent! Where can I find this? I would be most
interested in what your considerations for a good VM are.
(It'd probably be a good thing to reference people to when
they start arguing off-the-cuff about VMs, eh?)

-- 
Jason McMullan, Senior Linux Consultant
Linuxcare, Inc. 412.432.6457 tel, 412.656.3519 cell
jmcmullan@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Putting open source to work.

Re: What are the VM motivations??

[Henning P. Schmiedehausen]

Rik van Riel <riel@conectiva.com.br> writes:

>On Sun, 24 Jun 2001, Colonel wrote:

>> It's simple.  I want the old reliable behavior back, the one I found
>> in kernels from 1.1.41 thru 2.2.14.

>And which one would that be ?   Note that there have been
>4 different VM subsystems in that time and the kernel has
>made the transition from the buffer cache to the page cache
>in that period.

I'd say, what he tries to tell you is that he does not (and I don't
for this point, either) care, which one it is or how it is implemented
or whether you're using a page, buffer or crispy chips cache, as long
as the bugger works, does not lock up, does not lose memory and does
not kill innocent processes. If you need a roach to wire to the
computer to do it, fine, tell me how to wire it and I'll start
lobbying mainboard suppliers to provide six pin sockets for roach
plugging.

Just as all VMs up to 2.2.19 do (with a few notable exceptions around
the 2.2.14-2.2.16 range). These VM problems are my biggest stop sign
to move my production boxes to 2.4.x. My 2.2.x boxes have uptimes in
the hundreds of days, my 2.0.x boxes do, too:

henning@db1 21:05 ~ > uptime
  9:05pm  up 410 days, 17:12,  1 user,  load average: 0.15, 0.03, 0.01
henning@db1 21:05 ~ > uname -an
Linux db1 2.0.37 #1 Sat Mar 13 19:41:01 MET 1999 i686 unknown

(Heck if that USV in front of the Cobalt Qube wouldn't have died, I'd
have another 440+ days uptime to boast with.)

That's the stability most of us want to see in the _stable_ kernel
series. And I fully agree with the fact that I don't want to become a
VM kernel expert just to run the bugger. =:-)

Some of us like Linux not because it's free but because it's rock
solid. Because some of us use them to run our businesses on it. If I
want uptimes in the days range, I'd use Win2k and hire a monkey to
administrate the box, because it has a nice GUI (TM).

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: What are the VM motivations??

[Rik van Riel]

On Sun, 24 Jun 2001, Jason McMullan wrote:
> On Sun, Jun 24, 2001 at 03:26:13PM -0300, Rik van Riel wrote:
> > On Sun, 24 Jun 2001, Jason McMullan wrote:
> > > 	Uhh. That's not what I was ranting about. What I was
> > > ranting about is that we have never 'put to paper' the
> > > requirements ('motiviations') for a good VM, nor have we
> > > looked at said nonexistent list and figured out what instrumentation
> > > would be needed.
> >
> > But we have.  The fact that you missed the event doesn't
> > make it any less true.
>
> 	Excellent! Where can I find this? I would be most
> interested in what your considerations for a good VM are.
> (It'd probably be a good thing to reference people to when
> they start arguing off-the-cuff about VMs, eh?)

Over the last year there has been quite a bit of discussion
with Stephen Tweedie, Matt Dillon and more people. Parts of
it can be found on http://linux-mm.org/

The conclusion of most of this discussion is in my FREENIX
paper, which can be found at http://www.surriel.com/lectures/.


regards,

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: What are the VM motivations??

[Jason McMullan]

On Sun, Jun 24, 2001 at 04:29:09PM -0300, Rik van Riel wrote:
> Over the last year there has been quite a bit of discussion
> with Stephen Tweedie, Matt Dillon and more people. Parts of
> it can be found on http://linux-mm.org/
> 
> The conclusion of most of this discussion is in my FREENIX
> paper, which can be found at http://www.surriel.com/lectures/.

	[Just finished reading your paper, and hit the linux-mm.org site]

	Good overview of the Linux 2.4 VM. Although I do
have some questions...

	* When was the 'FREENIX' paper published? I could find
	  no date for it.

	* What workloads would you recommend for testing whether
	  a VM is 'well balanced' or not?

	* Would it be reasonable to have different 'default'
	  tunings for the kernel's VM based upon memory/swap
	  size/bandwidth? Personally, I feel that swap bandwidth
	  is an oft overlooked parameter in estimating VM performance.

-- 
Jason McMullan, Senior Linux Consultant
Linuxcare, Inc. 412.432.6457 tel, 412.656.3519 cell
jmcmullan@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Putting open source to work.

Re: What are the VM motivations??

[Stephen Satchell]

At 03:26 PM 6/24/01 -0300, Rik van Riel wrote:
>On Sun, 24 Jun 2001, Jason McMullan wrote:
>
> >       Uhh. That's not what I was ranting about. What I was
> > ranting about is that we have never 'put to paper' the
> > requirements ('motiviations') for a good VM, nor have we
> > looked at said nonexistent list and figured out what instrumentation
> > would be needed.
>
>But we have.  The fact that you missed the event doesn't
>make it any less true.

URL, please?

If the requirements/motivations were "put on paper" getting them on the Web 
is not a big deal.  If it's *only* on paper, I can give you my fax number 
and I'll be happy to put it up on the Web.

Satch

Re: What are the VM motivations??

[Jonathan Morton]

>The conclusion of most of this discussion is in my FREENIX
>paper, which can be found at http://www.surriel.com/lectures/.

Aha...  that paper answers a lot of the questions I had about how 
things work.  I seem to remember asking some of them, too, and didn't 
get an answer...  :P
-- 
--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)
website:  http://www.chromatix.uklinux.net/vnc/
geekcode: GCS$/E dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$
           V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)
tagline:  The key to knowledge is not to rely on people to teach you it.

Re: What are the VM motivations??

[Rik van Riel]

On Sun, 24 Jun 2001, Jason McMullan wrote:
> On Sun, Jun 24, 2001 at 04:29:09PM -0300, Rik van Riel wrote:
> > Over the last year there has been quite a bit of discussion
> > with Stephen Tweedie, Matt Dillon and more people. Parts of
> > it can be found on http://linux-mm.org/
> >
> > The conclusion of most of this discussion is in my FREENIX
> > paper, which can be found at http://www.surriel.com/lectures/.
>
> 	[Just finished reading your paper, and hit the linux-mm.org site]
>
> 	Good overview of the Linux 2.4 VM. Although I do
> have some questions...
>
> 	* When was the 'FREENIX' paper published? I could find
> 	  no date for it.

Next week.  Shhhhh... ;)

> 	* What workloads would you recommend for testing whether
> 	  a VM is 'well balanced' or not?

Whatever it is you always do. We're not interested in seeing
if the VM works well in unrealistic benchmarks, we want it to
run well in your normal workload.

> 	* Would it be reasonable to have different 'default'
> 	  tunings for the kernel's VM based upon memory/swap
> 	  size/bandwidth? Personally, I feel that swap bandwidth
> 	  is an oft overlooked parameter in estimating VM performance.

Maybe. Suggestions and patches on what to tune and how are
welcome. Extremely general handwaving generally doesn't add
much value to the discussion.

regards,

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: What are the VM motivations??

[Russell Leighton]

I read this thread as asking the question:

    If VM management is viewed as an optimization problem,
    then what exactly is the function that you are optimizing and what are the constraints?

If you could express that well with a even a very loose model, then
the code could be reviewed to see if it was really doing what was intended
and assumptions could be tested.

This seems like a reasonable request.

Jason McMullan wrote:

> On Sun, Jun 24, 2001 at 12:04:43PM -0300, Rik van Riel wrote:
> > OK.  I challenge you to come up with:
> >
> > 1) the set of inputs for the neural network
> > 2) the set of outputs
> > 3) the goal for training the thing
> >
> > I'm pretty fed up with people who want to "change the VM"
> > but never give any details of their ideas.
>
>         Uhh. That's not what I was ranting about. What I was
> ranting about is that we have never 'put to paper' the
> requirements ('motiviations') for a good VM, nor have we
> looked at said nonexistent list and figured out what instrumentation
> would be needed.
>
>         I don't now that much about VM, but I do know a bunch of
> people each scratching their own itch, and most of them not looking
> at the bigger picture. Linus, RvR, etc. excepted. Mostly. ;^)
>
>         The whole 'neural network' bit was mostly troll. Sorry,
> I got a little carried away at that point.
>
> --
> Jason McMullan, Senior Linux Consultant
> Linuxcare, Inc. 412.432.6457 tel, 412.656.3519 cell
> jmcmullan@linuxcare.com, http://www.linuxcare.com/
> Linuxcare. Putting open source to work.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

--
---------------------------------------------------
Russell Leighton    russell.leighton@247media.com
---------------------------------------------------

Re: What are the VM motivations??

[Daniel Phillips]

On Monday 25 June 2001 03:46, Russell Leighton wrote:
> I read this thread as asking the question:
>
>     If VM management is viewed as an optimization problem,
>     then what exactly is the function that you are optimizing and what are
> the constraints?
>
> If you could express that well with a even a very loose model, then
> the code could be reviewed to see if it was really doing what was intended
> and assumptions could be tested.

May I suggested an algorithm?

  - Write down what you think the optimization constraints are.
    (be specific, for example, enumerate all the flavors of page types -
    process code, process data, page cache file data, page cache swap
    cache, anonymous, shmem, etc.)

  - Write down what you think the current algorithms are.
    (again, be specific, use file names, function names, pseudocode and 
    snippets of existing code.)

  - Send it to Rik.  He'll tell you if it's right.

  - Because life is short and should not be boring, cc it to Andrea as well
    ;-)

This way you save everybody's time and ensure you understand what's really 
going on.  Not to mention coming up with a document worth its weight in 
dilithium crystals.

--
Daniel

Re: What are the VM motivations??

[Colonel]

In clouddancer.list.kernel, you wrote:
>
>On Sun, 24 Jun 2001, Colonel wrote:
>
>> It's simple.  I want the old reliable behavior back, the one I found
>> in kernels from 1.1.41 thru 2.2.14.
>
>And which one would that be ?   Note that there have been
>4 different VM subsystems in that time and the kernel has
>made the transition from the buffer cache to the page cache
>in that period.

Once again a typical Rik comment.  Completely misses the point and
picks something obscure to comment upon, after carefully removing the
remaining text.  PICK ONE!


>Please make up your mind before making feature requests ;)

Try OPENING yours.  There really is a world beyond your nose.


-- 
"Or heck, let's just make the VM a _real_ Neural Network, that self
trains itself to the load you put on the system. Hideously complex and
evil?  Well, why not wire up that roach on the floor, eating that stale
cheese doodle. It can't do any worse job on VM that some of the VM
patches I've seen..."  -- Jason McMullan

ditto

Re: What are the VM motivations??

[Colonel]

In clouddancer.list.kernel, you wrote:
>
>On Monday 25 June 2001 03:46, Russell Leighton wrote:
>> I read this thread as asking the question:
>>
>>     If VM management is viewed as an optimization problem,
>>     then what exactly is the function that you are optimizing and what are
>> the constraints?
>>
>> If you could express that well with a even a very loose model, then
>> the code could be reviewed to see if it was really doing what was intended
>> and assumptions could be tested.
>
>May I suggested an algorithm?
>
>  - Write down what you think the optimization constraints are.
>    (be specific, for example, enumerate all the flavors of page types -
>    process code, process data, page cache file data, page cache swap
>    cache, anonymous, shmem, etc.)
>
>  - Write down what you think the current algorithms are.
>    (again, be specific, use file names, function names, pseudocode and 
>    snippets of existing code.)
>
>  - Send it to Rik.  He'll tell you if it's right.

POST IT.  Give the rest of us some clues and the opportunity to check
evaluator's replies.

Re: What are the VM motivations??

[Daniel Phillips]

On Monday 25 June 2001 05:44, Colonel wrote:
> In clouddancer.list.kernel, you wrote:
> >On Monday 25 June 2001 03:46, Russell Leighton wrote:
> >> I read this thread as asking the question:
> >>
> >>     If VM management is viewed as an optimization problem,
> >>     then what exactly is the function that you are optimizing and what
> >> are the constraints?
> >>
> >> If you could express that well with a even a very loose model, then
> >> the code could be reviewed to see if it was really doing what was
> >> intended and assumptions could be tested.
> >
> >May I suggest an algorithm?
> >
> >  - Write down what you think the optimization constraints are.
> >    (be specific, for example, enumerate all the flavors of page types -
> >    process code, process data, page cache file data, page cache swap
> >    cache, anonymous, shmem, etc.)
> >
> >  - Write down what you think the current algorithms are.
> >    (again, be specific, use file names, function names, pseudocode and
> >    snippets of existing code.)
> >
> >  - Send it to Rik.  He'll tell you if it's right.
>
> POST IT.  Give the rest of us some clues and the opportunity to check
> evaluator's replies.

Well, if you try that strategy you'll find you never get around to posting it 
because you'll be too worried about getting it right.  The point is not to 
get it right, it's to get a starting point down on (virtual) paper.  I'd 
strongly suggest passing something like that around for comment privately 
first.

*Then* post it.

(and convince me you're not just talking)

--
Daniel

Re: What are the VM motivations??

[Colonel]

>> POST IT.  Give the rest of us some clues and the opportunity to check
>> evaluator's replies.
>
>Well, if you try that strategy you'll find you never get around to posting it 
>because you'll be too worried about getting it right.  The point is not to 
>get it right, it's to get a starting point down on (virtual) paper.  I'd 
>strongly suggest passing something like that around for comment privately 
>first.

Well, it does seem that some ego is involved here...

>
>*Then* post it.
>
>(and convince me you're not just talking)

As a physicist, I learned long ago that if you cannot use the back of
an envelope and explain your theory to a secretary, you do not
understand it.  It has only been revealed recently where some VM
design information may be hidden, and it's apparently out of date.
Maintenance requires good documentation, especially in critical areas.
I hope that your suggestion succeeds, and that a document is passed
around, but I'll bet a large amount that it doesn't happen.

If everyone is in the dark, it gets difficult to criticize the guy
with a match and leaves the matchholder in charge.  Some matchholders
want it that way.

Re: What are the VM motivations??

[Daniel Phillips]

On Monday 25 June 2001 17:46, Colonel wrote:
> >> POST IT.  Give the rest of us some clues and the opportunity to check
> >> evaluator's replies.
> >
> >Well, if you try that strategy you'll find you never get around to posting
> > it because you'll be too worried about getting it right.  The point is
> > not to get it right, it's to get a starting point down on (virtual)
> > paper.  I'd strongly suggest passing something like that around for
> > comment privately first.
>
> Well, it does seem that some ego is involved here...

Not in this case.  Do it whichever way you want, the point is: do it (don't 
just talk about it)

--
Daniel

Re: What are the VM motivations??

[Rik van Riel]

On Sun, 24 Jun 2001, Colonel wrote:

> >May I suggested an algorithm?
> >
> >  - Write down what you think the optimization constraints are.
> >    (be specific, for example, enumerate all the flavors of page types -
> >    process code, process data, page cache file data, page cache swap
> >    cache, anonymous, shmem, etc.)
> >
> >  - Write down what you think the current algorithms are.
> >    (again, be specific, use file names, function names, pseudocode and 
> >    snippets of existing code.)
> >
> >  - Send it to Rik.  He'll tell you if it's right.
> 
> POST IT.  Give the rest of us some clues and the opportunity to check
> evaluator's replies.

Why don't YOU post it?

All I've seen from you are complaints and insults, nothing
constructive yet.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: What are the VM motivations??

[Rik van Riel]

On Mon, 25 Jun 2001, Colonel wrote:

> As a physicist, I learned long ago that if you cannot use the back of
> an envelope and explain your theory to a secretary, you do not
> understand it.  It has only been revealed recently where some VM
> design information may be hidden, and it's apparently out of date.
> Maintenance requires good documentation, especially in critical areas.
> I hope that your suggestion succeeds, and that a document is passed
> around, but I'll bet a large amount that it doesn't happen.

Your contributions are welcome. I'm looking forward to your
updated documentation and patches.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: What are the VM motivations??

[Mark H. Wood]

I'm hopelessly behind on my mail, so this has probably been dealt with,
already, but here goes:  It's a userspace problem.

That is, any automagical VM tuner ought to be a daemon.  If the kernel
doesn't expose enough information or knobs to make a good VM tuner, add
what is needed.

Meanwhile, if anyone has a good handle on how to tune for interactivity
vs. power saving vs. server performance, it seems that a lot of sysadmin.s
need a good reference *from the sysadmin point of view* on how to tune
manually for different sorts of loads.  If the kernel is tunable in ways
that are understandable, it'll be fairly easy to tune by hand and the
daemon may not be needed at all.  And again, we have a userspace solution
that doesn't add much weight to the kernel.

The original post does make a valuable point:  resource allocation is a
problem with *no general solution*, because the goodness of a solution
depends on the values of the person applying it.

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
Make a good day.

Re: What are the VM motivations??

[Mark H. Wood]

On Sat, 23 Jun 2001, watermodem wrote:
[snip interior quotes]
> Would never work with the ac-series.  Not enough time
> to form a neural pattern between builds.  There is
> a semi-prior art here.  Unix on the Tandem (now Compaq)
> Helix shipped (and maybe still does) with a Neural Net
> for system sanity and tuning.  Only problem is that
> the learning period usually exceeds the average time
> between installing releases (communications customers).
> 8^)

For a less ambitious approach that worked fairly well for a lot of sites
(including mine) see also VMS Autogen.  It was nice, having a mechanical
"expert" looking over my shoulder, making suggestions, double-checking my
ideas, etc.  One run a week was usually enough to tune a system pretty
well, for a general timesharing/batch/modest services load.

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
Make a good day.