Good way to free as much memory as possible under 2.5.34?

Good way to free as much memory as possible under 2.5.34?

[Pavel Machek]

Hi!

/*
 * Try to free as much memory as possible, but do not OOM-kill anyone
 *
 * Notice: all userland should be stopped at this point, or livelock
is possible.
 */

This worked before -rmap came in, but it does not free anything
now. What needs to be done to fix it?

								Pavel 

static void free_some_memory(void)
{
        printk("Freeing memory: ");
        while
(try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
                printk(".");
        printk("|\n");
}

-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?

Re: Good way to free as much memory as possible under 2.5.34?

[Rik van Riel]

On Fri, 13 Sep 2002, Pavel Machek wrote:

> /*
>  * Try to free as much memory as possible, but do not OOM-kill anyone
>  *
>  * Notice: all userland should be stopped at this point, or livelock
> is possible.
>  */
>
> This worked before -rmap came in, but it does not free anything
> now. What needs to be done to fix it?

Actually, it still worked when -rmap came in, but it stopped working
when the LRU lists were made to be per-zone...

> static void free_some_memory(void)
> {
>         printk("Freeing memory: ");
>         while
> (try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
>                 printk(".");
>         printk("|\n");
> }

Why don't you just allocate memory ?

To prevent the OOM kill you can just check for a variable
in the OOM slow path.  No need to rely on any particular
behaviour of the VM.

regards,

Rik
-- 
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  september@surriel.com trac@trac.org

Re: Good way to free as much memory as possible under 2.5.34?

[Pavel Machek]

Hi!

> > /*
> >  * Try to free as much memory as possible, but do not OOM-kill anyone
> >  *
> >  * Notice: all userland should be stopped at this point, or livelock
> > is possible.
> >  */
> >
> > This worked before -rmap came in, but it does not free anything
> > now. What needs to be done to fix it?
> 
> Actually, it still worked when -rmap came in, but it stopped working
> when the LRU lists were made to be per-zone...
> 
> > static void free_some_memory(void)
> > {
> >         printk("Freeing memory: ");
> >         while
> > (try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
> >                 printk(".");
> >         printk("|\n");
> > }
> 
> Why don't you just allocate memory ?
> 
> To prevent the OOM kill you can just check for a variable
> in the OOM slow path.  No need to rely on any particular
> behaviour of the VM.

Allocating memory is pain because I have to free it afterwards. Yep I
have such code, but it is ugly. try_to_free_pages() really seems like
cleaner solution to me... if you only tell me how to fix it :-).

If it is not easy to fix, I can reintroduce memory eating code; but
I'd hate to special-case SWSUSP in VM. Otoh GFP_FAIL or something like
that might help me, right?
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Re: Good way to free as much memory as possible under 2.5.34?

[Andrew Morton]

Rik van Riel wrote:
> 
> On Fri, 13 Sep 2002, Pavel Machek wrote:
> 
> > /*
> >  * Try to free as much memory as possible, but do not OOM-kill anyone
> >  *
> >  * Notice: all userland should be stopped at this point, or livelock
> > is possible.
> >  */
> >
> > This worked before -rmap came in, but it does not free anything
> > now. What needs to be done to fix it?
> 
> Actually, it still worked when -rmap came in, but it stopped working
> when the LRU lists were made to be per-zone...

hmm, I missed that.  Yes, the zone balancing in try_to_free_pages() will
see that all zones are above ->pages_high and will just return.

> > static void free_some_memory(void)
> > {
> >         printk("Freeing memory: ");
> >         while
> > (try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
> >                 printk(".");
> >         printk("|\n");
> > }
> 
> Why don't you just allocate memory ?

That would work.
 
> To prevent the OOM kill you can just check for a variable
> in the OOM slow path.  No need to rely on any particular
> behaviour of the VM.
> 

Sure.

I'm not sure why swsusp needs "half of memory to be free"?  What's
the story there?

I'd recommend that you sit in a loop, allocating pages with an
allocation mode of

	__GFP_HIGHMEM | __GFP_WAIT

This will give you all the easily-reclaimable pages in the machine,
without trashing the page reserves (no __GFP_HIGH).

String all the pages together via page->list and when you have "enough",
free them all again.

Re: Good way to free as much memory as possible under 2.5.34?

[Rik van Riel]

On Fri, 13 Sep 2002, Pavel Machek wrote:

> Allocating memory is pain because I have to free it afterwards. Yep I
> have such code, but it is ugly. try_to_free_pages() really seems like
> cleaner solution to me... if you only tell me how to fix it :-).

"Fixing" the VM just so it behaves the way swsuspend wants is
out. If swsuspend relies on all other subsystems playing nicely,
I think it should be removed from the kernel.

I suspect only very few people will use swsuspend, so it should
not be intrusive.

regards,

Rik
-- 
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  september@surriel.com trac@trac.org

Re: Good way to free as much memory as possible under 2.5.34?

[Xavier Bestel]

Le ven 13/09/2002 à 23:33, Rik van Riel a écrit :

> I suspect only very few people will use swsuspend, so it should
> not be intrusive.

I don't think so.

Re: Good way to free as much memory as possible under 2.5.34?

[Andrew Morton]

Rik van Riel wrote:
> 
> On Fri, 13 Sep 2002, Pavel Machek wrote:
> 
> > Allocating memory is pain because I have to free it afterwards. Yep I
> > have such code, but it is ugly. try_to_free_pages() really seems like
> > cleaner solution to me... if you only tell me how to fix it :-).
> 
> "Fixing" the VM just so it behaves the way swsuspend wants is
> out. If swsuspend relies on all other subsystems playing nicely,
> I think it should be removed from the kernel.
> 

Yup.  Martin Bligh is cooking up a multi-page allocation API, so when that's
in place, swsusp need only do:

	LIST_HEAD(foo);
	alloc_many_pages(&foo, nr_pages, __GFP_HIGHMEM|__GFP_WAIT);
	free_many_pages(&foo);

So I suggest you do something local for the while, plan to use that later.

(Actually, the implementation would probably have a heart attack if you
asked for 100,000 pages so you may need to sit in a loop there; we'll see).

Re: Good way to free as much memory as possible under 2.5.34?

[William Lee Irwin III]

On Fri, Sep 13, 2002 at 02:53:07PM -0700, Andrew Morton wrote:
> So I suggest you do something local for the while, plan to use that later.
> (Actually, the implementation would probably have a heart attack if you
> asked for 100,000 pages so you may need to sit in a loop there; we'll see).

Actually, that's probably going to trip the NMI oopser.



Cheers,
Bill

Re: Good way to free as much memory as possible under 2.5.34?

[Rik van Riel]

On 13 Sep 2002, Xavier Bestel wrote:
> Le ven 13/09/2002 à 23:33, Rik van Riel a écrit :
>
> > I suspect only very few people will use swsuspend, so it should
> > not be intrusive.
>
> I don't think so.

You think many people will use it, or you think swsuspend
should be intrusive and have code in all other kernel
subsystems ? ;)

Rik
-- 
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  september@surriel.com trac@trac.org

Re: Good way to free as much memory as possible under 2.5.34?

[Xavier Bestel]

Le sam 14/09/2002 à 00:42, Rik van Riel a écrit :
> On 13 Sep 2002, Xavier Bestel wrote:
> > Le ven 13/09/2002 à 23:33, Rik van Riel a écrit :
> >
> > > I suspect only very few people will use swsuspend, so it should
> > > not be intrusive.
> >
> > I don't think so.
> 
> You think many people will use it, or you think swsuspend
> should be intrusive and have code in all other kernel
> subsystems ? ;)

I think people will use it. Either for laptops (when ACPI support will
match APM) or for desktop. Personnaly I'd use it for both if it were
functionnal.

Re: Good way to free as much memory as possible under 2.5.34?

[Pavel Machek]

Hi!

> > > Allocating memory is pain because I have to free it afterwards. Yep I
> > > have such code, but it is ugly. try_to_free_pages() really seems like
> > > cleaner solution to me... if you only tell me how to fix it :-).
> > 
> > "Fixing" the VM just so it behaves the way swsuspend wants is
> > out. If swsuspend relies on all other subsystems playing nicely,
> > I think it should be removed from the kernel.
> > 
> 
> Yup.  Martin Bligh is cooking up a multi-page allocation API, so when that's
> in place, swsusp need only do:
> 
> 	LIST_HEAD(foo);
> 	alloc_many_pages(&foo, nr_pages, __GFP_HIGHMEM|__GFP_WAIT);
> 	free_many_pages(&foo);
> 
> So I suggest you do something local for the while, plan to use that later.
> 
> (Actually, the implementation would probably have a heart attack if you
> asked for 100,000 pages so you may need to sit in a loop there;
> we'll see).

If nr_pages is > than number of pages really freable will it return
NULL or stall the calling process forever?

If it will return NULL I'm happy to use that... Otherwise its not too
usable because I do not want to OOM the machine.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Re: Good way to free as much memory as possible under 2.5.34?

[Andrew Morton]

Pavel Machek wrote:
> 
> Hi!
> 
> > > > Allocating memory is pain because I have to free it afterwards. Yep I
> > > > have such code, but it is ugly. try_to_free_pages() really seems like
> > > > cleaner solution to me... if you only tell me how to fix it :-).
> > >
> > > "Fixing" the VM just so it behaves the way swsuspend wants is
> > > out. If swsuspend relies on all other subsystems playing nicely,
> > > I think it should be removed from the kernel.
> > >
> >
> > Yup.  Martin Bligh is cooking up a multi-page allocation API, so when that's
> > in place, swsusp need only do:
> >
> >       LIST_HEAD(foo);
> >       alloc_many_pages(&foo, nr_pages, __GFP_HIGHMEM|__GFP_WAIT);
> >       free_many_pages(&foo);
> >
> > So I suggest you do something local for the while, plan to use that later.
> >
> > (Actually, the implementation would probably have a heart attack if you
> > asked for 100,000 pages so you may need to sit in a loop there;
> > we'll see).
> 
> If nr_pages is > than number of pages really freable will it return
> NULL or stall the calling process forever?

Good point.  In current Linus tree, it could be oom-killed.  Later,
lack of __GFP_FS will prevent that.  But it could loop forever.
I guess you'll need to drop the __GFP_WAIT and yield yourself, let
kswapd free the next batch of pages.

We need to do something more robust than this.  I'll cook something up.

Re: Good way to free as much memory as possible under 2.5.34?

[Hans-Peter Jansen]

Hi Rik,

On Friday 13 September 2002 23:33, Rik van Riel wrote:
> On Fri, 13 Sep 2002, Pavel Machek wrote:
> > Allocating memory is pain because I have to free it afterwards. Yep I
> > have such code, but it is ugly. try_to_free_pages() really seems like
> > cleaner solution to me... if you only tell me how to fix it :-).
>
> "Fixing" the VM just so it behaves the way swsuspend wants is
> out. If swsuspend relies on all other subsystems playing nicely,
> I think it should be removed from the kernel.

> I suspect only very few people will use swsuspend, so it should
> not be intrusive.

By now, it's only used by a minority (those, who get it going reliably),
but I bet, things change, when 2.6 is out. I would love and even celebrate
once swsuspend is working via nbd for my diskless setups. I consider
this as a real quantum leap (from a usability/energy saving point of view).

The question is: why is the VM not able to fulfill such a simple need in
a clean way?

> regards,
>
> Rik

Regards,
Hans-Peter

Re: Good way to free as much memory as possible under 2.5.34?

[Rik van Riel]

On Sat, 14 Sep 2002, Hans-Peter Jansen wrote:

> The question is: why is the VM not able to fulfill such a simple need in
> a clean way?

No. The question is: "why does swsuspend need to stick its fingers
into every other subsystem of the kernel, instead of trying to
remain vaguely modular ?"

If you have an answer, please let me know.

Rik
-- 
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  september@surriel.com trac@trac.org

Re: Good way to free as much memory as possible under 2.5.34?

[Pavel Machek]

Hi!

> > The question is: why is the VM not able to fulfill such a simple need in
> > a clean way?
> 
> No. The question is: "why does swsuspend need to stick its fingers
> into every other subsystem of the kernel, instead of trying to
> remain vaguely modular ?"

I'm not trying to stick swsusp fingers anywhere. It was suggested to
me to hack VM and I'm trying to avoid that.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.