[NEW]: Introducing shrink_all_memory from user space

[NEW]: Introducing shrink_all_memory from user space

[PINTU KUMAR]

Dear All,

This is regarding a small proposal for shrink_all_memory( ) function which is found in mm/vmscan.c.
For those who are not aware, this function helps in reclaiming specified amount of physical memory and returns number of freed pages.

Currently this function is under CONFIG_HIBERNATION flag, so cannot be used by others without enabling hibernation.
Moreover this function is not exported to the outside world, so no driver can use it directly without including EXPORT_SYMBOL(shrink_all_memory) and recompiling the kernel.
The purpose of using it under hibernation(kernel/power/snapshot.c) is to regain enough physical pages to create hibernation image.

The same can be useful for some drivers who wants to allocate large contiguous memory (in MBs) but his/her system is in very bad state and could not do so without rebooting.
Due to this bad state of the system the user space application will be badly hurt on performance, and there could be a need to quickly reclaim all physical memory from user space.
This could be very helpful for small embedded products and smart phones where rebooting is never a preferred choice.

With this support in kernel, a small utility can be developed in user space which user can run and reclaim as many physical pages and noticed that his system performance is increased without rebooting.

To make this work, I have performed a sample experiment on my ubuntu(10.x) machine running with kernel-3.3.0. 

Also I performed the similar experiment of one of our Samsung smart phones as well.

Following are the results from Ubuntu:

1) I downloaded kernel3.3.0 and made the respective changes in mm/vmscan.c. That is added EXPORT_SYMBOL(shrink_all_memory) under the function shrink_all_memory( ).
    (Note: CONFIG_HIBERNATION was already enabled for my system.)

2) Then I build the kernel and installed it on my Ubuntu PC.

3) After that I have developed a small kernel module (miscdevice: /dev/shrinkmem) to call shrink_all_memory( ) under my driver write( ) function.

4) Then from user space I just need to do this: 

    # echo 100 > /dev/shrinkmem   (To reclaim 100MB of physical memory without reboot)


The results that were obtained with this experiment is as follows:

1) At some point of time, the buddyinfo and free pages on my Ubuntu were as follows:
root@pintu-ubuntu:~/PintuHomeTest/KERNEL_ORG# cat /proc/buddyinfo ; free -tmNode 0, zone      DMA    468     23      0      0      0      0      0      0      0      0      0
Node 0, zone   Normal    898    161     38      8      0      0      0      0      0      0      0
                 total       used       free     shared    buffers     cached
Mem:           497        489          7          0         37        405
-/+ buffers/cache:         47        449
Swap:         1458        158       1300
Total:        1956        648       1308


2) After doing "echo 100 > /dev/shrinkmem" :
[19653.833916] [SHRINKMEM]: memsize(in MB) = 100
[19653.863618] <SHRINKMEM>: Number of Pages Freed: 26756
[19653.863664] [SHRINKMEM] : Device is Closed....

Node 0, zone      DMA    411    166     51      5      0      0      0      0      0      0      0
Node 0, zone   Normal   2915   3792   2475   1335    730     23      0      0      0      0      0
                 total       used       free     shared    buffers     cached
Mem:           497        323        173          0         37        238
-/+ buffers/cache:         47        449
Swap:         1458        158       1300
Total:        1956        481       1474


3) After running again with : echo 512 > /dev/shrinkmem
[21961.064534] [SHRINKMEM]: memsize(in MB) = 512
[21961.109497] <SHRINKMEM>: Number of Pages Freed: 61078
[21961.109562] [SHRINKMEM] : Device is Closed....
Node 0, zone      DMA    116     99     87     58     16      6      1      0      0      0      0
Node 0, zone   Normal   6697   6939   5529   3756   1442    203     17      0      0      0      0
                 total       used       free     shared    buffers     cached
Mem:           497         87        410          0         37          9
-/+ buffers/cache:         40        456
Swap:         1458        158       1300
Total:        1956        245       1711


4) Thus in both the cases huge number of free pages were reclaimed. 

5) After running this on my system, the performance was improved quickly.

6) I performed the same experiment on our Samsung Smart phones as well. And I have seen a drastic improve in performance after running this for 3/4 times.
    In case of phones it is more helpful as there is no swap space.

7) Your feedback and suggestion is important. Based on the feedback, I can plan to submit the patches officially after performing basic cleanups.


Future Work
==========
Our final goal is to use it during lowmem notifier where shrink_all_memory will be called automatically in background if the memory pressure falls below certain limit defined by the system.
For example we can measure the percentage memory fragmentation level of the system across each zone and if the fragmentation percentage goes above say 80-85% during lowmem notifier, we can invoke shrink_all_memory() in background.

This can be used by some application which requires large mmap or shared memory mapping.

This can be even using inside the multimedia drivers that requires large contiguous memory to check if that many memory pages can be reclaimed or not.


Please provide your valuable feedback and suggestion.


Thank you very much !


With Regards,
Pintu Kumar

Re: [NEW]: Introducing shrink_all_memory from user space

[santosh]

Can you please post your code here ?


--Santosh

On Sun, Apr 15, 2012 at 3:17 PM, PINTU KUMAR <pintu_agarwal@yahoo.com> wrote:
> Dear All,
>
> This is regarding a small proposal for shrink_all_memory( ) function which is found in mm/vmscan.c.
> For those who are not aware, this function helps in reclaiming specified amount of physical memory and returns number of freed pages.
>
> Currently this function is under CONFIG_HIBERNATION flag, so cannot be used by others without enabling hibernation.
> Moreover this function is not exported to the outside world, so no driver can use it directly without including EXPORT_SYMBOL(shrink_all_memory) and recompiling the kernel.
> The purpose of using it under hibernation(kernel/power/snapshot.c) is to regain enough physical pages to create hibernation image.
>
> The same can be useful for some drivers who wants to allocate large contiguous memory (in MBs) but his/her system is in very bad state and could not do so without rebooting.
> Due to this bad state of the system the user space application will be badly hurt on performance, and there could be a need to quickly reclaim all physical memory from user space.
> This could be very helpful for small embedded products and smart phones where rebooting is never a preferred choice.
>
> With this support in kernel, a small utility can be developed in user space which user can run and reclaim as many physical pages and noticed that his system performance is increased without rebooting.
>
> To make this work, I have performed a sample experiment on my ubuntu(10.x) machine running with kernel-3.3.0.
>
> Also I performed the similar experiment of one of our Samsung smart phones as well.
>
> Following are the results from Ubuntu:
>
> 1) I downloaded kernel3.3.0 and made the respective changes in mm/vmscan.c. That is added EXPORT_SYMBOL(shrink_all_memory) under the function shrink_all_memory( ).
>     (Note: CONFIG_HIBERNATION was already enabled for my system.)
>
> 2) Then I build the kernel and installed it on my Ubuntu PC.
>
> 3) After that I have developed a small kernel module (miscdevice: /dev/shrinkmem) to call shrink_all_memory( ) under my driver write( ) function.
>
> 4) Then from user space I just need to do this:
>
>     # echo 100 > /dev/shrinkmem   (To reclaim 100MB of physical memory without reboot)
>
>
> The results that were obtained with this experiment is as follows:
>
> 1) At some point of time, the buddyinfo and free pages on my Ubuntu were as follows:
> root@pintu-ubuntu:~/PintuHomeTest/KERNEL_ORG# cat /proc/buddyinfo ; free -tmNode 0, zone      DMA    468     23      0      0      0      0      0      0      0      0      0
> Node 0, zone   Normal    898    161     38      8      0      0      0      0      0      0      0
>                  total       used       free     shared    buffers     cached
> Mem:           497        489          7          0         37        405
> -/+ buffers/cache:         47        449
> Swap:         1458        158       1300
> Total:        1956        648       1308
>
>
> 2) After doing "echo 100 > /dev/shrinkmem" :
> [19653.833916] [SHRINKMEM]: memsize(in MB) = 100
> [19653.863618] <SHRINKMEM>: Number of Pages Freed: 26756
> [19653.863664] [SHRINKMEM] : Device is Closed....
>
> Node 0, zone      DMA    411    166     51      5      0      0      0      0      0      0      0
> Node 0, zone   Normal   2915   3792   2475   1335    730     23      0      0      0      0      0
>                  total       used       free     shared    buffers     cached
> Mem:           497        323        173          0         37        238
> -/+ buffers/cache:         47        449
> Swap:         1458        158       1300
> Total:        1956        481       1474
>
>
> 3) After running again with : echo 512 > /dev/shrinkmem
> [21961.064534] [SHRINKMEM]: memsize(in MB) = 512
> [21961.109497] <SHRINKMEM>: Number of Pages Freed: 61078
> [21961.109562] [SHRINKMEM] : Device is Closed....
> Node 0, zone      DMA    116     99     87     58     16      6      1      0      0      0      0
> Node 0, zone   Normal   6697   6939   5529   3756   1442    203     17      0      0      0      0
>                  total       used       free     shared    buffers     cached
> Mem:           497         87        410          0         37          9
> -/+ buffers/cache:         40        456
> Swap:         1458        158       1300
> Total:        1956        245       1711
>
>
> 4) Thus in both the cases huge number of free pages were reclaimed.
>
> 5) After running this on my system, the performance was improved quickly.
>
> 6) I performed the same experiment on our Samsung Smart phones as well. And I have seen a drastic improve in performance after running this for 3/4 times.
>     In case of phones it is more helpful as there is no swap space.
>
> 7) Your feedback and suggestion is important. Based on the feedback, I can plan to submit the patches officially after performing basic cleanups.
>
>
> Future Work
> ==========
> Our final goal is to use it during lowmem notifier where shrink_all_memory will be called automatically in background if the memory pressure falls below certain limit defined by the system.
> For example we can measure the percentage memory fragmentation level of the system across each zone and if the fragmentation percentage goes above say 80-85% during lowmem notifier, we can invoke shrink_all_memory() in background.
>
> This can be used by some application which requires large mmap or shared memory mapping.
>
> This can be even using inside the multimedia drivers that requires large contiguous memory to check if that many memory pages can be reclaimed or not.
>
>
> Please provide your valuable feedback and suggestion.
>
>
> Thank you very much !
>
>
> With Regards,
> Pintu Kumar
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [NEW]: Introducing shrink_all_memory from user space

[richard -rw- weinberger]

On Sun, Apr 15, 2012 at 11:47 AM, PINTU KUMAR <pintu_agarwal@yahoo.com> wrote:
> Please provide your valuable feedback and suggestion.

This is fundamentally flawed.
You're assuming that only one program will use this interface.
Linux is a multi/user-tasking system

If we expose it to user space *every* program/user will try too free
memory such that it
can use more.
Can you see the problem?

-- 
Thanks,
//richard

Re: [NEW]: Introducing shrink_all_memory from user space

[PINTU KUMAR]

> 
> This is fundamentally flawed.
> You're assuming that only one program will use this interface.
> Linux is a multi/user-tasking system

Dear Richard, Thank you for your comments.
I am sorry but this is meant only for *root* user.
That is only root user can do this: "echo 512 > /dev/shrinkmem"
If other user try to write in it: it says permission denied as follows:
pintu@pintu-ubuntu:~$ echo 512 > /dev/shrinkmem
-bash: /dev/shrinkmem: Permission denied

Moreover, this is mainly meant for mobile phones where there is only *one* user.

> 
> If we expose it to user space *every* program/user will try too free
> memory such that it
> can use more.
> Can you see the problem?
> 
As indicated above, every program/user cannot use it, as it requires root privileges.
Ok, you mean to say, every driver can call "shrink_all_memory" simultaneously??
Well, we can implement locking for that.
Anyways, I wrote a simple script to do this (echo 512 > /dev/shrinkmem) in a loop for 20 times from 2 different terminal (as root) and it works.
I cannot see any problem.
Can you elaborate more please?


Thanks,
Pintu

Re: [NEW]: Introducing shrink_all_memory from user space

[richard -rw- weinberger]

On Sun, Apr 15, 2012 at 1:47 PM, PINTU KUMAR <pintu_agarwal@yahoo.com> wrote:
> Moreover, this is mainly meant for mobile phones where there is only *one* user.

I see. Jet another awful hack.
Mobile phones are nothing special. They are computers.

>>
>> If we expose it to user space *every* program/user will try too free
>> memory such that it
>> can use more.
>> Can you see the problem?
>>
> As indicated above, every program/user cannot use it, as it requires root privileges.
> Ok, you mean to say, every driver can call "shrink_all_memory" simultaneously??
> Well, we can implement locking for that.
> Anyways, I wrote a simple script to do this (echo 512 > /dev/shrinkmem) in a loop for 20 times from 2 different terminal (as root) and it works.
> I cannot see any problem.

Every program which is allowed to use this interface will (ab)use it.
Anyway, by exposing this interface to user space (or kernel modules)
you'll confuse the VM system.

-- 
Thanks,
//richard

Re: [NEW]: Introducing shrink_all_memory from user space

[PINTU KUMAR]

> From: richard -rw- weinberger <richard.weinberger@gmail.com>
> Sent: Sunday, 15 April 2012 5:40 PM

> Every program which is allowed to use this interface will (ab)use it.
> Anyway, by exposing this interface to user space (or kernel modules)
> you'll confuse the VM system.

Dear Richard, thank you very much for all your feedbacks.
I can see that you forsee many problems in doing that (even as root) 
and in just one program as part of system utility.
I am sorry but if you can highlight few problems here it will be
helpful for me(and others) to understand.
Also I wanted to understand, why it is meant only for Hibernation?

Thank you so much !


Regards,
Pintu

Re: [NEW]: Introducing shrink_all_memory from user space

[Ying Han]

On Sun, Apr 15, 2012 at 2:47 AM, PINTU KUMAR <pintu_agarwal@yahoo.com> wrote:
> Dear All,
>
> This is regarding a small proposal for shrink_all_memory( ) function which is found in mm/vmscan.c.
> For those who are not aware, this function helps in reclaiming specified amount of physical memory and returns number of freed pages.
>
> Currently this function is under CONFIG_HIBERNATION flag, so cannot be used by others without enabling hibernation.
> Moreover this function is not exported to the outside world, so no driver can use it directly without including EXPORT_SYMBOL(shrink_all_memory) and recompiling the kernel.
> The purpose of using it under hibernation(kernel/power/snapshot.c) is to regain enough physical pages to create hibernation image.
>
> The same can be useful for some drivers who wants to allocate large contiguous memory (in MBs) but his/her system is in very bad state and could not do so without rebooting.
> Due to this bad state of the system the user space application will be badly hurt on performance, and there could be a need to quickly reclaim all physical memory from user space.
> This could be very helpful for small embedded products and smart phones where rebooting is never a preferred choice.
>
> With this support in kernel, a small utility can be developed in user space which user can run and reclaim as many physical pages and noticed that his system performance is increased without rebooting.
>
> To make this work, I have performed a sample experiment on my ubuntu(10.x) machine running with kernel-3.3.0.
>
> Also I performed the similar experiment of one of our Samsung smart phones as well.
>
> Following are the results from Ubuntu:
>
> 1) I downloaded kernel3.3.0 and made the respective changes in mm/vmscan.c. That is added EXPORT_SYMBOL(shrink_all_memory) under the function shrink_all_memory( ).
>     (Note: CONFIG_HIBERNATION was already enabled for my system.)
>
> 2) Then I build the kernel and installed it on my Ubuntu PC.
>
> 3) After that I have developed a small kernel module (miscdevice: /dev/shrinkmem) to call shrink_all_memory( ) under my driver write( ) function.
>
> 4) Then from user space I just need to do this:
>
>     # echo 100 > /dev/shrinkmem   (To reclaim 100MB of physical memory without reboot)
>
>
> The results that were obtained with this experiment is as follows:
>
> 1) At some point of time, the buddyinfo and free pages on my Ubuntu were as follows:
> root@pintu-ubuntu:~/PintuHomeTest/KERNEL_ORG# cat /proc/buddyinfo ; free -tmNode 0, zone      DMA    468     23      0      0      0      0      0      0      0      0      0
> Node 0, zone   Normal    898    161     38      8      0      0      0      0      0      0      0
>                  total       used       free     shared    buffers     cached
> Mem:           497        489          7          0         37        405
> -/+ buffers/cache:         47        449
> Swap:         1458        158       1300
> Total:        1956        648       1308
>
>
> 2) After doing "echo 100 > /dev/shrinkmem" :
> [19653.833916] [SHRINKMEM]: memsize(in MB) = 100
> [19653.863618] <SHRINKMEM>: Number of Pages Freed: 26756
> [19653.863664] [SHRINKMEM] : Device is Closed....
>
> Node 0, zone      DMA    411    166     51      5      0      0      0      0      0      0      0
> Node 0, zone   Normal   2915   3792   2475   1335    730     23      0      0      0      0      0
>                  total       used       free     shared    buffers     cached
> Mem:           497        323        173          0         37        238
> -/+ buffers/cache:         47        449
> Swap:         1458        158       1300
> Total:        1956        481       1474
>
>
> 3) After running again with : echo 512 > /dev/shrinkmem
> [21961.064534] [SHRINKMEM]: memsize(in MB) = 512
> [21961.109497] <SHRINKMEM>: Number of Pages Freed: 61078
> [21961.109562] [SHRINKMEM] : Device is Closed....
> Node 0, zone      DMA    116     99     87     58     16      6      1      0      0      0      0
> Node 0, zone   Normal   6697   6939   5529   3756   1442    203     17      0      0      0      0
>                  total       used       free     shared    buffers     cached
> Mem:           497         87        410          0         37          9
> -/+ buffers/cache:         40        456
> Swap:         1458        158       1300
> Total:        1956        245       1711
>
>
> 4) Thus in both the cases huge number of free pages were reclaimed.
>
> 5) After running this on my system, the performance was improved quickly.
>
> 6) I performed the same experiment on our Samsung Smart phones as well. And I have seen a drastic improve in performance after running this for 3/4 times.
>     In case of phones it is more helpful as there is no swap space.
>
> 7) Your feedback and suggestion is important. Based on the feedback, I can plan to submit the patches officially after performing basic cleanups.
>
>
> Future Work
> ==========
> Our final goal is to use it during lowmem notifier where shrink_all_memory will be called automatically in background if the memory pressure falls below certain limit defined by the system.
> For example we can measure the percentage memory fragmentation level of the system across each zone and if the fragmentation percentage goes above say 80-85% during lowmem notifier, we can invoke shrink_all_memory() in background.

Does it make sense to let kswapd reclaim pages at background w/ user
configured watermark?

--Ying

>
> This can be used by some application which requires large mmap or shared memory mapping.
>
> This can be even using inside the multimedia drivers that requires large contiguous memory to check if that many memory pages can be reclaimed or not.
>
>
> Please provide your valuable feedback and suggestion.
>
>
> Thank you very much !
>
>
> With Regards,
> Pintu Kumar
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
> Don't email: <a hrefmailto:"dont@kvack.org"> email@kvack.org </a>

Re: [NEW]: Introducing shrink_all_memory from user space

[KOSAKI Motohiro]

(4/15/12 5:47 AM), PINTU KUMAR wrote:
> Dear All,
>
> This is regarding a small proposal for shrink_all_memory( ) function which is found in mm/vmscan.c.
> For those who are not aware, this function helps in reclaiming specified amount of physical memory and returns number of freed pages.
>
> Currently this function is under CONFIG_HIBERNATION flag, so cannot be used by others without enabling hibernation.
> Moreover this function is not exported to the outside world, so no driver can use it directly without including EXPORT_SYMBOL(shrink_all_memory) and recompiling the kernel.
> The purpose of using it under hibernation(kernel/power/snapshot.c) is to regain enough physical pages to create hibernation image.

This is intended. current shrink_all_memory() is not designed for generic purpose. It doesn't care numa affinity etc..
In future, we may remove this function completely because actually hibernation don't depend on it. it only help to
improvement hibernation speed-up a bit.

Re: [NEW]: Introducing shrink_all_memory from user space

[Alan Cox]

> 5) After running this on my system, the performance was improved quickly.
> 
> 6) I performed the same experiment on our Samsung Smart phones as well. And I have seen a drastic improve in performance after running this for 3/4 times.
>     In case of phones it is more helpful as there is no swap space.
> 
> 7) Your feedback and suggestion is important. Based on the feedback, I can plan to submit the patches officially after performing basic cleanups.

So really I think this tells you two things

1. There are cases where the kernel paging subsystem is perhaps making
poor choices and should have forced out more read only pages.

2. For certain DMA allocation cases it might be a good idea to move the
interface out of the HIBERNATION config option and call it automatically
with the relevant memory allocator when requests for large linear
allocations would otherwise fail.

> This can be even using inside the multimedia drivers that requires large contiguous memory to check if that many memory pages can be reclaimed or not.

Yes - I agree. However the way that the memory is obtained and the use of
shrink_all_memory() should not be exposed as it breaks the abstraction.

If you can use it *within* the contiguous memory allocator so that the
driver does not know about shrink_all_memory, then this would be
interesting and potentially useful.

Alan

Re: [NEW]: Introducing shrink_all_memory from user space

[Alan Cox]

On Sun, 15 Apr 2012 14:10:00 +0200
richard -rw- weinberger <richard.weinberger@gmail.com> wrote:

> On Sun, Apr 15, 2012 at 1:47 PM, PINTU KUMAR <pintu_agarwal@yahoo.com> wrote:
> > Moreover, this is mainly meant for mobile phones where there is only *one* user.
> 
> I see. Jet another awful hack.
> Mobile phones are nothing special. They are computers

Correct - so if it is showing up useful situations then they are also
useful beyond mobile phone.

> Every program which is allowed to use this interface will (ab)use it.

If you expose it to userspace then you would want it very tightly
controlled and very much special case. Within the kernel using it
internally within things like CMA allocators seems to make more sense.

I think you overestimate the abuse. It's an interface which pushes clean
pages that can be cheaply recovered out of memory. It doesn't guarantee
the caller reaps the benefit of that, and the vm will continue to try and
share out any new resource fairly.

Alan

Re: [NEW]: Introducing shrink_all_memory from user space

[PINTU KUMAR]

>________________________________
>From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>To: richard -rw- weinberger <richard.weinberger@gmail.com>  
>Sent: Thursday, 19 April 2012 1:40 AM
>Subject: Re: [NEW]: Introducing shrink_all_memory from user space
>
>On Sun, 15 Apr 2012 14:10:00 +0200
>richard -rw- weinberger <richard.weinberger@gmail.com> wrote:
>
>> On Sun, Apr 15, 2012 at 1:47 PM, PINTU KUMAR <pintu_agarwal@yahoo.com> wrote:
>> > Moreover, this is mainly meant for mobile phones where there is only *one* user.
>> 
>> I see. Jet another awful hack.
>> Mobile phones are nothing special. They are computers
>
>Correct - so if it is showing up useful situations then they are also
>useful beyond mobile phone.
>
>> Every program which is allowed to use this interface will (ab)use it.
>
>If you expose it to userspace then you would want it very tightly
>controlled and very much special case. Within the kernel using it
>internally within things like CMA allocators seems to make more sense.
>
>I think you overestimate the abuse. It's an interface which pushes clean
>pages that can be cheaply recovered out of memory. It doesn't guarantee
>the caller reaps the benefit of that, and the vm will continue to try and
>share out any new resource fairly.
>
>Alan

Dear Alan, thank you very much for your comments and suggestion.
My plan is to develop a kind of system utility (like defragment) which we can run from user space (as root).
 
And yes you are right, my future plan is also to use it for CMA as it also suffers from memory fragmentation.
Now I think CMA uses memory compaction solution to reclaim pages for its allocation. Similarly we can use this on top of compaction for better results.
 
And we can even call this from low memory notifier whenever memory pressure falls below watermark and regain memory state as it was before.
 
Well more experiments and findings are in progress.
 
 
 
Thanks,
Pintu

Re: Introducing Aggressive Low Memory Booster [1]

[Bartlomiej Zolnierkiewicz]

On Sunday 13 January 2013 16:32:57 PINTU KUMAR wrote:
> Hi,
> 
> Here I am trying to introduce a new feature in kernel called "Aggressive Low Memory Booster".
> The main advantage of this will be to boost the available free memory of the system to "certain level" during extremely low memory condition.
> 
> Please provide your comments to improve further.

Could you please post the code somewhere so it can be reviewed?

Thanks,
--
Bartlomiej Zolnierkiewicz
Samsung Poland R&D Center

> Can it be used along with vmpressure_fd ???
> 
> 
> It can be invoked as follows:
>     a) Automatically by kernel memory management when the memory threshold falls below 10MB.
>     b) From user space program/scripts by passing the "required amount of memory to be reclaimed".
>     Example: echo 100 > /dev/shrinkmem
>     c) using sys interface - /sys/kernel/debug/shrinkallmem
>     d) using an ioctl call and returning number of pages reclaimed.
>     e) using a new system call - shrinkallmem(&nrpages);
>     f) During CMA to reclaim and shrink a specific CMA regions.
> 
> 
> I have developed a kernel module to verify the (b) part.
> 
> Here is the snapshot of the write call:
> +static ssize_t shrinkmem_write(struct file *file, const char *buff,
> +                                size_t length, loff_t *pos)
> +{
> +        int ret = -1;
> +        unsigned long memsize = 0;
> +        unsigned long nr_reclaim = 0;
> +        unsigned long pages = 0;
> +        ret = kstrtoul_from_user(buff, length, 0, &memsize);
> +        if (ret < 0) {
> +                printk(KERN_ERR "[SHRINKMEM]: kstrtoul_from_user: Failed !\n");
> +                return -1;
> +        }
> +        printk(KERN_INFO "[SHRINKMEM]: memsize(in MB) = %ld\n",
> +                                (unsigned long)memsize);
> +        memsize = memsize*(1024UL*1024UL);
> +        nr_reclaim = memsize / PAGE_SIZE;
> +        pages = shrink_all_memory(nr_reclaim);
> +        printk(KERN_INFO "<SHRINKMEM>: Number of Pages Freed: %lu\n", pages);
> +        return pages;
> +}
> Please note: This requires CONFIG_HIBERNATION to be permanently enabled in the kernel.
> 
> 
> Several experiments have been performed on Ubuntu(kernel 3.3) to verify it under low memory conditions.
> 
> Following are some results obtained:
> -------------------------------------
> 
> Node 0, zone      DMA    290    115      0      0      0      0      0      0      0      0      0
> Node 0, zone   Normal    304    540    116     13      2      2      0      0      0      0      0
> =========================
>              total       used       free     shared    buffers     cached
> Mem:           497        487         10          0         63        303
> -/+ buffers/cache:        120        376
> Swap:         1458         34       1424
> Total:        1956        522       1434
> =========================
> Total Memory Freed: 342 MB
> Total Memory Freed: 53 MB
> Total Memory Freed: 23 MB
> Total Memory Freed: 10 MB
> Total Memory Freed: 15 MB
> Total Memory Freed: -1 MB
> Node 0, zone      DMA      6      6      7      8     10      9      7      4      1      0      0
> Node 0, zone   Normal   2129   2612   2166   1723   1260    759    359    108     10      0      0
> =========================
>              total       used       free     shared    buffers     cached
> Mem:           497         47        449          0          0          5
> -/+ buffers/cache:         41        455
> Swap:         1458         97       1361
> Total:        1956        145       1811
> =========================
> 
> It was verified using a sample shell script "reclaim_memory.sh" which keeps recovering memory by doing "echo 500 > /dev/shrinkmem" until no further reclaim is possible.
> 
> The experiments were performed with various scenarios as follows:
> a) Just after the boot up - (could recover around 150MB with 512MB RAM)
> b) After running many applications include youtube videos, large tar files download - 
> 
>    [until free mem becomes < 10MB]
>    [Could recover around 300MB in one shot]
> c) Run reclaim, while download is in progress and video still playing - (Not applications killed)
> 
> d) revoke all background applications again, after running reclaim - (No impact, normal behavior)
>    [Just it took little extra time to launch, as if it was launched for first time]
> 
> 
> Please see more discussions on this in the last year mailing list:
> 
> https://lkml.org/lkml/2012/4/15/35 
> 
> 
> Thank You!
> With regards,
> Pintu Kumar
> Samsung - India

Re: Introducing Aggressive Low Memory Booster [1]

[PINTU KUMAR]

Hi,

Can anybody provide any inputs/suggestions/improvements on the following.

According to my experiments these proved to be a useful utility during low memory condition on the embedded devices.
Is there something wrong I am doing?

Please provide your suggestions.

Thanks,
Pintu



>________________________________
> From: PINTU KUMAR <pintu_agarwal@yahoo.com>
>To: "linux-mm@kvack.org" <linux-mm@kvack.org>; "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org> 
>Cc: "linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>; "pintu.k@samsung.com" <pintu.k@samsung.com>; Anton Vorontsov <anton.vorontsov@linaro.org>; Alan Cox <alan@lxorguk.ukuu.org.uk>; richard -rw- weinberger <richard.weinberger@gmail.com>; "patches@linaro.org" <patches@linaro.org>; Mel Gorman <mgorman@suse.de>; Wanpeng Li <liwanp@linux.vnet.ibm.com> 
>Sent: Sunday, 13 January 2013 9:02 PM
>Subject: Introducing Aggressive Low Memory Booster [1]
> 
>
>Hi,
>
>
>Here I am trying to introduce a new feature in kernel called "Aggressive Low Memory Booster".
>The main advantage of this will be to boost the available free memory of the system to "certain level" during extremely low memory condition.
>
>
>Please provide your comments to improve further.
>Can it be used along with vmpressure_fd ???
>
>
>
>It can be invoked as follows:
>    a) Automatically by kernel memory management when the memory threshold falls below 10MB.
>    b) From user space program/scripts by passing the "required amount of memory to be reclaimed".
>    Example: echo 100 > /dev/shrinkmem
>    c) using sys interface - /sys/kernel/debug/shrinkallmem
>    d) using an ioctl call and returning number of pages reclaimed.
>    e) using a new system call - shrinkallmem(&nrpages);
>    f) During CMA to reclaim and shrink a specific CMA regions.
>
>
>
>I have developed a kernel module to verify the (b) part.
>
>
>Here is the snapshot of the write call:
>+static ssize_t shrinkmem_write(struct file *file, const char *buff,
>+                                size_t length, loff_t *pos)
>+{
>+        int ret = -1;
>+        unsigned long memsize = 0;
>+        unsigned long nr_reclaim = 0;
>+        unsigned long pages = 0;
>+        ret = kstrtoul_from_user(buff, length, 0, &memsize);
>+        if (ret < 0) {
>+                printk(KERN_ERR "[SHRINKMEM]: kstrtoul_from_user: Failed !\n");
>+                return
-1;
>+        }
>+        printk(KERN_INFO "[SHRINKMEM]: memsize(in MB) = %ld\n",
>+                                (unsigned long)memsize);
>+        memsize = memsize*(1024UL*1024UL);
>+        nr_reclaim = memsize / PAGE_SIZE;
>+        pages = shrink_all_memory(nr_reclaim);
>+        printk(KERN_INFO "<SHRINKMEM>: Number of Pages Freed: %lu\n", pages);
>+        return pages;
>+}
>Please note: This requires CONFIG_HIBERNATION to be permanently enabled in the kernel.
>
>
>Several experiments have been performed on Ubuntu(kernel 3.3) to verify it under low memory conditions.
>
>
>Following are some results obtained:
>-------------------------------------
>
>Node 0, zone      DMA    290    115      0      0      0      0      0      0      0      0      0
>Node 0, zone   Normal    304    540    116     13      2      2      0      0      0      0      0
>=========================
>             total      
used       free     shared    buffers     cached
>Mem:           497        487         10          0         63        303
>-/+ buffers/cache:        120        376
>Swap:         1458         34       1424
>Total:        1956        522       1434
>=========================
>Total Memory Freed: 342 MB
>Total Memory Freed: 53 MB
>Total
Memory Freed: 23 MB
>Total Memory Freed: 10 MB
>Total Memory Freed: 15 MB
>Total Memory Freed: -1 MB
>Node 0, zone      DMA      6      6      7      8     10      9      7      4      1      0      0
>Node 0, zone   Normal   2129   2612   2166   1723   1260    759    359    108     10      0      0
>=========================
>             total      
used       free     shared    buffers     cached
>Mem:           497         47        449          0          0          5
>-/+ buffers/cache:         41        455
>Swap:         1458         97       1361
>Total:        1956        145       1811
>=========================
>
>
>It was verified using a sample shell script "reclaim_memory.sh" which keeps recovering memory by doing "echo 500 > /dev/shrinkmem" until no further reclaim is possible.
>
>
>The experiments were performed with various scenarios as follows:
>a) Just after the boot up - (could recover around 150MB with 512MB RAM)
>b) After running many applications include youtube videos, large tar files download - 
>
>   [until free mem becomes < 10MB]
>   [Could recover around 300MB in one shot]
>c) Run reclaim, while download is in progress and video still playing - (Not applications killed)
>
>d) revoke all background applications again, after running reclaim - (No impact, normal behavior)
>   [Just it took little extra time to launch, as if it was launched for first time]
>
>
>
>
>Please see more discussions on this in the last year mailing list:
>
>https://lkml.org/lkml/2012/4/15/35
>
>
>
>Thank You!
>With regards,
>Pintu Kumar
>Samsung - India
>
>
>
>
>
>