* transfer physical memory page to swap disk
@ 2020-01-15 12:31 Sumit Kumar
2020-01-15 12:42 ` aleix sanchis ramírez
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Sumit Kumar @ 2020-01-15 12:31 UTC (permalink / raw)
To: kernelnewbies
[-- Attachment #1.1: Type: text/plain, Size: 580 bytes --]
Hi,
Does C/C++ provide any API / system call that enables user to force the
application to transfer some its physical pages to swap disk ? If so, is it
also possible to obtain the least used page using some API ?
AFAIK, linux kernel is supposed to do this as part of memory management. I
want to know if the kernel also exposes some API to enable users to control
their application's memory management. I have many applications running at
a time that cause too much memory consumption. I believe that experimenting
with memory management can help.
--
Thanks and Regards,
Sumit
[-- Attachment #1.2: Type: text/html, Size: 690 bytes --]
[-- Attachment #2: Type: text/plain, Size: 170 bytes --]
_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
^ permalink raw reply [flat|nested] 11+ messages in thread* RE: transfer physical memory page to swap disk 2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar @ 2020-01-15 12:42 ` aleix sanchis ramírez 2020-01-15 12:53 ` Anupam Kapoor 2020-01-19 16:59 ` Bernd Petrovitsch 2 siblings, 0 replies; 11+ messages in thread From: aleix sanchis ramírez @ 2020-01-15 12:42 UTC (permalink / raw) To: Sumit Kumar, kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 953 bytes --] > Hi, >Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ? I don't think such a system call exists. There might be some mechanisms that may cause your memory to be swapped out as a side effect, but I doubt you have much control over which page gets swapped. > AFAIK, linux kernel is supposed to do this as part of memory management. I want to know if the kernel also exposes some API to enable users to control their application's memory management. I have many applications running at a time that cause too much memory consumption. I believe that experimenting with memory management can help. I really don't think this kind of management can, or should, be done from user-space. If you have too much memory consumption, try to increase system memory or optimize your applications. [-- Attachment #1.2: Type: text/html, Size: 1407 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar 2020-01-15 12:42 ` aleix sanchis ramírez @ 2020-01-15 12:53 ` Anupam Kapoor 2020-01-18 15:10 ` Valdis Klētnieks 2020-01-19 16:59 ` Bernd Petrovitsch 2 siblings, 1 reply; 11+ messages in thread From: Anupam Kapoor @ 2020-01-15 12:53 UTC (permalink / raw) To: Sumit Kumar; +Cc: kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 471 bytes --] On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote: > Hi, > Does C/C++ provide any API / system call that enables user to force the > application to transfer some its physical pages to swap disk ? If so, is it > also possible to obtain the least used page using some API ? > would madvise not serve your needs ? — kind regards anupam -- In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. [-- Attachment #1.2: Type: text/html, Size: 1039 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-15 12:53 ` Anupam Kapoor @ 2020-01-18 15:10 ` Valdis Klētnieks 2020-01-19 10:01 ` Sumit Kumar 2020-01-19 10:55 ` Anupam Kapoor 0 siblings, 2 replies; 11+ messages in thread From: Valdis Klētnieks @ 2020-01-18 15:10 UTC (permalink / raw) To: Anupam Kapoor; +Cc: Sumit Kumar, kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 679 bytes --] On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said: > On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote: > > Does C/C++ provide any API / system call that enables user to force the > > application to transfer some its physical pages to swap disk ? If so, is it > > also possible to obtain the least used page using some API ? > would madvise not serve your needs ? There's this word "force" in the question. The problem is that madvise() is *advice* to the kernel, not a strict guarantee. There's a difference between "If you need to move pages to disk, consider these pages first" and "Move them to disk now, whether you really wanted to or not". [-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-18 15:10 ` Valdis Klētnieks @ 2020-01-19 10:01 ` Sumit Kumar 2020-01-19 10:55 ` Anupam Kapoor 1 sibling, 0 replies; 11+ messages in thread From: Sumit Kumar @ 2020-01-19 10:01 UTC (permalink / raw) To: Valdis Klētnieks, Anupam Kapoor, aleixsanchis; +Cc: kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 907 bytes --] Thanks a lot for your responses Aleix, Anupam and Valdis. -- Thanks and Regards, Sumit On Sat, 18 Jan 2020 at 20:40, Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote: > On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said: > > On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> > wrote: > > > Does C/C++ provide any API / system call that enables user to force the > > > application to transfer some its physical pages to swap disk ? If so, > is it > > > also possible to obtain the least used page using some API ? > > would madvise not serve your needs ? > > There's this word "force" in the question. > > The problem is that madvise() is *advice* to the kernel, not a strict > guarantee. There's a difference between "If you need to move pages > to disk, consider these pages first" and "Move them to disk now, whether > you really wanted to or not". > > [-- Attachment #1.2: Type: text/html, Size: 1377 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-18 15:10 ` Valdis Klētnieks 2020-01-19 10:01 ` Sumit Kumar @ 2020-01-19 10:55 ` Anupam Kapoor 2020-01-19 11:14 ` Valdis Klētnieks 1 sibling, 1 reply; 11+ messages in thread From: Anupam Kapoor @ 2020-01-19 10:55 UTC (permalink / raw) To: Valdis Klētnieks; +Cc: Sumit Kumar, kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 2048 bytes --] On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said: >> On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote: >>> Does C/C++ provide any API / system call that enables user to force the >>> application to transfer some its physical pages to swap disk ? If so, is it >>> also possible to obtain the least used page using some API ? >> would madvise not serve your needs ? > There's this word "force" in the question. > The problem is that madvise() is *advice* to the kernel, not a strict > guarantee. There's a difference between "If you need to move pages > to disk, consider these pages first" and "Move them to disk now, whether > you really wanted to or not". ofcourse (afaik) there is no way for the application to force the *kernel* to do something like this. but if _all_ that is required is randomly unmapping some marked application pages, _that_ can be naively 'done' by the application itself :) for example, have a list of unmappable pages and an unmapper thread which does the unmapping for you (picking any element in the list at random) -- kind regards anupam In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. On Sat, Jan 18, 2020 at 3:10 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote: > On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said: > > On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> > wrote: > > > Does C/C++ provide any API / system call that enables user to force the > > > application to transfer some its physical pages to swap disk ? If so, > is it > > > also possible to obtain the least used page using some API ? > > would madvise not serve your needs ? > > There's this word "force" in the question. > > The problem is that madvise() is *advice* to the kernel, not a strict > guarantee. There's a difference between "If you need to move pages > to disk, consider these pages first" and "Move them to disk now, whether > you really wanted to or not". > > [-- Attachment #1.2: Type: text/html, Size: 4081 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-19 10:55 ` Anupam Kapoor @ 2020-01-19 11:14 ` Valdis Klētnieks 2020-01-19 12:45 ` Anupam Kapoor 0 siblings, 1 reply; 11+ messages in thread From: Valdis Klētnieks @ 2020-01-19 11:14 UTC (permalink / raw) To: Anupam Kapoor; +Cc: Sumit Kumar, kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 385 bytes --] On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said: > but if _all_ that is required is randomly unmapping some marked > application pages, _that_ can be naively 'done' by the application > itself :) Note that in this case, "naively" includes "not remembering to consider that the page being unmapped may have contained data we'd rather have kept by flushing the page to disk" :) [-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-19 11:14 ` Valdis Klētnieks @ 2020-01-19 12:45 ` Anupam Kapoor 2020-01-19 13:18 ` Valdis Klētnieks 0 siblings, 1 reply; 11+ messages in thread From: Anupam Kapoor @ 2020-01-19 12:45 UTC (permalink / raw) To: Valdis Klētnieks; +Cc: Sumit Kumar, kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 1398 bytes --] >> but if _all_ that is required is randomly unmapping some marked >> application pages, _that_ can be naively 'done' by the application >> itself :) > Note that in this case, "naively" includes "not remembering to consider > that the page being unmapped may have contained data we'd rather > have kept by flushing the page to disk" :) but is it that bad ? before marking a page unmappable, the application has full control over what it wants to do with the data, and can choose to dump it to the appropriate destination. or if enough information is available, the unmapper thread can itself play that role. thinking some more about it, application has full control over the unmap/resurrect behavior. though latter might not be as transparent... -- kind regards anupam In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. On Sun, Jan 19, 2020 at 11:14 AM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote: > On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said: > > > but if _all_ that is required is randomly unmapping some marked > > application pages, _that_ can be naively 'done' by the application > > itself :) > > Note that in this case, "naively" includes "not remembering to consider > that the page being unmapped may have contained data we'd rather > have kept by flushing the page to disk" :) > [-- Attachment #1.2: Type: text/html, Size: 2593 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-19 12:45 ` Anupam Kapoor @ 2020-01-19 13:18 ` Valdis Klētnieks 2020-01-19 16:01 ` Anupam Kapoor 0 siblings, 1 reply; 11+ messages in thread From: Valdis Klētnieks @ 2020-01-19 13:18 UTC (permalink / raw) To: Anupam Kapoor; +Cc: Sumit Kumar, kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 1213 bytes --] On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said: > > Note that in this case, "naively" includes "not remembering to consider > > that the page being unmapped may have contained data we'd rather > > have kept by flushing the page to disk" :) > > but is it that bad ? > > before marking a page unmappable, the application has full control > over what it wants to do with the data, and can choose to dump it > to the appropriate destination. Yes, but now you're getting into more code that has to be written, including code to marshal things like binary trees into a savable format, and more code to read them back at a later time. Plus all the fun if the tree has hundreds of thousands or millions of entries, and how to deal with it if some parts of the tree have been released and saved to disk, or if the 4K page contained members of several different data structures - in other words, you probably just decided to write your own backing store, garbage collector, and virtual object manager for your heap. As I said - it's a naive approach that ends up following the 90/10 rule: the easy 90% of it takes the first 90% of the time to code it, and the difficult 10% takes the other 90% of the time... :) [-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-19 13:18 ` Valdis Klētnieks @ 2020-01-19 16:01 ` Anupam Kapoor 0 siblings, 0 replies; 11+ messages in thread From: Anupam Kapoor @ 2020-01-19 16:01 UTC (permalink / raw) To: Valdis Klētnieks; +Cc: Sumit Kumar, kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 1909 bytes --] On Sun, 19 Jan 2020 at 6:48 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote: > On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said: > > > > Note that in this case, "naively" includes "not remembering to consider > > > that the page being unmapped may have contained data we'd rather > > > have kept by flushing the page to disk" :) > > > > but is it that bad ? > > > > before marking a page unmappable, the application has full control > > over what it wants to do with the data, and can choose to dump it > > to the appropriate destination. > > Yes, but now you're getting into more code that has to be written, > including > code to marshal things like binary trees into a savable format, and more > code > to read them back at a later time. Plus all the fun if the tree has > hundreds of thousands > or millions of entries, and how to deal with it if some parts of the tree > have been > released and saved to disk, or if the 4K page contained members of several > different > data structures - in other words, you probably just decided to write your > own backing store, > garbage collector, and virtual object manager for your heap. > > As I said - it's a naive approach that ends up following the 90/10 rule: > the easy 90% of it takes the first 90% of the time to code it, and the > difficult > 10% takes the other 90% of the time... :) well sure, if you try to replicate everything that exists below libc, then there is little hope. however if your application’s data can be serialized/deserialized, then i _suspect_ it might not be too much of work. for example, if i am maintaining l2 forwarding table entries then it might be possible to have, on an average fixed number of pages representing this cache... — kind regards anupam > > -- In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. [-- Attachment #1.2: Type: text/html, Size: 2749 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: transfer physical memory page to swap disk 2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar 2020-01-15 12:42 ` aleix sanchis ramírez 2020-01-15 12:53 ` Anupam Kapoor @ 2020-01-19 16:59 ` Bernd Petrovitsch 2 siblings, 0 replies; 11+ messages in thread From: Bernd Petrovitsch @ 2020-01-19 16:59 UTC (permalink / raw) To: Sumit Kumar; +Cc: kernelnewbies Hi all! To answer it from a somewhat different angle: On Wed, 2020-01-15 at 18:01 +0530, Sumit Kumar wrote: > Hi, > Does C/C++ provide any API / system call that enables user to force the > application to transfer some its physical pages to swap disk ? If so, is it Short answer: No (and you do not want such a thing in the first place). Longer answer: I don't think any normal application would force some memory pages to the swap space (as it slows down) - quite the contrary. You can use open(..., O_DIRECT) to minimize cache effects if that may help). WRF, you can implement pseudo-swapping with that within your application. madvise() has been mentioned in other mails. One application has (usually) no knowledge about the rest of the system - neither the amount of RAM nor the workload (which may change over time) - or is at least written that way. So it's IMHO better to leave the kernels heuristics work system-wide instead of "micro-optimizing" in some application which will sooner or later interfere with the kernels heuristics. And - always - the next question is: how could this API be abused/creatively used for the own benefit/...? BTW: what's a real intended real-world application? > also possible to obtain the least used page using some API ? > > AFAIK, linux kernel is supposed to do this as part of memory management. I Yes - if there actually is swap space available (und don't underestimate the number of systems running without a swap space and CONFIG_SWAP=n in /boot/*config* - desktop and servers are not everything in the world, more like a small part ....). > want to know if the kernel also exposes some API to enable users to control > their application's memory management. I have many applications running at > a time that cause too much memory consumption. I believe that experimenting > with memory management can help. The application can (and should) control it's memory management anyways - just allocate as much as the application needs and free it when it's no longer used. For larger memeory areas (possibly with a self-build malloc/free-equivalent or memory pools), the application can mmap() it and munmap() it simply when it's done. You can experiment with setting the process limits via setrlimit() to smaller or larger values. MfG, Bernd -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2020-01-19 17:00 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar 2020-01-15 12:42 ` aleix sanchis ramírez 2020-01-15 12:53 ` Anupam Kapoor 2020-01-18 15:10 ` Valdis Klētnieks 2020-01-19 10:01 ` Sumit Kumar 2020-01-19 10:55 ` Anupam Kapoor 2020-01-19 11:14 ` Valdis Klētnieks 2020-01-19 12:45 ` Anupam Kapoor 2020-01-19 13:18 ` Valdis Klētnieks 2020-01-19 16:01 ` Anupam Kapoor 2020-01-19 16:59 ` Bernd Petrovitsch
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).