transfer physical memory page to swap disk

transfer physical memory page to swap disk

[Sumit Kumar]

[-- Attachment #1.1: Type: text/plain, Size: 580 bytes --]

Hi,
Does C/C++ provide any API / system call that enables user to force the
application to transfer some its physical pages to swap disk ? If so, is it
also possible to obtain the least used page using some API ?

AFAIK, linux kernel is supposed to do this as part of memory management. I
want to know if the kernel also exposes some API to enable users to control
their application's memory management. I have many applications running at
a time that cause too much memory consumption. I believe that experimenting
with memory management can help.

 --
Thanks and Regards,
Sumit

[-- Attachment #1.2: Type: text/html, Size: 690 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

RE: transfer physical memory page to swap disk

[aleix sanchis ramírez]

[-- Attachment #1.1: Type: text/plain, Size: 953 bytes --]


> Hi,
>Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ?

I don't think such a system call exists. There might be some mechanisms that may cause your memory to be swapped out as a side effect, but I doubt you have much control over which page gets swapped.

> AFAIK, linux kernel is supposed to do this as part of memory management. I want to know if the kernel also exposes some API to enable users to control their application's memory management. I have many applications running at a time that cause too much memory consumption. I believe that experimenting with memory management can help.

I really don't think this kind of management can, or should, be done from user-space. If you have too much memory consumption, try to increase system memory or optimize your applications.

[-- Attachment #1.2: Type: text/html, Size: 1407 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Anupam Kapoor]

[-- Attachment #1.1: Type: text/plain, Size: 471 bytes --]

On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:

> Hi,
> Does C/C++ provide any API / system call that enables user to force the
> application to transfer some its physical pages to swap disk ? If so, is it
> also possible to obtain the least used page using some API ?
>

would madvise not serve your needs ?

—
kind regards
anupam
-- 
In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

[-- Attachment #1.2: Type: text/html, Size: 1039 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 679 bytes --]

On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
> On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
> > Does C/C++ provide any API / system call that enables user to force the
> > application to transfer some its physical pages to swap disk ? If so, is it
> > also possible to obtain the least used page using some API ?
> would madvise not serve your needs ?

There's this word "force" in the question.

The problem is that madvise() is *advice* to the kernel, not a strict
guarantee.  There's a difference between "If you need to move pages
to disk, consider these pages first" and "Move them to disk now, whether
you really wanted to or not".


[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Sumit Kumar]

[-- Attachment #1.1: Type: text/plain, Size: 907 bytes --]

Thanks a lot for your responses Aleix, Anupam and Valdis.

--
Thanks and Regards,
Sumit

On Sat, 18 Jan 2020 at 20:40, Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
> > On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com>
> wrote:
> > > Does C/C++ provide any API / system call that enables user to force the
> > > application to transfer some its physical pages to swap disk ? If so,
> is it
> > > also possible to obtain the least used page using some API ?
> > would madvise not serve your needs ?
>
> There's this word "force" in the question.
>
> The problem is that madvise() is *advice* to the kernel, not a strict
> guarantee.  There's a difference between "If you need to move pages
> to disk, consider these pages first" and "Move them to disk now, whether
> you really wanted to or not".
>
>

[-- Attachment #1.2: Type: text/html, Size: 1377 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Anupam Kapoor]

[-- Attachment #1.1: Type: text/plain, Size: 2048 bytes --]

On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
>> On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
>>> Does C/C++ provide any API / system call that enables user to force the
>>> application to transfer some its physical pages to swap disk ? If so,
is it
>>> also possible to obtain the least used page using some API ?

>> would madvise not serve your needs ?

> There's this word "force" in the question.

> The problem is that madvise() is *advice* to the kernel, not a strict
> guarantee.  There's a difference between "If you need to move pages
> to disk, consider these pages first" and "Move them to disk now, whether
> you really wanted to or not".

ofcourse (afaik) there is no way for the application to force the
*kernel* to do something like this.

but if _all_ that is required is randomly unmapping some marked
application pages, _that_ can be naively 'done' by the application
itself :)

for example, have a list of unmappable pages and an unmapper thread
which does the unmapping for you (picking any element in the list
at random)

--
kind regards
anupam

In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.


On Sat, Jan 18, 2020 at 3:10 PM Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
> > On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com>
> wrote:
> > > Does C/C++ provide any API / system call that enables user to force the
> > > application to transfer some its physical pages to swap disk ? If so,
> is it
> > > also possible to obtain the least used page using some API ?
> > would madvise not serve your needs ?
>
> There's this word "force" in the question.
>
> The problem is that madvise() is *advice* to the kernel, not a strict
> guarantee.  There's a difference between "If you need to move pages
> to disk, consider these pages first" and "Move them to disk now, whether
> you really wanted to or not".
>
>

[-- Attachment #1.2: Type: text/html, Size: 4081 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 385 bytes --]

On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said:

> but if _all_ that is required is randomly unmapping some marked
> application pages, _that_ can be naively 'done' by the application
> itself :)

Note that in this case, "naively" includes "not remembering to consider
that the page being unmapped may have contained data we'd rather
have kept by flushing the page to disk" :)

[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Anupam Kapoor]

[-- Attachment #1.1: Type: text/plain, Size: 1398 bytes --]

>> but if _all_ that is required is randomly unmapping some marked
>> application pages, _that_ can be naively 'done' by the application
>> itself :)

> Note that in this case, "naively" includes "not remembering to consider
> that the page being unmapped may have contained data we'd rather
> have kept by flushing the page to disk" :)

but is it that bad ?

before marking a page unmappable, the application has full control
over what it wants to do with the data, and can choose to dump it
to the appropriate destination.

or if enough information is available, the unmapper thread can itself
play that role.

thinking some more about it, application has full control over the
unmap/resurrect behavior. though latter might not be as
transparent...

--
kind regards
anupam

In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.


On Sun, Jan 19, 2020 at 11:14 AM Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said:
>
> > but if _all_ that is required is randomly unmapping some marked
> > application pages, _that_ can be naively 'done' by the application
> > itself :)
>
> Note that in this case, "naively" includes "not remembering to consider
> that the page being unmapped may have contained data we'd rather
> have kept by flushing the page to disk" :)
>

[-- Attachment #1.2: Type: text/html, Size: 2593 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 1213 bytes --]

On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said:

> > Note that in this case, "naively" includes "not remembering to consider
> > that the page being unmapped may have contained data we'd rather
> > have kept by flushing the page to disk" :)
>
> but is it that bad ?
>
> before marking a page unmappable, the application has full control
> over what it wants to do with the data, and can choose to dump it
> to the appropriate destination.

Yes, but now you're getting into more code that has to be written, including
code to marshal things like binary trees into a savable format, and more code
to read them back at a later time. Plus all the fun if the tree has hundreds of thousands
or millions of entries, and how to deal with it if some parts of the tree have been
released and saved to disk, or if the 4K page contained members of several different
data structures - in other words, you probably just decided to write your own backing store,
garbage collector, and virtual object manager for your heap.

As I said - it's a naive approach that ends up following the 90/10 rule:
the easy 90% of it takes the first 90% of the time to code it, and the difficult
10% takes the other 90% of the time... :)

[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Anupam Kapoor]

[-- Attachment #1.1: Type: text/plain, Size: 1909 bytes --]

On Sun, 19 Jan 2020 at 6:48 PM Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said:
>
> > > Note that in this case, "naively" includes "not remembering to consider
> > > that the page being unmapped may have contained data we'd rather
> > > have kept by flushing the page to disk" :)
> >
> > but is it that bad ?
> >
> > before marking a page unmappable, the application has full control
> > over what it wants to do with the data, and can choose to dump it
> > to the appropriate destination.
>
> Yes, but now you're getting into more code that has to be written,
> including
> code to marshal things like binary trees into a savable format, and more
> code
> to read them back at a later time. Plus all the fun if the tree has
> hundreds of thousands
> or millions of entries, and how to deal with it if some parts of the tree
> have been
> released and saved to disk, or if the 4K page contained members of several
> different
> data structures - in other words, you probably just decided to write your
> own backing store,
> garbage collector, and virtual object manager for your heap.
>
> As I said - it's a naive approach that ends up following the 90/10 rule:
> the easy 90% of it takes the first 90% of the time to code it, and the
> difficult
> 10% takes the other 90% of the time... :)


well sure, if you try to replicate everything that exists below libc, then
there is little hope.

however if your application’s data can be serialized/deserialized, then i
_suspect_ it might not be too much of work.

for example, if i am maintaining l2 forwarding table entries then it might
be possible to have, on an average fixed number of pages representing this
cache...

—
kind regards
anupam

>
> --
In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

[-- Attachment #1.2: Type: text/html, Size: 2749 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: transfer physical memory page to swap disk

[Bernd Petrovitsch]

Hi all!

To answer it from a somewhat different angle:

On Wed, 2020-01-15 at 18:01 +0530, Sumit Kumar wrote:
> Hi,
> Does C/C++ provide any API / system call that enables user to force the
> application to transfer some its physical pages to swap disk ? If so, is it

Short answer: No (and you do not want such a thing in the first
place).

Longer answer: I don't think any normal application would force
some memory pages to the swap space (as it slows down) - quite
the contrary.
You can use open(..., O_DIRECT) to minimize cache effects
if that may help). WRF, you can implement pseudo-swapping
with that within your application.
madvise() has been mentioned in other mails.
One application has (usually) no knowledge about the rest of
the system - neither the amount of RAM nor the workload
(which may change over time) - or is at least written
that way.
So it's IMHO better to leave the kernels heuristics work
system-wide instead of "micro-optimizing" in some
application which will sooner or later interfere with
the kernels heuristics.

And - always - the next question is: how could
this API be abused/creatively used for the own
benefit/...?

BTW: what's a real intended real-world application?

> also possible to obtain the least used page using some API ?
> 
> AFAIK, linux kernel is supposed to do this as part of memory management. I

Yes - if there actually is swap space available (und don't
underestimate the number of systems running without a swap
space and CONFIG_SWAP=n in /boot/*config* - desktop and
servers are not everything in the world, more like a
small part ....).

> want to know if the kernel also exposes some API to enable users to control
> their application's memory management. I have many applications running at
> a time that cause too much memory consumption. I believe that experimenting
> with memory management can help.

The application can (and should) control it's memory management
anyways - just allocate as much as the application needs and
free it when it's no longer used.
For larger memeory areas (possibly with a self-build
malloc/free-equivalent or memory pools), the application
can mmap() it and munmap() it simply when it's done.

You can experiment with setting the process limits via
setrlimit() to smaller or larger values.

MfG,
	Bernd
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
                     LUGA : http://www.luga.at


_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies