Kernel Newbies archive on lore.kernel.org
 help / color / Atom feed
* transfer physical memory page to swap disk
@ 2020-01-15 12:31 Sumit Kumar
  2020-01-15 12:42 ` aleix sanchis ramírez
                   ` (2 more replies)
  0 siblings, 3 replies; 11+ messages in thread
From: Sumit Kumar @ 2020-01-15 12:31 UTC (permalink / raw)
  To: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 580 bytes --]

Hi,
Does C/C++ provide any API / system call that enables user to force the
application to transfer some its physical pages to swap disk ? If so, is it
also possible to obtain the least used page using some API ?

AFAIK, linux kernel is supposed to do this as part of memory management. I
want to know if the kernel also exposes some API to enable users to control
their application's memory management. I have many applications running at
a time that cause too much memory consumption. I believe that experimenting
with memory management can help.

 --
Thanks and Regards,
Sumit

[-- Attachment #1.2: Type: text/html, Size: 690 bytes --]

<div dir="ltr">Hi, <div>Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ? </div><div><br></div><div>AFAIK, linux kernel is supposed to do this as part of memory management. I want to know if the kernel also exposes some API to enable users to control their application&#39;s memory management. I have many applications running at a time that cause too much memory consumption. I believe that experimenting with memory management can help.</div><div><br></div><div> --</div><div>Thanks and Regards,</div><div>Sumit</div></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: transfer physical memory page to swap disk
  2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar
@ 2020-01-15 12:42 ` aleix sanchis ramírez
  2020-01-15 12:53 ` Anupam Kapoor
  2020-01-19 16:59 ` Bernd Petrovitsch
  2 siblings, 0 replies; 11+ messages in thread
From: aleix sanchis ramírez @ 2020-01-15 12:42 UTC (permalink / raw)
  To: Sumit Kumar, kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 953 bytes --]


> Hi,
>Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ?

I don't think such a system call exists. There might be some mechanisms that may cause your memory to be swapped out as a side effect, but I doubt you have much control over which page gets swapped.

> AFAIK, linux kernel is supposed to do this as part of memory management. I want to know if the kernel also exposes some API to enable users to control their application's memory management. I have many applications running at a time that cause too much memory consumption. I believe that experimenting with memory management can help.

I really don't think this kind of management can, or should, be done from user-space. If you have too much memory consumption, try to increase system memory or optimize your applications.

[-- Attachment #1.2: Type: text/html, Size: 1407 bytes --]

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div><br>
</div>
<div>&gt; Hi,&nbsp; </div>
<div>
<div>
<div dir="ltr">
<div>&gt;Does C/C&#43;&#43; provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ?
<br>
</div>
<div><br>
</div>
<div>I don't think such a system call exists. There might be some mechanisms that may cause your memory to be swapped out as a side effect, but I doubt you have much control over which page gets swapped.<br>
</div>
<div><br>
</div>
<div>&gt; AFAIK, linux kernel is supposed to do this as part of memory management. I want to know if the kernel also exposes some API to enable users to control their application's memory management. I have many applications running at a time that cause too much
 memory&nbsp;consumption. I believe that experimenting with memory management can help.</div>
<div><br>
</div>
<div>I really don't think this kind of management can, or should, be done from user-space. If you have too much memory consumption, try to increase system memory or optimize your applications.<br>
</div>
</div>
</div>
</div>
</body>
</html>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar
  2020-01-15 12:42 ` aleix sanchis ramírez
@ 2020-01-15 12:53 ` Anupam Kapoor
  2020-01-18 15:10   ` Valdis Klētnieks
  2020-01-19 16:59 ` Bernd Petrovitsch
  2 siblings, 1 reply; 11+ messages in thread
From: Anupam Kapoor @ 2020-01-15 12:53 UTC (permalink / raw)
  To: Sumit Kumar; +Cc: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 471 bytes --]

On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:

> Hi,
> Does C/C++ provide any API / system call that enables user to force the
> application to transfer some its physical pages to swap disk ? If so, is it
> also possible to obtain the least used page using some API ?
>

would madvise not serve your needs ?

—
kind regards
anupam
-- 
In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

[-- Attachment #1.2: Type: text/html, Size: 1039 bytes --]

<div><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar &lt;<a href="mailto:sumit686215@gmail.com">sumit686215@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi, <div>Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ? </div></div></blockquote><div dir="auto"><br></div><div dir="auto">would madvise not serve your needs ?</div><div dir="auto"><br></div><div dir="auto">—</div><div dir="auto">kind regards</div><div dir="auto">anupam </div></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font face="monospace, monospace">In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. </font></div></div></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-15 12:53 ` Anupam Kapoor
@ 2020-01-18 15:10   ` Valdis Klētnieks
  2020-01-19 10:01     ` Sumit Kumar
  2020-01-19 10:55     ` Anupam Kapoor
  0 siblings, 2 replies; 11+ messages in thread
From: Valdis Klētnieks @ 2020-01-18 15:10 UTC (permalink / raw)
  To: Anupam Kapoor; +Cc: Sumit Kumar, kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 679 bytes --]

On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
> On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
> > Does C/C++ provide any API / system call that enables user to force the
> > application to transfer some its physical pages to swap disk ? If so, is it
> > also possible to obtain the least used page using some API ?
> would madvise not serve your needs ?

There's this word "force" in the question.

The problem is that madvise() is *advice* to the kernel, not a strict
guarantee.  There's a difference between "If you need to move pages
to disk, consider these pages first" and "Move them to disk now, whether
you really wanted to or not".


[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-18 15:10   ` Valdis Klētnieks
@ 2020-01-19 10:01     ` Sumit Kumar
  2020-01-19 10:55     ` Anupam Kapoor
  1 sibling, 0 replies; 11+ messages in thread
From: Sumit Kumar @ 2020-01-19 10:01 UTC (permalink / raw)
  To: Valdis Klētnieks, Anupam Kapoor, aleixsanchis; +Cc: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 907 bytes --]

Thanks a lot for your responses Aleix, Anupam and Valdis.

--
Thanks and Regards,
Sumit

On Sat, 18 Jan 2020 at 20:40, Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
> > On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com>
> wrote:
> > > Does C/C++ provide any API / system call that enables user to force the
> > > application to transfer some its physical pages to swap disk ? If so,
> is it
> > > also possible to obtain the least used page using some API ?
> > would madvise not serve your needs ?
>
> There's this word "force" in the question.
>
> The problem is that madvise() is *advice* to the kernel, not a strict
> guarantee.  There's a difference between "If you need to move pages
> to disk, consider these pages first" and "Move them to disk now, whether
> you really wanted to or not".
>
>

[-- Attachment #1.2: Type: text/html, Size: 1377 bytes --]

<div dir="ltr">Thanks a lot for your responses Aleix, Anupam and Valdis. <div><br></div><div>--</div><div>Thanks and Regards,</div><div>Sumit</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 18 Jan 2020 at 20:40, Valdis Klētnieks &lt;<a href="mailto:valdis.kletnieks@vt.edu">valdis.kletnieks@vt.edu</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:<br>
&gt; On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar &lt;<a href="mailto:sumit686215@gmail.com" target="_blank">sumit686215@gmail.com</a>&gt; wrote:<br>
&gt; &gt; Does C/C++ provide any API / system call that enables user to force the<br>
&gt; &gt; application to transfer some its physical pages to swap disk ? If so, is it<br>
&gt; &gt; also possible to obtain the least used page using some API ?<br>
&gt; would madvise not serve your needs ?<br>
<br>
There&#39;s this word &quot;force&quot; in the question.<br>
<br>
The problem is that madvise() is *advice* to the kernel, not a strict<br>
guarantee.  There&#39;s a difference between &quot;If you need to move pages<br>
to disk, consider these pages first&quot; and &quot;Move them to disk now, whether<br>
you really wanted to or not&quot;.<br>
<br>
</blockquote></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-18 15:10   ` Valdis Klētnieks
  2020-01-19 10:01     ` Sumit Kumar
@ 2020-01-19 10:55     ` Anupam Kapoor
  2020-01-19 11:14       ` Valdis Klētnieks
  1 sibling, 1 reply; 11+ messages in thread
From: Anupam Kapoor @ 2020-01-19 10:55 UTC (permalink / raw)
  To: Valdis Klētnieks; +Cc: Sumit Kumar, kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 2048 bytes --]

On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
>> On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
>>> Does C/C++ provide any API / system call that enables user to force the
>>> application to transfer some its physical pages to swap disk ? If so,
is it
>>> also possible to obtain the least used page using some API ?

>> would madvise not serve your needs ?

> There's this word "force" in the question.

> The problem is that madvise() is *advice* to the kernel, not a strict
> guarantee.  There's a difference between "If you need to move pages
> to disk, consider these pages first" and "Move them to disk now, whether
> you really wanted to or not".

ofcourse (afaik) there is no way for the application to force the
*kernel* to do something like this.

but if _all_ that is required is randomly unmapping some marked
application pages, _that_ can be naively 'done' by the application
itself :)

for example, have a list of unmappable pages and an unmapper thread
which does the unmapping for you (picking any element in the list
at random)

--
kind regards
anupam

In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.


On Sat, Jan 18, 2020 at 3:10 PM Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
> > On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com>
> wrote:
> > > Does C/C++ provide any API / system call that enables user to force the
> > > application to transfer some its physical pages to swap disk ? If so,
> is it
> > > also possible to obtain the least used page using some API ?
> > would madvise not serve your needs ?
>
> There's this word "force" in the question.
>
> The problem is that madvise() is *advice* to the kernel, not a strict
> guarantee.  There's a difference between "If you need to move pages
> to disk, consider these pages first" and "Move them to disk now, whether
> you really wanted to or not".
>
>

[-- Attachment #1.2: Type: text/html, Size: 4081 bytes --]

<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:<br>&gt;&gt; On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar &lt;<a href="mailto:sumit686215@gmail.com">sumit686215@gmail.com</a>&gt; wrote:<br>&gt;&gt;&gt; Does C/C++ provide any API / system call that enables user to force the<br>&gt;&gt;&gt; application to transfer some its physical pages to swap disk ? If so, is it<br>&gt;&gt;&gt; also possible to obtain the least used page using some API ?<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">&gt;&gt; would madvise not serve your needs ?<br><br>&gt; There&#39;s this word &quot;force&quot; in the question.<br><br>&gt; The problem is that madvise() is *advice* to the kernel, not a strict<br>&gt; guarantee.  There&#39;s a difference between &quot;If you need to move pages<br>&gt; to disk, consider these pages first&quot; and &quot;Move them to disk now, whether<br>&gt; you really wanted to or not&quot;.<br clear="all"></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">ofcourse (afaik) there is no way for the application to force the<br>*kernel* to do something like this. </div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">but if _all_ that is required is randomly unmapping some marked </div><div class="gmail_default" style="font-family:monospace,monospace">application pages, _that_ can be naively &#39;done&#39; by the application </div><div class="gmail_default" style="font-family:monospace,monospace">itself :) </div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">for example, have a list of unmappable pages and an unmapper thread </div><div class="gmail_default" style="font-family:monospace,monospace">which does the unmapping for you (picking any element in the list </div><div class="gmail_default" style="font-family:monospace,monospace">at random) <br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">--</div><div class="gmail_default" style="font-family:monospace,monospace">kind regards</div><div class="gmail_default" style="font-family:monospace,monospace">anupam</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font face="monospace, monospace">In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. </font></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 18, 2020 at 3:10 PM Valdis Klētnieks &lt;<a href="mailto:valdis.kletnieks@vt.edu">valdis.kletnieks@vt.edu</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:<br>
&gt; On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar &lt;<a href="mailto:sumit686215@gmail.com" target="_blank">sumit686215@gmail.com</a>&gt; wrote:<br>
&gt; &gt; Does C/C++ provide any API / system call that enables user to force the<br>
&gt; &gt; application to transfer some its physical pages to swap disk ? If so, is it<br>
&gt; &gt; also possible to obtain the least used page using some API ?<br>
&gt; would madvise not serve your needs ?<br>
<br>
There&#39;s this word &quot;force&quot; in the question.<br>
<br>
The problem is that madvise() is *advice* to the kernel, not a strict<br>
guarantee.  There&#39;s a difference between &quot;If you need to move pages<br>
to disk, consider these pages first&quot; and &quot;Move them to disk now, whether<br>
you really wanted to or not&quot;.<br>
<br>
</blockquote></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-19 10:55     ` Anupam Kapoor
@ 2020-01-19 11:14       ` Valdis Klētnieks
  2020-01-19 12:45         ` Anupam Kapoor
  0 siblings, 1 reply; 11+ messages in thread
From: Valdis Klētnieks @ 2020-01-19 11:14 UTC (permalink / raw)
  To: Anupam Kapoor; +Cc: Sumit Kumar, kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 385 bytes --]

On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said:

> but if _all_ that is required is randomly unmapping some marked
> application pages, _that_ can be naively 'done' by the application
> itself :)

Note that in this case, "naively" includes "not remembering to consider
that the page being unmapped may have contained data we'd rather
have kept by flushing the page to disk" :)

[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-19 11:14       ` Valdis Klētnieks
@ 2020-01-19 12:45         ` Anupam Kapoor
  2020-01-19 13:18           ` Valdis Klētnieks
  0 siblings, 1 reply; 11+ messages in thread
From: Anupam Kapoor @ 2020-01-19 12:45 UTC (permalink / raw)
  To: Valdis Klētnieks; +Cc: Sumit Kumar, kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 1398 bytes --]

>> but if _all_ that is required is randomly unmapping some marked
>> application pages, _that_ can be naively 'done' by the application
>> itself :)

> Note that in this case, "naively" includes "not remembering to consider
> that the page being unmapped may have contained data we'd rather
> have kept by flushing the page to disk" :)

but is it that bad ?

before marking a page unmappable, the application has full control
over what it wants to do with the data, and can choose to dump it
to the appropriate destination.

or if enough information is available, the unmapper thread can itself
play that role.

thinking some more about it, application has full control over the
unmap/resurrect behavior. though latter might not be as
transparent...

--
kind regards
anupam

In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.


On Sun, Jan 19, 2020 at 11:14 AM Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said:
>
> > but if _all_ that is required is randomly unmapping some marked
> > application pages, _that_ can be naively 'done' by the application
> > itself :)
>
> Note that in this case, "naively" includes "not remembering to consider
> that the page being unmapped may have contained data we'd rather
> have kept by flushing the page to disk" :)
>

[-- Attachment #1.2: Type: text/html, Size: 2593 bytes --]

<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">&gt;&gt; but if _all_ that is required is randomly unmapping some marked<br>&gt;&gt; application pages, _that_ can be naively &#39;done&#39; by the application<br>&gt;&gt; itself :)<br><br>&gt; Note that in this case, &quot;naively&quot; includes &quot;not remembering to consider<br>&gt; that the page being unmapped may have contained data we&#39;d rather<br>&gt; have kept by flushing the page to disk&quot; :)<br clear="all"></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">but is it that bad ? <br><br>before marking a page unmappable, the application has full control<br>over what it wants to do with the data, and can choose to dump it<br>to the appropriate destination.<br><br>or if enough information is available, the unmapper thread can itself<br>play that role.<br><br>thinking some more about it, application has full control over the<br>unmap/resurrect behavior. though latter might not be as<br>transparent...<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">--</div><div class="gmail_default" style="font-family:monospace,monospace">kind regards</div><div class="gmail_default" style="font-family:monospace,monospace">anupam</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font face="monospace, monospace">In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. </font></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jan 19, 2020 at 11:14 AM Valdis Klētnieks &lt;<a href="mailto:valdis.kletnieks@vt.edu">valdis.kletnieks@vt.edu</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said:<br>
<br>
&gt; but if _all_ that is required is randomly unmapping some marked<br>
&gt; application pages, _that_ can be naively &#39;done&#39; by the application<br>
&gt; itself :)<br>
<br>
Note that in this case, &quot;naively&quot; includes &quot;not remembering to consider<br>
that the page being unmapped may have contained data we&#39;d rather<br>
have kept by flushing the page to disk&quot; :)<br>
</blockquote></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-19 12:45         ` Anupam Kapoor
@ 2020-01-19 13:18           ` Valdis Klētnieks
  2020-01-19 16:01             ` Anupam Kapoor
  0 siblings, 1 reply; 11+ messages in thread
From: Valdis Klētnieks @ 2020-01-19 13:18 UTC (permalink / raw)
  To: Anupam Kapoor; +Cc: Sumit Kumar, kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 1213 bytes --]

On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said:

> > Note that in this case, "naively" includes "not remembering to consider
> > that the page being unmapped may have contained data we'd rather
> > have kept by flushing the page to disk" :)
>
> but is it that bad ?
>
> before marking a page unmappable, the application has full control
> over what it wants to do with the data, and can choose to dump it
> to the appropriate destination.

Yes, but now you're getting into more code that has to be written, including
code to marshal things like binary trees into a savable format, and more code
to read them back at a later time. Plus all the fun if the tree has hundreds of thousands
or millions of entries, and how to deal with it if some parts of the tree have been
released and saved to disk, or if the 4K page contained members of several different
data structures - in other words, you probably just decided to write your own backing store,
garbage collector, and virtual object manager for your heap.

As I said - it's a naive approach that ends up following the 90/10 rule:
the easy 90% of it takes the first 90% of the time to code it, and the difficult
10% takes the other 90% of the time... :)

[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-19 13:18           ` Valdis Klētnieks
@ 2020-01-19 16:01             ` Anupam Kapoor
  0 siblings, 0 replies; 11+ messages in thread
From: Anupam Kapoor @ 2020-01-19 16:01 UTC (permalink / raw)
  To: Valdis Klētnieks; +Cc: Sumit Kumar, kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 1909 bytes --]

On Sun, 19 Jan 2020 at 6:48 PM Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said:
>
> > > Note that in this case, "naively" includes "not remembering to consider
> > > that the page being unmapped may have contained data we'd rather
> > > have kept by flushing the page to disk" :)
> >
> > but is it that bad ?
> >
> > before marking a page unmappable, the application has full control
> > over what it wants to do with the data, and can choose to dump it
> > to the appropriate destination.
>
> Yes, but now you're getting into more code that has to be written,
> including
> code to marshal things like binary trees into a savable format, and more
> code
> to read them back at a later time. Plus all the fun if the tree has
> hundreds of thousands
> or millions of entries, and how to deal with it if some parts of the tree
> have been
> released and saved to disk, or if the 4K page contained members of several
> different
> data structures - in other words, you probably just decided to write your
> own backing store,
> garbage collector, and virtual object manager for your heap.
>
> As I said - it's a naive approach that ends up following the 90/10 rule:
> the easy 90% of it takes the first 90% of the time to code it, and the
> difficult
> 10% takes the other 90% of the time... :)


well sure, if you try to replicate everything that exists below libc, then
there is little hope.

however if your application’s data can be serialized/deserialized, then i
_suspect_ it might not be too much of work.

for example, if i am maintaining l2 forwarding table entries then it might
be possible to have, on an average fixed number of pages representing this
cache...

—
kind regards
anupam

>
> --
In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

[-- Attachment #1.2: Type: text/html, Size: 2749 bytes --]

<div><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 19 Jan 2020 at 6:48 PM Valdis Klētnieks &lt;<a href="mailto:valdis.kletnieks@vt.edu">valdis.kletnieks@vt.edu</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said:<br>
<br>
&gt; &gt; Note that in this case, &quot;naively&quot; includes &quot;not remembering to consider<br>
&gt; &gt; that the page being unmapped may have contained data we&#39;d rather<br>
&gt; &gt; have kept by flushing the page to disk&quot; :)<br>
&gt;<br>
&gt; but is it that bad ?<br>
&gt;<br>
&gt; before marking a page unmappable, the application has full control<br>
&gt; over what it wants to do with the data, and can choose to dump it<br>
&gt; to the appropriate destination.<br>
<br>
Yes, but now you&#39;re getting into more code that has to be written, including<br>
code to marshal things like binary trees into a savable format, and more code<br>
to read them back at a later time. Plus all the fun if the tree has hundreds of thousands<br>
or millions of entries, and how to deal with it if some parts of the tree have been<br>
released and saved to disk, or if the 4K page contained members of several different<br>
data structures - in other words, you probably just decided to write your own backing store,<br>
garbage collector, and virtual object manager for your heap.<br>
<br>
As I said - it&#39;s a naive approach that ends up following the 90/10 rule:<br>
the easy 90% of it takes the first 90% of the time to code it, and the difficult<br>
10% takes the other 90% of the time... :)</blockquote><div dir="auto"><br></div><div dir="auto">well sure, if you try to replicate everything that exists below libc, then there is little hope.</div><div dir="auto"><br></div><div dir="auto">however if your application’s data can be serialized/deserialized, then i _suspect_ it might not be too much of work. </div><div dir="auto"><br></div><div dir="auto">for example, if i am maintaining l2 forwarding table entries then it might be possible to have, on an average fixed number of pages representing this cache...</div><div dir="auto"><br></div><div dir="auto">—</div><div dir="auto">kind regards </div><div dir="auto">anupam </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font face="monospace, monospace">In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. </font></div></div></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: transfer physical memory page to swap disk
  2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar
  2020-01-15 12:42 ` aleix sanchis ramírez
  2020-01-15 12:53 ` Anupam Kapoor
@ 2020-01-19 16:59 ` Bernd Petrovitsch
  2 siblings, 0 replies; 11+ messages in thread
From: Bernd Petrovitsch @ 2020-01-19 16:59 UTC (permalink / raw)
  To: Sumit Kumar; +Cc: kernelnewbies

Hi all!

To answer it from a somewhat different angle:

On Wed, 2020-01-15 at 18:01 +0530, Sumit Kumar wrote:
> Hi,
> Does C/C++ provide any API / system call that enables user to force the
> application to transfer some its physical pages to swap disk ? If so, is it

Short answer: No (and you do not want such a thing in the first
place).

Longer answer: I don't think any normal application would force
some memory pages to the swap space (as it slows down) - quite
the contrary.
You can use open(..., O_DIRECT) to minimize cache effects
if that may help). WRF, you can implement pseudo-swapping
with that within your application.
madvise() has been mentioned in other mails.
One application has (usually) no knowledge about the rest of
the system - neither the amount of RAM nor the workload
(which may change over time) - or is at least written
that way.
So it's IMHO better to leave the kernels heuristics work
system-wide instead of "micro-optimizing" in some
application which will sooner or later interfere with
the kernels heuristics.

And - always - the next question is: how could
this API be abused/creatively used for the own
benefit/...?

BTW: what's a real intended real-world application?

> also possible to obtain the least used page using some API ?
> 
> AFAIK, linux kernel is supposed to do this as part of memory management. I

Yes - if there actually is swap space available (und don't
underestimate the number of systems running without a swap
space and CONFIG_SWAP=n in /boot/*config* - desktop and
servers are not everything in the world, more like a
small part ....).

> want to know if the kernel also exposes some API to enable users to control
> their application's memory management. I have many applications running at
> a time that cause too much memory consumption. I believe that experimenting
> with memory management can help.

The application can (and should) control it's memory management
anyways - just allocate as much as the application needs and
free it when it's no longer used.
For larger memeory areas (possibly with a self-build
malloc/free-equivalent or memory pools), the application
can mmap() it and munmap() it simply when it's done.

You can experiment with setting the process limits via
setrlimit() to smaller or larger values.

MfG,
	Bernd
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
                     LUGA : http://www.luga.at


_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, back to index

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-15 12:31 transfer physical memory page to swap disk Sumit Kumar
2020-01-15 12:42 ` aleix sanchis ramírez
2020-01-15 12:53 ` Anupam Kapoor
2020-01-18 15:10   ` Valdis Klētnieks
2020-01-19 10:01     ` Sumit Kumar
2020-01-19 10:55     ` Anupam Kapoor
2020-01-19 11:14       ` Valdis Klētnieks
2020-01-19 12:45         ` Anupam Kapoor
2020-01-19 13:18           ` Valdis Klētnieks
2020-01-19 16:01             ` Anupam Kapoor
2020-01-19 16:59 ` Bernd Petrovitsch

Kernel Newbies archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kernelnewbies/0 kernelnewbies/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kernelnewbies kernelnewbies/ https://lore.kernel.org/kernelnewbies \
		kernelnewbies@kernelnewbies.org
	public-inbox-index kernelnewbies

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernelnewbies.kernelnewbies


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git