Kernel Newbies archive on lore.kernel.org
 help / Atom feed
* Understanding memory fragmentation in Linux kernel
@ 2018-12-31  6:02 Amit Agarwal
  2018-12-31  6:22 ` Manish Katiyar
  0 siblings, 1 reply; 6+ messages in thread
From: Amit Agarwal @ 2018-12-31  6:02 UTC (permalink / raw)
  To: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 624 bytes --]

Hi All, 

I am trying to understand memory fragmentation and how to understand
and/or analyze the same. 

Is there some detailed documentation on pagetypeinfo and buddyinfo files
present in the proc directory? Am I looking at right files to understand
if the memory is fragmented. 

Problem Description: 

One of the applications does a few 200-300Mb calloc's during start-up.
On one of the production servers, the application does not start up
although there should be enough RAM available for application to start.
I am suspecting that this is because of fragmentation and would like to
confirm the same.

-- 
Thanks,
-aka

[-- Attachment #1.2: Type: text/html, Size: 966 bytes --]

<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 10pt; font-family: Verdana,Geneva,sans-serif'>
<p>Hi All,</p>
<p><br /></p>
<p>I am trying to understand memory fragmentation and how to understand and/or analyze the same.</p>
<p>Is there some detailed documentation on pagetypeinfo and buddyinfo files present in the proc directory? Am I looking at right files to understand if the memory is fragmented.</p>
<p><br /></p>
<p>Problem Description:</p>
<p>One of the applications does a few 200-300Mb calloc's during start-up. On one of the production servers, the application does not start up although there should be enough RAM available for application to start. I am suspecting that this is because of fragmentation and would like to confirm the same.</p>
<div>-- <br />
<div class="pre" style="margin: 0; padding: 0; font-family: monospace">Thanks,<br /> -aka</div>
</div>
</body></html>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Understanding memory fragmentation in Linux kernel
  2018-12-31  6:02 Understanding memory fragmentation in Linux kernel Amit Agarwal
@ 2018-12-31  6:22 ` Manish Katiyar
  2018-12-31  6:28   ` Amit Agarwal
  0 siblings, 1 reply; 6+ messages in thread
From: Manish Katiyar @ 2018-12-31  6:22 UTC (permalink / raw)
  To: Amit Agarwal; +Cc: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 1143 bytes --]

On Sun, Dec 30, 2018, 10:03 PM Amit Agarwal <amit@amit-agarwal.co.in wrote:

> Hi All,
>
>
> I am trying to understand memory fragmentation and how to understand
> and/or analyze the same.
>
> Is there some detailed documentation on pagetypeinfo and buddyinfo files
> present in the proc directory? Am I looking at right files to understand if
> the memory is fragmented.
>
>
> Problem Description:
>
> One of the applications does a few 200-300Mb calloc's during start-up. On
> one of the production servers, the application does not start up although
> there should be enough RAM available for application to start. I am
> suspecting that this is because of fragmentation and would like to confirm
> the same.
>

How do you know it is because of memory ? If you have sufficient RAM then
it should be able to allocate. Does you application fail with ENOMEM ?

Without knowing the error code from application it's hard to suggest
anything.


-- 
> Thanks,
> -aka
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

[-- Attachment #1.2: Type: text/html, Size: 2227 bytes --]

<div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr">On Sun, Dec 30, 2018, 10:03 PM Amit Agarwal &lt;<a href="mailto:amit@amit-agarwal.co.in">amit@amit-agarwal.co.in</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:10pt;font-family:Verdana,Geneva,sans-serif">
<p>Hi All,</p>
<p><br></p>
<p>I am trying to understand memory fragmentation and how to understand and/or analyze the same.</p>
<p>Is there some detailed documentation on pagetypeinfo and buddyinfo files present in the proc directory? Am I looking at right files to understand if the memory is fragmented.</p>
<p><br></p>
<p>Problem Description:</p>
<p>One of the applications does a few 200-300Mb calloc&#39;s during start-up. On one of the production servers, the application does not start up although there should be enough RAM available for application to start. I am suspecting that this is because of fragmentation and would like to confirm the same.</p></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">How do you know it is because of memory ? If you have sufficient RAM then it should be able to allocate. Does you application fail with ENOMEM ?</div><div dir="auto"><br></div><div dir="auto">Without knowing the error code from application it&#39;s hard to suggest anything.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:10pt;font-family:Verdana,Geneva,sans-serif">
<div>-- <br>
<div class="m_-8541251558628996521pre" style="margin:0;padding:0;font-family:monospace">Thanks,<br> -aka</div>
</div>
</div>
_______________________________________________<br>
Kernelnewbies mailing list<br>
<a href="mailto:Kernelnewbies@kernelnewbies.org" target="_blank" rel="noreferrer">Kernelnewbies@kernelnewbies.org</a><br>
<a href="https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies" rel="noreferrer noreferrer" target="_blank">https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies</a><br>
</blockquote></div></div></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Understanding memory fragmentation in Linux kernel
  2018-12-31  6:22 ` Manish Katiyar
@ 2018-12-31  6:28   ` Amit Agarwal
  2018-12-31  6:54     ` Manish Katiyar
  2019-01-13 18:36     ` Richard Siegfried
  0 siblings, 2 replies; 6+ messages in thread
From: Amit Agarwal @ 2018-12-31  6:28 UTC (permalink / raw)
  To: Manish Katiyar; +Cc: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 739 bytes --]

On 2018-12-31 06:22, Manish Katiyar wrote:

> How do you know it is because of memory ? If you have sufficient RAM then it should be able to allocate. Does you application fail with ENOMEM ? 
> Without knowing the error code from application it's hard to suggest anything.

There is no debugging enabled on site and there is nothing we can do now
to enable it and it is happening in only one site. But from whatever
logs I have, it is failing in calloc. 

I am not requesting help in resolving the issue with application (I will
be able to debug that), but only need help in understanding the memory
fragmentation part. ( I gave details of original problem so that I can
explain what I am trying to understand and why). 

-- 
Thanks, 
-aka

[-- Attachment #1.2: Type: text/html, Size: 1299 bytes --]

<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 10pt; font-family: Verdana,Geneva,sans-serif'>
<p>On 2018-12-31 06:22, Manish Katiyar wrote:</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div dir="auto"><br />
<div dir="auto">How do you know it is because of memory ? If you have sufficient RAM then it should be able to allocate. Does you application fail with ENOMEM ?</div>
<div dir="auto">Without knowing the error code from application it's hard to suggest anything.</div>
</div>
</blockquote>
<div>&nbsp;</div>
<div>There is no debugging enabled on site and there is nothing we can do now to enable it and it is happening in only one site. But from whatever logs I have, it is failing in calloc.</div>
<div>&nbsp;</div>
<div>I am not requesting help in resolving the issue with application (I will be able to debug that), but only need help in understanding the memory fragmentation part. ( I gave details of original problem so that I can explain what I am trying to understand and why).</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>--</div>
<div>Thanks,</div>
<div>-aka</div>

</body></html>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Understanding memory fragmentation in Linux kernel
  2018-12-31  6:28   ` Amit Agarwal
@ 2018-12-31  6:54     ` Manish Katiyar
  2018-12-31  7:14       ` Amit Agarwal
  2019-01-13 18:36     ` Richard Siegfried
  1 sibling, 1 reply; 6+ messages in thread
From: Manish Katiyar @ 2018-12-31  6:54 UTC (permalink / raw)
  To: Amit Agarwal; +Cc: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 931 bytes --]

On Sun, Dec 30, 2018, 10:28 PM Amit Agarwal <amit@amit-agarwal.co.in wrote:

> On 2018-12-31 06:22, Manish Katiyar wrote:
>
>
> How do you know it is because of memory ? If you have sufficient RAM then
> it should be able to allocate. Does you application fail with ENOMEM ?
> Without knowing the error code from application it's hard to suggest
> anything.
>
>
> There is no debugging enabled on site and there is nothing we can do now
> to enable it and it is happening in only one site. But from whatever logs I
> have, it is failing in calloc.
>
> I am not requesting help in resolving the issue with application (I will
> be able to debug that), but only need help in understanding the memory
> fragmentation part. ( I gave details of original problem so that I can
> explain what I am trying to understand and wh
>

Maybe you can try

echo m > /proc/sysrq-trigger


Or looking at /proc/buddyinfo



>
> --
> Thanks,
> -aka
>

[-- Attachment #1.2: Type: text/html, Size: 2726 bytes --]

<div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr">On Sun, Dec 30, 2018, 10:28 PM Amit Agarwal &lt;<a href="mailto:amit@amit-agarwal.co.in">amit@amit-agarwal.co.in</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:10pt;font-family:Verdana,Geneva,sans-serif">
<p>On 2018-12-31 06:22, Manish Katiyar wrote:</p>
<blockquote type="cite" style="padding:0 0.4em;border-left:#1010ff 2px solid;margin:0">
<div dir="auto"><br>
<div dir="auto">How do you know it is because of memory ? If you have sufficient RAM then it should be able to allocate. Does you application fail with ENOMEM ?</div>
<div dir="auto">Without knowing the error code from application it&#39;s hard to suggest anything.</div>
</div>
</blockquote>
<div> </div>
<div>There is no debugging enabled on site and there is nothing we can do now to enable it and it is happening in only one site. But from whatever logs I have, it is failing in calloc.</div>
<div> </div>
<div>I am not requesting help in resolving the issue with application (I will be able to debug that), but only need help in understanding the memory fragmentation part. ( I gave details of original problem so that I can explain what I am trying to understand and wh</div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Maybe you can try</div><div dir="auto"><br></div><div dir="auto"><pre style="margin-top:0px;margin-bottom:0.8em;padding:10px;border:0px;line-height:inherit;font-family:consolas,menlo,monaco,&quot;lucida console&quot;,&quot;liberation mono&quot;,&quot;dejavu sans mono&quot;,&quot;bitstream vera sans mono&quot;,&quot;courier new&quot;,monospace;font-size:13px;vertical-align:baseline;background-color:rgb(240,240,240);max-height:300px;color:rgb(12,13,14)"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:consolas,menlo,monaco,&quot;lucida console&quot;,&quot;liberation mono&quot;,&quot;dejavu sans mono&quot;,&quot;bitstream vera sans mono&quot;,&quot;courier new&quot;,monospace;vertical-align:baseline;max-height:300px">echo m &gt; /proc/sysrq-trigger</code></pre></div><div dir="auto"><br></div><div dir="auto">Or looking at /proc/buddyinfo</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:10pt;font-family:Verdana,Geneva,sans-serif">
<div> </div>
<div> </div>
<div>--</div>
<div>Thanks,</div>
<div>-aka</div>

</div>
</blockquote></div></div></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Understanding memory fragmentation in Linux kernel
  2018-12-31  6:54     ` Manish Katiyar
@ 2018-12-31  7:14       ` Amit Agarwal
  0 siblings, 0 replies; 6+ messages in thread
From: Amit Agarwal @ 2018-12-31  7:14 UTC (permalink / raw)
  To: Manish Katiyar; +Cc: kernelnewbies


On Sun, 2018-12-30 at 22:54 -0800, Manish Katiyar wrote:
> 
> Maybe you can try
> 
> echo m > /proc/sysrq-trigger
I will check this and see if this can provide something useful for me.
Did not know about this.

> 
> Or looking at /proc/buddyinfo
I looked at buddyinfo and hence asked for more documentation on that.
Does the pages that buddyinfo list under any "order at" are contigeous
or are they located somewhere in memory? For ex: If buddyinfo shows 2
pages for 0th order, does it mean that there are 2 contigeous pages of
2M each?

> 

-- 
Thanks,
-aka



_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Understanding memory fragmentation in Linux kernel
  2018-12-31  6:28   ` Amit Agarwal
  2018-12-31  6:54     ` Manish Katiyar
@ 2019-01-13 18:36     ` Richard Siegfried
  1 sibling, 0 replies; 6+ messages in thread
From: Richard Siegfried @ 2019-01-13 18:36 UTC (permalink / raw)
  To: kernelnewbies

[-- Attachment #1.1.1: Type: text/plain, Size: 702 bytes --]

On 31/12/2018 07:28, Amit Agarwal wrote:
> I am not requesting help in resolving the issue with application (I will
> be able to debug that), but only need help in understanding the memory
> fragmentation part. ( I gave details of original problem so that I can
> explain what I am trying to understand and why). 
To get a deeper understanding of memomry managment (and fragmentation)
on linux I would recommend the Memory Managment case study chapter of
"Modern Operating Systems" from Andrew S Tanenbaum.

It has a own Case Study Study Part with Chapters for several modern
Operating Systems, one being Linux. There it covers its Memory managment
in nice detail.

Thanks,
-- Richard


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-31  6:02 Understanding memory fragmentation in Linux kernel Amit Agarwal
2018-12-31  6:22 ` Manish Katiyar
2018-12-31  6:28   ` Amit Agarwal
2018-12-31  6:54     ` Manish Katiyar
2018-12-31  7:14       ` Amit Agarwal
2019-01-13 18:36     ` Richard Siegfried

Kernel Newbies archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kernelnewbies/0 kernelnewbies/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kernelnewbies kernelnewbies/ https://lore.kernel.org/kernelnewbies \
		kernelnewbies@kernelnewbies.org kernelnewbies@archiver.kernel.org
	public-inbox-index kernelnewbies


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernelnewbies.kernelnewbies


AGPL code for this site: git clone https://public-inbox.org/ public-inbox