Understanding memory fragmentation in Linux kernel

Understanding memory fragmentation in Linux kernel

[Amit Agarwal]

[-- Attachment #1.1: Type: text/plain, Size: 624 bytes --]

Hi All, 

I am trying to understand memory fragmentation and how to understand
and/or analyze the same. 

Is there some detailed documentation on pagetypeinfo and buddyinfo files
present in the proc directory? Am I looking at right files to understand
if the memory is fragmented. 

Problem Description: 

One of the applications does a few 200-300Mb calloc's during start-up.
On one of the production servers, the application does not start up
although there should be enough RAM available for application to start.
I am suspecting that this is because of fragmentation and would like to
confirm the same.

-- 
Thanks,
-aka

[-- Attachment #1.2: Type: text/html, Size: 966 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Understanding memory fragmentation in Linux kernel

[Manish Katiyar]

[-- Attachment #1.1: Type: text/plain, Size: 1143 bytes --]

On Sun, Dec 30, 2018, 10:03 PM Amit Agarwal <amit@amit-agarwal.co.in wrote:

> Hi All,
>
>
> I am trying to understand memory fragmentation and how to understand
> and/or analyze the same.
>
> Is there some detailed documentation on pagetypeinfo and buddyinfo files
> present in the proc directory? Am I looking at right files to understand if
> the memory is fragmented.
>
>
> Problem Description:
>
> One of the applications does a few 200-300Mb calloc's during start-up. On
> one of the production servers, the application does not start up although
> there should be enough RAM available for application to start. I am
> suspecting that this is because of fragmentation and would like to confirm
> the same.
>

How do you know it is because of memory ? If you have sufficient RAM then
it should be able to allocate. Does you application fail with ENOMEM ?

Without knowing the error code from application it's hard to suggest
anything.


-- 
> Thanks,
> -aka
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

[-- Attachment #1.2: Type: text/html, Size: 2227 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Understanding memory fragmentation in Linux kernel

[Amit Agarwal]

[-- Attachment #1.1: Type: text/plain, Size: 739 bytes --]

On 2018-12-31 06:22, Manish Katiyar wrote:

> How do you know it is because of memory ? If you have sufficient RAM then it should be able to allocate. Does you application fail with ENOMEM ? 
> Without knowing the error code from application it's hard to suggest anything.

There is no debugging enabled on site and there is nothing we can do now
to enable it and it is happening in only one site. But from whatever
logs I have, it is failing in calloc. 

I am not requesting help in resolving the issue with application (I will
be able to debug that), but only need help in understanding the memory
fragmentation part. ( I gave details of original problem so that I can
explain what I am trying to understand and why). 

-- 
Thanks, 
-aka

[-- Attachment #1.2: Type: text/html, Size: 1299 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Understanding memory fragmentation in Linux kernel

[Manish Katiyar]

[-- Attachment #1.1: Type: text/plain, Size: 931 bytes --]

On Sun, Dec 30, 2018, 10:28 PM Amit Agarwal <amit@amit-agarwal.co.in wrote:

> On 2018-12-31 06:22, Manish Katiyar wrote:
>
>
> How do you know it is because of memory ? If you have sufficient RAM then
> it should be able to allocate. Does you application fail with ENOMEM ?
> Without knowing the error code from application it's hard to suggest
> anything.
>
>
> There is no debugging enabled on site and there is nothing we can do now
> to enable it and it is happening in only one site. But from whatever logs I
> have, it is failing in calloc.
>
> I am not requesting help in resolving the issue with application (I will
> be able to debug that), but only need help in understanding the memory
> fragmentation part. ( I gave details of original problem so that I can
> explain what I am trying to understand and wh
>

Maybe you can try

echo m > /proc/sysrq-trigger


Or looking at /proc/buddyinfo



>
> --
> Thanks,
> -aka
>

[-- Attachment #1.2: Type: text/html, Size: 2726 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Understanding memory fragmentation in Linux kernel

[Amit Agarwal]

On Sun, 2018-12-30 at 22:54 -0800, Manish Katiyar wrote:
> 
> Maybe you can try
> 
> echo m > /proc/sysrq-trigger
I will check this and see if this can provide something useful for me.
Did not know about this.

> 
> Or looking at /proc/buddyinfo
I looked at buddyinfo and hence asked for more documentation on that.
Does the pages that buddyinfo list under any "order at" are contigeous
or are they located somewhere in memory? For ex: If buddyinfo shows 2
pages for 0th order, does it mean that there are 2 contigeous pages of
2M each?

> 

-- 
Thanks,
-aka



_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Understanding memory fragmentation in Linux kernel

[Richard Siegfried]

[-- Attachment #1.1.1: Type: text/plain, Size: 702 bytes --]

On 31/12/2018 07:28, Amit Agarwal wrote:
> I am not requesting help in resolving the issue with application (I will
> be able to debug that), but only need help in understanding the memory
> fragmentation part. ( I gave details of original problem so that I can
> explain what I am trying to understand and why). 
To get a deeper understanding of memomry managment (and fragmentation)
on linux I would recommend the Memory Managment case study chapter of
"Modern Operating Systems" from Andrew S Tanenbaum.

It has a own Case Study Study Part with Chapters for several modern
Operating Systems, one being Linux. There it covers its Memory managment
in nice detail.

Thanks,
-- Richard


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies