malloc problem to allocate very large blocks

malloc problem to allocate very large blocks

[Tung-Han Hsieh]

Hello,

I am developing applications which requires more than 2GB memory.
But I found that in my Linux system the malloc() cannot allocate
more than 2GB memory. Here is the details of my system:

CPU:    Pentium 4 2.53 GHz
RAM:    2 GB
Swap:   512 MB
OS:	Debian-3.0 stable
Kernel: 2.4.20
gcc:	2.95.4 20011002
glibc:  2.2.5-6

In theory, in a 32-bits machine the maximum allocatable memory
is up to 4GB. But in the following very simple testing program:

=====================================================================
#include <stdio.h>
#include <stdlib.h>

main()
{
    size_t l;
    char *s1=NULL, *s2=NULL;

    l = 1024*1024*1024;

    s1 = malloc(l);
    s2 = malloc(l);
    if (! s1) printf("s1 malloc failed\n");
    if (! s2) printf("s2 malloc failed\n");
}
=====================================================================

only the block for s1 can be allocated. Further, if I change the
program to

=====================================================================
#include <stdio.h>
#include <stdlib.h>

main()
{
    size_t l;
    char *s1=NULL;

    l = 2*1024*1024*1024;

    s1 = malloc(l);
    if (! s1) printf("s1 malloc failed\n");
}
=====================================================================

the gcc complier complain to me that "foo.c:9: warning: integer overflow
in expression" during the compilation (I use: "gcc foo.c" to compile it),
and the block for s1 cannot be allocated at all. I am wondering if there
is any way to overcome the 2GB limit.

Thank you very much for your reply in advance.


Best Regards,

T.H.Hsieh

Re: malloc problem to allocate very large blocks

[jw schultz]

On Mon, Jul 28, 2003 at 02:44:28PM +0800, Tung-Han Hsieh wrote:
> Hello,
> 
> I am developing applications which requires more than 2GB memory.
> But I found that in my Linux system the malloc() cannot allocate
> more than 2GB memory. Here is the details of my system:

Malloc is a library function, not a syscall.

That said i'm not surprised.  Malloc front-ends sbrk and
sometimes mmap.  Sbrk uses a ptrdiff_t to indicate the size
desired.  ptrdiff_t is signed and on 32 bit platforms should
be 32 bits.  Therefore, the maximum you can allocate at one
time with sbrk is going to be 2GB.  That malloc would
inherit this limitation is to be expected.

If you need to allocate that much memory in one chunk, don't
use malloc.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Re: malloc problem to allocate very large blocks

[jw schultz]

On Mon, Jul 28, 2003 at 01:37:30AM -0700, jw schultz wrote:
> On Mon, Jul 28, 2003 at 02:44:28PM +0800, Tung-Han Hsieh wrote:
> > Hello,
> > 
> > I am developing applications which requires more than 2GB memory.
> > But I found that in my Linux system the malloc() cannot allocate
> > more than 2GB memory. Here is the details of my system:
> 
> Malloc is a library function, not a syscall.
> 
> That said i'm not surprised.  Malloc front-ends sbrk and
> sometimes mmap.  Sbrk uses a ptrdiff_t to indicate the size
> desired.  ptrdiff_t is signed and on 32 bit platforms should
> be 32 bits.  Therefore, the maximum you can allocate at one
> time with sbrk is going to be 2GB.  That malloc would
> inherit this limitation is to be expected.
> 
> If you need to allocate that much memory in one chunk, don't
> use malloc.

And regarding trying to get it in multiple chunks.  That is
likely to fail with sbrk due to the way the address space is
assigned to the various memory areas.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Re: malloc problem to allocate very large blocks

[Nagendra Singh Tomar]

AFAIK malloc will not return you memory more than the total virtual memory 
(RAM+swap) in the system. So if you want more than 2GB allocations from 
malloc, make sure you have at least 2GB virtual mem, keeping aside some 
space for the kernel.

--tomar

On Mon, 28 Jul 2003, Tung-Han Hsieh wrote:

> Hello,
> 
> I am developing applications which requires more than 2GB memory.
> But I found that in my Linux system the malloc() cannot allocate
> more than 2GB memory. Here is the details of my system:
> 
> CPU:    Pentium 4 2.53 GHz
> RAM:    2 GB
> Swap:   512 MB
> OS:	Debian-3.0 stable
> Kernel: 2.4.20
> gcc:	2.95.4 20011002
> glibc:  2.2.5-6
> 
> In theory, in a 32-bits machine the maximum allocatable memory
> is up to 4GB. But in the following very simple testing program:
> 
> =====================================================================
> #include <stdio.h>
> #include <stdlib.h>
> 
> main()
> {
>     size_t l;
>     char *s1=NULL, *s2=NULL;
> 
>     l = 1024*1024*1024;
> 
>     s1 = malloc(l);
>     s2 = malloc(l);
>     if (! s1) printf("s1 malloc failed\n");
>     if (! s2) printf("s2 malloc failed\n");
> }
> =====================================================================
> 
> only the block for s1 can be allocated. Further, if I change the
> program to
> 
> =====================================================================
> #include <stdio.h>
> #include <stdlib.h>
> 
> main()
> {
>     size_t l;
>     char *s1=NULL;
> 
>     l = 2*1024*1024*1024;
> 
>     s1 = malloc(l);
>     if (! s1) printf("s1 malloc failed\n");
> }
> =====================================================================
> 
> the gcc complier complain to me that "foo.c:9: warning: integer overflow
> in expression" during the compilation (I use: "gcc foo.c" to compile
> it),
> and the block for s1 cannot be allocated at all. I am wondering if there
> is any way to overcome the 2GB limit.
> 
> Thank you very much for your reply in advance.
> 
> 
> Best Regards,
> 
> T.H.Hsieh
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: malloc problem to allocate very large blocks

[Nagendra Singh Tomar]

I missed the system details, you already have >2GB virtual memory.

--tomar
On Tue, 29 Jul 2003, Tomar, Nagendra wrote:

> AFAIK malloc will not return you memory more than the total virtual
> memory 
> (RAM+swap) in the system. So if you want more than 2GB allocations from 
> malloc, make sure you have at least 2GB virtual mem, keeping aside some 
> space for the kernel.
> 
> --tomar
> 
> On Mon, 28 Jul 2003, Tung-Han Hsieh wrote:
> 
> > Hello,
> > 
> > I am developing applications which requires more than 2GB memory.
> > But I found that in my Linux system the malloc() cannot allocate
> > more than 2GB memory. Here is the details of my system:
> > 
> > CPU:    Pentium 4 2.53 GHz
> > RAM:    2 GB
> > Swap:   512 MB
> > OS:	Debian-3.0 stable
> > Kernel: 2.4.20
> > gcc:	2.95.4 20011002
> > glibc:  2.2.5-6
> > 
> > In theory, in a 32-bits machine the maximum allocatable memory
> > is up to 4GB. But in the following very simple testing program:
> > 
> > =====================================================================
> > #include <stdio.h>
> > #include <stdlib.h>
> > 
> > main()
> > {
> >     size_t l;
> >     char *s1=NULL, *s2=NULL;
> > 
> >     l = 1024*1024*1024;
> > 
> >     s1 = malloc(l);
> >     s2 = malloc(l);
> >     if (! s1) printf("s1 malloc failed\n");
> >     if (! s2) printf("s2 malloc failed\n");
> > }
> > =====================================================================
> > 
> > only the block for s1 can be allocated. Further, if I change the
> > program to
> > 
> > =====================================================================
> > #include <stdio.h>
> > #include <stdlib.h>
> > 
> > main()
> > {
> >     size_t l;
> >     char *s1=NULL;
> > 
> >     l = 2*1024*1024*1024;
> > 
> >     s1 = malloc(l);
> >     if (! s1) printf("s1 malloc failed\n");
> > }
> > =====================================================================
> > 
> > the gcc complier complain to me that "foo.c:9: warning: integer
> overflow
> > in expression" during the compilation (I use: "gcc foo.c" to compile
> > it),
> > and the block for s1 cannot be allocated at all. I am wondering if
> there
> > is any way to overcome the 2GB limit.
> > 
> > Thank you very much for your reply in advance.
> > 
> > 
> > Best Regards,
> > 
> > T.H.Hsieh
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> linux-kernel"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> > 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: malloc problem to allocate very large blocks

[Alan Cox]

On Maw, 2003-07-29 at 05:58, Nagendra Singh Tomar wrote:
> AFAIK malloc will not return you memory more than the total virtual memory 
> (RAM+swap) in the system. So if you want more than 2GB allocations from 
> malloc, make sure you have at least 2GB virtual mem, keeping aside some 
> space for the kernel.

On the default memory settings it may do. However a request for 2Gb of
memory requires there is a free 2Gb of address space to map it into -
which may not be true because of things like shared libraries. 

The actual total allocatable limit for x86 is a bit under 3Gb, but you 
won't get that as one linear allocation. (1Gb is kernel mappings)