* malloc problem to allocate very large blocks
@ 2003-07-28 6:44 Tung-Han Hsieh
2003-07-28 8:37 ` jw schultz
2003-07-29 4:58 ` Nagendra Singh Tomar
0 siblings, 2 replies; 6+ messages in thread
From: Tung-Han Hsieh @ 2003-07-28 6:44 UTC (permalink / raw)
To: linux-kernel; +Cc: jamagallon
Hello,
I am developing applications which requires more than 2GB memory.
But I found that in my Linux system the malloc() cannot allocate
more than 2GB memory. Here is the details of my system:
CPU: Pentium 4 2.53 GHz
RAM: 2 GB
Swap: 512 MB
OS: Debian-3.0 stable
Kernel: 2.4.20
gcc: 2.95.4 20011002
glibc: 2.2.5-6
In theory, in a 32-bits machine the maximum allocatable memory
is up to 4GB. But in the following very simple testing program:
=====================================================================
#include <stdio.h>
#include <stdlib.h>
main()
{
size_t l;
char *s1=NULL, *s2=NULL;
l = 1024*1024*1024;
s1 = malloc(l);
s2 = malloc(l);
if (! s1) printf("s1 malloc failed\n");
if (! s2) printf("s2 malloc failed\n");
}
=====================================================================
only the block for s1 can be allocated. Further, if I change the
program to
=====================================================================
#include <stdio.h>
#include <stdlib.h>
main()
{
size_t l;
char *s1=NULL;
l = 2*1024*1024*1024;
s1 = malloc(l);
if (! s1) printf("s1 malloc failed\n");
}
=====================================================================
the gcc complier complain to me that "foo.c:9: warning: integer overflow
in expression" during the compilation (I use: "gcc foo.c" to compile it),
and the block for s1 cannot be allocated at all. I am wondering if there
is any way to overcome the 2GB limit.
Thank you very much for your reply in advance.
Best Regards,
T.H.Hsieh
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: malloc problem to allocate very large blocks
2003-07-28 6:44 malloc problem to allocate very large blocks Tung-Han Hsieh
@ 2003-07-28 8:37 ` jw schultz
2003-07-28 9:14 ` jw schultz
2003-07-29 4:58 ` Nagendra Singh Tomar
1 sibling, 1 reply; 6+ messages in thread
From: jw schultz @ 2003-07-28 8:37 UTC (permalink / raw)
To: linux-kernel
On Mon, Jul 28, 2003 at 02:44:28PM +0800, Tung-Han Hsieh wrote:
> Hello,
>
> I am developing applications which requires more than 2GB memory.
> But I found that in my Linux system the malloc() cannot allocate
> more than 2GB memory. Here is the details of my system:
Malloc is a library function, not a syscall.
That said i'm not surprised. Malloc front-ends sbrk and
sometimes mmap. Sbrk uses a ptrdiff_t to indicate the size
desired. ptrdiff_t is signed and on 32 bit platforms should
be 32 bits. Therefore, the maximum you can allocate at one
time with sbrk is going to be 2GB. That malloc would
inherit this limitation is to be expected.
If you need to allocate that much memory in one chunk, don't
use malloc.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: jw@pegasys.ws
Remember Cernan and Schmitt
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: malloc problem to allocate very large blocks
2003-07-28 8:37 ` jw schultz
@ 2003-07-28 9:14 ` jw schultz
0 siblings, 0 replies; 6+ messages in thread
From: jw schultz @ 2003-07-28 9:14 UTC (permalink / raw)
To: linux-kernel
On Mon, Jul 28, 2003 at 01:37:30AM -0700, jw schultz wrote:
> On Mon, Jul 28, 2003 at 02:44:28PM +0800, Tung-Han Hsieh wrote:
> > Hello,
> >
> > I am developing applications which requires more than 2GB memory.
> > But I found that in my Linux system the malloc() cannot allocate
> > more than 2GB memory. Here is the details of my system:
>
> Malloc is a library function, not a syscall.
>
> That said i'm not surprised. Malloc front-ends sbrk and
> sometimes mmap. Sbrk uses a ptrdiff_t to indicate the size
> desired. ptrdiff_t is signed and on 32 bit platforms should
> be 32 bits. Therefore, the maximum you can allocate at one
> time with sbrk is going to be 2GB. That malloc would
> inherit this limitation is to be expected.
>
> If you need to allocate that much memory in one chunk, don't
> use malloc.
And regarding trying to get it in multiple chunks. That is
likely to fail with sbrk due to the way the address space is
assigned to the various memory areas.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: jw@pegasys.ws
Remember Cernan and Schmitt
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: malloc problem to allocate very large blocks
2003-07-28 6:44 malloc problem to allocate very large blocks Tung-Han Hsieh
2003-07-28 8:37 ` jw schultz
@ 2003-07-29 4:58 ` Nagendra Singh Tomar
2003-07-29 5:02 ` Nagendra Singh Tomar
2003-07-29 11:52 ` Alan Cox
1 sibling, 2 replies; 6+ messages in thread
From: Nagendra Singh Tomar @ 2003-07-29 4:58 UTC (permalink / raw)
To: Tung-Han Hsieh; +Cc: linux-kernel, jamagallon
AFAIK malloc will not return you memory more than the total virtual memory
(RAM+swap) in the system. So if you want more than 2GB allocations from
malloc, make sure you have at least 2GB virtual mem, keeping aside some
space for the kernel.
--tomar
On Mon, 28 Jul 2003, Tung-Han Hsieh wrote:
> Hello,
>
> I am developing applications which requires more than 2GB memory.
> But I found that in my Linux system the malloc() cannot allocate
> more than 2GB memory. Here is the details of my system:
>
> CPU: Pentium 4 2.53 GHz
> RAM: 2 GB
> Swap: 512 MB
> OS: Debian-3.0 stable
> Kernel: 2.4.20
> gcc: 2.95.4 20011002
> glibc: 2.2.5-6
>
> In theory, in a 32-bits machine the maximum allocatable memory
> is up to 4GB. But in the following very simple testing program:
>
> =====================================================================
> #include <stdio.h>
> #include <stdlib.h>
>
> main()
> {
> size_t l;
> char *s1=NULL, *s2=NULL;
>
> l = 1024*1024*1024;
>
> s1 = malloc(l);
> s2 = malloc(l);
> if (! s1) printf("s1 malloc failed\n");
> if (! s2) printf("s2 malloc failed\n");
> }
> =====================================================================
>
> only the block for s1 can be allocated. Further, if I change the
> program to
>
> =====================================================================
> #include <stdio.h>
> #include <stdlib.h>
>
> main()
> {
> size_t l;
> char *s1=NULL;
>
> l = 2*1024*1024*1024;
>
> s1 = malloc(l);
> if (! s1) printf("s1 malloc failed\n");
> }
> =====================================================================
>
> the gcc complier complain to me that "foo.c:9: warning: integer overflow
> in expression" during the compilation (I use: "gcc foo.c" to compile
> it),
> and the block for s1 cannot be allocated at all. I am wondering if there
> is any way to overcome the 2GB limit.
>
> Thank you very much for your reply in advance.
>
>
> Best Regards,
>
> T.H.Hsieh
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: malloc problem to allocate very large blocks
2003-07-29 4:58 ` Nagendra Singh Tomar
@ 2003-07-29 5:02 ` Nagendra Singh Tomar
2003-07-29 11:52 ` Alan Cox
1 sibling, 0 replies; 6+ messages in thread
From: Nagendra Singh Tomar @ 2003-07-29 5:02 UTC (permalink / raw)
To: Tomar, Nagendra; +Cc: Tung-Han Hsieh, linux-kernel, jamagallon
I missed the system details, you already have >2GB virtual memory.
--tomar
On Tue, 29 Jul 2003, Tomar, Nagendra wrote:
> AFAIK malloc will not return you memory more than the total virtual
> memory
> (RAM+swap) in the system. So if you want more than 2GB allocations from
> malloc, make sure you have at least 2GB virtual mem, keeping aside some
> space for the kernel.
>
> --tomar
>
> On Mon, 28 Jul 2003, Tung-Han Hsieh wrote:
>
> > Hello,
> >
> > I am developing applications which requires more than 2GB memory.
> > But I found that in my Linux system the malloc() cannot allocate
> > more than 2GB memory. Here is the details of my system:
> >
> > CPU: Pentium 4 2.53 GHz
> > RAM: 2 GB
> > Swap: 512 MB
> > OS: Debian-3.0 stable
> > Kernel: 2.4.20
> > gcc: 2.95.4 20011002
> > glibc: 2.2.5-6
> >
> > In theory, in a 32-bits machine the maximum allocatable memory
> > is up to 4GB. But in the following very simple testing program:
> >
> > =====================================================================
> > #include <stdio.h>
> > #include <stdlib.h>
> >
> > main()
> > {
> > size_t l;
> > char *s1=NULL, *s2=NULL;
> >
> > l = 1024*1024*1024;
> >
> > s1 = malloc(l);
> > s2 = malloc(l);
> > if (! s1) printf("s1 malloc failed\n");
> > if (! s2) printf("s2 malloc failed\n");
> > }
> > =====================================================================
> >
> > only the block for s1 can be allocated. Further, if I change the
> > program to
> >
> > =====================================================================
> > #include <stdio.h>
> > #include <stdlib.h>
> >
> > main()
> > {
> > size_t l;
> > char *s1=NULL;
> >
> > l = 2*1024*1024*1024;
> >
> > s1 = malloc(l);
> > if (! s1) printf("s1 malloc failed\n");
> > }
> > =====================================================================
> >
> > the gcc complier complain to me that "foo.c:9: warning: integer
> overflow
> > in expression" during the compilation (I use: "gcc foo.c" to compile
> > it),
> > and the block for s1 cannot be allocated at all. I am wondering if
> there
> > is any way to overcome the 2GB limit.
> >
> > Thank you very much for your reply in advance.
> >
> >
> > Best Regards,
> >
> > T.H.Hsieh
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> linux-kernel"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
> >
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: malloc problem to allocate very large blocks
2003-07-29 4:58 ` Nagendra Singh Tomar
2003-07-29 5:02 ` Nagendra Singh Tomar
@ 2003-07-29 11:52 ` Alan Cox
1 sibling, 0 replies; 6+ messages in thread
From: Alan Cox @ 2003-07-29 11:52 UTC (permalink / raw)
To: nagendra_tomar; +Cc: Tung-Han Hsieh, Linux Kernel Mailing List, jamagallon
On Maw, 2003-07-29 at 05:58, Nagendra Singh Tomar wrote:
> AFAIK malloc will not return you memory more than the total virtual memory
> (RAM+swap) in the system. So if you want more than 2GB allocations from
> malloc, make sure you have at least 2GB virtual mem, keeping aside some
> space for the kernel.
On the default memory settings it may do. However a request for 2Gb of
memory requires there is a free 2Gb of address space to map it into -
which may not be true because of things like shared libraries.
The actual total allocatable limit for x86 is a bit under 3Gb, but you
won't get that as one linear allocation. (1Gb is kernel mappings)
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2003-07-29 11:59 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-07-28 6:44 malloc problem to allocate very large blocks Tung-Han Hsieh
2003-07-28 8:37 ` jw schultz
2003-07-28 9:14 ` jw schultz
2003-07-29 4:58 ` Nagendra Singh Tomar
2003-07-29 5:02 ` Nagendra Singh Tomar
2003-07-29 11:52 ` Alan Cox
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).