Any limitations on bigmem usage?

Any limitations on bigmem usage?

[Matt Nelson]

I am about to embark on a data processing software project that will require a 
LOT of memory (about, ohhh, 6GB or so), and I was wondering if there are any 
limitations to how one can use very large chunks of memory under Linux. 
Specifically, is there anything to prevent me from malloc()ing 6GB of memory, 
then accessing that memory as I would any other buffer?  FYI, the application
will be buffering a stream of data, then performing a variety of calculations on 
large blocks of data at a time, before writing it out to a socket.

I've been eyeing an 8-way Intel box with gobs of memory, but if there are subtle 
issues with using that much memory, I need to know now.

I haven't seen this specifcally addressed, so I figured I should ask you folk. 
Any insights/comments/reccomendations would be greatly appreciated.

Thanks,

Matt

-- 
Matthew Nelson
Dynamics Technology, Inc.
21311 Hawthorne Blvd., Suite 300, Torrance, CA 90503-5610
Voice: (310) 543-5433   FAX: (310) 543-2117   Email: mnelson@dynatec.com

Re: Any limitations on bigmem usage?

[Doug McNaught]

Matt Nelson <mnelson@dynatec.com> writes:

> I am about to embark on a data processing software project that will require a
> LOT of memory (about, ohhh, 6GB or so), and I was wondering if there are any
> limitations to how one can use very large chunks of memory under
> Linux. Specifically, is there anything to prevent me from malloc()ing 6GB of
> memory, then accessing that memory as I would any other buffer?  FYI, the
> application
> 
> will be buffering a stream of data, then performing a variety of calculations
> on large blocks of data at a time, before writing it out to a socket.

Pointers on IA32 are still 32 bits, so (as I understand it) each
process can address a maximum of 4GB.  You would have to allocate
multiple chunks (in shared memory or mmap()ed files, so they're
persistent) and map them in and out as needed (which could get hairy).

> I've been eyeing an 8-way Intel box with gobs of memory, but if there are
> subtle issues with using that much memory, I need to know now.

Best bet is probably to buy an Alpha machine, if you don't want to
spend time working around the 4GB limitation.

-Doug
-- 
The rain man gave me two cures; he said jump right in,
The first was Texas medicine--the second was just railroad gin,
And like a fool I mixed them, and it strangled up my mind,
Now people just get uglier, and I got no sense of time...          --Dylan

Re: Any limitations on bigmem usage?

[Doug McNaught]

Doug McNaught <doug@wireboard.com> writes:

> Matt Nelson <mnelson@dynatec.com> writes:
> 
> > I am about to embark on a data processing software project that
> > will require a LOT of memory (about, ohhh, 6GB or so), and I was
> > wondering if there are any limitations to how one can use very
> > large chunks of memory under Linux. Specifically, is there
> > anything to prevent me from malloc()ing 6GB of memory, then
> > accessing that memory as I would any other buffer?  FYI, the
> > application will be buffering a stream of data, then performing a
> > variety of calculations on large blocks of data at a time, before
> > writing it out to a socket.
> 
> Pointers on IA32 are still 32 bits, so (as I understand it) each
> process can address a maximum of 4GB.  You would have to allocate
> multiple chunks (in shared memory or mmap()ed files, so they're
> persistent) and map them in and out as needed (which could get hairy).

Sorry to followup to myself, but I just had another thought--if you
can split your task up into multiple processe such that no process
needs to address more than 4GB, an IA32 machine will work fine.  It
will probably still take more programming work than the naive approach
(malloc() huge chink, use it) that you could use on a 64-bit
machine...

-Doug
-- 
The rain man gave me two cures; he said jump right in,
The first was Texas medicine--the second was just railroad gin,
And like a fool I mixed them, and it strangled up my mind,
Now people just get uglier, and I got no sense of time...          --Dylan

Re: Any limitations on bigmem usage?

[Matt Nelson]

Thanks to all that replied.  I've never needed to use so much memory before, and 
was ignorant to how much magic was implemented in the 64G support on IA32. 
Unfortunately, there's not quite enough magic in there for my needs... now to 
find an affordable SMP Alpha system....

Thanks again,

-Matt

Matt Nelson wrote:

> I am about to embark on a data processing software project that will 
> require a LOT of memory (about, ohhh, 6GB or so), and I was wondering if 
> there are any limitations to how one can use very large chunks of memory 
> under Linux. Specifically, is there anything to prevent me from 
> malloc()ing 6GB of memory, then accessing that memory as I would any 
> other buffer?  FYI, the application
> will be buffering a stream of data, then performing a variety of 
> calculations on large blocks of data at a time, before writing it out to 
> a socket.
> 
> I've been eyeing an 8-way Intel box with gobs of memory, but if there 
> are subtle issues with using that much memory, I need to know now.
> 
> I haven't seen this specifcally addressed, so I figured I should ask you 
> folk. Any insights/comments/reccomendations would be greatly appreciated.
> 
> Thanks,
> 
> Matt
> 


-- 
Matthew Nelson
Dynamics Technology, Inc.
21311 Hawthorne Blvd., Suite 300, Torrance, CA 90503-5610
Voice: (310) 543-5433   FAX: (310) 543-2117   Email: mnelson@dynatec.com

Re: Any limitations on bigmem usage?

[Alan Cox]

> Specifically, is there anything to prevent me from malloc()ing 6GB of memory, 
> then accessing that memory as I would any other buffer?  FYI, the application

X86 has no nice way to address over 4Gb of RAM. You can do paging games with
multiple banks and shmat (ie like using DOS expanded ram but with bigger
pieces).

> I've been eyeing an 8-way Intel box with gobs of memory, but if there are subtle 
> issues with using that much memory, I need to know now.

If your algorithm can work well with say 2Gb windows on the data and only change
window evey so often (in computing terms) then it should be ok, if its access
is random you need to look at a 64bit box like an Alpha, Sparc64 or eventually
IA64

Re: Any limitations on bigmem usage?

[Rob Landley]

On Tuesday 12 June 2001 12:29, Alan Cox wrote:

> If your algorithm can work well with say 2Gb windows on the data and only
> change window evey so often (in computing terms) then it should be ok, if
> its access is random you need to look at a 64bit box like an Alpha, Sparc64
> or eventually IA64

No, eventually Sledgehammer.

You know that IA64 will never ship, because the performance will always suck. 
 The design is fundamentally flawed.  The real bottleneck is the memory bus.  
These days they're clock multiplying the inside of the processor by a factor 
of what, twelve?  Fun as long as you're entirely in L1 cache and aren't task 
switching.  But that's basicaly an embedded system.

CISC instructions are effectively compressed.  When you exhaust your cache 
your next 64 bit gulp can get more than 2 instructions, you can get more like 
5 (which still means you're getting about 1/4 the performance of in-cache 
execution, although L2 and L3 caches help here, of course..)

But optimizing for something that isn't actually your bottleneck is just 
dumb, and that's exactly what Intel did with the move to VLIW/EPIC/IA64.  3 
times 64 bit instructions is 192, whereas Pentium can fit more than that in a 
single 64 bit chunk.  Brilliant.  You need what, a 6x larger cache just to 
break even with the amount of time you're running in-cache?  And of course 
the compiler is GOING to put NOPs in that because it won't always be able to 
figure out something for the second and third cores to do this clock, 
regardless of how good a compiler it is.  That's just beautiful.

This is why they were so desperate for RAMBUS.  They KNOW the memory bus is 
killing them, they were grasping at straws to fix it.  (Currently they're 
saying that a future vaporware version of iTanium will fix everything, but 
it's a fundamental design flaw: the new instruction set they've chosen is WAY 
less efficient going across the memory bus, and that's the real limiting 
factor in performance.)

Transmeta at least sucks CISC in from main memory to keep the northbridge 
bottleneck optimized.  And they have a big cache, and they're still using 32 
bit instructions so they only need 96 bytes per chunk (or atom or whatever 
they're calling it these days).

Sledgehammer is the interesting x86 64 bit instruction set.  You still have 
the cisc/risc hybrid that made pentium work.  (And, of course, backwards 
compatability that's inherent in the design rather than bolted onto the side 
with duct tape and crazy glue.)  Yeah the circuitry to make it work is 
probably fairly insane, but at least the Athlon design team made all the racy 
logic go away so they can migrate the design to new manufacturing processes 
without four months of redesign work.  (And no, making insanely long 
pipelines in Pentium 4 is not a major improvement there either.  Cache 
exhaustion still kills you, so does branch misprediction.  Stalling a longer 
pipeline is more expensive.)

I saw an 800 mhz iTanium which benchmarked about the same (running 64 bit 
code) as a Penium III 300 mhz running 32 bit code.  That's just sad.  Go 
ahead and blame the compiler.  That's still just sad.  And a design flaw in 
the new instruction set is not something that can be easily optimized away.

Rob

Re: Any limitations on bigmem usage?

[Rik van Riel]

On Tue, 12 Jun 2001, Rob Landley wrote:

> Brilliant.  You need what, a 6x larger cache just to break even with
> the amount of time you're running in-cache? 

That's going to be hard, since the cache will also need to be
faster in order to feed the CPU core.  Making a cache both
larger AND faster at the same time will need some smart people.

> And of course the compiler is GOING to put NOPs in that because it
> won't always be able to figure out something for the second and third
> cores to do this clock, regardless of how good a compiler it is.  

Compilers are also notoriously bad at runtime optimisations.

> That's just beautiful.

I also never expected Intel to dispose of themselves in such
a cute way.

cheers,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

RE: Any limitations on bigmem usage?

[Holzrichter, Bruce]

On Tue, 12 Jun 2001, Rob Landley wrote:

> Brilliant.  You need what, a 6x larger cache just to break even with
> the amount of time you're running in-cache? 

This may be the wrong platform for this question, but after reading Rob
Landley's note on performance on Itanium and architecture concerns, I am
interested in Kernel hackers who have had to write code for Itanium's
comments on the same, if you are not bound by NDA's.  Correct me if I am
wrong, but I thought I saw the announcement that Itanium is shipping.  Have
you tested Itanium performance?  We have an preproduction unit with quad
Itanium's.  I have not had time to benchmark against other units, I am
interested in performance items.  Feel free to drop me a line off list if
you can.

B.

Re: Any limitations on bigmem usage?

[Ralf Baechle]

On Tue, Jun 12, 2001 at 02:34:40PM -0400, Holzrichter, Bruce wrote:

> > Brilliant.  You need what, a 6x larger cache just to break even with
> > the amount of time you're running in-cache? 
> 
> This may be the wrong platform for this question, but after reading Rob
> Landley's note on performance on Itanium and architecture concerns, I am
> interested in Kernel hackers who have had to write code for Itanium's
> comments on the same, if you are not bound by NDA's.  Correct me if I am
> wrong, but I thought I saw the announcement that Itanium is shipping.  Have
> you tested Itanium performance?  We have an preproduction unit with quad
> Itanium's.  I have not had time to benchmark against other units, I am
> interested in performance items.  Feel free to drop me a line off list if
> you can.

A number of Specbench numbers of Itanium systems is now available.  Itanium
performs relativly bad for the integer numbers compared to the entire
competition but is a true fp killer.  As a developer I hate that compiling
code for Itanium due to the extra complexity of optimization and code
generation is way slower than for others CPUs.  So all in all Itanium is
a two edged sword.

  Ralf