Re: What is the truth about Linux 2.4's RAM limitations?

Re: What is the truth about Linux 2.4's RAM limitations?

[Jesse Pollard]

Timur Tabi <ttabi@interactivesi.com>:
> Jesse Pollard wrote:
> >>So what are the limits without using PAE? Here I'm still having a little
> >>problem finding definitive answers but ...
> >>
> >3 GB. Final answers are in the FAQ, and have been discussed before. You can
> >also look in the Intel 80x86 CPU specifications.
> >
> >The only way to exceed current limits is via some form of segment register usage
> >which will require a different compiler and a replacement of the memory
> >architecture of x86 Linux implementation.
> >
> 
> Are you talking about using 48-bit pointers?
> 
> (48-bit pointers, aka 16:32 pointers, on x86 are basically "far 32-bit 
> pointers".  That is, each pointer is stored as a 48-bit value, where 16 
> bits are for the selector/segment, and 32 bits are for the offset.

That sounds right - I'm not yet fully familiar with the low level intel
x86 design yet. There is also (based on list email) a limit to how
many page tables can be active. Two is desirable (one system, one user)
but the x86 design only has one. This causes Linux (and maybe others too)
to split the 32 bit range into a 3G (user) and 1G (system) address ranges
to allow the page cache/cpu cache to work in a more optimum manner. If
the entire page table were given to a user, then a full cache flush would
have to be done on every context switch and system call. That would be
very slow, but would allow a full 4G address for the user.

The use of 48 bit addresses has the same problem. Doing the remapping for
the segment + offset requires flushing the cache as well (the cache seems
to be between the segment registers and the page tables - not sure, not
necessarily coreect... I still have to get the new CPU specs...)

Any body want to offer a full reference? Or a tutorial on Intel addressing
capability?.


-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: What is the truth about Linux 2.4's RAM limitations?

[Jonathan Lundell]

At 1:12 PM -0500 2001-07-10, Jesse Pollard wrote:
>Any body want to offer a full reference? Or a tutorial on Intel addressing
>capability?.

For IA-32, try Chapter 3 of "IA-32 Intel Architecture Software 
Developer's Manual Volume 3: System Programming Guide", which you can 
find at http://developer.intel.com/design/PentiumIII/manuals/

-- 
/Jonathan Lundell.

Re: What is the truth about Linux 2.4's RAM limitations?

[Brian Gerst]

Jesse Pollard wrote:
> 
> Timur Tabi <ttabi@interactivesi.com>:
> > Jesse Pollard wrote:
> > >>So what are the limits without using PAE? Here I'm still having a little
> > >>problem finding definitive answers but ...
> > >>
> > >3 GB. Final answers are in the FAQ, and have been discussed before. You can
> > >also look in the Intel 80x86 CPU specifications.
> > >
> > >The only way to exceed current limits is via some form of segment register usage
> > >which will require a different compiler and a replacement of the memory
> > >architecture of x86 Linux implementation.
> > >
> >
> > Are you talking about using 48-bit pointers?
> >
> > (48-bit pointers, aka 16:32 pointers, on x86 are basically "far 32-bit
> > pointers".  That is, each pointer is stored as a 48-bit value, where 16
> > bits are for the selector/segment, and 32 bits are for the offset.
> 
> That sounds right - I'm not yet fully familiar with the low level intel
> x86 design yet. There is also (based on list email) a limit to how
> many page tables can be active. Two is desirable (one system, one user)
> but the x86 design only has one. This causes Linux (and maybe others too)
> to split the 32 bit range into a 3G (user) and 1G (system) address ranges
> to allow the page cache/cpu cache to work in a more optimum manner. If
> the entire page table were given to a user, then a full cache flush would
> have to be done on every context switch and system call. That would be
> very slow, but would allow a full 4G address for the user.

A full cache flush would be needed at every entry into the kernel,
including hardware interrupts.  Very poor for performance.

> The use of 48 bit addresses has the same problem. Doing the remapping for
> the segment + offset requires flushing the cache as well (the cache seems
> to be between the segment registers and the page tables - not sure, not
> necessarily coreect... I still have to get the new CPU specs...)
> 
> Any body want to offer a full reference? Or a tutorial on Intel addressing
> capability?.

Using segmentation does not give you access to any more memory without
dirty hacks using fault handlers.  The segment base is added to the
offset to get a linear address (truncated to 32 bits).  This linear
address is fed through the page tables to get the physical address.

--

				Brian Gerst

Re: What is the truth about Linux 2.4's RAM limitations?

[Chris Wedgwood]

On Tue, Jul 10, 2001 at 02:28:54PM -0400, Brian Gerst wrote:

    Jesse Pollard wrote:

        > If the entire page table were given to a user, then a full cache
        > flush would have to be done on every context switch and system
        > call. That would be very slow, but would allow a full 4G address
        > for the user.

    A full cache flush would be needed at every entry into the kernel,
    including hardware interrupts.  Very poor for performance.

Why would a cache flush be necessary at all? I assume ia32 caches
where physically not virtually mapped?




   --cw

Re: What is the truth about Linux 2.4's RAM limitations?

[Brian Gerst]

Chris Wedgwood wrote:
> 
> On Tue, Jul 10, 2001 at 02:28:54PM -0400, Brian Gerst wrote:
> 
>     Jesse Pollard wrote:
> 
>         > If the entire page table were given to a user, then a full cache
>         > flush would have to be done on every context switch and system
>         > call. That would be very slow, but would allow a full 4G address
>         > for the user.
> 
>     A full cache flush would be needed at every entry into the kernel,
>     including hardware interrupts.  Very poor for performance.
> 
> Why would a cache flush be necessary at all? I assume ia32 caches
> where physically not virtually mapped?

I meant TLB flush, sorry.

--

				Brian Gerst

Re: What is the truth about Linux 2.4's RAM limitations?

[alad]

Jesse Pollard <pollard@tomcat.admin.navo.hpc.mil> on 07/11/2001 01:08:02 AM

To:   root@chaos.analogic.com, Timur Tabi <ttabi@interactivesi.com>
cc:   linux-kernel@vger.kernel.org (bcc: Amol Lad/HSS)

Subject:  Re: What is the truth about Linux 2.4's RAM limitations?




"Richard B. Johnson" <root@chaos.analogic.com>
...
> In Unix and Unix variants, it is by design, provided that the
> kernel exist within every process address space. Early Nixes
> like Ultrix, simply called the kernel entry point. Since it
> was protected, this trapped to the real kernel and the page-fault
> handler actually performed the work on behalf of the caller.
>
> Unlike some OS (like VMS), a context-switch does not occur
> when the kernel provides services for the calling task.
> Therefore, it was most reasonable to have the kernel exist within
> each tasks address space. With modern processors, it doesn't make
> very much difference, you could have user space start at virtual
> address 0 and extend to virtual address 0xffffffff. However, this would
> not be Unix. It would also force the kernel to use additional
> CPU cycles when addressing a tasks virtual address space,
> i.e., when data are copied to/from user to kernel space.

I believe the VAX/VMS implementation shared OS and user space:

     p0   - user application       0
     p1   - system shared libraries      0x3fffffff
     p2   - kernel            0x7fffffff
          rest was I/O, cache memory     0xffffffff

It was a hardware design, not a function of the software.

UNIX origins were on a PDP-11. there were two sets of addressing registers
1 kernel, 1 user  (except on 11/45 - 1 kernel, 1 user, 1 "executive"
(never used except in some really strange form of extented shared library)

A full context switch was required. Kernel had to map a single 4KW window
to the user space for access to the parameters. Another 4KW window was used
to map the IO space. The remaining 6 mapping registers were used for supporting
the kernel virtual address. BTW, 1 KW = 2K Bytes, a mapping register could
map anything from 16 bytes to 8K bytes, if I remember correctly. The PDP 11
with memory management only had 16 mapping registers (8 user, 8 kernel) with
a maximum address of 64K bytes (16 bit addresses... my how far we've come).
The base hardware could only handle a maximum of 256 K bytes. More recent
cpu's expanded the size of the mapping registers (more bits/register) but did
not increase the number of registers. The last system (PDP-11/70 level) could
handle 4 MB of physical memory, but with all of the restrictions of the small
systems, just more processes were handled.

It was not possible to share memory between kernel/user other than that one
4KW window. The Linux 3G/1G split is a design choice for speed. It would still
be Linux even if it did 4G/0, just a different MM architecture with a lot
more overhead on intel x86 hardware.
>>>>> Can you please write what exactly the 'overhead' is and how the same
overhead is not there in 3G/1G split

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: What is the truth about Linux 2.4's RAM limitations?

[Jesse Pollard]

> On Tue, Jul 10, 2001 at 02:28:54PM -0400, Brian Gerst wrote:
> 
>     Jesse Pollard wrote:
> 
>         > If the entire page table were given to a user, then a full cache
>         > flush would have to be done on every context switch and system
>         > call. That would be very slow, but would allow a full 4G address
>         > for the user.
> 
>     A full cache flush would be needed at every entry into the kernel,
>     including hardware interrupts.  Very poor for performance.
> 
> Why would a cache flush be necessary at all? I assume ia32 caches
> where physically not virtually mapped?

Because the entire virtual mapping is replaced by that of the kernel.
This would invalidate the entire cache table. It was also pointed out
that this would have to be done for every interrupt too.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: What is the truth about Linux 2.4's RAM limitations?

[Jonathan Lundell]

At 4:49 PM -0500 2001-07-10, Jesse Pollard wrote:
>  >     A full cache flush would be needed at every entry into the kernel,
>>      including hardware interrupts.  Very poor for performance.
>>
>>  Why would a cache flush be necessary at all? I assume ia32 caches
>>  where physically not virtually mapped?
>
>Because the entire virtual mapping is replaced by that of the kernel.
>This would invalidate the entire cache table. It was also pointed out
>that this would have to be done for every interrupt too.

If the cache were physically indexed and tagged, this would not be 
the case; changes in mapping would be irrelevant. If someone has a 
reference that describes IA-32 cache tags in detail, I'd like to know 
about it.

TLBs are another story, though on some other architectures they're 
not a problem either. UltraSPARC, for example, maps the entire kernel 
with a couple or three reserved TLB entries (at least Solaris does). 
Of course, SPARC has hardware contexts, which are helpful in this 
regard.
-- 
/Jonathan Lundell.

Re: What is the truth about Linux 2.4's RAM limitations?

[Jesse Pollard]

"Richard B. Johnson" <root@chaos.analogic.com>
...
> In Unix and Unix variants, it is by design, provided that the
> kernel exist within every process address space. Early Nixes
> like Ultrix, simply called the kernel entry point. Since it
> was protected, this trapped to the real kernel and the page-fault
> handler actually performed the work on behalf of the caller.
> 
> Unlike some OS (like VMS), a context-switch does not occur
> when the kernel provides services for the calling task.
> Therefore, it was most reasonable to have the kernel exist within
> each tasks address space. With modern processors, it doesn't make
> very much difference, you could have user space start at virtual
> address 0 and extend to virtual address 0xffffffff. However, this would
> not be Unix. It would also force the kernel to use additional
> CPU cycles when addressing a tasks virtual address space,
> i.e., when data are copied to/from user to kernel space.

I believe the VAX/VMS implementation shared OS and user space:

	p0	- user application		0
	p1	- system shared libraries	0x3fffffff
	p2	- kernel			0x7fffffff
		rest was I/O, cache memory	0xffffffff

It was a hardware design, not a function of the software.

UNIX origins were on a PDP-11. there were two sets of addressing registers
1 kernel, 1 user  (except on 11/45 - 1 kernel, 1 user, 1 "executive"
(never used except in some really strange form of extented shared library)

A full context switch was required. Kernel had to map a single 4KW window
to the user space for access to the parameters. Another 4KW window was used
to map the IO space. The remaining 6 mapping registers were used for supporting
the kernel virtual address. BTW, 1 KW = 2K Bytes, a mapping register could
map anything from 16 bytes to 8K bytes, if I remember correctly. The PDP 11
with memory management only had 16 mapping registers (8 user, 8 kernel) with
a maximum address of 64K bytes (16 bit addresses... my how far we've come).
The base hardware could only handle a maximum of 256 K bytes. More recent
cpu's expanded the size of the mapping registers (more bits/register) but did
not increase the number of registers. The last system (PDP-11/70 level) could
handle 4 MB of physical memory, but with all of the restrictions of the small
systems, just more processes were handled.

It was not possible to share memory between kernel/user other than that one
4KW window. The Linux 3G/1G split is a design choice for speed. It would still
be Linux even if it did 4G/0, just a different MM architecture with a lot
more overhead on intel x86 hardware.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: What is the truth about Linux 2.4's RAM limitations?

[Mark H. Wood]

On Tue, 10 Jul 2001, Jesse Pollard wrote:
> "Richard B. Johnson" <root@chaos.analogic.com>
[snip]
> > Unlike some OS (like VMS), a context-switch does not occur
> > when the kernel provides services for the calling task.
> > Therefore, it was most reasonable to have the kernel exist within
> > each tasks address space. With modern processors, it doesn't make
> > very much difference, you could have user space start at virtual
> > address 0 and extend to virtual address 0xffffffff. However, this would
> > not be Unix. It would also force the kernel to use additional
> > CPU cycles when addressing a tasks virtual address space,
> > i.e., when data are copied to/from user to kernel space.
>
> I believe the VAX/VMS implementation shared OS and user space:
>
> 	p0	- user application		0
> 	p1	- system shared libraries	0x3fffffff
> 	p2	- kernel			0x7fffffff
> 		rest was I/O, cache memory	0xffffffff
>
> It was a hardware design, not a function of the software.

Correct, except that "p2" is called S0.  IIRC it is the top quarter of the
address space, and there's a reserved S1 region.  The command interpreter
is also mapped into P2.  The very top of memory is reserved for device
registers.

> UNIX origins were on a PDP-11. there were two sets of addressing registers
> 1 kernel, 1 user  (except on 11/45 - 1 kernel, 1 user, 1 "executive"
> (never used except in some really strange form of extented shared library)

Early '11s didn't have the Ispace/Dspace split.  My PDP11 databook is at
home, and I don't trust my memory far enough to say which model introduced
the split.  My recollection is that the earliest implementation was an
add-on board which monitored the Unibus address lines and fired interrupts
when it thought its programmed memory access rules were violated.

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
Make a good day.

Re: What is the truth about Linux 2.4's RAM limitations?

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> 
> Where I just started work we run large processes for simulations and
> testing of semi-conductors.  Currently we use Solaris because of past
> limitations on the amount of RAM that a single process can address under
> Linux.  Recently we tried to run some tests on a Dell dual Xeon 1.7GHz
> with 4GB of RAM running Redhat 7.1 box (with the stock Redhat SMP kernel).
> Speedwise it kicked the crap out of our Sunblade (Dual 750MHz with 8GB of
> RAM)but we had problems with process dying right around 2.3GB (according
> to top).
> 
> So I started to investigate, and quickly discovered that there is no good
> source for finding this sort of information on line.  At least not that I
> could find.  Nearly every piece of information I found conflicted in at
> least some small way with another piece of information I found.
> So I ask here in the hopes of a definitive answer.
> 
>  * What is the maximum amount of RAM that a *single* process can address
>    under a 2.4 kernel, with PAE enabled?  Without?

3GB tops. PAE only allows more processes. A single process cannot under
any circumstances reach more than 4G, but due to other limits 3 is the max.

>  * And, what (if any) paramaters can effect this (recompiling the app
>    etc)?

You need a 64 bit CPU.

> What I think I know so far is listed below.  I welcome being flamed, told
> that I'm stupid and that I should have looked "here" so long as said
> messages also contain pointers to definitive information :-)
> 
> Linux 2.4 does support greater then 4GB of RAM with these caveats ...
> 
>  * It does this by supporting Intel's PAE (Physical Address Extension)
>    features which are in all Pentium Pro and newer CPU's.
>  * The PAE extensions allow up to a maximum of 64GB of RAM that the OS
>    (not a process) can address.
>  * It does this via indirect pointers to the higher memory locations, so
>    there is a CPU and RAM hit for using this.
>  * Benchmarks seem to indicated around 3-6% CPU hit just for using the PAE
>    extensions (ie. it applies regardless of whether you are actually
>    accessing memory locations greater then 4GB).
>  * If the kernel is compiled to use PAE, Linux will not boot on a computer
>    whose hardware doesn't support PAE.
>  * PAE does not increase Linux's ability for *single* processes to see
>    greater then 3GB of RAM (see below).
> 
> So what are the limits without using PAE? Here I'm still having a little
> problem finding definitive answers but ...

3 GB. Final answers are in the FAQ, and have been discussed before. You can
also look in the Intel 80x86 CPU specifications.

The only way to exceed current limits is via some form of segment register usage
which will require a different compiler and a replacement of the memory
architecture of x86 Linux implementation.

> 
>  * With PAE compiled into the kernel the OS can address a maximum of 4GB
>    of RAM.
>  * With 2.4 kernels (with a large memory configuration) a single process
>    can address up to the total amount of RAM in the machine minus 1GB
>    (reserved for the kernel), to a maximum 3GB.
>  * By default the kernel reserves 1GB for it's own use, however I think
>    that this is a tunable parameter so if we have 4GB of RAM in a box we
>    can tune it so that most of that should be available to the processes
>    (?).
> 
> I have documented the below information on my web site, and will post
> whatever answers I recieve there:
> 
> 	http://www.spack.org/index.cgi/LinuxRamLimits

And it isn't really a Linux limit. It is the hardware. If you need more
virtual space, get a 64 bit processor.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: What is the truth about Linux 2.4's RAM limitations?

[Timur Tabi]

Jesse Pollard wrote:

>>So what are the limits without using PAE? Here I'm still having a little
>>problem finding definitive answers but ...
>>
>3 GB. Final answers are in the FAQ, and have been discussed before. You can
>also look in the Intel 80x86 CPU specifications.
>
>The only way to exceed current limits is via some form of segment register usage
>which will require a different compiler and a replacement of the memory
>architecture of x86 Linux implementation.
>

Are you talking about using 48-bit pointers?

(48-bit pointers, aka 16:32 pointers, on x86 are basically "far 32-bit 
pointers".  That is, each pointer is stored as a 48-bit value, where 16 
bits are for the selector/segment, and 32 bits are for the offset.

-- 
Timur Tabi
Interactive Silicon

Re: What is the truth about Linux 2.4's RAM limitations?

[Andi Kleen]

Adam Shand <larry@spack.org> writes:

> 
>  * And, what (if any) paramaters can effect this (recompiling the app
>    etc)?

The kernel parameter is a constant called __PAGE_OFFSET which you can set.
You also need to edit arch/i386/vmlinux.lds

The reason why your simulation stopped at 2.3GB is likely that the malloc
allocation hit the shared libraries (check with /proc/<pid>/maps). Ways
around that are telling malloc to use mmap more aggressively (see the
malloc documentation in info libc) or moving the shared libraries up 
by changing a kernel constant called TASK_UNMAPPED_BASE. 

-Andi

What is the truth about Linux 2.4's RAM limitations?

[Adam Shand]

Where I just started work we run large processes for simulations and
testing of semi-conductors.  Currently we use Solaris because of past
limitations on the amount of RAM that a single process can address under
Linux.  Recently we tried to run some tests on a Dell dual Xeon 1.7GHz
with 4GB of RAM running Redhat 7.1 box (with the stock Redhat SMP kernel).
Speedwise it kicked the crap out of our Sunblade (Dual 750MHz with 8GB of
RAM)but we had problems with process dying right around 2.3GB (according
to top).

So I started to investigate, and quickly discovered that there is no good
source for finding this sort of information on line.  At least not that I
could find.  Nearly every piece of information I found conflicted in at
least some small way with another piece of information I found.
So I ask here in the hopes of a definitive answer.

 * What is the maximum amount of RAM that a *single* process can address
   under a 2.4 kernel, with PAE enabled?  Without?

 * And, what (if any) paramaters can effect this (recompiling the app
   etc)?

What I think I know so far is listed below.  I welcome being flamed, told
that I'm stupid and that I should have looked "here" so long as said
messages also contain pointers to definitive information :-)

Linux 2.4 does support greater then 4GB of RAM with these caveats ...

 * It does this by supporting Intel's PAE (Physical Address Extension)
   features which are in all Pentium Pro and newer CPU's.
 * The PAE extensions allow up to a maximum of 64GB of RAM that the OS
   (not a process) can address.
 * It does this via indirect pointers to the higher memory locations, so
   there is a CPU and RAM hit for using this.
 * Benchmarks seem to indicated around 3-6% CPU hit just for using the PAE
   extensions (ie. it applies regardless of whether you are actually
   accessing memory locations greater then 4GB).
 * If the kernel is compiled to use PAE, Linux will not boot on a computer
   whose hardware doesn't support PAE.
 * PAE does not increase Linux's ability for *single* processes to see
   greater then 3GB of RAM (see below).

So what are the limits without using PAE? Here I'm still having a little
problem finding definitive answers but ...

 * With PAE compiled into the kernel the OS can address a maximum of 4GB
   of RAM.
 * With 2.4 kernels (with a large memory configuration) a single process
   can address up to the total amount of RAM in the machine minus 1GB
   (reserved for the kernel), to a maximum 3GB.
 * By default the kernel reserves 1GB for it's own use, however I think
   that this is a tunable parameter so if we have 4GB of RAM in a box we
   can tune it so that most of that should be available to the processes
   (?).

I have documented the below information on my web site, and will post
whatever answers I recieve there:

	http://www.spack.org/index.cgi/LinuxRamLimits

Thanks,
Adam.

Re: What is the truth about Linux 2.4's RAM limitations?

[Brian Gerst]

Adam Shand wrote:
> 
> Where I just started work we run large processes for simulations and
> testing of semi-conductors.  Currently we use Solaris because of past
> limitations on the amount of RAM that a single process can address under
> Linux.  Recently we tried to run some tests on a Dell dual Xeon 1.7GHz
> with 4GB of RAM running Redhat 7.1 box (with the stock Redhat SMP kernel).
> Speedwise it kicked the crap out of our Sunblade (Dual 750MHz with 8GB of
> RAM)but we had problems with process dying right around 2.3GB (according
> to top).
> 
> So I started to investigate, and quickly discovered that there is no good
> source for finding this sort of information on line.  At least not that I
> could find.  Nearly every piece of information I found conflicted in at
> least some small way with another piece of information I found.
> So I ask here in the hopes of a definitive answer.
> 
>  * What is the maximum amount of RAM that a *single* process can address
>    under a 2.4 kernel, with PAE enabled?  Without?

3GB in either mode for user virtual address space.

>  * And, what (if any) paramaters can effect this (recompiling the app
>    etc)?

None.

> What I think I know so far is listed below.  I welcome being flamed, told
> that I'm stupid and that I should have looked "here" so long as said
> messages also contain pointers to definitive information :-)
> 
> Linux 2.4 does support greater then 4GB of RAM with these caveats ...
> 
>  * It does this by supporting Intel's PAE (Physical Address Extension)
>    features which are in all Pentium Pro and newer CPU's.
>  * The PAE extensions allow up to a maximum of 64GB of RAM that the OS
>    (not a process) can address.

64GB of PHYSICAL memory.  The processor is still limited to 4GB of
VIRTUAL memory per page table (per process) which must be shared between
user space and kernel space.  Linux uses a 3:1 split.

>  * It does this via indirect pointers to the higher memory locations, so
>    there is a CPU and RAM hit for using this.

If the kernel needs to access physical memory above the 960MB limit it
must remap it into a virtual address window on demand.  This leads to
TLB misses and poor caching.

>  * Benchmarks seem to indicated around 3-6% CPU hit just for using the PAE
>    extensions (ie. it applies regardless of whether you are actually
>    accessing memory locations greater then 4GB).

This is because the page table entries are 64-bit values, and we all
know how well the x86 does 64-bit stuff.

>  * If the kernel is compiled to use PAE, Linux will not boot on a computer
>    whose hardware doesn't support PAE.

This is by design.  It would be too much overhead to allow runtime
decision on PAE support.

>  * PAE does not increase Linux's ability for *single* processes to see
>    greater then 3GB of RAM (see below).

Correct.  PAE is only for addressing more physical memory.

> So what are the limits without using PAE? Here I'm still having a little
> problem finding definitive answers but ...
> 
>  * With PAE compiled into the kernel the OS can address a maximum of 4GB
>    of RAM.

Almost correct.  There needs to be a hole somewhere for memory mapped
devices, so not all of the physical addresses below 4GB are memory.

>  * With 2.4 kernels (with a large memory configuration) a single process
>    can address up to the total amount of RAM in the machine minus 1GB
>    (reserved for the kernel), to a maximum 3GB.

Not true.  A user process always has 3GB of address space, even if some
of that must be allocated from swap.

>  * By default the kernel reserves 1GB for it's own use, however I think
>    that this is a tunable parameter so if we have 4GB of RAM in a box we
>    can tune it so that most of that should be available to the processes
>    (?).

Again, the kernel reserves 1GB of VIRTUAL address space.  It uses this
to directly map up to 960 MB of physical memory.  The remaining 64 MB is
used for vmalloc, ioremap, and kmap.  Technically, you could change the
user:kernel split to 3.5:1.5, but doing that reduces the amount of
memory the kernel can directly map, leading to reduced performance (ie.
less memory for caches, etc.)

> 
> I have documented the below information on my web site, and will post
> whatever answers I recieve there:
> 
>         http://www.spack.org/index.cgi/LinuxRamLimits
> 
> Thanks,
> Adam.

To summarize:

- The x86 architecture is limited by the design of its addressing modes
and page tables to accessing 4GB of virtual memory.  This is a hard
limit.
- PAE mode (Physical Address Extensions) allows the processor to address
64GB of physical memory via the page tables, but does not change the
size of the virtual address space.
- User processes have 3GB of virtual address space to use.  This is true
even if there is less physical memory, becuase of swap and memory mapped
files (including executable code and shared libraries).
- The kernel has the remaining 1GB of virtual memory.
  - It directly maps up to 960MB of memory.
  - It leaves a 64MB window for dynamic allocations, such as vmalloc,
ioremap, and kmap.
- The user:kernel split can be changed, but this leaves the kernel with
less memory to work with, reducing performance and increasing the risk
of out of memory conditions.
- The only way the kernel can use "high" memory (physical memory over
960MB) is through user space mappings or via vmalloc or kmap on demand.
- There is no sane way around this architectural limit.  Any workrounds
would involve switching page tables and flushing kernel TLB entries
which would kill performance.
- On the user side, you can work within this limit by using shared
memory or mmap more agressively, to map in data on demand and release it
when it's not needed.
- 64-bit architectures aren't as limited in memory usage.  I think (but
I'm not positive) that some non-x86 32-bit architectures allow for
seperate user and kernel page tables in the hardware, expanding the
limits.

--

				Brian Gerst

Re: What is the truth about Linux 2.4's RAM limitations?

[Rik van Riel]

On Mon, 9 Jul 2001, Adam Shand wrote:

>  * What is the maximum amount of RAM that a *single* process can address
>    under a 2.4 kernel, with PAE enabled?  Without?

3 GB, 3GB.  This is basically a hardware limit.

>  * And, what (if any) paramaters can effect this (recompiling the app
>    etc)?

Alpha, AMD Hammer, ...  ;)


regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: What is the truth about Linux 2.4's RAM limitations?

[Matti Aarnio]

  This really should be a Linux-Kernel FAQ item...
  ( http://www.tux.org/lkml/ )

On Mon, Jul 09, 2001 at 01:01:17PM -0700, Adam Shand wrote:
> Linux 2.4 does support greater then 4GB of RAM with these caveats ...
> 
>  * It does this by supporting Intel's PAE (Physical Address Extension)
>    features which are in all Pentium Pro and newer CPU's.

   Carefull out there.

   Check intel 386 family processors address mapping manuals.
   You will see that the address calculated by combining "base"
   and "segment" are in fact TRUNCATED at 32-bits before feeding
   them to page mapping machinery --> There are no tricks at all
   to have SIMULTANEOUS access to more than 4 GB of memory without
   playing MMU mapping tricks -- which are painfully slow...

   That is the origin of i386 architecture 4 GB virtual memory limit.

   All that PAE mode does is to allow that 4 GB virtual space to be
   mapped into larger physical space.

   Compound that with unability to have separate user and kernel
   mappings active at the same time (unlike e.g. Motorola 68000
   family MMUs do), and the userspace can't have even that 4GB,
   but is limited to at least 3.5/0.5 (user/kernel) split, more
   commonly to 3.0/1.0 split.

   With 64-bit processors (those that do addresses in 64-bit mode)
   there are a plenty of bits to "waste" in these mappings.  E.g.
   one can have a 1:1 mapping in between kernel and user, with
   2^63 bits address space for each.  The number is mindbogglingly
   large..  ( 2G * 4G = 8 000 000 T = 8 000 E )
   (In reality e.g. Alphas do use "only" 43-bits of addresses in
    these mappings, but even those are 1000+ times larger than 4G.)

> Thanks,
> Adam.

/Matti Aarnio

Re: What is the truth about Linux 2.4's RAM limitations?

[Chris Wedgwood]

On Tue, Jul 10, 2001 at 01:17:55AM +0300, Matti Aarnio wrote:

       Compound that with unability to have separate user and kernel
       mappings active at the same time (unlike e.g. Motorola 68000
       family MMUs do), and the userspace can't have even that 4GB,
       but is limited to at least 3.5/0.5 (user/kernel) split, more
       commonly to 3.0/1.0 split.

How does FreeBSD do this? What about other OSs? Do they map out most
of userland on syscall entry and map it in as required for their
equivalents to copy_to/from_user? (Taking the performance hit in doing
so?)



  --cw

Re: What is the truth about Linux 2.4's RAM limitations?

[Timur Tabi]

Chris Wedgwood wrote:

>How does FreeBSD do this? What about other OSs? Do they map out most
>of userland on syscall entry and map it in as required for their
>equivalents to copy_to/from_user? (Taking the performance hit in doing
>so?)
>

I don't know about *BSD, but in Windows NT/2000, even drivers run in 
virtual space.  The OS is not monolithic, so address spaces are general 
not "shared" as they are in Linux.

-- 
Timur Tabi
Interactive Silicon

Re: What is the truth about Linux 2.4's RAM limitations?

[Richard B. Johnson]

On Tue, 10 Jul 2001, Timur Tabi wrote:

> Chris Wedgwood wrote:
> 
> >How does FreeBSD do this? What about other OSs? Do they map out most
> >of userland on syscall entry and map it in as required for their
> >equivalents to copy_to/from_user? (Taking the performance hit in doing
> >so?)
> >
> 
> I don't know about *BSD, but in Windows NT/2000, even drivers run in 
> virtual space.  The OS is not monolithic, so address spaces are general 
  ^^^^^^^^^^^^^
> not "shared" as they are in Linux.
       ^^^^^^^

Everything in the kernel is virtual. There are some 1:1 mappings in
low-address space, but there is a PTE for everything addressed anywhere.
It has to be this way or paging would not work.  Linux is also not
as you say "monolithic". We have installable device drivers, you know.
They are logically within the kernel's virtual address space, but
may be anywhere in physical memory. Also, although the kernel is
uncompressed into memory slightly above 1 megabyte upon startup,
it could exist anywhere in physical memory. It's just easier to
make the 1:1 mappings exist where the kernel was started in 32-bit
linear mode, before paging was enabled.

The kernel has the capability (by design) of addressing anything it
wants. So, if this is what you mean by "shared", I guess you imply
that Windows can't address anything it wants? Of course it can.

In Unix and Unix variants, it is by design, provided that the
kernel exist within every process address space. Early Nixes
like Ultrix, simply called the kernel entry point. Since it
was protected, this trapped to the real kernel and the page-fault
handler actually performed the work on behalf of the caller.

Unlike some OS (like VMS), a context-switch does not occur
when the kernel provides services for the calling task.
Therefore, it was most reasonable to have the kernel exist within
each tasks address space. With modern processors, it doesn't make
very much difference, you could have user space start at virtual
address 0 and extend to virtual address 0xffffffff. However, this would
not be Unix. It would also force the kernel to use additional
CPU cycles when addressing a tasks virtual address space,
i.e., when data are copied to/from user to kernel space.

> 
> -- 
> Timur Tabi
> Interactive Silicon
> 

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: What is the truth about Linux 2.4's RAM limitations?

[Timur Tabi]

Richard B. Johnson wrote:

>The kernel has the capability (by design) of addressing anything it
>wants. So, if this is what you mean by "shared", I guess you imply
>that Windows can't address anything it wants? Of course it can.
>

Well, I may have oversimplified it a bit.  My point was that a given 
Windows driver can't just take a 32-bit pointer and pass it to another 
driver and have it just work like that.

-- 
Timur Tabi
Interactive Silicon

Re: What is the truth about Linux 2.4's RAM limitations?

[Jonathan Lundell]

At 1:35 PM -0400 2001-07-10, Richard B. Johnson wrote:
>Unlike some OS (like VMS), a context-switch does not occur
>when the kernel provides services for the calling task.
>Therefore, it was most reasonable to have the kernel exist within
>each tasks address space. With modern processors, it doesn't make
>very much difference, you could have user space start at virtual
>address 0 and extend to virtual address 0xffffffff. However, this would
>not be Unix. It would also force the kernel to use additional
>CPU cycles when addressing a tasks virtual address space,
>i.e., when data are copied to/from user to kernel space.

Certainly the shared space is convenient, but in what sense would a 
separate kernel space "not be Unix"? I'm quite sure that back in the 
AT&T days that there were Unix ports with separate kernel (vs user) 
address spaces, as well as processors with special instructions for 
doing the copies (move to/from user space). Having separate system & 
user base page table pointers makes this relatively practical.
-- 
/Jonathan Lundell.

Re: What is the truth about Linux 2.4's RAM limitations?

[Richard B. Johnson]

On Tue, 10 Jul 2001, Jonathan Lundell wrote:

> At 1:35 PM -0400 2001-07-10, Richard B. Johnson wrote:
> >Unlike some OS (like VMS), a context-switch does not occur
> >when the kernel provides services for the calling task.
> >Therefore, it was most reasonable to have the kernel exist within
> >each tasks address space. With modern processors, it doesn't make
> >very much difference, you could have user space start at virtual
> >address 0 and extend to virtual address 0xffffffff. However, this would
> >not be Unix. It would also force the kernel to use additional
> >CPU cycles when addressing a tasks virtual address space,
> >i.e., when data are copied to/from user to kernel space.
> 
> Certainly the shared space is convenient, but in what sense would a 
> separate kernel space "not be Unix"? I'm quite sure that back in the 

I explained why it would not be Unix.


> AT&T days that there were Unix ports with separate kernel (vs user) 
> address spaces, as well as processors with special instructions for 

No. The difference between kernel and user address space is protection.
Let's say that you decided to revamp all the user space to go from
0 to 2^32-1. You call the kernel with a pointer to a buffer into
which you need to write kernel data:

You will need to set a selector that will access both user and
kernel data at the same time. Since the user address space is
paged, this will not be possible, you get one or the other, but
not both. Therefore, you need to use two selectors. In the case
of ix86 stuff, you could set DS = KERNEL_DS and ES = a separately
calculated selector that represents the base address of the caller's
virtual address space. Note that you can't just use the caller's
DS or ES because they can be changed by the caller.

Then you can move data from DS:OFFSET to ES:OFFSET, but not the
other way. If you need to move data the other way, you need DS: = a
separately calculated selector that represents the base address of the
caller's virtual address space, and ES = KERNEL_DS. Then you can copy from
ES:OFFSET to ES:OFFSET (as before), but the data goes the other way.

With the same virtual address space for kernel and user, you
don't need any of this stuff. The only reason we have special
functions (macros) for copy to/from, is to protect the kernel
from crashing when the user-mode caller gives it a bad address.

It all tasks were cooperative, you could use memcpy() perfectly
fine (or rep movsl ; rep movsw ; rep movsb).

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: What is the truth about Linux 2.4's RAM limitations?

[Jonathan Lundell]

At 2:45 PM -0400 2001-07-10, Richard B. Johnson wrote:
>  > AT&T days that there were Unix ports with separate kernel (vs user)
>>  address spaces, as well as processors with special instructions for
>
>No. The difference between kernel and user address space is protection.
>Let's say that you decided to revamp all the user space to go from
>0 to 2^32-1. You call the kernel with a pointer to a buffer into
>which you need to write kernel data:
>
>You will need to set a selector that will access both user and
>kernel data at the same time. Since the user address space is
>paged, this will not be possible, you get one or the other, but
>not both. Therefore, you need to use two selectors. In the case
>of ix86 stuff, you could set DS = KERNEL_DS and ES = a separately
>calculated selector that represents the base address of the caller's
>virtual address space. Note that you can't just use the caller's
>DS or ES because they can be changed by the caller.

Sure, for IA-32, it's a royal pain. But Unix runs, after all, on 
other CPUs. In point of simple historical fact, there have been Unix 
ports with separate kernel and user address spaces, and there are 
CPUs that, unlike IA-32, can simultaneously address the two spaces, 
by virtue of having separate page table pointers and instructions for 
copying between the two spaces.

To the extent that we're talking about Linux on IA-32, though, I 
entirely agree that the cost of expanding user space from 3GB to 4GB 
far outweighs any benefit. Especially with 64-bit architectures 
coming online.
-- 
/Jonathan Lundell.

Re: What is the truth about Linux 2.4's RAM limitations?

[Jesse Pollard]

On Tue, 10 Jul 2001, Richard B. Johnson wrote:
>On Tue, 10 Jul 2001, Jonathan Lundell wrote:
>
>> At 1:35 PM -0400 2001-07-10, Richard B. Johnson wrote:
>> >Unlike some OS (like VMS), a context-switch does not occur
>> >when the kernel provides services for the calling task.
>> >Therefore, it was most reasonable to have the kernel exist within
>> >each tasks address space. With modern processors, it doesn't make
>> >very much difference, you could have user space start at virtual
>> >address 0 and extend to virtual address 0xffffffff. However, this would
>> >not be Unix. It would also force the kernel to use additional
>> >CPU cycles when addressing a tasks virtual address space,
>> >i.e., when data are copied to/from user to kernel space.
>> 
>> Certainly the shared space is convenient, but in what sense would a 
>> separate kernel space "not be Unix"? I'm quite sure that back in the 
>
>I explained why it would not be Unix.
>
>
>> AT&T days that there were Unix ports with separate kernel (vs user) 
>> address spaces, as well as processors with special instructions for 
>
>No. The difference between kernel and user address space is protection.
>Let's say that you decided to revamp all the user space to go from
>0 to 2^32-1. You call the kernel with a pointer to a buffer into
>which you need to write kernel data:
>
>You will need to set a selector that will access both user and
>kernel data at the same time. Since the user address space is
>paged, this will not be possible, you get one or the other, but
>not both. Therefore, you need to use two selectors. In the case
>of ix86 stuff, you could set DS = KERNEL_DS and ES = a separately
>calculated selector that represents the base address of the caller's
>virtual address space. Note that you can't just use the caller's
>DS or ES because they can be changed by the caller.

Sure you can - But you do have to modify the page tables for the
kernel access. You also have to verity that the page is valid
to the user, that the offset to the location to modify is
valid from both the users context and the kernel context.

>Then you can move data from DS:OFFSET to ES:OFFSET, but not the
>other way. If you need to move data the other way, you need DS: = a
>separately calculated selector that represents the base address of the
>caller's virtual address space, and ES = KERNEL_DS. Then you can copy from
>ES:OFFSET to ES:OFFSET (as before), but the data goes the other way.

See, it can be done, but the changing page tables are a PITA. It's slow.

>With the same virtual address space for kernel and user, you
>don't need any of this stuff. The only reason we have special
>functions (macros) for copy to/from, is to protect the kernel
>from crashing when the user-mode caller gives it a bad address.

wrong - In either case, the parameters to the system call (and the
return values) have to be evaluated for proper usage and security.
That is NOT unique and can be done in many ways.

They provide the same protection as that provided by
other hardware. Even using page remapping will work, if implemented
properly. it just isn't as fast as using a single virtual mapping.

Been there, done that.

Multi-tasking real time systems do this a LOT - and because the
amount of data passed may be quite small it is frequently
more efficient. (largest I ever did was about 256 bytes - ocean going
autopilot/seismic surveys). The determining factor is:
1. correctness, 2. speed, 3. ease of implementation, 4. hardware
support.

>It all tasks were cooperative, you could use memcpy() perfectly
>fine (or rep movsl ; rep movsw ; rep movsb).

You still must verify that the source/destination are reasonably valid.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: jesse@cats-chateau.net

Any opinions expressed are solely my own.

Re: What is the truth about Linux 2.4's RAM limitations?

[Malcolm Beattie]

Richard B. Johnson writes:
> On Tue, 10 Jul 2001, Timur Tabi wrote:
> 
> > Chris Wedgwood wrote:
> > 
> > >How does FreeBSD do this? What about other OSs? Do they map out most
> > >of userland on syscall entry and map it in as required for their
> > >equivalents to copy_to/from_user? (Taking the performance hit in doing
> > >so?)
> > >
> > 
> > I don't know about *BSD, but in Windows NT/2000, even drivers run in 
> > virtual space.  The OS is not monolithic, so address spaces are general 
>   ^^^^^^^^^^^^^
> > not "shared" as they are in Linux.
>        ^^^^^^^
> 
> Therefore, it was most reasonable to have the kernel exist within
> each tasks address space. With modern processors, it doesn't make
> very much difference, you could have user space start at virtual
> address 0 and extend to virtual address 0xffffffff. However, this would
> not be Unix. It would also force the kernel to use additional
> CPU cycles when addressing a tasks virtual address space,
> i.e., when data are copied to/from user to kernel space.

This is rather misleading and Intel-architecture-specific rather than
Unix-specific. For example, Linux on S/390 uses a complete 2Gb address
space (31 bits; the limit of addressability on the 32-bit S/390
architecture) for the current task and a separate 2GB address space for
the kernel. The kernel is not mapped into the "current" address space
but features of the architecture which provide for separate concurrent
address spaces via special registers are used. Copies between kernel
space and user space use special instructions which reference these
address space registers automagically.

--Malcolm

-- 
Malcolm Beattie <mbeattie@sable.ox.ac.uk>   <-- This email address will break
Unix Systems Programmer                   when I quit OUCS on Jul 20th. Send
Oxford University Computing Services    private mail to mbeattie@clueful.co.uk
                                      I'll sort out my IBM email address soon.

Re: What is the truth about Linux 2.4's RAM limitations?

[jlnance]

On Mon, Jul 09, 2001 at 01:01:17PM -0700, Adam Shand wrote:
> 
> Where I just started work we run large processes for simulations and
> testing of semi-conductors.  Currently we use Solaris because of past
> limitations on the amount of RAM that a single process can address under
> Linux.  Recently we tried to run some tests on a Dell dual Xeon 1.7GHz
> with 4GB of RAM running Redhat 7.1 box (with the stock Redhat SMP kernel).
> Speedwise it kicked the crap out of our Sunblade (Dual 750MHz with 8GB of
> RAM)but we had problems with process dying right around 2.3GB (according
> to top).

I have had the same experience.  I work with machines that run large single
processes, and while the Linux machines are much faster than the Solaris
machines we have, the Linux machines do not handle large processes very well.

I asked some questions on this list similar to yours and got a lot of useful
information.  One important thing that did not come up, probably because its
not considered a kernel issue, is that the malloc() in glibc has some
limitations that keep you from using all 3G of virtual address space that
your process has.  Here is a description of the problem, its from memory so
the numbers may not be exact, but it will give you the idea.  When you run
a program, your address space gets divided up into several regions.  Your
program itself gets mapped into low memory a few megs above 0x0.  Your programs
stack gets mapped into the top of memory at 0xc0000000 and it grows down
tword the program text.  Files or shared libraries are mmaped in starting
at 0x40000000 (1G) and each new mmap() occurs at a higher address so that
the mapped area grows tword the stack.
its something like this:

       0x00000000 ---> Unmapped
       0x08048000 ---> Text/Data
       0x40000000 ---> Shared libs and mmaped
       0xc0000000 ---> Stack

Now when you call malloc() it can get its memory in one of two ways.  For small
requests (less than 128K), it uses the sbrk() call which gets memory space
that grows up from the top up the text/data area.  When this area grows up
into the shared libs, sbrk() fails and malloc can not give you any more memory,
even though you have about 2G of address space available above the shared
libs.

For large requests malloc() calls mmap(), which starts allocating at the top
of your shared libs and will fail when this area hits the stack.  You can
get close to 2G of memory this way, but that still leaves almost 1G of unused
memory below 0x40000000.

There are some environment variables that let you tune exactly when malloc()
calls mmap() and when it calls sbrk().  These did not help me though, because
our program allocates most of its memory in small blocks that are too small
to be mmap()ed effectivly.

I thought I could get around this by statically linking my program.  That did
not work because there still seems to be a single page mapped in at 0x40000000.

Next I tried the hoard allocator.  That worked better than the glibc malloc
because it always uses mmap() so I got close to 2G of memory rather than
slightly less than 1G.

Next I patched the hoard allocator so that instead of calling mmap(0, ...)
it would call it as something like mmap(0x10000000, ...) which causes it
to start allocating memory that would normally be reserved for sbrk().
I think I could get close to 2.5G from the hoard malloc after this patch.
This change got incorporated into the latest hoard, but I had problems
building the latest hoard, so you may want to wait for a future release.

Anyway, I just wanted to let you know that there are some glibc issues that
are more restrictive than the kernel issues.

Hope this helps,

Jim

Re: What is the truth about Linux 2.4's RAM limitations?

[Michael Bacarella]

On Mon, Jul 09, 2001 at 11:01:51PM -0400, jlnance@intrex.net wrote:
> Next I patched the hoard allocator so that instead of calling mmap(0, ...)
> it would call it as something like mmap(0x10000000, ...) which causes it
> to start allocating memory that would normally be reserved for sbrk().
> I think I could get close to 2.5G from the hoard malloc after this patch.
> This change got incorporated into the latest hoard, but I had problems
> building the latest hoard, so you may want to wait for a future release.
> 
> Anyway, I just wanted to let you know that there are some glibc issues that
> are more restrictive than the kernel issues.

dietlibc is pretty funky:

http://www.fefe.de/dietlibc/

I'm not sure if it's memory usage patterns are all that different from
glibc (as I haven't checked). If so, it may still be easier to
hack than something as uh.. established.. as glibc.

-- 
Michael Bacarella <mbac@nyct.net>
Technical Staff / System Development,
New York Connect.Net, Ltd.

Re: What is the truth about Linux 2.4's RAM limitations?

[Ingo Oeser]

On Mon, Jul 09, 2001 at 11:01:51PM -0400, jlnance@intrex.net wrote:
> stack gets mapped into the top of memory at 0xc0000000 and it grows down
> tword the program text.  Files or shared libraries are mmaped in starting
> at 0x40000000 (1G) and each new mmap() occurs at a higher address so that
> the mapped area grows tword the stack.
> its something like this:
> 
>        0x00000000 ---> Unmapped
>        0x08048000 ---> Text/Data
[1]
>        0x40000000 ---> Shared libs and mmaped
>        0xc0000000 ---> Stack

Does anybody know, why we leave that much Room unmapped in [1]?

Having at least one page unmapped is ok to catch programmer
errors, but the first 132MB seems a little bit too large...

I still don't know why an allocator ONLY using mmap() and no
brk()/sbrk() shouldn't use mmap(getpagesize(),...).

PS: I used this information already to help someone having these
   kind of problems. 
   

Thanks & Regards

Ingo Oeser