Re: BIOS detects 4 GB RAM, but kernel does not

Re: BIOS detects 4 GB RAM, but kernel does not

[Robert Hancock]

ravibt@gmail.com wrote:
> Hello,
> 
>  On a dual core Pentium 4 EM64T machine (Intel Desktop Board D915GAV),
> we used four 1GB RAM (DDR 400) modules. The BIOS (EV91510A.86A.0444)
> detected all the four 1 GB modules, but once the OS is booted, only
> ~3.1GB is available for usage (from dmesg: "Memory: 3210516k/3267772k
> available"; see below). The kernel used is version 2.6.9-22.ELsmp
> coming with 'CentOS release 4.2 Final'.
> [The four RAM modules have been tested OK with the 'memtest'].
> 
>  Using "mem=4096m" while booting the kernel also did not help. Searched
> through the old messages and it looks like in most of the cases
> enabling some memory-hole related option in BIOS is suggested, but in
> this case probably the BIOS is fine. Not sure if some kernel
> configuration option is missing or if someother version of the kernel
> needs to be used.
> 
> This being a 64 bit machine, we expected memory-remap to be happening.
> Is there a way in which ~900 MB of RAM can be made usable?
> Any pointers will be of great help.
> 
> Please let me know if more information is needed than the following
> transcripts (/proc/iomem and dmesg):

Essentially I don't think there is much you can do about this on this 
board. The memory space starting at around 3.2GB is being used by the 
memory-mapped IO regions for the PCI and PCI Express devices and 
motherboard resources and therefore "covers up" the RAM in that part of 
the address space. The solution to this is for the system to remap the 
affected memory above the 4GB mark, which is possible with Athlon 
64/Opteron CPUs and on some of the Intel server chipsets. However, I 
don't think any Intel desktop chipsets support this for some 
unfathomable reason.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: BIOS detects 4 GB RAM, but kernel does not

[Robert Hancock]

Jean-David Beyer wrote:
> You need to compile with
> 
> # CONFIG_NOHIGHMEM is not set
> # CONFIG_HIGHMEM4G is not set
> CONFIG_HIGHMEM64G=y
> CONFIG_HIGHMEM=y
> CONFIG_HIGHPTE=y
> CONFIG_X86_PAE=y
> CONFIG_HIGHIO=y
> CONFIG_X86_4G=y
> CONFIG_X86_SWITCH_PAGETABLES=y
> CONFIG_X86_4G_VM_LAYOUT=y
> CONFIG_X86_UACCESS_INDIRECT=y
> CONFIG_X86_HIGH_ENTRY=y
> 
> in your .config (for *86 machines) to get 4G of user and 4G of system space.
> I so or know how much sense this would make for a 4G of RAM machine. I never
> tried it when this machine had only 4G, but that is what is required to get
> all 4G in a single user process now that I have 8G.

Those options are not applicable on an x86-64 kernel.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: BIOS detects 4 GB RAM, but kernel does not

[Jan Engelhardt]

>
> Essentially I don't think there is much you can do about this on this board.
> The memory space starting at around 3.2GB is being used by the memory-mapped IO
> regions for the PCI and PCI Express devices and motherboard resources and
> therefore "covers up" the RAM in that part of the address space. The solution
> to this is for the system to remap the affected memory above the 4GB mark,
> which is possible with Athlon 64/Opteron CPUs and on some of the Intel server
> chipsets. However, I don't think any Intel desktop chipsets support this for
> some unfathomable reason.

Maybe PAE can help?


Jan Engelhardt
-- 

Re: BIOS detects 4 GB RAM, but kernel does not

[Robert Hancock]

Jan Engelhardt wrote:
>> Essentially I don't think there is much you can do about this on this board.
>> The memory space starting at around 3.2GB is being used by the memory-mapped IO
>> regions for the PCI and PCI Express devices and motherboard resources and
>> therefore "covers up" the RAM in that part of the address space. The solution
>> to this is for the system to remap the affected memory above the 4GB mark,
>> which is possible with Athlon 64/Opteron CPUs and on some of the Intel server
>> chipsets. However, I don't think any Intel desktop chipsets support this for
>> some unfathomable reason.
> 
> Maybe PAE can help?

No.. it's not a matter of being unable to reach the address space, but 
the memory used by the IO regions is essentially overridden and 
unavailable to the kernel. PAE is not applicable on x86-64 anyway..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: BIOS detects 4 GB RAM, but kernel does not

[Robert Hancock]

iforone wrote:
> see why above (mostly)... That doesn't necessarily mean that some kind
> of certain 'config' doesn't need to be compiled into the kernel. Yet
> I'm not sure exactly which options are necessary, if any, nor am I sure
> you can achieve your goal using what *seems* (Robert Hancock's
> reponses) to be essentially an Intel DualCore 32bit CPU(s). I'm having
> a bit of trouble understanding (or believing) that an EM64T system
> isn't capable of seeing more than 4GB RAM (although the Mobo max for
> that Desktop mobo is 4GB) -- BUT then again, that is not a Server grade
> Mobo either -- it's a Desktop mobo (perhaps using an
> unsupporting(cheaper) Chipset(?). Yet; the BIOS detects all 4GB (which
> shoots down my chipset theory) - so maybe it's a kerenl specific issue
> (harkens back to General S's response about 2.6.15+ kernels only).
> In my earlier post, I hadn't realized you were using a 64bit kernel.

The BIOS can see that all 4GB of memory is there, but it has no way of 
moving it out of the way to make room for the PCI/PCI-E memory-mapped IO 
space, so it has to map that IO space over the RAM in that part of the 
address space, rendering it inaccessible. There's no way that the kernel 
can fix this, regardless of whether you are using 32 or 64 bit or what 
configuration options are set.

Athlon 64/Opteron CPUs have support for moving this part of the RAM 
above 4GB to allow it to be used. This is part of the CPU's on-die 
memory controller so no special chipset support is needed. On Intel 
systems this support has to be provided by the chipset, and on the 
desktop boards, it's not.

You can see what's going on from the BIOS e820 memory map that's printed 
in the dmesg output at the start of bootup. If you calculate out the 
amount of address space reported as "usable" then you will get your 
value of 3.2GB or so which is all the kernel has access to. If the 
system supported memory remapping then you would see another region 
starting at 0x0000000100000000 (4GB) which would account for this 
missing 800MB or so.

Probably the main reason Intel didn't bother including this support in 
the desktop boards is that current non-server versions of Windows (at 
least 32-bit) won't use any memory that is mapped above 4GB anyway even 
though PAE is enabled - a purely artificial limit that MS put in place 
to discourage using desktop Windows on such large memory machines..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: BIOS detects 4 GB RAM, but kernel does not

[Robert Hancock]

iforone wrote:
> If we take a quick look at (part of) the OP's 'dmesg' output;
>  ACPI: RSDT (v001 INTEL  D915GAV  0x20050429 MSFT 0x00000097) @
>  0x00000000c7730000
>  ACPI: FADT (v002 INTEL  D915GAV  0x20050429 MSFT 0x00000097) @
>  0x00000000c7730200
>  ACPI: MADT (v001 INTEL  D915GAV  0x20050429 MSFT 0x00000097) @
>  0x00000000c7730390
>  ACPI: MCFG (v001 INTEL  D915GAV  0x20050429 MSFT 0x00000097) @
>  0x00000000c7730400
>  ACPI: ASF! (v016 LEGEND I865PASF 0x00000001 INTL 0x02002026) @
>  0x00000000c7736020
>  ACPI: TCPA (v001 INTEL  TBLOEMID 0x00000001 MSFT 0x00000097) @
>  0x00000000c77360c0
>  ACPI: WDDT (v001 INTEL  OEMWDDT  0x00000001 INTL 0x02002026) @
>  0x00000000c77360f4
>  ACPI: DSDT (v001 INTEL  D915GAV  0x00000001 INTL 0x02002026) @
>  0x0000000000000000
>  No NUMA configuration found
>  Faking a node at 0000000000000000-00000000c772f000
>  Bootmem setup node 0 0000000000000000-00000000c772f000
>  No mptable found.
>  On node 0 totalpages: 816943
> 
> ===============
> 
> We see MSFT (Microsoft) all over the place as well as INTL (Intel ?) --
> I suppose this machine may be a Dellio (perhaps HP or Compaq)? or
> atleast one that has one of those stickers on the case that says
> "Designed for use with MS Windows [version]"

I believe that indicates that the Microsoft ACPI compiler was used to 
generate the ACPI tables in the BIOS..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: BIOS detects 4 GB RAM, but kernel does not

[Robert Hancock]

iforone wrote:
>> Probably the main reason Intel didn't bother including this support in
>> the desktop boards is that current non-server versions of Windows (at
>> least 32-bit) won't use any memory that is mapped above 4GB anyway even
>> though PAE is enabled - a purely artificial limit that MS put in place
>> to discourage using desktop Windows on such large memory machines..
> 
> Agreed -- I _was_ going to inquire about XP x64 versions, and PAE also,
> and I've read a bit about it (as microshaft has written on it's site),
> but not in quite a while. I recall the semi-related  NoExec (NX) bit
> stuff (though I forgot the mnemonic difference between AMD64 and Intel
> concerning this "bit", which if set disallows execution of any code
> above the 4GB boundary, IIRC).
> 
> to follow-up: Do the XP x64 versions do something else artificially to
> enable addressing up to 16GB of RAM or thereabouts. Or - is it that PAE
> (Physical Address Extensions) stuff again that allows or it?

PAE is what allows an OS in 32-bit mode to access memory located above 
4GB in the address space. On a 64-bit OS the CPU can access all the 
memory directly so PAE is not needed or possible. Also, PAE is often 
enabled on 32-bit systems even with less than 4GB of RAM as it allows 
using the hardware NX bit, which was only added to the PAE versions of 
the 32-bit page tables.

That's a separate issue, though, from the original subject of the thread 
(memory being mapped over by IO space), and from the artificial limit on 
accessing memory above 4GB in at least 32-bit consumer Windows.

> 
> More importantly -- I have (an as-yet-to-be-assembled system) : AMD64
> s754 3000+ with a crappy mATX mobo here (VIA KTm800/8237) Chipset --
> The RAM limit is 2GB total (2 x 1GB DIMM slots only). Do you think this
> el-cheapo mobo would have problems accessing over 4GB *if* the Mobo was
> designed for 4GB ?? IOW-- a Mobo perhaps such as General S uses (MSI,
> ASUStek, etc) -- or do you know if there's something different between
> the s754 and s939 models that I'm unaware of (besides the No Dual
> Channel RAM in s754, since it's only 64bit Single-Channel capable, not
> 128bit).

I don't think most S754 boards support physically installing that much 
RAM, so it may be moot. If you could, you could likely access memory 
above 4GB, though I'm not certain if the S754 CPUs support the memory 
hole remapping to avoid some of that RAM being lost.

> Thanks for the continuing discussion -- I'm glad I didn't follow Intel
> recently and become deceived again - as I'm sure MANY have -- you'd
> think buying a spanking new Pentium-D (8xx) and a 'decent' Intel
> Desktop mobo would allow access to more than 4GB RAM ...but no.... :-(
> (especially when the specs for the mobo claim "4GB" - heck - might as
> well remove 1 x 1GB DIMM, you'd only lose 200MB (yikes)).

Yeah, there is not much point in installing more than 3GB in such a board..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: BIOS detects 4 GB RAM, but kernel does not

[Andi Kleen]

Robert Hancock <hancockr@shaw.ca> writes:

> Athlon 64/Opteron CPUs have support for moving this part of the RAM
> above 4GB to allow it to be used. This is part of the CPU's on-die
> memory controller so no special chipset support is needed.

In cheap boards >3.5GB RAM configurations are usually not officially
supported by the vendor (= not tested) and there are systems where it
doesn't work when enabled in the BIOS (doesn't work = kernel crashes
randomly when accessing bad memory ranges)

I guess it's a subtle hint that above 3GB of RAM you should be using
ECC DIMMs anyways, which need a more expensive workstation class
board.

-Andi