How to detect the amount of RAM on non-EFI platforms?

How to detect the amount of RAM on non-EFI platforms?

[Cyan]

Hi,

I am modifying GRUB for a Linux LiveOS, which boots into separate images depends on
how much RAM presents on the system. GRUB has a memory map initialized, so I think
it is relatively easy to do so.

However, I discovered this method only works on EFI systems, if I load it on i386-pc
(not a real i386 PC, just using BIOS to boot), the memory usage returns 0.

Here's my code[1]:

grub-core/commands/phymem.c

grub_uint64_t total_mem_bytes = 0ULL;

static int
traverse_mmap_hook (grub_uint64_t addr, grub_uint64_t size, grub_memory_type_t type,
                    void *data __attribute__ ((unused)))
{
  total_mem_bytes += size;
}

grub_err_t
grub_cmd_phymem (...) {
	grub_mmap_iterate (traverse_mmap_hook, NULL);
}

Then I convert the integer into a string and export to the environment variables,
and print it out.

On EFI systems, I can get the correct amount of system memory:

grub> insmod phymem
grub> phymem
The total system memory is 16384 MiB.

However on BIOS systems, the result is zero:

grub> insmod phymem
grub> phymem
The total system memory is 0 MiB.

So, how can I detect the amount if RAM on non-EFI systems? If I recalled it right,
GRUB can boot kernels which complies with Multiboot specification, so the memory
info should be always present. If I missed something, please point it out.

Thanks,
Cinhi Young

Re: How to detect the amount of RAM on non-EFI platforms?

[Cyan Yang]

Hi again,

On 2023/7/11 10:08, Cyan via Grub-devel wrote:

> static int
> traverse_mmap_hook (grub_uint64_t addr, grub_uint64_t size, grub_memory_type_t type,
>                     void *data __attribute__ ((unused)))
> {
>   total_mem_bytes += size;
> }
> 

I discovered a huge problem in my hook function: missing a return at the end of the hook function.
Without this return the hook function will execute only once. 
And, regions with type GRUB_MEMORY_RESERVED should be skipped. So the hook function body should look like this:

if (type != GRUB_MEMORY_RESERVED) {
  total_mem_bytes += size;
}
return GRUB_ERR_NONE;

After this change the result looks fine.

Regards,
Cinhi

Re: How to detect the amount of RAM on non-EFI platforms?

[Daniel Kiper]

On Tue, Jul 11, 2023 at 09:37:15PM +0800, Cyan Yang via Grub-devel wrote:
> Hi again,
>
> On 2023/7/11 10:08, Cyan via Grub-devel wrote:
>
> > static int
> > traverse_mmap_hook (grub_uint64_t addr, grub_uint64_t size, grub_memory_type_t type,
> >                     void *data __attribute__ ((unused)))
> > {
> >   total_mem_bytes += size;
> > }
>
> I discovered a huge problem in my hook function: missing a return at the end of the hook function.
> Without this return the hook function will execute only once.
> And, regions with type GRUB_MEMORY_RESERVED should be skipped. So the hook function body should look like this:
>
> if (type != GRUB_MEMORY_RESERVED) {
>   total_mem_bytes += size;
> }
> return GRUB_ERR_NONE;

You are mixing enum and int types. The function should return 0 here.

Anyway, I think it would be nice if you could upstream this module.
Potentially other people may find it useful.

Daniel

Re: How to detect the amount of RAM on non-EFI platforms?

[Cyan]

Hi Daniel,

On 7/14/23 02:38, Daniel Kiper wrote:
> On Tue, Jul 11, 2023 at 09:37:15PM +0800, Cyan Yang via Grub-devel wrote:
>
> You are mixing enum and int types. The function should return 0 here.
>

Yes, the hook function should return zero instead of enum vaule
GRUB_ERR_NONE.


> Anyway, I think it would be nice if you could upstream this module.
> Potentially other people may find it useful.
>

Thanks for the suggestion, and yes I was planning to upstream this
module. However, a serious issue have been discovered during the initial
test, made me concerning that this way of presenting the amount of
physical memory should not be considered as the primary way:

When running on a VM it did not report the real amount of system memory.
A few hundred KiB of memory was left undetected, probably reserved by
firmware or hardware.

Here's an example, on a Virtualbox VM:

grub> phymem
The total system memory is 8191 MiB (8589539328 bytes).

while allocated RAM for the VM is 8192MiB (8589934592 Bytes).

Users usually expect the real amount, which should be a multiple of e.g.
2MiB or even bigger. My code[1] is currently rounding the result to the
closest multiple of 2MiB or 128MiB, which depends on platform. The
number is totally an assumption, and there's a chance that some systems
can reserve more than that.

So basically there are two options:

1. We detect the amount of system memory by traversing the GRUB memory
map, return it as is, do not process the result. Downside of this is,
that users who write scripts should pay attention to the possible
"cutout", to avoid using a multiple of 1MiB while doing comparisons:

# This should be avoided if we choose the first option
if [ $total_mem -lt 8192 ] ; then
	
2. Provide a more accurate way of detecting the amount of physical
memory. Downside of this is there will be more code to write, and more
lot more work to do, as the detection logic probably is platform dependent.

Thanks,
Cinhi

[1]: https://github.com/AOSC-Dev/grub/tree/grub-2.06-aosc

Re: How to detect the amount of RAM on non-EFI platforms?

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

Le jeu. 13 juil. 2023, 20:39, Daniel Kiper <dkiper@net-space.pl> a écrit :

> On Tue, Jul 11, 2023 at 09:37:15PM +0800, Cyan Yang via Grub-devel wrote:
> > Hi again,
> >
> > On 2023/7/11 10:08, Cyan via Grub-devel wrote:
> >
> > > static int
> > > traverse_mmap_hook (grub_uint64_t addr, grub_uint64_t size,
> grub_memory_type_t type,
> > >                     void *data __attribute__ ((unused)))
> > > {
> > >   total_mem_bytes += size;
> > > }
> >
> > I discovered a huge problem in my hook function: missing a return at the
> end of the hook function.
> > Without this return the hook function will execute only once.
> > And, regions with type GRUB_MEMORY_RESERVED should be skipped. So the
> hook function body should look like this:
> >
> > if (type != GRUB_MEMORY_RESERVED) {
> >   total_mem_bytes += size;
> > }
> > return GRUB_ERR_NONE;
>
> You are mixing enum and int types. The function should return 0 here.
>
That actually makes an interesting point. I think we will need an enum type
for hook returns with 2 values: GRUB_HOOK_CONTINUE and GRUB_HOOK_STOP but
obviously it's outside of scope of your patch

>
> Anyway, I think it would be nice if you could upstream this module.
> Potentially other people may find it useful.
>
> Daniel
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 2291 bytes --]

Re: How to detect the amount of RAM on non-EFI platforms?

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2526 bytes --]

Le ven. 14 juil. 2023, 05:25, Cyan via Grub-devel <grub-devel@gnu.org> a
écrit :

> Hi Daniel,
>
> On 7/14/23 02:38, Daniel Kiper wrote:
> > On Tue, Jul 11, 2023 at 09:37:15PM +0800, Cyan Yang via Grub-devel wrote:
> >
> > You are mixing enum and int types. The function should return 0 here.
> >
>
> Yes, the hook function should return zero instead of enum vaule
> GRUB_ERR_NONE.
>
>
> > Anyway, I think it would be nice if you could upstream this module.
> > Potentially other people may find it useful.
> >
>
> Thanks for the suggestion, and yes I was planning to upstream this
> module. However, a serious issue have been discovered during the initial
> test, made me concerning that this way of presenting the amount of
> physical memory should not be considered as the primary way:
>
> When running on a VM it did not report the real amount of system memory.
> A few hundred KiB of memory was left undetected, probably reserved by
> firmware or hardware.
>
> Here's an example, on a Virtualbox VM:
>
> grub> phymem
> The total system memory is 8191 MiB (8589539328 bytes).
>
> while allocated RAM for the VM is 8192MiB (8589934592 Bytes).
>
> Users usually expect the real amount, which should be a multiple of e.g.
> 2MiB or even bigger. My code[1] is currently rounding the result to the
> closest multiple of 2MiB or 128MiB, which depends on platform. The
> number is totally an assumption, and there's a chance that some systems
> can reserve more than that.
>
> So basically there are two options:
>
> 1. We detect the amount of system memory by traversing the GRUB memory
> map, return it as is, do not process the result. Downside of this is,
> that users who write scripts should pay attention to the possible
> "cutout", to avoid using a multiple of 1MiB while doing comparisons:
>
> # This should be avoided if we choose the first option
> if [ $total_mem -lt 8192 ] ; then
>
> 2. Provide a more accurate way of detecting the amount of physical
> memory. Downside of this is there will be more code to write, and more
> lot more work to do, as the detection logic probably is platform dependent.
>
Just add an option to consider "reserved" memory as available. User's
choice. Fire you it's a single if.

>
> Thanks,
> Cinhi
>
> [1]: https://github.com/AOSC-Dev/grub/tree/grub-2.06-aosc
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 3473 bytes --]

Re: How to detect the amount of RAM on non-EFI platforms?

[Cyan Yang]

Hi,

On 2023/7/14 22:04, Vladimir 'phcoder' Serbinenko wrote:
>
> Just add an option to consider "reserved" memory as available. User's
> choice. Fire you it's a single if.
>
Thanks for the tip, but I don't think it will work out - if we add up
the reserved section, the result will be more than actual amount. I ran
a test to show this.

Consider a VM with EFI firmware (x86_64-efi) with 16,384MiB of RAM, I
built and ran the module with memory map info printed out. Here's the
result (output is way too long):

grub> phymem
Memory map 0001 @ 0x0000000000000000, with size of 0x87000
Memory map 0001 @ 0x0000000000087000, with size of 0x1000
Memory map 0001 @ 0x0000000000088000, with size of 0x18000
Memory map 0001 @ 0x0000000000100000, with size of 0x706000
Memory map 0001 @ 0x0000000000806000, with size of 0x1000
Memory map 0001 @ 0x0000000000807000, with size of 0x4000
Memory map 0001 @ 0x000000000080b000, with size of 0x1000
Memory map 0001 @ 0x000000000080c000, with size of 0x14000
Memory map 0001 @ 0x0000000000820000, with size of 0xce0000
Memory map 0001 @ 0x0000000001500000, with size of 0x5b548000
Memory map 0001 @ 0x000000005ca48000, with size of 0x1f576000
Memory map 0001 @ 0x000000007bfbe000, with size of 0x20000
Memory map 0001 @ 0x000000007bfde000, with size of 0x22f3000
Memory map 0001 @ 0x000000007e2d1000, with size of 0x25000
Memory map 0001 @ 0x000000007e2f6000, with size of 0x26000
Memory map 0001 @ 0x000000007e31c000, with size of 0x10000
Memory map 0001 @ 0x000000007e32c000, with size of 0x1000
Memory map 0001 @ 0x000000007e32d000, with size of 0x1000
Memory map 0001 @ 0x000000007e32e000, with size of 0x79e000
Memory map 0001 @ 0x000000007eacc000, with size of 0xa7000
Memory map 0001 @ 0x000000007eb73000, with size of 0x26000
Memory map 0001 @ 0x000000007eb99000, with size of 0x9e000
Memory map 0001 @ 0x000000007ec37000, with size of 0x3c000
Memory map 0001 @ 0x000000007ec73000, with size of 0x5000
Memory map 0001 @ 0x000000007ec78000, with size of 0x1000
Memory map 0001 @ 0x000000007ec79000, with size of 0x20000
Memory map 0001 @ 0x000000007ec99000, with size of 0x1000
Memory map 0001 @ 0x000000007ec9a000, with size of 0x7000
Memory map 0001 @ 0x000000007eca1000, with size of 0x2000
Memory map 0001 @ 0x000000007eca3000, with size of 0xa000
Memory map 0001 @ 0x000000007ecad000, with size of 0x1000
Memory map 0001 @ 0x000000007ecae000, with size of 0x11000
Memory map 0001 @ 0x000000007ecbf000, with size of 0x2000
Memory map 0001 @ 0x000000007ecc1000, with size of 0x4000
Memory map 0001 @ 0x000000007ecc5000, with size of 0x4000
Memory map 0001 @ 0x000000007ecc9000, with size of 0x1a000
Memory map 0001 @ 0x000000007ece3000, with size of 0x5000
Memory map 0001 @ 0x000000007ece8000, with size of 0x14000
Memory map 0001 @ 0x000000007ecfc000, with size of 0xb000
Memory map 0001 @ 0x000000007ed07000, with size of 0x15000
Memory map 0001 @ 0x000000007ed1c000, with size of 0x1000
Memory map 0001 @ 0x000000007ed1d000, with size of 0x7000
Memory map 0001 @ 0x000000007ed24000, with size of 0x7000
Memory map 0001 @ 0x000000007ed2b000, with size of 0x1c000
Memory map 0001 @ 0x000000007ed47000, with size of 0x2000
Memory map 0001 @ 0x000000007ed49000, with size of 0x14000
Memory map 0001 @ 0x000000007ed5d000, with size of 0x7000
Memory map 0001 @ 0x000000007ed64000, with size of 0x6000
Memory map 0001 @ 0x000000007ed6a000, with size of 0x1000
Memory map 0001 @ 0x000000007ed6b000, with size of 0x1d000
Memory map 0001 @ 0x000000007ed88000, with size of 0x3000
Memory map 0001 @ 0x000000007ed8b000, with size of 0xe000
Memory map 0001 @ 0x000000007ed99000, with size of 0x2000
Memory map 0001 @ 0x000000007ed9b000, with size of 0x14000
Memory map 0001 @ 0x000000007edaf000, with size of 0x2000
Memory map 0001 @ 0x000000007edb1000, with size of 0xd000
Memory map 0001 @ 0x000000007edbe000, with size of 0x3000
Memory map 0001 @ 0x000000007edc1000, with size of 0x6000
Memory map 0001 @ 0x000000007edc7000, with size of 0x2000
Memory map 0001 @ 0x000000007edc9000, with size of 0x1000
Memory map 0001 @ 0x000000007edca000, with size of 0x1000
Memory map 0001 @ 0x000000007edcb000, with size of 0x5000
Memory map 0001 @ 0x000000007edd0000, with size of 0x5000
Memory map 0001 @ 0x000000007edd5000, with size of 0x12000
Memory map 0001 @ 0x000000007ede7000, with size of 0x2000
Memory map 0001 @ 0x000000007ede9000, with size of 0x6000
Memory map 0001 @ 0x000000007edef000, with size of 0x2000
Memory map 0001 @ 0x000000007edf1000, with size of 0xf000
Memory map 0001 @ 0x000000007ee00000, with size of 0x200000
Memory map 0001 @ 0x000000007f000000, with size of 0xc000
Memory map 0001 @ 0x000000007f00c000, with size of 0x5000
Memory map 0001 @ 0x000000007f011000, with size of 0x12000
Memory map 0001 @ 0x000000007f023000, with size of 0x2000
Memory map 0001 @ 0x000000007f025000, with size of 0x1000
Memory map 0001 @ 0x000000007f026000, with size of 0x1000
Memory map 0001 @ 0x000000007f027000, with size of 0x7000
Memory map 0001 @ 0x000000007f02e000, with size of 0x7000
Memory map 0001 @ 0x000000007f035000, with size of 0x4000
Memory map 0001 @ 0x000000007f039000, with size of 0x6000
Memory map 0001 @ 0x000000007f03f000, with size of 0x4000
Memory map 0001 @ 0x000000007f043000, with size of 0x2000
Memory map 0001 @ 0x000000007f045000, with size of 0x2b000
Memory map 0001 @ 0x000000007f070000, with size of 0xa000
Memory map 0001 @ 0x000000007f07a000, with size of 0xa000
Memory map 0001 @ 0x000000007f084000, with size of 0x4000
Memory map 0002 @ 0x000000007f088000, with size of 0xc1000
Memory map 0001 @ 0x000000007f149000, with size of 0x15000
Memory map 0001 @ 0x000000007f15e000, with size of 0x3000
Memory map 0001 @ 0x000000007f161000, with size of 0x3000
Memory map 0001 @ 0x000000007f164000, with size of 0x2000
Memory map 0001 @ 0x000000007f166000, with size of 0x5000
Memory map 0001 @ 0x000000007f16b000, with size of 0x1000
Memory map 0001 @ 0x000000007f16c000, with size of 0x1000
Memory map 0001 @ 0x000000007f16d000, with size of 0x1000
Memory map 0001 @ 0x000000007f16e000, with size of 0x1000
Memory map 0001 @ 0x000000007f16f000, with size of 0x1000
Memory map 0001 @ 0x000000007f170000, with size of 0x1000
Memory map 0001 @ 0x000000007f171000, with size of 0x2000
Memory map 0001 @ 0x000000007f173000, with size of 0x3000
Memory map 0001 @ 0x000000007f176000, with size of 0x2000
Memory map 0001 @ 0x000000007f178000, with size of 0x1000
Memory map 0001 @ 0x000000007f179000, with size of 0x42000
Memory map 0001 @ 0x000000007f1bb000, with size of 0xc000
Memory map 0001 @ 0x000000007f1c7000, with size of 0x1000
Memory map 0001 @ 0x000000007f1c8000, with size of 0x1000
Memory map 0001 @ 0x000000007f1c9000, with size of 0x1000
Memory map 0001 @ 0x000000007f1ca000, with size of 0x1000
Memory map 0001 @ 0x000000007f1cb000, with size of 0x2000
Memory map 0001 @ 0x000000007f1cd000, with size of 0x6000
Memory map 0001 @ 0x000000007f1d3000, with size of 0x400000
Memory map 0001 @ 0x000000007f5d3000, with size of 0x4000
Memory map 0001 @ 0x000000007f5d7000, with size of 0x1000
Memory map 0001 @ 0x000000007f5d8000, with size of 0x2000
Memory map 0001 @ 0x000000007f5da000, with size of 0x2000
Memory map 0001 @ 0x000000007f5dc000, with size of 0x1000
Memory map 0001 @ 0x000000007f5dd000, with size of 0x1000
Memory map 0001 @ 0x000000007f5de000, with size of 0x9000
Memory map 0001 @ 0x000000007f5e7000, with size of 0x3000
Memory map 0001 @ 0x000000007f5ea000, with size of 0x3000
Memory map 0001 @ 0x000000007f5ed000, with size of 0x302000
Memory map 0002 @ 0x000000007f8ef000, with size of 0x100000
Memory map 0014 @ 0x000000007f9ef000, with size of 0x100000
Memory map 0002 @ 0x000000007faef000, with size of 0x80000
Memory map 0003 @ 0x000000007fb6f000, with size of 0x10000
Memory map 0004 @ 0x000000007fb7f000, with size of 0x80000
Memory map 0001 @ 0x000000007fbff000, with size of 0x201000
Memory map 0001 @ 0x000000007fe00000, with size of 0xe9000
Memory map 0001 @ 0x000000007fee9000, with size of 0x20000
Memory map 0001 @ 0x000000007ff09000, with size of 0x1a000
Memory map 0001 @ 0x000000007ff23000, with size of 0x42000
Memory map 0001 @ 0x000000007ff65000, with size of 0x17000
Memory map 0002 @ 0x000000007ff7c000, with size of 0x84000
Memory map 0001 @ 0x0000000100000000, with size of 0x380000000
Memory map 0002 @ 0x00000000b0000000, with size of 0x10000000
Memory map 0002 @ 0x00000000ffc00000, with size of 0x400000
The total system memory is 16643 MiB (17452105728 Bytes).

As you can see, the reported amount will be more than actual amount.
This is even twice the amount if a machine has only a hundred of RAM.

BTW, I am considering to move the thread to discuss the way to precisely
detect the amount of system memory, as the original issue was resolved.

Thanks,
Cinhi Young