[SPDK] Questions about hugepages requiement

[SPDK] Questions about hugepages requiement

[Oscar.Huang]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]


Hi,

The SPDK document says "Before running an SPDK application, some hugepages must be allocated...". But at least how much is sufficient for SPDK to run?

If I give a large HUGEMEM number while run scripts/setup.sh, I could see errors like this:

# ERROR: requested 1024 hugepages but only 672 could be allocated.
# Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.


So I usually use a much smaller HUGEMEM number, like 128, and my SPDK application can still run. But I see this message:

EAL: No free hugepages reported in hugepages-1048576Kb

What does this message imply? Is there any risk though my application can run?

It looks like scripts/setup.sh allocates 2-MB hugepages. Why does it complain about 1-GB hugepages?


Thanks
-Oscar

Re: [SPDK] Questions about hugepages requiement

[Howell, Seth]

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

Hi Oscar,


I don't have a specific number for you in terms of how many hugepages an arbitrary application will need. More or less will be required depending on which parts of SPDK you use and initialize, such as the NVMe-oF target or the bdev layer.

The message - EAL: No free hugepages reported in hugepages-1048576Kb - is simply informational. DPDK checks for both classes of hugepages when allocating DMA safe memory. I believe it checks 1G first, then 2MB printing out messages for each. You are only in trouble if you have no hugepages in both of those sizes.

A little background about hugepage usage in SPDK.

In older versions of SPDK and DPDK, it was required for applications to allocate all of their hugepage memory at startup. This was controlled by passing a -s parameter to an SPDK application. This option is still available if you want to pin a minimum amount of memory to your application, but it is not necessary.

Since DPDK 18.05, the environment abstraction layer has supported runtime allocations of hugepages from the system. So the application will continue to attempt to allocate more hugepages from the system until the system runs out of memory or you reach hugepage allocation limits. These dynamically allocated hugepages will be freed back to the system when they are no longer required by your SPDK application.

Thanks,

Seth


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oscar.Huang(a)microchip.com
Sent: Wednesday, April 24, 2019 5:19 PM
To: spdk(a)lists.01.org
Subject: [SPDK] Questions about hugepages requiement


Hi,

The SPDK document says "Before running an SPDK application, some hugepages must be allocated...". But at least how much is sufficient for SPDK to run?

If I give a large HUGEMEM number while run scripts/setup.sh, I could see errors like this:

# ERROR: requested 1024 hugepages but only 672 could be allocated.
# Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.


So I usually use a much smaller HUGEMEM number, like 128, and my SPDK application can still run. But I see this message:

EAL: No free hugepages reported in hugepages-1048576Kb

What does this message imply? Is there any risk though my application can run?

It looks like scripts/setup.sh allocates 2-MB hugepages. Why does it complain about 1-GB hugepages?


Thanks
-Oscar
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Questions about hugepages requiement

[E.W.Z.]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Don’t know about the minimal memory size, but here my 5 cents.

>> EAL: No free hugepages reported in hugepages-1048576Kb

As far as I remember it is just a warning, they check 1Gb pages availability, and if you opt not to use these it just gives a warning and then checks 2Mb hugepages.

>> ERROR: requested 1024 hugepages but only 672 could be allocated
To avoid this fragmentation message you can configure your boot options in GRUB to have hugepages allocated during boot

Sent from Mail for Windows 10

From: Oscar.Huang(a)microchip.com
Sent: Thursday, April 25, 2019 19:19
To: spdk(a)lists.01.org
Subject: [SPDK] Questions about hugepages requiement


Hi,

The SPDK document says "Before running an SPDK application, some hugepages must be allocated...". But at least how much is sufficient for SPDK to run?

If I give a large HUGEMEM number while run scripts/setup.sh, I could see errors like this:

# ERROR: requested 1024 hugepages but only 672 could be allocated.
# Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.


So I usually use a much smaller HUGEMEM number, like 128, and my SPDK application can still run. But I see this message:

EAL: No free hugepages reported in hugepages-1048576Kb

What does this message imply? Is there any risk though my application can run?

It looks like scripts/setup.sh allocates 2-MB hugepages. Why does it complain about 1-GB hugepages?


Thanks
-Oscar
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Questions about hugepages requiement

[Lance Hartmann ORACLE]

[-- Attachment #1: Type: text/plain, Size: 2116 bytes --]





> On Apr 16, 2019, at 7:07 PM, Oscar.Huang(a)microchip.com wrote:
> 
> Hi,
> 
> The SPDK document says "Before running an SPDK application, some hugepages must be allocated...". But at least how much is sufficient for SPDK to run?


The short answer is:  it depends.   Take for example, SPDK's nvme perf application.   When launching the application, one can specify a variable number of I/O queues, I/O size, queue depth, etc.; the larger those values, the more memory required.


> 
> If I give a large HUGEMEM number while run scripts/setup.sh, I could see errors like this:
> 
> # ERROR: requested 1024 hugepages but only 672 could be allocated.
> # Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.


Usually one plans ahead of time -- i.e. before, or just after -- boot when knowing they will be using a potentially large number of hugepages.   hugepages can be allocated via the kernel command-line or, as the SPDK's setup script does, via a write to an entry in the proc file system.   If a system has been up for a while, there may not be enough chunks of contiguous free memory to allocate significant numbers of hugepages.


> 
> 
> So I usually use a much smaller HUGEMEM number, like 128, and my SPDK application can still run. But I see this message:
> 
> EAL: No free hugepages reported in hugepages-1048576Kb
> 
> What does this message imply? Is there any risk though my application can run?
> 
> It looks like scripts/setup.sh allocates 2-MB hugepages. Why does it complain about 1-GB hugepages?


Linux running on Intel processors can support multiple page sizes concurrently.   Unless otherwise configured, Linux usually relies on the default hugepage size of 2MB for those Intel processors capable of hugepage support.   However, one can, if desired, also allocate 1GB-sized hugepages when the processor also supports that size.   The message you saw merely was reporting that no 1GB hugepages were allocated.   It wasn't really a "complaint", per se.

> 
> 
> Thanks
> -Oscar

--
Lance Hartmann

[SPDK] Questions about hugepages requiement

[Oscar.Huang]

[-- Attachment #1: Type: text/plain, Size: 812 bytes --]

Hi,

The SPDK document says "Before running an SPDK application, some hugepages must be allocated...". But at least how much is sufficient for SPDK to run?

If I give a large HUGEMEM number while run scripts/setup.sh, I could see errors like this:

# ERROR: requested 1024 hugepages but only 672 could be allocated.
# Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.


So I usually use a much smaller HUGEMEM number, like 128, and my SPDK application can still run. But I see this message:

EAL: No free hugepages reported in hugepages-1048576Kb

What does this message imply? Is there any risk though my application can run?

It looks like scripts/setup.sh allocates 2-MB hugepages. Why does it complain about 1-GB hugepages?


Thanks
-Oscar