[SPDK] Hugepage allocation, and issue with non contiguous memory

[SPDK] Hugepage allocation, and issue with non contiguous memory

[Shahar Salzman]

[-- Attachment #1: Type: text/plain, Size: 1252 bytes --]

Hi,


On our system, we make extensive use of hugepages, so only a fraction of the hugepages are for spdk usage, and the memory allocated may be fragmented at the hugepage level.

Initially we used "--socket-mem=2048,0", but init time was very long, probably since dpdk built its hugepage info from all the hugepages on the system.


Currently I am working around the long init time by this patch to dpdk:

diff --git a/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c b/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
index 18858e2..f7e8199 100644
--- a/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
+++ b/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
@@ -97,6 +97,10 @@ get_num_hugepages(const char *subdir)
        if (num_pages > UINT32_MAX)
                num_pages = UINT32_MAX;

+#define MAX_NUM_HUGEPAGES (2048)
+        if (num_pages > MAX_NUM_HUGEPAGES)
+                num_pages = MAX_NUM_HUGEPAGES;
+
        return num_pages;
 }



For the fragmentation I am running a small program that initializes dpdk before the rest of the hugepage owners start allocating their pages.

Is there a better way to limit the # of pages that dpdk works on, and to preallocate a contiguous amount of hugepages?

Shahar

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2462 bytes --]

Re: [SPDK] Hugepage allocation, and issue with non contiguous memory

[Shahar Salzman]

[-- Attachment #1: Type: text/plain, Size: 2290 bytes --]

Hi Ben,


Thanks for the response, I'll stay tuned for updates on the dpdk side.

One small question, are there places where large memory blocks are expected to be contiguous? Before preallocating memory, we would sometimes fail to open an RDMA session since there was not enough contiguous memory, but it seems that this is not a requirement, since the large allocated block is then split into a mempool.


Shahar

________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Walker, Benjamin <benjamin.walker(a)intel.com>
Sent: Wednesday, January 31, 2018 7:00:51 PM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] Hugepage allocation, and issue with non contiguous memory

On Mon, 2018-01-29 at 12:11 +0000, Shahar Salzman wrote:
> On our system, we make extensive use of hugepages, so only a fraction of the
> hugepages are for spdk usage, and the memory allocated may be fragmented at
> the hugepage level.
> Initially we used "--socket-mem=2048,0", but init time was very long, probably
> since dpdk built its hugepage info from all the hugepages on the system.
>  For the fragmentation I am running a small program that initializes dpdk
> before the rest of the hugepage owners start allocating their pages.
>
> Is there a better way to limit the # of pages that dpdk works on, and to
> preallocate a contiguous amount of hugepages?

This is a common scenario that many users of SPDK have run into. However, SPDK
is just using DPDK's memory allocation framework, so SPDK can't solve it by
itself. We've been working with the DPDK team for nearly a year now to capture
all of the challenges that SPDK users have had and to turn them into
improvements to the core memory management system in DPDK. There is currently a
very large patch series that makes memory allocation in DPDK dynamic. See this
thread:

https://dpdk.org/ml/archives/dev/2017-December/084302.html

I don't know when that will land, but as soon as it does land SPDK will take
advantage of it. That's the "right" way to fix this, in the long term.

In the short term, what you're doing is probably the best practice.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3510 bytes --]

Re: [SPDK] Hugepage allocation, and issue with non contiguous memory

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]

On Mon, 2018-01-29 at 12:11 +0000, Shahar Salzman wrote:
> On our system, we make extensive use of hugepages, so only a fraction of the
> hugepages are for spdk usage, and the memory allocated may be fragmented at
> the hugepage level.
> Initially we used "--socket-mem=2048,0", but init time was very long, probably
> since dpdk built its hugepage info from all the hugepages on the system.
>  For the fragmentation I am running a small program that initializes dpdk
> before the rest of the hugepage owners start allocating their pages.
> 
> Is there a better way to limit the # of pages that dpdk works on, and to
> preallocate a contiguous amount of hugepages?

This is a common scenario that many users of SPDK have run into. However, SPDK
is just using DPDK's memory allocation framework, so SPDK can't solve it by
itself. We've been working with the DPDK team for nearly a year now to capture
all of the challenges that SPDK users have had and to turn them into
improvements to the core memory management system in DPDK. There is currently a
very large patch series that makes memory allocation in DPDK dynamic. See this
thread:

https://dpdk.org/ml/archives/dev/2017-December/084302.html

I don't know when that will land, but as soon as it does land SPDK will take
advantage of it. That's the "right" way to fix this, in the long term.

In the short term, what you're doing is probably the best practice.