Re: [SPDK] spdk exits with failure if hugemem is very large

Re: [SPDK] spdk exits with failure if hugemem is very large

[Lance Hartmann ORACLE]

[-- Attachment #1: Type: text/plain, Size: 2027 bytes --]


> On Oct 16, 2018, at 4:28 AM, wuzhouhui <wuzhouhui14(a)mails.ucas.ac.cn> wrote:
> 
> 
>> -----Original Messages-----
>> From: wuzhouhui <wuzhouhui14(a)mails.ucas.ac.cn>
>> Sent Time: 2018-10-16 17:18:47 (Tuesday)
>> To: "storage performance development kit" <spdk(a)lists.01.org>
>> Cc: 
>> Subject: [CASS SPAM]Re: [SPDK] [CASS SPAM]Re: [CASS SPAM]Re: spdk exits with failure if hugemem is very large
>> 
>>> -----Original Messages-----
>>> From: wuzhouhui <wuzhouhui14(a)mails.ucas.ac.cn>
>>> Sent Time: 2018-10-16 09:32:09 (Tuesday)
>>> To: "storage performance development kit" <spdk(a)lists.01.org>
>>> Cc: 
>>> Subject: [CASS SPAM]Re: [SPDK] [CASS SPAM]Re: spdk exits with failure if hugemem is very large
>>> 
>>>> -----原始邮件-----
>>>> 发件人: "Stojaczyk, Dariusz" <dariusz.stojaczyk(a)intel.com>
>>>> 发送时间: 2018-10-15 21:18:40 (星期一)
>>>> 收件人: "Storage Performance Development Kit" <spdk(a)lists.01.org>
>>>> 抄送: 
>>>> 主题: [CASS SPAM]Re: [SPDK] spdk exits with failure if hugemem is very large
>>>> 
>>>> I could guess you're hitting a max virtual memory limit. 
>> The manual of mmap() says:
>> 
>> ENOMEM  The  process's  maximum number of mappings would have been exceeded. 
>>        This error can also occur for munmap(), when unmapping a region in 
>>        the middle of an existing mapping, since this results in two smaller  
>>        mappings  on either side of the region being unmapped.
>> 
>> I also write a demo to mmap() 69632 files, each file is 8-bytes long. The demo
>> report error when mmap() 65515th file. So it is the limit of number of mappings
>> that kernel enforced, although I haven't check kernel code.
> Ok, I find it. The /proc/sys/vm/max_map_count controls this limit. The default
> value is 65535.


I had run into this problem on a system with a lot of memory as well.   Increasing vm.max_map_count to a larger number resolved the problem for us.

--
Lance Hartmann
lance.hartmann(a)oracle.com

[SPDK] spdk exits with failure if hugemem is very large

[wuzhouhui]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

The dpdk will map all hugepage and unmap unneeded even though the vhost-target
just needs 1024MB hugepage. Map all hugepage will cost a lot of time and may
cause mmap() in create_shared_memory() failed with ENOMEM. In my system, the
/proc/sys/vm/nr_hugepages is 69632, and vhost-target will always failed:

Starting SPDK v18.10-pre / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: vhost --no-shconf -c 0x1 -m 1024 --legacy-mem --file-prefix=spdk_pid162034 ]
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/spdk_pid162034/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Failed to create shared memory!
EAL: FATAL: Cannot init memory

EAL: Cannot init memory

Failed to initialize DPDK
Unable to initialize SPDK env

And I found that the option -g of vhost-target can help app bypass this issue.

Can we resolve this issue by optimizing dpdk?

Re: [SPDK] spdk exits with failure if hugemem is very large

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 1865 bytes --]

I could guess you're hitting a max virtual memory limit. 

Do you mind sharing the output of `ulimit -a`?

This is a limitation of the legacy memory management in DPDK. There's already a new, dynamic memory management in DPDK which overcomes this problem, but we're still getting spdk ready for it. Once we get all patches merged your case should start working automatically, without tweaking any ulimits.

D.

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of wuzhouhui
> Sent: Monday, October 15, 2018 12:14 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] spdk exits with failure if hugemem is very large
> 
> The dpdk will map all hugepage and unmap unneeded even though the
> vhost-target
> just needs 1024MB hugepage. Map all hugepage will cost a lot of time and
> may
> cause mmap() in create_shared_memory() failed with ENOMEM. In my
> system, the
> /proc/sys/vm/nr_hugepages is 69632, and vhost-target will always failed:
> 
> Starting SPDK v18.10-pre / DPDK 18.05.0 initialization...
> [ DPDK EAL parameters: vhost --no-shconf -c 0x1 -m 1024 --legacy-mem --
> file-prefix=spdk_pid162034 ]
> EAL: Detected 64 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/spdk_pid162034/mp_socket
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: Failed to create shared memory!
> EAL: FATAL: Cannot init memory
> 
> EAL: Cannot init memory
> 
> Failed to initialize DPDK
> Unable to initialize SPDK env
> 
> And I found that the option -g of vhost-target can help app bypass this issue.
> 
> Can we resolve this issue by optimizing dpdk?
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk