* Re: [SPDK] spdk in docker container
@ 2018-04-25 7:38 Stojaczyk, DariuszX
0 siblings, 0 replies; 10+ messages in thread
From: Stojaczyk, DariuszX @ 2018-04-25 7:38 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 9660 bytes --]
DPDK is putting a lot of effort into that currently. I'm also looking forward for their work.
Meanwhile, here's a patch that brings our physical hugepage contingency requirement to ~3MB.
[event/reactor: take into account 1 extra slot in event mempools](https://review.gerrithub.io/c/spdk/spdk/+/408925)
Since you had a couple of contiguous pages on your system, it should fix your problem. Please let me know.
I also made another patch for lifting the contingency requirement entirely, but I have some concerns about it and I'm not sure if it should be merged. Posting just FYI:
[event/reactor: take into account ring metadata size in event mempools](https://review.gerrithub.io/c/spdk/spdk/+/408926)
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
Sent: Wednesday, April 25, 2018 8:17 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] spdk in docker container
Hi,
The hugepage fragmentation issue is a real pain, even on real systems.
In our systems, there are several hugepage users so they can get fragmented. We are currently jumping through loops to allow spdk to be brought up when there are enough contiguous hugepages, so lifting this requirement would be a blessing for non docker systems as well.
Shahar
________________________________
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Sent: Wednesday, April 25, 2018 9:11:24 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] spdk in docker container
Hi,
My two cents. I've been using SPDK in docker containers for a while. Everything was working as expected, indeed I've been using 1GiB hugepages defined in kernel startup options. Works seamlessly.
On Wed, Apr 25, 2018 at 8:55 AM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi,
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
I just noticed SPDK has a requirement of at least 5MB physically contiguous memory, which you apparently don't have. I'll try to lift this requirement and send out some patches. For now you can try setting hugepages=N option in host kernel boot options, so that the memory will be reserved before anything else and will likely be physically contiguous. If the memory is still fragmented, you could try switching to 1GB hugepages.
Out of curiosity - did you try to run the same app on the host system directly? We haven't tested SPDK in containers officially. Maybe this fragmentation is caused by docker somehow.
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Wednesday, April 25, 2018 2:16 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] spdk in docker container
Thanks for the quick response. I tried both reducing memory (setting mem_size option to 16 from 256) as well as dropping caches (echo 3 > /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo large amount of memory seems to be free.
Thanks,
PR
cat /proc/meminfo
MemTotal: 263597240 kB
MemFree: 241112892 kB
MemAvailable: 241329340 kB
Buffers: 0 kB
Cached: 3244912 kB
SwapCached: 86024 kB
Active: 13700764 kB
Inactive: 3663916 kB
Active(anon): 13193844 kB
Inactive(anon): 3315936 kB
Active(file): 506920 kB
Inactive(file): 347980 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 1696376 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 14034892 kB
Mapped: 333672 kB
Shmem: 2389980 kB
Slab: 847508 kB
SReclaimable: 245704 kB
SUnreclaim: 601804 kB
KernelStack: 59520 kB
PageTables: 151484 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 134944344 kB
Committed_AS: 46069756 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 804760 kB
VmallocChunk: 34224542720 kB
HardwareCorrupted: 0 kB
AnonHugePages: 11982848 kB
HugePages_Total: 1024
HugePages_Free: 1016
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 706604 kB
DirectMap2M: 73385984 kB
DirectMap1G: 196083712 kB
On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:daniel.verkamp(a)intel.com>> wrote:
Hi,
The significant part of the first error message below is this:
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
> Please either increase it or request less amount of memory.
> EAL: FATAL: Cannot init memory
This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time. You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments. You can also try reducing the size of memory requested by your app - by default, DPDK will attempt to allocate all available hugepages.
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] spdk in docker container
Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.
Thanks,
PR
========
After running scripts/setup.sh
========
cat /proc/sys/vm/nr_hugepages
1024
cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 8429568 kB
Node 0 HugePages_Total: 640
Node 0 HugePages_Free: 512
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 3663872 kB
Node 1 HugePages_Total: 384
Node 1 HugePages_Free: 384
Node 1 HugePages_Surp: 0
=======
Run example app without mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Can only reserve 259 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
=======
with mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
RING: Cannot reserve memory
reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.
RING: Cannot reserve memory
reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed
app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option.
===========
with -no-huge option
===========
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]
EAL: Detected 72 lcore(s)
EAL: Probing VFIO support...
EAL: Started without hugepages support, physical addresses not available
Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2
invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
Bus error (core dumped)
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 53768 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-29 11:47 Shahar Salzman
0 siblings, 0 replies; 10+ messages in thread
From: Shahar Salzman @ 2018-04-29 11:47 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 14849 bytes --]
Hi guys,
Wouldn't these patches allow us to understand if this is a fragmentation issue?
I am running with the first (which is just a port of the part of the dpdk procinfo) patch to understand failures in mempool creations, but seems like the second patch can help understand if this is a fragmentation isssue.
Shahar
commit ca02d6eaf02e1c3261b94d778e6e0c44fc5c38c3
Author: shahar salzman <shahar.salzman(a)kaminario.com>
Date: Sun Apr 29 13:54:22 2018 +0300
lib/env: print memory stats on dpdk init failure
Signed-off-by: shahar salzman <shahar.salzman(a)kaminario.com>
diff --git a/lib/env_dpdk/init.c b/lib/env_dpdk/init.c
index ec16a00..9113c3f 100644
--- a/lib/env_dpdk/init.c
+++ b/lib/env_dpdk/init.c
@@ -326,6 +326,7 @@ int spdk_env_init(const struct spdk_env_opts *opts)
free(dpdk_args);
if (rc < 0) {
+ spdk_memory_info_dump(stderr);
fprintf(stderr, "Failed to initialize DPDK\n");
return -1;
}
commit 88e34b0d361f3fe1c92e80737642a56135b2b00a
Author: shahar salzman <shahar.salzman(a)kaminario.com>
Date: Tue Jan 23 10:55:40 2018 +0200
lib/env: interface to print memory statistics
Signed-off-by: shahar salzman <shahar.salzman(a)kaminario.com>
diff --git a/include/spdk/env.h b/include/spdk/env.h
index 81bfd29..7093532 100644
--- a/include/spdk/env.h
+++ b/include/spdk/env.h
@@ -213,6 +213,12 @@ int spdk_memzone_free(const char *name);
*/
void spdk_memzone_dump(FILE *f);
+/**
+ * All the memory information (malloc stats, memory segments, memory zones, tail
+ * queues).
+ */
+void spdk_memory_info_dump(FILE *f);
+
struct spdk_mempool;
#define SPDK_MEMPOOL_DEFAULT_CACHE_SIZE SIZE_MAX
diff --git a/lib/env_dpdk/env.c b/lib/env_dpdk/env.c
index da76969..467f178 100644
--- a/lib/env_dpdk/env.c
+++ b/lib/env_dpdk/env.c
@@ -159,6 +159,22 @@ spdk_memzone_dump(FILE *f)
rte_memzone_dump(f);
}
+void
+spdk_memory_info_dump(FILE *f)
+{
+ fprintf(f, "----------- MALLOC_STATS -----------\n");
+ rte_malloc_dump_stats(f, "stats");
+ fprintf(f, "--------- END_MALLOC_STATS ---------\n");
+
+ fprintf(f, "----------- MEMORY_SEGMENTS -----------\n");
+ rte_dump_physmem_layout(f);
+ fprintf(f, "--------- END_MEMORY_SEGMENTS ---------\n");
+
+ fprintf(f, "------------ MEMORY_ZONES -------------\n");
+ rte_memzone_dump(f);
+ fprintf(f, "---------- END_MEMORY_ZONES -----------\n");
+}
+
struct spdk_mempool *
spdk_mempool_create_ctor(const char *name, size_t count,
size_t ele_size, size_t cache_size, int socket_id,
________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com>
Sent: Friday, April 27, 2018 7:56:18 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] spdk in docker container
> Sorry for the delay. I tried the patch and it still doesnt work.
So it fails with the same error msg? Does the second patch I posted make any difference?
https://review.gerrithub.io/c/spdk/spdk/+/408926 (event/reactor: take into account ring metadata size in event mempools)
> I somehow dont think it is fragmentation issue (is there a way to validate this theory)?
You could try booting with 1GB hugepages.
There's also another way around to get your setup possibly working. It'd be to use '-g' command line option. It's available on latest SPDK master and will land in the upcoming 18.04 release. There's a brief description of that option here - http://www.spdk.io/doc/virtio.html#virtio_2mb
This has a strict limitation, though. You will only be required to use a single NUMA node for SPDK. If that's fine for you, then go ahead.
We will be doing a lot of memory management work for SPDK 18.07 release and I'd expect a setup like yours to be fully supported by then.
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of PR PR
Sent: Thursday, April 26, 2018 8:01 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] spdk in docker container
Sorry for the delay. I tried the patch and it still doesnt work. I ran the test on baremetal as well and it fails there as well. This machine is not heavily loaded and with 256GB of memory, I somehow dont think it is fragmentation issue (is there a way to validate this theory)? Is there some other setting that need to be fixed for hugepages? Currently ulimit -l says 64 but i have seen spdk work fine in another machine with same setting. Any pointers for things to check will be helpful.
Thanks,
PR
On Wed, Apr 25, 2018 at 12:38 AM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
DPDK is putting a lot of effort into that currently. I'm also looking forward for their work.
Meanwhile, here's a patch that brings our physical hugepage contingency requirement to ~3MB.
[event/reactor: take into account 1 extra slot in event mempools](https://review.gerrithub.io/c/spdk/spdk/+/408925)
Since you had a couple of contiguous pages on your system, it should fix your problem. Please let me know.
I also made another patch for lifting the contingency requirement entirely, but I have some concerns about it and I'm not sure if it should be merged. Posting just FYI:
[event/reactor: take into account ring metadata size in event mempools](https://review.gerrithub.io/c/spdk/spdk/+/408926)
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Shahar Salzman
Sent: Wednesday, April 25, 2018 8:17 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] spdk in docker container
Hi,
The hugepage fragmentation issue is a real pain, even on real systems.
In our systems, there are several hugepage users so they can get fragmented. We are currently jumping through loops to allow spdk to be brought up when there are enough contiguous hugepages, so lifting this requirement would be a blessing for non docker systems as well.
Shahar
________________________________
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Sent: Wednesday, April 25, 2018 9:11:24 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] spdk in docker container
Hi,
My two cents. I've been using SPDK in docker containers for a while. Everything was working as expected, indeed I've been using 1GiB hugepages defined in kernel startup options. Works seamlessly.
On Wed, Apr 25, 2018 at 8:55 AM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi,
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
I just noticed SPDK has a requirement of at least 5MB physically contiguous memory, which you apparently don't have. I'll try to lift this requirement and send out some patches. For now you can try setting hugepages=N option in host kernel boot options, so that the memory will be reserved before anything else and will likely be physically contiguous. If the memory is still fragmented, you could try switching to 1GB hugepages.
Out of curiosity - did you try to run the same app on the host system directly? We haven't tested SPDK in containers officially. Maybe this fragmentation is caused by docker somehow.
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Wednesday, April 25, 2018 2:16 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] spdk in docker container
Thanks for the quick response. I tried both reducing memory (setting mem_size option to 16 from 256) as well as dropping caches (echo 3 > /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo large amount of memory seems to be free.
Thanks,
PR
cat /proc/meminfo
MemTotal: 263597240 kB
MemFree: 241112892 kB
MemAvailable: 241329340 kB
Buffers: 0 kB
Cached: 3244912 kB
SwapCached: 86024 kB
Active: 13700764 kB
Inactive: 3663916 kB
Active(anon): 13193844 kB
Inactive(anon): 3315936 kB
Active(file): 506920 kB
Inactive(file): 347980 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 1696376 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 14034892 kB
Mapped: 333672 kB
Shmem: 2389980 kB
Slab: 847508 kB
SReclaimable: 245704 kB
SUnreclaim: 601804 kB
KernelStack: 59520 kB
PageTables: 151484 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 134944344 kB
Committed_AS: 46069756 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 804760 kB
VmallocChunk: 34224542720 kB
HardwareCorrupted: 0 kB
AnonHugePages: 11982848 kB
HugePages_Total: 1024
HugePages_Free: 1016
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 706604 kB
DirectMap2M: 73385984 kB
DirectMap1G: 196083712 kB
On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:daniel.verkamp(a)intel.com>> wrote:
Hi,
The significant part of the first error message below is this:
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
> Please either increase it or request less amount of memory.
> EAL: FATAL: Cannot init memory
This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time. You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments. You can also try reducing the size of memory requested by your app – by default, DPDK will attempt to allocate all available hugepages.
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] spdk in docker container
Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.
Thanks,
PR
========
After running scripts/setup.sh
========
cat /proc/sys/vm/nr_hugepages
1024
cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 8429568 kB
Node 0 HugePages_Total: 640
Node 0 HugePages_Free: 512
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 3663872 kB
Node 1 HugePages_Total: 384
Node 1 HugePages_Free: 384
Node 1 HugePages_Surp: 0
=======
Run example app without mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Can only reserve 259 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
=======
with mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
RING: Cannot reserve memory
reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.
RING: Cannot reserve memory
reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed
app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option.
===========
with —no-huge option
===========
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]
EAL: Detected 72 lcore(s)
EAL: Probing VFIO support...
EAL: Started without hugepages support, physical addresses not available
Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2
invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
Bus error (core dumped)
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 69297 bytes --]
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-27 16:56 Stojaczyk, DariuszX
0 siblings, 0 replies; 10+ messages in thread
From: Stojaczyk, DariuszX @ 2018-04-27 16:56 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 11754 bytes --]
> Sorry for the delay. I tried the patch and it still doesnt work.
So it fails with the same error msg? Does the second patch I posted make any difference?
https://review.gerrithub.io/c/spdk/spdk/+/408926 (event/reactor: take into account ring metadata size in event mempools)
> I somehow dont think it is fragmentation issue (is there a way to validate this theory)?
You could try booting with 1GB hugepages.
There's also another way around to get your setup possibly working. It'd be to use '-g' command line option. It's available on latest SPDK master and will land in the upcoming 18.04 release. There's a brief description of that option here - http://www.spdk.io/doc/virtio.html#virtio_2mb
This has a strict limitation, though. You will only be required to use a single NUMA node for SPDK. If that's fine for you, then go ahead.
We will be doing a lot of memory management work for SPDK 18.07 release and I'd expect a setup like yours to be fully supported by then.
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of PR PR
Sent: Thursday, April 26, 2018 8:01 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] spdk in docker container
Sorry for the delay. I tried the patch and it still doesnt work. I ran the test on baremetal as well and it fails there as well. This machine is not heavily loaded and with 256GB of memory, I somehow dont think it is fragmentation issue (is there a way to validate this theory)? Is there some other setting that need to be fixed for hugepages? Currently ulimit -l says 64 but i have seen spdk work fine in another machine with same setting. Any pointers for things to check will be helpful.
Thanks,
PR
On Wed, Apr 25, 2018 at 12:38 AM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
DPDK is putting a lot of effort into that currently. I'm also looking forward for their work.
Meanwhile, here's a patch that brings our physical hugepage contingency requirement to ~3MB.
[event/reactor: take into account 1 extra slot in event mempools](https://review.gerrithub.io/c/spdk/spdk/+/408925)
Since you had a couple of contiguous pages on your system, it should fix your problem. Please let me know.
I also made another patch for lifting the contingency requirement entirely, but I have some concerns about it and I'm not sure if it should be merged. Posting just FYI:
[event/reactor: take into account ring metadata size in event mempools](https://review.gerrithub.io/c/spdk/spdk/+/408926)
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Shahar Salzman
Sent: Wednesday, April 25, 2018 8:17 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] spdk in docker container
Hi,
The hugepage fragmentation issue is a real pain, even on real systems.
In our systems, there are several hugepage users so they can get fragmented. We are currently jumping through loops to allow spdk to be brought up when there are enough contiguous hugepages, so lifting this requirement would be a blessing for non docker systems as well.
Shahar
________________________________
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Sent: Wednesday, April 25, 2018 9:11:24 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] spdk in docker container
Hi,
My two cents. I've been using SPDK in docker containers for a while. Everything was working as expected, indeed I've been using 1GiB hugepages defined in kernel startup options. Works seamlessly.
On Wed, Apr 25, 2018 at 8:55 AM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi,
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
I just noticed SPDK has a requirement of at least 5MB physically contiguous memory, which you apparently don't have. I'll try to lift this requirement and send out some patches. For now you can try setting hugepages=N option in host kernel boot options, so that the memory will be reserved before anything else and will likely be physically contiguous. If the memory is still fragmented, you could try switching to 1GB hugepages.
Out of curiosity - did you try to run the same app on the host system directly? We haven't tested SPDK in containers officially. Maybe this fragmentation is caused by docker somehow.
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Wednesday, April 25, 2018 2:16 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] spdk in docker container
Thanks for the quick response. I tried both reducing memory (setting mem_size option to 16 from 256) as well as dropping caches (echo 3 > /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo large amount of memory seems to be free.
Thanks,
PR
cat /proc/meminfo
MemTotal: 263597240 kB
MemFree: 241112892 kB
MemAvailable: 241329340 kB
Buffers: 0 kB
Cached: 3244912 kB
SwapCached: 86024 kB
Active: 13700764 kB
Inactive: 3663916 kB
Active(anon): 13193844 kB
Inactive(anon): 3315936 kB
Active(file): 506920 kB
Inactive(file): 347980 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 1696376 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 14034892 kB
Mapped: 333672 kB
Shmem: 2389980 kB
Slab: 847508 kB
SReclaimable: 245704 kB
SUnreclaim: 601804 kB
KernelStack: 59520 kB
PageTables: 151484 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 134944344 kB
Committed_AS: 46069756 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 804760 kB
VmallocChunk: 34224542720 kB
HardwareCorrupted: 0 kB
AnonHugePages: 11982848 kB
HugePages_Total: 1024
HugePages_Free: 1016
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 706604 kB
DirectMap2M: 73385984 kB
DirectMap1G: 196083712 kB
On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:daniel.verkamp(a)intel.com>> wrote:
Hi,
The significant part of the first error message below is this:
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
> Please either increase it or request less amount of memory.
> EAL: FATAL: Cannot init memory
This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time. You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments. You can also try reducing the size of memory requested by your app – by default, DPDK will attempt to allocate all available hugepages.
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] spdk in docker container
Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.
Thanks,
PR
========
After running scripts/setup.sh
========
cat /proc/sys/vm/nr_hugepages
1024
cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 8429568 kB
Node 0 HugePages_Total: 640
Node 0 HugePages_Free: 512
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 3663872 kB
Node 1 HugePages_Total: 384
Node 1 HugePages_Free: 384
Node 1 HugePages_Surp: 0
=======
Run example app without mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Can only reserve 259 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
=======
with mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
RING: Cannot reserve memory
reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.
RING: Cannot reserve memory
reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed
app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option.
===========
with —no-huge option
===========
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]
EAL: Detected 72 lcore(s)
EAL: Probing VFIO support...
EAL: Started without hugepages support, physical addresses not available
Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2
invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
Bus error (core dumped)
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 68333 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-26 18:00 PR PR
0 siblings, 0 replies; 10+ messages in thread
From: PR PR @ 2018-04-26 18:00 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 10905 bytes --]
Sorry for the delay. I tried the patch and it still doesnt work. I ran the
test on baremetal as well and it fails there as well. This machine is not
heavily loaded and with 256GB of memory, I somehow dont think it is
fragmentation issue (is there a way to validate this theory)? Is there some
other setting that need to be fixed for hugepages? Currently ulimit -l says
64 but i have seen spdk work fine in another machine with same setting. Any
pointers for things to check will be helpful.
Thanks,
PR
On Wed, Apr 25, 2018 at 12:38 AM, Stojaczyk, DariuszX <
dariuszx.stojaczyk(a)intel.com> wrote:
> DPDK is putting a lot of effort into that currently. I'm also looking
> forward for their work.
>
>
>
> Meanwhile, here's a patch that brings our physical hugepage contingency
> requirement to ~3MB.
>
>
>
> [event/reactor: take into account 1 extra slot in event mempools](
> https://review.gerrithub.io/c/spdk/spdk/+/408925)
>
>
>
> Since you had a couple of contiguous pages on your system, it should fix
> your problem. Please let me know.
>
>
>
> I also made another patch for lifting the contingency requirement
> entirely, but I have some concerns about it and I'm not sure if it should
> be merged. Posting just FYI:
>
>
>
> [event/reactor: take into account ring metadata size in event mempools](
> https://review.gerrithub.io/c/spdk/spdk/+/408926)
>
>
>
> D.
>
>
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Shahar
> Salzman
> *Sent:* Wednesday, April 25, 2018 8:17 AM
>
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] spdk in docker container
>
>
>
> Hi,
>
>
>
> The hugepage fragmentation issue is a real pain, even on real systems.
>
> In our systems, there are several hugepage users so they can get
> fragmented. We are currently jumping through loops to allow spdk to be
> brought up when there are enough contiguous hugepages, so lifting this
> requirement would be a blessing for non docker systems as well.
>
>
>
> Shahar
> ------------------------------
>
> *From:* SPDK <spdk-bounces(a)lists.01.org> on behalf of Ernest Zed <
> kreuzerkrieg(a)gmail.com>
> *Sent:* Wednesday, April 25, 2018 9:11:24 AM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] spdk in docker container
>
>
>
> Hi,
>
> My two cents. I've been using SPDK in docker containers for a while.
> Everything was working as expected, indeed I've been using 1GiB hugepages
> defined in kernel startup options. Works seamlessly.
>
>
>
> On Wed, Apr 25, 2018 at 8:55 AM, Stojaczyk, DariuszX <
> dariuszx.stojaczyk(a)intel.com> wrote:
>
> Hi,
>
>
>
> > EAL: Can only reserve 259 pages from 1024 requested
>
> > Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
>
>
> I just noticed SPDK has a requirement of at least 5MB physically
> contiguous memory, which you apparently don't have. I'll try to lift this
> requirement and send out some patches. For now you can try setting
> hugepages=N option in host kernel boot options, so that the memory will be
> reserved before anything else and will likely be physically contiguous. If
> the memory is still fragmented, you could try switching to 1GB hugepages.
>
>
>
> Out of curiosity - did you try to run the same app on the host system
> directly? We haven't tested SPDK in containers officially. Maybe this
> fragmentation is caused by docker somehow.
>
>
>
> D.
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *PR PR
> *Sent:* Wednesday, April 25, 2018 2:16 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] spdk in docker container
>
>
>
> Thanks for the quick response. I tried both reducing memory (setting
> mem_size option to 16 from 256) as well as dropping caches (echo 3 >
> /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo
> large amount of memory seems to be free.
>
>
>
> Thanks,
>
> PR
>
>
>
> cat /proc/meminfo
>
> MemTotal: 263597240 kB
>
> MemFree: 241112892 kB
>
> MemAvailable: 241329340 kB
>
> Buffers: 0 kB
>
> Cached: 3244912 kB
>
> SwapCached: 86024 kB
>
> Active: 13700764 kB
>
> Inactive: 3663916 kB
>
> Active(anon): 13193844 kB
>
> Inactive(anon): 3315936 kB
>
> Active(file): 506920 kB
>
> Inactive(file): 347980 kB
>
> Unevictable: 0 kB
>
> Mlocked: 0 kB
>
> SwapTotal: 4194300 kB
>
> SwapFree: 1696376 kB
>
> Dirty: 92 kB
>
> Writeback: 0 kB
>
> AnonPages: 14034892 kB
>
> Mapped: 333672 kB
>
> Shmem: 2389980 kB
>
> Slab: 847508 kB
>
> SReclaimable: 245704 kB
>
> SUnreclaim: 601804 kB
>
> KernelStack: 59520 kB
>
> PageTables: 151484 kB
>
> NFS_Unstable: 0 kB
>
> Bounce: 0 kB
>
> WritebackTmp: 0 kB
>
> CommitLimit: 134944344 kB
>
> Committed_AS: 46069756 kB
>
> VmallocTotal: 34359738367 kB
>
> VmallocUsed: 804760 kB
>
> VmallocChunk: 34224542720 kB
>
> HardwareCorrupted: 0 kB
>
> AnonHugePages: 11982848 kB
>
> HugePages_Total: 1024
>
> HugePages_Free: 1016
>
> HugePages_Rsvd: 0
>
> HugePages_Surp: 0
>
> Hugepagesize: 2048 kB
>
> DirectMap4k: 706604 kB
>
> DirectMap2M: 73385984 kB
>
> DirectMap1G: 196083712 kB
>
>
>
> On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com>
> wrote:
>
> Hi,
>
>
>
> The significant part of the first error message below is this:
>
>
>
> > EAL: Can only reserve 259 pages from 1024 requested
>
> > Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
> > Please either increase it or request less amount of memory.
>
> > EAL: FATAL: Cannot init memory
>
>
>
> This is saying that DPDK was unable to allocate hugepages to meet your
> requested memory size in the maximum number of segments (256) configured at
> DPDK compile time. You can try rebooting or dropping caches (echo 3 >
> /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK
> will be able to find larger contiguous memory regions and thus require
> fewer total memory segments. You can also try reducing the size of memory
> requested by your app – by default, DPDK will attempt to allocate all
> available hugepages.
>
>
>
> Thanks,
>
> -- Daniel
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *PR PR
> *Sent:* Tuesday, April 24, 2018 4:51 PM
> *To:* spdk(a)lists.01.org
> *Subject:* [SPDK] spdk in docker container
>
>
>
> Hi, I am trying to run one of the spdk example (reactor_perf) in docker
> container. Our build system uses bazel so after compiling spdk natively, I
> have created a bazel target for linking with spdk (I have set the
> alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The
> container is run with privileged option on docker cmdline and has /dev bind
> mounted from host. Before running example app, I do run the
> scripts/setup.sh which succeeds. The example app (reactor_perf) is copied
> to new workspace location and compiled as a bazel target. For some reason,
> it fails to allocate huge pages. Some relevant outputs below. Appreciate
> any pointers.
>
>
>
> Thanks,
>
> PR
>
>
>
> ========
>
> After running scripts/setup.sh
>
> ========
>
>
>
> cat /proc/sys/vm/nr_hugepages
>
> 1024
>
> cat /sys/devices/system/node/node*/meminfo | fgrep Huge
>
> Node 0 AnonHugePages: 8429568 kB
>
> Node 0 HugePages_Total: 640
>
> Node 0 HugePages_Free: 512
>
> Node 0 HugePages_Surp: 0
>
> Node 1 AnonHugePages: 3663872 kB
>
> Node 1 HugePages_Total: 384
>
> Node 1 HugePages_Free: 384
>
> Node 1 HugePages_Surp: 0
>
>
>
> =======
>
> Run example app without mem_size option
>
> =======
>
> /ws/bazel-bin/spdk/spdk_test -t 10
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci
> --file-prefix=spdk_pid28875 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: Can only reserve 259 pages from 1024 requested
>
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
> Please either increase it or request less amount of memory.
>
> EAL: FATAL: Cannot init memory
>
>
>
> EAL: Cannot init memory
>
>
>
> Failed to initialize DPDK
>
> app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
>
>
>
> =======
>
> with mem_size option
>
> =======
>
> /ws/bazel-bin/spdk/spdk_test -t 10
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci
> --file-prefix=spdk_pid29246 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
>
> reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is
> 0x1
>
> RING: Cannot reserve memory
>
> reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed
> on preferred socket 0.
>
> RING: Cannot reserve memory
>
> reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation
> failed
>
> app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
>
>
>
>
>
> ps: I tried disabling huge pages but spdk seems to be crashing. Looks like
> this is not an option.
>
> ===========
>
> with —no-huge option
>
> ===========
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge
> --file-prefix=spdk_pid23800 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: Probing VFIO support...
>
> EAL: Started without hugepages support, physical addresses not available
>
> Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info.
> Error code: 2
>
> invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
>
> app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
>
> reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is
> 0x1
>
> Bus error (core dumped)
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 56987 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-25 6:17 Shahar Salzman
0 siblings, 0 replies; 10+ messages in thread
From: Shahar Salzman @ 2018-04-25 6:17 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 8635 bytes --]
Hi,
The hugepage fragmentation issue is a real pain, even on real systems.
In our systems, there are several hugepage users so they can get fragmented. We are currently jumping through loops to allow spdk to be brought up when there are enough contiguous hugepages, so lifting this requirement would be a blessing for non docker systems as well.
Shahar
________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com>
Sent: Wednesday, April 25, 2018 9:11:24 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] spdk in docker container
Hi,
My two cents. I've been using SPDK in docker containers for a while. Everything was working as expected, indeed I've been using 1GiB hugepages defined in kernel startup options. Works seamlessly.
On Wed, Apr 25, 2018 at 8:55 AM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi,
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
I just noticed SPDK has a requirement of at least 5MB physically contiguous memory, which you apparently don't have. I'll try to lift this requirement and send out some patches. For now you can try setting hugepages=N option in host kernel boot options, so that the memory will be reserved before anything else and will likely be physically contiguous. If the memory is still fragmented, you could try switching to 1GB hugepages.
Out of curiosity - did you try to run the same app on the host system directly? We haven't tested SPDK in containers officially. Maybe this fragmentation is caused by docker somehow.
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Wednesday, April 25, 2018 2:16 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] spdk in docker container
Thanks for the quick response. I tried both reducing memory (setting mem_size option to 16 from 256) as well as dropping caches (echo 3 > /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo large amount of memory seems to be free.
Thanks,
PR
cat /proc/meminfo
MemTotal: 263597240 kB
MemFree: 241112892 kB
MemAvailable: 241329340 kB
Buffers: 0 kB
Cached: 3244912 kB
SwapCached: 86024 kB
Active: 13700764 kB
Inactive: 3663916 kB
Active(anon): 13193844 kB
Inactive(anon): 3315936 kB
Active(file): 506920 kB
Inactive(file): 347980 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 1696376 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 14034892 kB
Mapped: 333672 kB
Shmem: 2389980 kB
Slab: 847508 kB
SReclaimable: 245704 kB
SUnreclaim: 601804 kB
KernelStack: 59520 kB
PageTables: 151484 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 134944344 kB
Committed_AS: 46069756 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 804760 kB
VmallocChunk: 34224542720 kB
HardwareCorrupted: 0 kB
AnonHugePages: 11982848 kB
HugePages_Total: 1024
HugePages_Free: 1016
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 706604 kB
DirectMap2M: 73385984 kB
DirectMap1G: 196083712 kB
On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:daniel.verkamp(a)intel.com>> wrote:
Hi,
The significant part of the first error message below is this:
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
> Please either increase it or request less amount of memory.
> EAL: FATAL: Cannot init memory
This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time. You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments. You can also try reducing the size of memory requested by your app – by default, DPDK will attempt to allocate all available hugepages.
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] spdk in docker container
Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.
Thanks,
PR
========
After running scripts/setup.sh
========
cat /proc/sys/vm/nr_hugepages
1024
cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 8429568 kB
Node 0 HugePages_Total: 640
Node 0 HugePages_Free: 512
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 3663872 kB
Node 1 HugePages_Total: 384
Node 1 HugePages_Free: 384
Node 1 HugePages_Surp: 0
=======
Run example app without mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Can only reserve 259 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
=======
with mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
RING: Cannot reserve memory
reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.
RING: Cannot reserve memory
reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed
app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option.
===========
with —no-huge option
===========
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]
EAL: Detected 72 lcore(s)
EAL: Probing VFIO support...
EAL: Started without hugepages support, physical addresses not available
Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2
invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
Bus error (core dumped)
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 48352 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-25 6:11 Ernest Zed
0 siblings, 0 replies; 10+ messages in thread
From: Ernest Zed @ 2018-04-25 6:11 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 8353 bytes --]
Hi,
My two cents. I've been using SPDK in docker containers for a while.
Everything was working as expected, indeed I've been using 1GiB hugepages
defined in kernel startup options. Works seamlessly.
On Wed, Apr 25, 2018 at 8:55 AM, Stojaczyk, DariuszX <
dariuszx.stojaczyk(a)intel.com> wrote:
> Hi,
>
>
>
> > EAL: Can only reserve 259 pages from 1024 requested
>
> > Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
>
>
> I just noticed SPDK has a requirement of at least 5MB physically
> contiguous memory, which you apparently don't have. I'll try to lift this
> requirement and send out some patches. For now you can try setting
> hugepages=N option in host kernel boot options, so that the memory will be
> reserved before anything else and will likely be physically contiguous. If
> the memory is still fragmented, you could try switching to 1GB hugepages.
>
>
>
> Out of curiosity - did you try to run the same app on the host system
> directly? We haven't tested SPDK in containers officially. Maybe this
> fragmentation is caused by docker somehow.
>
>
>
> D.
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *PR PR
> *Sent:* Wednesday, April 25, 2018 2:16 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] spdk in docker container
>
>
>
> Thanks for the quick response. I tried both reducing memory (setting
> mem_size option to 16 from 256) as well as dropping caches (echo 3 >
> /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo
> large amount of memory seems to be free.
>
>
>
> Thanks,
>
> PR
>
>
>
> cat /proc/meminfo
>
> MemTotal: 263597240 kB
>
> MemFree: 241112892 kB
>
> MemAvailable: 241329340 kB
>
> Buffers: 0 kB
>
> Cached: 3244912 kB
>
> SwapCached: 86024 kB
>
> Active: 13700764 kB
>
> Inactive: 3663916 kB
>
> Active(anon): 13193844 kB
>
> Inactive(anon): 3315936 kB
>
> Active(file): 506920 kB
>
> Inactive(file): 347980 kB
>
> Unevictable: 0 kB
>
> Mlocked: 0 kB
>
> SwapTotal: 4194300 kB
>
> SwapFree: 1696376 kB
>
> Dirty: 92 kB
>
> Writeback: 0 kB
>
> AnonPages: 14034892 kB
>
> Mapped: 333672 kB
>
> Shmem: 2389980 kB
>
> Slab: 847508 kB
>
> SReclaimable: 245704 kB
>
> SUnreclaim: 601804 kB
>
> KernelStack: 59520 kB
>
> PageTables: 151484 kB
>
> NFS_Unstable: 0 kB
>
> Bounce: 0 kB
>
> WritebackTmp: 0 kB
>
> CommitLimit: 134944344 kB
>
> Committed_AS: 46069756 kB
>
> VmallocTotal: 34359738367 kB
>
> VmallocUsed: 804760 kB
>
> VmallocChunk: 34224542720 kB
>
> HardwareCorrupted: 0 kB
>
> AnonHugePages: 11982848 kB
>
> HugePages_Total: 1024
>
> HugePages_Free: 1016
>
> HugePages_Rsvd: 0
>
> HugePages_Surp: 0
>
> Hugepagesize: 2048 kB
>
> DirectMap4k: 706604 kB
>
> DirectMap2M: 73385984 kB
>
> DirectMap1G: 196083712 kB
>
>
>
> On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com>
> wrote:
>
> Hi,
>
>
>
> The significant part of the first error message below is this:
>
>
>
> > EAL: Can only reserve 259 pages from 1024 requested
>
> > Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
> > Please either increase it or request less amount of memory.
>
> > EAL: FATAL: Cannot init memory
>
>
>
> This is saying that DPDK was unable to allocate hugepages to meet your
> requested memory size in the maximum number of segments (256) configured at
> DPDK compile time. You can try rebooting or dropping caches (echo 3 >
> /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK
> will be able to find larger contiguous memory regions and thus require
> fewer total memory segments. You can also try reducing the size of memory
> requested by your app – by default, DPDK will attempt to allocate all
> available hugepages.
>
>
>
> Thanks,
>
> -- Daniel
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *PR PR
> *Sent:* Tuesday, April 24, 2018 4:51 PM
> *To:* spdk(a)lists.01.org
> *Subject:* [SPDK] spdk in docker container
>
>
>
> Hi, I am trying to run one of the spdk example (reactor_perf) in docker
> container. Our build system uses bazel so after compiling spdk natively, I
> have created a bazel target for linking with spdk (I have set the
> alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The
> container is run with privileged option on docker cmdline and has /dev bind
> mounted from host. Before running example app, I do run the
> scripts/setup.sh which succeeds. The example app (reactor_perf) is copied
> to new workspace location and compiled as a bazel target. For some reason,
> it fails to allocate huge pages. Some relevant outputs below. Appreciate
> any pointers.
>
>
>
> Thanks,
>
> PR
>
>
>
> ========
>
> After running scripts/setup.sh
>
> ========
>
>
>
> cat /proc/sys/vm/nr_hugepages
>
> 1024
>
> cat /sys/devices/system/node/node*/meminfo | fgrep Huge
>
> Node 0 AnonHugePages: 8429568 kB
>
> Node 0 HugePages_Total: 640
>
> Node 0 HugePages_Free: 512
>
> Node 0 HugePages_Surp: 0
>
> Node 1 AnonHugePages: 3663872 kB
>
> Node 1 HugePages_Total: 384
>
> Node 1 HugePages_Free: 384
>
> Node 1 HugePages_Surp: 0
>
>
>
> =======
>
> Run example app without mem_size option
>
> =======
>
> /ws/bazel-bin/spdk/spdk_test -t 10
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci
> --file-prefix=spdk_pid28875 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: Can only reserve 259 pages from 1024 requested
>
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
> Please either increase it or request less amount of memory.
>
> EAL: FATAL: Cannot init memory
>
>
>
> EAL: Cannot init memory
>
>
>
> Failed to initialize DPDK
>
> app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
>
>
>
> =======
>
> with mem_size option
>
> =======
>
> /ws/bazel-bin/spdk/spdk_test -t 10
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci
> --file-prefix=spdk_pid29246 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
>
> reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is
> 0x1
>
> RING: Cannot reserve memory
>
> reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed
> on preferred socket 0.
>
> RING: Cannot reserve memory
>
> reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation
> failed
>
> app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
>
>
>
>
>
> ps: I tried disabling huge pages but spdk seems to be crashing. Looks like
> this is not an option.
>
> ===========
>
> with —no-huge option
>
> ===========
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge
> --file-prefix=spdk_pid23800 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: Probing VFIO support...
>
> EAL: Started without hugepages support, physical addresses not available
>
> Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info.
> Error code: 2
>
> invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
>
> app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
>
> reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is
> 0x1
>
> Bus error (core dumped)
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 46732 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-25 5:55 Stojaczyk, DariuszX
0 siblings, 0 replies; 10+ messages in thread
From: Stojaczyk, DariuszX @ 2018-04-25 5:55 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 7266 bytes --]
Hi,
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
I just noticed SPDK has a requirement of at least 5MB physically contiguous memory, which you apparently don't have. I'll try to lift this requirement and send out some patches. For now you can try setting hugepages=N option in host kernel boot options, so that the memory will be reserved before anything else and will likely be physically contiguous. If the memory is still fragmented, you could try switching to 1GB hugepages.
Out of curiosity - did you try to run the same app on the host system directly? We haven't tested SPDK in containers officially. Maybe this fragmentation is caused by docker somehow.
D.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of PR PR
Sent: Wednesday, April 25, 2018 2:16 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] spdk in docker container
Thanks for the quick response. I tried both reducing memory (setting mem_size option to 16 from 256) as well as dropping caches (echo 3 > /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo large amount of memory seems to be free.
Thanks,
PR
cat /proc/meminfo
MemTotal: 263597240 kB
MemFree: 241112892 kB
MemAvailable: 241329340 kB
Buffers: 0 kB
Cached: 3244912 kB
SwapCached: 86024 kB
Active: 13700764 kB
Inactive: 3663916 kB
Active(anon): 13193844 kB
Inactive(anon): 3315936 kB
Active(file): 506920 kB
Inactive(file): 347980 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 1696376 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 14034892 kB
Mapped: 333672 kB
Shmem: 2389980 kB
Slab: 847508 kB
SReclaimable: 245704 kB
SUnreclaim: 601804 kB
KernelStack: 59520 kB
PageTables: 151484 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 134944344 kB
Committed_AS: 46069756 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 804760 kB
VmallocChunk: 34224542720 kB
HardwareCorrupted: 0 kB
AnonHugePages: 11982848 kB
HugePages_Total: 1024
HugePages_Free: 1016
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 706604 kB
DirectMap2M: 73385984 kB
DirectMap1G: 196083712 kB
On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:daniel.verkamp(a)intel.com>> wrote:
Hi,
The significant part of the first error message below is this:
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
> Please either increase it or request less amount of memory.
> EAL: FATAL: Cannot init memory
This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time. You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments. You can also try reducing the size of memory requested by your app – by default, DPDK will attempt to allocate all available hugepages.
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] spdk in docker container
Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.
Thanks,
PR
========
After running scripts/setup.sh
========
cat /proc/sys/vm/nr_hugepages
1024
cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 8429568 kB
Node 0 HugePages_Total: 640
Node 0 HugePages_Free: 512
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 3663872 kB
Node 1 HugePages_Total: 384
Node 1 HugePages_Free: 384
Node 1 HugePages_Surp: 0
=======
Run example app without mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Can only reserve 259 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
=======
with mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
RING: Cannot reserve memory
reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.
RING: Cannot reserve memory
reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed
app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option.
===========
with —no-huge option
===========
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]
EAL: Detected 72 lcore(s)
EAL: Probing VFIO support...
EAL: Started without hugepages support, physical addresses not available
Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2
invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
Bus error (core dumped)
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 47149 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-25 0:16 PR PR
0 siblings, 0 replies; 10+ messages in thread
From: PR PR @ 2018-04-25 0:16 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6577 bytes --]
Thanks for the quick response. I tried both reducing memory (setting
mem_size option to 16 from 256) as well as dropping caches (echo 3 >
/proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo
large amount of memory seems to be free.
Thanks,
PR
cat /proc/meminfo
MemTotal: 263597240 kB
MemFree: 241112892 kB
MemAvailable: 241329340 kB
Buffers: 0 kB
Cached: 3244912 kB
SwapCached: 86024 kB
Active: 13700764 kB
Inactive: 3663916 kB
Active(anon): 13193844 kB
Inactive(anon): 3315936 kB
Active(file): 506920 kB
Inactive(file): 347980 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 1696376 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 14034892 kB
Mapped: 333672 kB
Shmem: 2389980 kB
Slab: 847508 kB
SReclaimable: 245704 kB
SUnreclaim: 601804 kB
KernelStack: 59520 kB
PageTables: 151484 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 134944344 kB
Committed_AS: 46069756 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 804760 kB
VmallocChunk: 34224542720 kB
HardwareCorrupted: 0 kB
AnonHugePages: 11982848 kB
HugePages_Total: 1024
HugePages_Free: 1016
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 706604 kB
DirectMap2M: 73385984 kB
DirectMap1G: 196083712 kB
On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com>
wrote:
> Hi,
>
>
>
> The significant part of the first error message below is this:
>
>
>
> > EAL: Can only reserve 259 pages from 1024 requested
>
> > Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
> > Please either increase it or request less amount of memory.
>
> > EAL: FATAL: Cannot init memory
>
>
>
> This is saying that DPDK was unable to allocate hugepages to meet your
> requested memory size in the maximum number of segments (256) configured at
> DPDK compile time. You can try rebooting or dropping caches (echo 3 >
> /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK
> will be able to find larger contiguous memory regions and thus require
> fewer total memory segments. You can also try reducing the size of memory
> requested by your app – by default, DPDK will attempt to allocate all
> available hugepages.
>
>
>
> Thanks,
>
> -- Daniel
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *PR PR
> *Sent:* Tuesday, April 24, 2018 4:51 PM
> *To:* spdk(a)lists.01.org
> *Subject:* [SPDK] spdk in docker container
>
>
>
> Hi, I am trying to run one of the spdk example (reactor_perf) in docker
> container. Our build system uses bazel so after compiling spdk natively, I
> have created a bazel target for linking with spdk (I have set the
> alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The
> container is run with privileged option on docker cmdline and has /dev bind
> mounted from host. Before running example app, I do run the
> scripts/setup.sh which succeeds. The example app (reactor_perf) is copied
> to new workspace location and compiled as a bazel target. For some reason,
> it fails to allocate huge pages. Some relevant outputs below. Appreciate
> any pointers.
>
>
>
> Thanks,
>
> PR
>
>
>
> ========
>
> After running scripts/setup.sh
>
> ========
>
>
>
> cat /proc/sys/vm/nr_hugepages
>
> 1024
>
> cat /sys/devices/system/node/node*/meminfo | fgrep Huge
>
> Node 0 AnonHugePages: 8429568 kB
>
> Node 0 HugePages_Total: 640
>
> Node 0 HugePages_Free: 512
>
> Node 0 HugePages_Surp: 0
>
> Node 1 AnonHugePages: 3663872 kB
>
> Node 1 HugePages_Total: 384
>
> Node 1 HugePages_Free: 384
>
> Node 1 HugePages_Surp: 0
>
>
>
> =======
>
> Run example app without mem_size option
>
> =======
>
> /ws/bazel-bin/spdk/spdk_test -t 10
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci
> --file-prefix=spdk_pid28875 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> EAL: Can only reserve 259 pages from 1024 requested
>
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
>
> Please either increase it or request less amount of memory.
>
> EAL: FATAL: Cannot init memory
>
>
>
> EAL: Cannot init memory
>
>
>
> Failed to initialize DPDK
>
> app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
>
>
>
> =======
>
> with mem_size option
>
> =======
>
> /ws/bazel-bin/spdk/spdk_test -t 10
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci
> --file-prefix=spdk_pid29246 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
>
> reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is
> 0x1
>
> RING: Cannot reserve memory
>
> reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed
> on preferred socket 0.
>
> RING: Cannot reserve memory
>
> reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation
> failed
>
> app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
>
>
>
>
>
> ps: I tried disabling huge pages but spdk seems to be crashing. Looks like
> this is not an option.
>
> ===========
>
> with —no-huge option
>
> ===========
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge
> --file-prefix=spdk_pid23800 ]
>
> EAL: Detected 72 lcore(s)
>
> EAL: Probing VFIO support...
>
> EAL: Started without hugepages support, physical addresses not available
>
> Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info.
> Error code: 2
>
> invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
>
> app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
>
> reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is
> 0x1
>
> Bus error (core dumped)
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 31865 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [SPDK] spdk in docker container
@ 2018-04-24 23:59 Verkamp, Daniel
0 siblings, 0 replies; 10+ messages in thread
From: Verkamp, Daniel @ 2018-04-24 23:59 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4387 bytes --]
Hi,
The significant part of the first error message below is this:
> EAL: Can only reserve 259 pages from 1024 requested
> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
> Please either increase it or request less amount of memory.
> EAL: FATAL: Cannot init memory
This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time. You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments. You can also try reducing the size of memory requested by your app – by default, DPDK will attempt to allocate all available hugepages.
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To: spdk(a)lists.01.org
Subject: [SPDK] spdk in docker container
Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.
Thanks,
PR
========
After running scripts/setup.sh
========
cat /proc/sys/vm/nr_hugepages
1024
cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 8429568 kB
Node 0 HugePages_Total: 640
Node 0 HugePages_Free: 512
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 3663872 kB
Node 1 HugePages_Total: 384
Node 1 HugePages_Free: 384
Node 1 HugePages_Surp: 0
=======
Run example app without mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Can only reserve 259 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
=======
with mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
RING: Cannot reserve memory
reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.
RING: Cannot reserve memory
reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed
app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option.
===========
with —no-huge option
===========
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]
EAL: Detected 72 lcore(s)
EAL: Probing VFIO support...
EAL: Started without hugepages support, physical addresses not available
Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2
invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
Bus error (core dumped)
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 28505 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* [SPDK] spdk in docker container
@ 2018-04-24 23:51 PR PR
0 siblings, 0 replies; 10+ messages in thread
From: PR PR @ 2018-04-24 23:51 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3367 bytes --]
Hi, I am trying to run one of the spdk example (reactor_perf) in docker
container. Our build system uses bazel so after compiling spdk natively, I
have created a bazel target for linking with spdk (I have set the
alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The
container is run with privileged option on docker cmdline and has /dev bind
mounted from host. Before running example app, I do run the
scripts/setup.sh which succeeds. The example app (reactor_perf) is copied
to new workspace location and compiled as a bazel target. For some reason,
it fails to allocate huge pages. Some relevant outputs below. Appreciate
any pointers.
Thanks,
PR
========
After running scripts/setup.sh
========
cat /proc/sys/vm/nr_hugepages
1024
cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 8429568 kB
Node 0 HugePages_Total: 640
Node 0 HugePages_Free: 512
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 3663872 kB
Node 1 HugePages_Total: 384
Node 1 HugePages_Free: 384
Node 1 HugePages_Surp: 0
=======
Run example app without mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci
--file-prefix=spdk_pid28875 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Can only reserve 259 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env
=======
with mem_size option
=======
/ws/bazel-bin/spdk/spdk_test -t 10
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci
--file-prefix=spdk_pid29246 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
RING: Cannot reserve memory
reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed
on preferred socket 0.
RING: Cannot reserve memory
reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation
failed
app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.
ps: I tried disabling huge pages but spdk seems to be crashing. Looks like
this is not an option.
===========
with —no-huge option
===========
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge
--file-prefix=spdk_pid23800 ]
EAL: Detected 72 lcore(s)
EAL: Probing VFIO support...
EAL: Started without hugepages support, physical addresses not available
Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info.
Error code: 2
invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
Bus error (core dumped)
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 19522 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-04-29 11:47 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-25 7:38 [SPDK] spdk in docker container Stojaczyk, DariuszX
-- strict thread matches above, loose matches on Subject: below --
2018-04-29 11:47 Shahar Salzman
2018-04-27 16:56 Stojaczyk, DariuszX
2018-04-26 18:00 PR PR
2018-04-25 6:17 Shahar Salzman
2018-04-25 6:11 Ernest Zed
2018-04-25 5:55 Stojaczyk, DariuszX
2018-04-25 0:16 PR PR
2018-04-24 23:59 Verkamp, Daniel
2018-04-24 23:51 PR PR
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.