* ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) @ 2018-05-31 16:32 Glimcher, Boris 2018-05-31 18:48 ` Logan Gunthorpe 2018-05-31 19:31 ` Dave Jiang 0 siblings, 2 replies; 6+ messages in thread From: Glimcher, Boris @ 2018-05-31 16:32 UTC (permalink / raw) To: linux-ntb [-- Attachment #1: Type: text/plain, Size: 2387 bytes --] Hi, I'm trying to enlarge buffers I use in ntb_tool by using cma: $ dmesg | grep cma [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable [ 0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable [ 0.000000] Memory: 6057140K/1073389456K available (10252K kernel code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K reserved, 33554432K cma-reserved) grep -i cma /proc/meminfo CmaTotal: 33554432 kB CmaFree: 33501788 kB After loading drivers I allocate memory (uses dma_alloc_coherent): echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 I get this stack: [ 313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes) [ 313.224546] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520 [ 313.224550] Call Trace: [ 313.224560] dump_stack+0x5c/0x83 [ 313.224567] swiotlb_alloc_coherent+0x14c/0x160 [ 313.224572] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool] [ 313.224578] ? perf_trace_mm_compaction_migratepages+0x5a/0x120 [ 313.224584] full_proxy_write+0x5a/0xa0 [ 313.224589] __vfs_write+0x33/0x170 [ 313.224594] ? handle_mm_fault+0xc4/0x1d0 [ 313.224599] ? __audit_syscall_entry+0xaf/0x100 [ 313.224601] vfs_write+0xad/0x1a0 [ 313.224603] SyS_write+0x52/0xc0 [ 313.224608] do_syscall_64+0x61/0x1a0 [ 313.224614] entry_SYSCALL64_slow_path+0x25/0x25 [ 313.224616] RIP: 0033:0x7f79ac50e840 [ 313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007f79ac50e840 [ 313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: 0000000000000001 [ 313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: 00007f79ace27740 [ 313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: 00007f79ac7e2400 [ 313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000 Any help please? -- Best regards, Boris [-- Attachment #2: Type: text/html, Size: 6659 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) 2018-05-31 16:32 ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) Glimcher, Boris @ 2018-05-31 18:48 ` Logan Gunthorpe 2018-05-31 19:42 ` Glimcher, Boris 2018-05-31 19:31 ` Dave Jiang 1 sibling, 1 reply; 6+ messages in thread From: Logan Gunthorpe @ 2018-05-31 18:48 UTC (permalink / raw) To: Glimcher, Boris, linux-ntb On 31/05/18 10:32 AM, Glimcher, Boris wrote: > I’m trying to enlarge buffers I use in ntb_tool by using cma: Hmm, not sure what the problem is. Is this x86? It would also be useful to see a complete dmesg trace. Logan ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) 2018-05-31 18:48 ` Logan Gunthorpe @ 2018-05-31 19:42 ` Glimcher, Boris 0 siblings, 0 replies; 6+ messages in thread From: Glimcher, Boris @ 2018-05-31 19:42 UTC (permalink / raw) To: Logan Gunthorpe, linux-ntb Logan, Yes, it is x86 machine. When using smaller buffer size everything works fine: $ echo 8192 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 $ cat /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 Peer MW 1 Information: Physical Address 0x00001af400000000[p] Window Size 17179869184 Alignment 4096 Size Alignment 4096 Size Max 17179869184 Ready Y Allocated Size 8192 Dmesg: [ 211.417152] switchtec: loading out-of-tree module taints kernel. [ 211.417222] switchtec: module verification failed: signature and/or required key missing - tainting kernel [ 211.418519] switchtec 0000:3b:00.1: enabling device (0140 -> 0142) [ 211.421294] switchtec switchtec0: Management device registered. [ 211.421556] switchtec 0000:86:00.1: enabling device (0140 -> 0142) [ 211.424389] switchtec switchtec1: Management device registered. [ 211.424439] switchtec: loaded. [ 211.428558] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec0: Partition ID 0 of 4 [ 211.428713] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec0: MWs: 2 direct, 32 lut [ 211.428828] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec0: Peer MWs: 2 direct, 32 lut [ 211.480112] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:00.0 -> BB:01.0 [ 211.480134] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 3A:00.0 -> BB:01.1 [ 211.532022] switchtec switchtec0: Using crosslink configuration [ 211.532099] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR0 addr: 0 [ 211.532135] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR2 addr: 1000000000 [ 211.532172] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR4 addr: 2000000000 [ 211.792034] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.0 -> BB:02.0 [ 211.792056] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.1 -> BB:02.1 [ 211.844037] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec0: dbs: shift 0/0, mask 0fffffffffffffff [ 211.948006] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec0: Shared MW Ready [ 211.948029] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec0: irqs - event: 2, db: 0, msgs: 1 [ 211.948080] switchtec switchtec0: NTB device registered [ 211.948131] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec1: Partition ID 1 of 4 [ 211.948253] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec1: MWs: 2 direct, 32 lut [ 211.948373] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec1: Peer MWs: 2 direct, 32 lut [ 212.000049] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:00.0 -> BB:01.4 [ 212.000072] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 85:00.0 -> BB:01.5 [ 212.051993] switchtec switchtec1: Using crosslink configuration [ 212.052069] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR0 addr: 0 [ 212.052106] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR2 addr: 1000000000 [ 212.052143] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR4 addr: 2000000000 [ 212.312006] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.4 -> BB:02.4 [ 212.312028] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.5 -> BB:02.5 [ 212.363990] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec1: dbs: shift 0/0, mask 0fffffffffffffff [ 212.468009] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec1: Shared MW Ready [ 212.468034] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec1: irqs - event: 2, db: 0, msgs: 1 [ 212.468115] switchtec switchtec1: NTB device registered [ 212.474696] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec0: enabling link [ 212.474700] ntb_tool:tool_link_event:170: ntb_tool 0000:3b:00.1: link is down speed 0 width 0 [ 212.474771] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec1: enabling link [ 212.474774] ntb_tool:tool_link_event:170: ntb_tool 0000:86:00.1: link is down speed 0 width 0 [ 212.493665] ntb_hw_switchtec:switchtec_ntb_mw_set_trans:319: switchtec switchtec0: MW 1: part 0 addr 0x0000000034c02000 size 0x0000000000002000 $ echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 [ 113.848900] switchtec: loading out-of-tree module taints kernel. [ 113.848947] switchtec: module verification failed: signature and/or required key missing - tainting kernel [ 113.850239] switchtec 0000:3b:00.1: enabling device (0140 -> 0142) [ 113.852935] switchtec switchtec0: Management device registered. [ 113.853166] switchtec 0000:86:00.1: enabling device (0140 -> 0142) [ 113.856024] switchtec switchtec1: Management device registered. [ 113.856089] switchtec: loaded. [ 113.860198] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec0: Partition ID 0 of 4 [ 113.860348] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec0: MWs: 2 direct, 32 lut [ 113.860464] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec0: Peer MWs: 2 direct, 32 lut [ 113.911797] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:00.0 -> BB:01.0 [ 113.911819] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 3A:00.0 -> BB:01.1 [ 113.963711] switchtec switchtec0: Using crosslink configuration [ 113.963787] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR0 addr: 0 [ 113.963824] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR2 addr: 1000000000 [ 113.963860] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR4 addr: 2000000000 [ 114.223723] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.0 -> BB:02.0 [ 114.223745] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.1 -> BB:02.1 [ 114.275731] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec0: dbs: shift 0/0, mask 0fffffffffffffff [ 114.379693] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec0: Shared MW Ready [ 114.379715] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec0: irqs - event: 2, db: 0, msgs: 1 [ 114.379768] switchtec switchtec0: NTB device registered [ 114.379819] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec1: Partition ID 1 of 4 [ 114.379940] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec1: MWs: 2 direct, 32 lut [ 114.380060] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec1: Peer MWs: 2 direct, 32 lut [ 114.431741] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:00.0 -> BB:01.4 [ 114.431764] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 85:00.0 -> BB:01.5 [ 114.483686] switchtec switchtec1: Using crosslink configuration [ 114.483754] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR0 addr: 0 [ 114.483792] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR2 addr: 1000000000 [ 114.483829] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR4 addr: 2000000000 [ 114.743705] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.4 -> BB:02.4 [ 114.743727] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.5 -> BB:02.5 [ 114.795686] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec1: dbs: shift 0/0, mask 0fffffffffffffff [ 114.899664] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec1: Shared MW Ready [ 114.899688] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec1: irqs - event: 2, db: 0, msgs: 1 [ 114.899731] switchtec switchtec1: NTB device registered [ 114.906116] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec0: enabling link [ 114.906120] ntb_tool:tool_link_event:170: ntb_tool 0000:3b:00.1: link is down speed 0 width 0 [ 114.906185] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec1: enabling link [ 114.906188] ntb_tool:tool_link_event:170: ntb_tool 0000:86:00.1: link is down speed 0 width 0 [ 114.925778] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes) [ 114.925779] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520 [ 114.925781] CPU: 30 PID: 1862 Comm: ntb_driver_setu Tainted: G OE 4.14.17 #2 [ 114.925783] Call Trace: [ 114.925792] dump_stack+0x5c/0x83 [ 114.925798] swiotlb_alloc_coherent+0x14c/0x160 [ 114.925804] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool] [ 114.925808] ? perf_trace_mm_compaction_migratepages+0x5a/0x120 [ 114.925814] full_proxy_write+0x5a/0xa0 [ 114.925820] __vfs_write+0x33/0x170 [ 114.925824] ? handle_mm_fault+0xc4/0x1d0 [ 114.925830] ? __audit_syscall_entry+0xaf/0x100 [ 114.925831] vfs_write+0xad/0x1a0 [ 114.925833] SyS_write+0x52/0xc0 [ 114.925838] do_syscall_64+0x61/0x1a0 [ 114.925844] entry_SYSCALL64_slow_path+0x25/0x25 [ 114.925846] RIP: 0033:0x7effb38e6840 [ 114.925847] RSP: 002b:00007ffefbae83f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 114.925849] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007effb38e6840 [ 114.925849] RDX: 0000000000000009 RSI: 00007effb4205000 RDI: 0000000000000001 [ 114.925850] RBP: 00007effb4205000 R08: 000000000000000a R09: 00007effb41ff740 [ 114.925851] R10: 00007ffefbae81a0 R11: 0000000000000246 R12: 00007effb3bba400 [ 114.925852] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000 -- Best regards, Boris Let me google that for you -----Original Message----- From: linux-ntb@googlegroups.com [mailto:linux-ntb@googlegroups.com] On Behalf Of Logan Gunthorpe Sent: Thursday, May 31, 2018 21:49 To: Glimcher, Boris; linux-ntb@googlegroups.com Subject: Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) On 31/05/18 10:32 AM, Glimcher, Boris wrote: > I’m trying to enlarge buffers I use in ntb_tool by using cma: Hmm, not sure what the problem is. Is this x86? It would also be useful to see a complete dmesg trace. Logan -- You received this message because you are subscribed to the Google Groups "linux-ntb" group. To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com. To post to this group, send email to linux-ntb@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/52648710-cf4a-9072-c03b-9a0f3fe7c19d%40deltatee.com. For more options, visit https://groups.google.com/d/optout. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) 2018-05-31 16:32 ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) Glimcher, Boris 2018-05-31 18:48 ` Logan Gunthorpe @ 2018-05-31 19:31 ` Dave Jiang 2018-05-31 19:44 ` Glimcher, Boris 1 sibling, 1 reply; 6+ messages in thread From: Dave Jiang @ 2018-05-31 19:31 UTC (permalink / raw) To: Glimcher, Boris, linux-ntb [-- Attachment #1: Type: text/plain, Size: 3540 bytes --] On 5/31/2018 9:32 AM, Glimcher, Boris wrote: > > Hi, > > I’m trying to enlarge buffers I use in ntb_tool by using cma: > > $ dmesg | grep cma > > [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 > root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto > rhgb quiet cma=32G hugepages=100000 intel_pstate=disable > > [ 0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000 > > [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 > root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto > rhgb quiet cma=32G hugepages=100000 intel_pstate=disable > > [ 0.000000] Memory: 6057140K/1073389456K available (10252K kernel > code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K > reserved, 33554432K cma-reserved) > > grep -i cma /proc/meminfo > > CmaTotal: 33554432 kB > > CmaFree: 33501788 kB > > After loading drivers I allocate memory (uses dma_alloc_coherent): > > echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 > > I get this stack: > > [ 313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: > 20971520 bytes) > > [ 313.224546] swiotlb: coherent allocation failed for device > 0000:3b:00.1 size=20971520 > So it's quite difficult to allocate CMA memory on x86 without iommu enabled as I discovered. Anything larger than 4M and you are pretty much doing it by luck. When you turn the iommu on it does not go through swiotlb and that seems to work fairly well. > [ 313.224550] Call Trace: > > [ 313.224560] dump_stack+0x5c/0x83 > > [ 313.224567] swiotlb_alloc_coherent+0x14c/0x160 > > [ 313.224572] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool] > > [ 313.224578] ? perf_trace_mm_compaction_migratepages+0x5a/0x120 > > [ 313.224584] full_proxy_write+0x5a/0xa0 > > [ 313.224589] __vfs_write+0x33/0x170 > > [ 313.224594] ? handle_mm_fault+0xc4/0x1d0 > > [ 313.224599] ? __audit_syscall_entry+0xaf/0x100 > > [ 313.224601] vfs_write+0xad/0x1a0 > > [ 313.224603] SyS_write+0x52/0xc0 > > [ 313.224608] do_syscall_64+0x61/0x1a0 > > [ 313.224614] entry_SYSCALL64_slow_path+0x25/0x25 > > [ 313.224616] RIP: 0033:0x7f79ac50e840 > > [ 313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: > 0000000000000001 > > [ 313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: > 00007f79ac50e840 > > [ 313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: > 0000000000000001 > > [ 313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: > 00007f79ace27740 > > [ 313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: > 00007f79ac7e2400 > > [ 313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: > 0000000000000000 > > Any help please? > > -- > > Best regards, Boris > > -- > You received this message because you are subscribed to the Google > Groups "linux-ntb" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to linux-ntb+unsubscribe@googlegroups.com > <mailto:linux-ntb+unsubscribe@googlegroups.com>. > To post to this group, send email to linux-ntb@googlegroups.com > <mailto:linux-ntb@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com > <https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout. [-- Attachment #2: Type: text/html, Size: 8550 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) 2018-05-31 19:31 ` Dave Jiang @ 2018-05-31 19:44 ` Glimcher, Boris 2018-05-31 20:15 ` Dave Jiang 0 siblings, 1 reply; 6+ messages in thread From: Glimcher, Boris @ 2018-05-31 19:44 UTC (permalink / raw) To: Dave Jiang, linux-ntb [-- Attachment #1: Type: text/plain, Size: 4595 bytes --] Dave, How can I enable IOMMU? Are you referring to “NTB: PCI Quirk to Enable Switchtec NT Functionality with IOMMU On” patch? -- Best regards, Boris Let me google that for you<http://lmgtfy.com/> From: linux-ntb@googlegroups.com [mailto:linux-ntb@googlegroups.com] On Behalf Of Dave Jiang Sent: Thursday, May 31, 2018 22:31 To: Glimcher, Boris; linux-ntb@googlegroups.com Subject: Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) On 5/31/2018 9:32 AM, Glimcher, Boris wrote: Hi, I’m trying to enlarge buffers I use in ntb_tool by using cma: $ dmesg | grep cma [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable [ 0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable [ 0.000000] Memory: 6057140K/1073389456K available (10252K kernel code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K reserved, 33554432K cma-reserved) grep -i cma /proc/meminfo CmaTotal: 33554432 kB CmaFree: 33501788 kB After loading drivers I allocate memory (uses dma_alloc_coherent): echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 I get this stack: [ 313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes) [ 313.224546] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520 So it's quite difficult to allocate CMA memory on x86 without iommu enabled as I discovered. Anything larger than 4M and you are pretty much doing it by luck. When you turn the iommu on it does not go through swiotlb and that seems to work fairly well. [ 313.224550] Call Trace: [ 313.224560] dump_stack+0x5c/0x83 [ 313.224567] swiotlb_alloc_coherent+0x14c/0x160 [ 313.224572] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool] [ 313.224578] ? perf_trace_mm_compaction_migratepages+0x5a/0x120 [ 313.224584] full_proxy_write+0x5a/0xa0 [ 313.224589] __vfs_write+0x33/0x170 [ 313.224594] ? handle_mm_fault+0xc4/0x1d0 [ 313.224599] ? __audit_syscall_entry+0xaf/0x100 [ 313.224601] vfs_write+0xad/0x1a0 [ 313.224603] SyS_write+0x52/0xc0 [ 313.224608] do_syscall_64+0x61/0x1a0 [ 313.224614] entry_SYSCALL64_slow_path+0x25/0x25 [ 313.224616] RIP: 0033:0x7f79ac50e840 [ 313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007f79ac50e840 [ 313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: 0000000000000001 [ 313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: 00007f79ace27740 [ 313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: 00007f79ac7e2400 [ 313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000 Any help please? -- Best regards, Boris -- You received this message because you are subscribed to the Google Groups "linux-ntb" group. To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com<mailto:linux-ntb+unsubscribe@googlegroups.com>. To post to this group, send email to linux-ntb@googlegroups.com<mailto:linux-ntb@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com<https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "linux-ntb" group. To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com<mailto:linux-ntb+unsubscribe@googlegroups.com>. To post to this group, send email to linux-ntb@googlegroups.com<mailto:linux-ntb@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/1f327d5f-5bd9-12ee-2569-8f66eb0034c2%40intel.com<https://groups.google.com/d/msgid/linux-ntb/1f327d5f-5bd9-12ee-2569-8f66eb0034c2%40intel.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. [-- Attachment #2: Type: text/html, Size: 11251 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) 2018-05-31 19:44 ` Glimcher, Boris @ 2018-05-31 20:15 ` Dave Jiang 0 siblings, 0 replies; 6+ messages in thread From: Dave Jiang @ 2018-05-31 20:15 UTC (permalink / raw) To: Glimcher, Boris, linux-ntb On 05/31/2018 12:44 PM, Glimcher, Boris wrote: > Dave, > > > > How can I enable IOMMU? Turn VTd on in your BIOS. Set intel_iommu=on as a kernel boot parameter. If you are using Intel CPU. > > Are you referring to “NTB: PCI Quirk to Enable Switchtec NT > Functionality with IOMMU On” patch? > > > > -- > > Best regards, Boris > > Let me google that for you <http://lmgtfy.com/> > > > > *From:*linux-ntb@googlegroups.com [mailto:linux-ntb@googlegroups.com] > *On Behalf Of *Dave Jiang > *Sent:* Thursday, May 31, 2018 22:31 > *To:* Glimcher, Boris; linux-ntb@googlegroups.com > *Subject:* Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) > > > > > > On 5/31/2018 9:32 AM, Glimcher, Boris wrote: > > Hi, > > > > I’m trying to enlarge buffers I use in ntb_tool by using cma: > > $ dmesg | grep cma > > [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 > root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto > rhgb quiet cma=32G hugepages=100000 intel_pstate=disable > > [ 0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000 > > [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 > root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto > rhgb quiet cma=32G hugepages=100000 intel_pstate=disable > > [ 0.000000] Memory: 6057140K/1073389456K available (10252K kernel > code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K > reserved, 33554432K cma-reserved) > > > > grep -i cma /proc/meminfo > > CmaTotal: 33554432 kB > > CmaFree: 33501788 kB > > > > After loading drivers I allocate memory (uses dma_alloc_coherent): > > echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 > > > > I get this stack: > > > > [ 313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: > 20971520 bytes) > > [ 313.224546] swiotlb: coherent allocation failed for device > 0000:3b:00.1 size=20971520 > > So it's quite difficult to allocate CMA memory on x86 without iommu > enabled as I discovered. Anything larger than 4M and you are pretty much > doing it by luck. When you turn the iommu on it does not go through > swiotlb and that seems to work fairly well. > > > > [ 313.224550] Call Trace: > > [ 313.224560] dump_stack+0x5c/0x83 > > [ 313.224567] swiotlb_alloc_coherent+0x14c/0x160 > > [ 313.224572] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool] > > [ 313.224578] ? perf_trace_mm_compaction_migratepages+0x5a/0x120 > > [ 313.224584] full_proxy_write+0x5a/0xa0 > > [ 313.224589] __vfs_write+0x33/0x170 > > [ 313.224594] ? handle_mm_fault+0xc4/0x1d0 > > [ 313.224599] ? __audit_syscall_entry+0xaf/0x100 > > [ 313.224601] vfs_write+0xad/0x1a0 > > [ 313.224603] SyS_write+0x52/0xc0 > > [ 313.224608] do_syscall_64+0x61/0x1a0 > > [ 313.224614] entry_SYSCALL64_slow_path+0x25/0x25 > > [ 313.224616] RIP: 0033:0x7f79ac50e840 > > [ 313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: > 0000000000000001 > > [ 313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: > 00007f79ac50e840 > > [ 313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: > 0000000000000001 > > [ 313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: > 00007f79ace27740 > > [ 313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: > 00007f79ac7e2400 > > [ 313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: > 0000000000000000 > > > > Any help please? > > > > -- > > Best regards, Boris > > > > -- > You received this message because you are subscribed to the Google > Groups "linux-ntb" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to linux-ntb+unsubscribe@googlegroups.com > <mailto:linux-ntb+unsubscribe@googlegroups.com>. > To post to this group, send email to linux-ntb@googlegroups.com > <mailto:linux-ntb@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com > <https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google > Groups "linux-ntb" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to linux-ntb+unsubscribe@googlegroups.com > <mailto:linux-ntb+unsubscribe@googlegroups.com>. > To post to this group, send email to linux-ntb@googlegroups.com > <mailto:linux-ntb@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/linux-ntb/1f327d5f-5bd9-12ee-2569-8f66eb0034c2%40intel.com > <https://groups.google.com/d/msgid/linux-ntb/1f327d5f-5bd9-12ee-2569-8f66eb0034c2%40intel.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout. > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-05-31 20:15 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-31 16:32 ntb_tool cannot dma_alloc_coherent with CMA (over switchtec) Glimcher, Boris 2018-05-31 18:48 ` Logan Gunthorpe 2018-05-31 19:42 ` Glimcher, Boris 2018-05-31 19:31 ` Dave Jiang 2018-05-31 19:44 ` Glimcher, Boris 2018-05-31 20:15 ` Dave Jiang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.