Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec)

[Dave Jiang <dave.jiang@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 3540 bytes --]


On 5/31/2018 9:32 AM, Glimcher, Boris wrote:
>
> Hi,
>
> I’m trying to enlarge buffers I use in ntb_tool by using cma:
>
> $ dmesg | grep cma
>
> [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 
> root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto 
> rhgb quiet cma=32G hugepages=100000 intel_pstate=disable
>
> [    0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000
>
> [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 
> root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto 
> rhgb quiet cma=32G hugepages=100000 intel_pstate=disable
>
> [    0.000000] Memory: 6057140K/1073389456K available (10252K kernel 
> code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K 
> reserved, 33554432K cma-reserved)
>
> grep -i cma /proc/meminfo
>
> CmaTotal: 33554432 kB
>
> CmaFree: 33501788 kB
>
> After loading drivers I allocate memory (uses dma_alloc_coherent):
>
> echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1
>
> I get this stack:
>
> [  313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 
> 20971520 bytes)
>
> [  313.224546] swiotlb: coherent allocation failed for device 
> 0000:3b:00.1 size=20971520
>
So it's quite difficult to allocate CMA memory on x86 without iommu 
enabled as I discovered. Anything larger than 4M and you are pretty much 
doing it by luck. When you turn the iommu on it does not go through 
swiotlb and that seems to work fairly well.


> [  313.224550] Call Trace:
>
> [  313.224560]  dump_stack+0x5c/0x83
>
> [  313.224567] swiotlb_alloc_coherent+0x14c/0x160
>
> [  313.224572] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool]
>
> [  313.224578]  ? perf_trace_mm_compaction_migratepages+0x5a/0x120
>
> [  313.224584] full_proxy_write+0x5a/0xa0
>
> [  313.224589]  __vfs_write+0x33/0x170
>
> [  313.224594]  ? handle_mm_fault+0xc4/0x1d0
>
> [  313.224599]  ? __audit_syscall_entry+0xaf/0x100
>
> [  313.224601]  vfs_write+0xad/0x1a0
>
> [  313.224603]  SyS_write+0x52/0xc0
>
> [  313.224608]  do_syscall_64+0x61/0x1a0
>
> [  313.224614] entry_SYSCALL64_slow_path+0x25/0x25
>
> [  313.224616] RIP: 0033:0x7f79ac50e840
>
> [  313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: 
> 0000000000000001
>
> [  313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 
> 00007f79ac50e840
>
> [  313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: 
> 0000000000000001
>
> [  313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: 
> 00007f79ace27740
>
> [  313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: 
> 00007f79ac7e2400
>
> [  313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: 
> 0000000000000000
>
> Any help please?
>
> --
>
> Best regards, Boris
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to linux-ntb+unsubscribe@googlegroups.com 
> <mailto:linux-ntb+unsubscribe@googlegroups.com>.
> To post to this group, send email to linux-ntb@googlegroups.com 
> <mailto:linux-ntb@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com 
> <https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[-- Attachment #2: Type: text/html, Size: 8550 bytes --]