CUDA not working with ib_write_bw

CUDA not working with ib_write_bw

[Steve Wise]

Hey Gil,

I'm trying to test iWARP RDMA <-> GPU memory and I compiled CUDA into the top-o-tree perftest repo.  My Nvidia setup is working
because I have verified it with another gpu rdma package (donard from pmc).  But when using ib_write_bw the server gets an error
registering the gpu memory with the device.  Below is the output from ib_write_bw.  I instrumented the kernel registration path and
I find that get_user_pages() is returning -14 (-EFAULT) when called by ib_umem_get(). 

Q:  Is this supposed to work with the upstream RDMA drivers?   I'm using a 3.16.3 kernel.org kernel.

Thanks,

Steve
---

[root@stevo1 perftest]# ./ib_write_bw -R --use_cuda

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : cxgb4_1
 Number of qps   : 1            Transport type : IW
 Connection type : RC           Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 Gid index       : 0
 Max inline data : 0[B]
 rdma_cm QPs     : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 Waiting for client rdma_cm QP to connect
 Please run the same command with the IB/RoCE interface IP
---------------------------------------------------------------------------------------
initializing CUDA
There is 1 device supporting CUDA
[pid = 14124, dev = 0] device name = [Tesla K20Xm]
creating CUDA Ctx
making it the current CUDA Ctx
cuMemAlloc() of a 131072 bytes GPU buffer
allocated GPU buffer address at 0000001304260000 pointer=0x1304260000
Couldn't allocate MR
 Unable to create the resources needed by comm struct
Unable to perform rdma_client function
[root@stevo1 perftest]#

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: CUDA not working with ib_write_bw

[Steve Wise]

I want to RDMA to/from gpu memory from a remote peer.  So the data arrives at the RDMA device from the wire, and is DMA'd directly
to GPU memory of the peer adapter in the system.



> -----Original Message-----
> From: Zhangfengwei [mailto:fngw.zhang-hv44wF8Li93QT0dZR+AlfA@public.gmane.org]
> Sent: Wednesday, February 04, 2015 12:42 AM
> To: Steve Wise; gilr-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Stephen Bates
> Subject: 答复: CUDA not working with ib_write_bw
> 
> Hi Steve，
> 
> Do you want to transfer the data to the gpu buffer directly ? I guess the DMA seems not to do this all by itself.
> 
> -----邮件原件-----
> 发件人: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] 代表 Steve Wise
> 发送时间: 2015年2月2日 23:59
> 收件人: gilr-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
> 抄送: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Stephen Bates
> 主题: CUDA not working with ib_write_bw
> 
> Hey Gil,
> 
> I'm trying to test iWARP RDMA <-> GPU memory and I compiled CUDA into the top-o-tree perftest repo.  My Nvidia setup is working
> because I have verified it with another gpu rdma package (donard from pmc).  But when using ib_write_bw the server gets an error
> registering the gpu memory with the device.  Below is the output from ib_write_bw.  I instrumented the kernel registration path
and I find
> that get_user_pages() is returning -14 (-EFAULT) when called by ib_umem_get().
> 
> Q:  Is this supposed to work with the upstream RDMA drivers?   I'm using a 3.16.3 kernel.org kernel.
> 
> Thanks,
> 
> Steve
> ---
> 
> [root@stevo1 perftest]# ./ib_write_bw -R --use_cuda
> 
> ************************************
> * Waiting for client to connect... *
> ************************************
> ---------------------------------------------------------------------------------------
>                     RDMA_Write BW Test
>  Dual-port       : OFF          Device         : cxgb4_1
>  Number of qps   : 1            Transport type : IW
>  Connection type : RC           Using SRQ      : OFF
>  CQ Moderation   : 100
>  Mtu             : 1024[B]
>  Link type       : Ethernet
>  Gid index       : 0
>  Max inline data : 0[B]
>  rdma_cm QPs     : ON
>  Data ex. method : rdma_cm
> ---------------------------------------------------------------------------------------
>  Waiting for client rdma_cm QP to connect  Please run the same command with the IB/RoCE interface IP
> ---------------------------------------------------------------------------------------
> initializing CUDA
> There is 1 device supporting CUDA
> [pid = 14124, dev = 0] device name = [Tesla K20Xm] creating CUDA Ctx making it the current CUDA Ctx
> cuMemAlloc() of a 131072 bytes GPU buffer allocated GPU buffer address at 0000001304260000 pointer=0x1304260000 Couldn't allocate
> MR  Unable to create the resources needed by comm struct Unable to perform rdma_client function
> [root@stevo1 perftest]#
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More
> majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html