From: "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
To: gilr-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Stephen Bates <Stephen.Bates-PwyqCcigF0Q@public.gmane.org>
Subject: CUDA not working with ib_write_bw
Date: Mon, 2 Feb 2015 09:59:10 -0600 [thread overview]
Message-ID: <000f01d03f01$2eb315b0$8c194110$@opengridcomputing.com> (raw)
Hey Gil,
I'm trying to test iWARP RDMA <-> GPU memory and I compiled CUDA into the top-o-tree perftest repo. My Nvidia setup is working
because I have verified it with another gpu rdma package (donard from pmc). But when using ib_write_bw the server gets an error
registering the gpu memory with the device. Below is the output from ib_write_bw. I instrumented the kernel registration path and
I find that get_user_pages() is returning -14 (-EFAULT) when called by ib_umem_get().
Q: Is this supposed to work with the upstream RDMA drivers? I'm using a 3.16.3 kernel.org kernel.
Thanks,
Steve
---
[root@stevo1 perftest]# ./ib_write_bw -R --use_cuda
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : cxgb4_1
Number of qps : 1 Transport type : IW
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
---------------------------------------------------------------------------------------
initializing CUDA
There is 1 device supporting CUDA
[pid = 14124, dev = 0] device name = [Tesla K20Xm]
creating CUDA Ctx
making it the current CUDA Ctx
cuMemAlloc() of a 131072 bytes GPU buffer
allocated GPU buffer address at 0000001304260000 pointer=0x1304260000
Couldn't allocate MR
Unable to create the resources needed by comm struct
Unable to perform rdma_client function
[root@stevo1 perftest]#
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next reply other threads:[~2015-02-02 15:59 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-02-02 15:59 Steve Wise [this message]
[not found] ` <E51087C8852F7244A3C5126F7988F0D36A0DAF90@nkgeml511-mbx.china.huawei.com>
[not found] ` <E51087C8852F7244A3C5126F7988F0D36A0DAF90-gH8YcUvTLkOF0yysJWCP+gK1hpo4iccwjNknBlVQO8k@public.gmane.org>
2015-02-04 15:16 ` CUDA not working with ib_write_bw Steve Wise
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='000f01d03f01$2eb315b0$8c194110$@opengridcomputing.com' \
--to=swise-7bpotxp6k4+p2yhjcf5u+vpxobypeauw@public.gmane.org \
--cc=Stephen.Bates-PwyqCcigF0Q@public.gmane.org \
--cc=gilr-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.