* fio: rdma_create_event_channel fail
@ 2016-03-31 16:44 Robert LeBlanc
2016-04-03 16:00 ` Jens Axboe
0 siblings, 1 reply; 9+ messages in thread
From: Robert LeBlanc @ 2016-03-31 16:44 UTC (permalink / raw)
To: fio
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
When trying to use the RDMA engine and the example job for the server
with the port set to an arbitrary value I get:
# /home/rleblanc/fio/fio /home/rleblanc/fio/examples/rdmaio-server.fio
receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
fio-2.8-14-g23a8
Starting 1 process
fio: rdma_create_event_channel fail
fio: io engine init failed. Perhaps try reducing io depth?
fio: pid=18588, err=1/
Setting I/O depth=1 only removed the corresponding message from the output.
# uname -r
4.1.15.bs.ufd
# rpm -qa | grep -E "ibverbs|rdmacm"
libibverbs-devel-1.1.8-8.el7.x86_64
librdmacm-1.0.21-1.el7.x86_64
libibverbs-1.1.8-8.el7.x86_64
librdmacm-devel-1.0.21-1.el7.x86_64
I'm not sure what is wrong as this is the first time I've tried using
the RDMA engine.
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
EcOB
=f1X7
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-03-31 16:44 fio: rdma_create_event_channel fail Robert LeBlanc
@ 2016-04-03 16:00 ` Jens Axboe
2016-04-03 17:57 ` Logan Gunthorpe
0 siblings, 1 reply; 9+ messages in thread
From: Jens Axboe @ 2016-04-03 16:00 UTC (permalink / raw)
To: Robert LeBlanc, fio; +Cc: Logan Gunthorpe
CC'ing Logan, who might have an idea.
On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> When trying to use the RDMA engine and the example job for the server
> with the port set to an arbitrary value I get:
>
> # /home/rleblanc/fio/fio /home/rleblanc/fio/examples/rdmaio-server.fio
> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
> fio-2.8-14-g23a8
> Starting 1 process
> fio: rdma_create_event_channel fail
> fio: io engine init failed. Perhaps try reducing io depth?
> fio: pid=18588, err=1/
>
> Setting I/O depth=1 only removed the corresponding message from the output.
> # uname -r
> 4.1.15.bs.ufd
>
> # rpm -qa | grep -E "ibverbs|rdmacm"
> libibverbs-devel-1.1.8-8.el7.x86_64
> librdmacm-1.0.21-1.el7.x86_64
> libibverbs-1.1.8-8.el7.x86_64
> librdmacm-devel-1.0.21-1.el7.x86_64
>
> I'm not sure what is wrong as this is the first time I've tried using
> the RDMA engine.
> - ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
> EcOB
> =f1X7
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Jens Axboe
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-04-03 16:00 ` Jens Axboe
@ 2016-04-03 17:57 ` Logan Gunthorpe
2016-04-03 22:38 ` Robert LeBlanc
2016-04-03 22:39 ` Robert LeBlanc
0 siblings, 2 replies; 9+ messages in thread
From: Logan Gunthorpe @ 2016-04-03 17:57 UTC (permalink / raw)
To: Jens Axboe, Robert LeBlanc, fio
Hi Robert,
It looks like rdma_create_event_channel has failed which is a pretty
basic part of the RDMA initialization. To me, this likely indicates your
RDMA setup is broken. Do commands like ibstatus and ibv_devices report
active interfaces? Do other RDMA test programs like ib_write_bw, etc
(from the perftest package) work?
It may be worth printing the errno with that error message, I may have
time to make a patch to that effect tomorrow.
Logan
On 03/04/16 10:00 AM, Jens Axboe wrote:
> CC'ing Logan, who might have an idea.
>
>
> On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> When trying to use the RDMA engine and the example job for the server
>> with the port set to an arbitrary value I get:
>>
>> # /home/rleblanc/fio/fio /home/rleblanc/fio/examples/rdmaio-server.fio
>> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
>> fio-2.8-14-g23a8
>> Starting 1 process
>> fio: rdma_create_event_channel fail
>> fio: io engine init failed. Perhaps try reducing io depth?
>> fio: pid=18588, err=1/
>>
>> Setting I/O depth=1 only removed the corresponding message from the
>> output.
>> # uname -r
>> 4.1.15.bs.ufd
>>
>> # rpm -qa | grep -E "ibverbs|rdmacm"
>> libibverbs-devel-1.1.8-8.el7.x86_64
>> librdmacm-1.0.21-1.el7.x86_64
>> libibverbs-1.1.8-8.el7.x86_64
>> librdmacm-devel-1.0.21-1.el7.x86_64
>>
>> I'm not sure what is wrong as this is the first time I've tried using
>> the RDMA engine.
>> - ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>> -----BEGIN PGP SIGNATURE-----
>> Version: Mailvelope v1.3.6
>> Comment: https://www.mailvelope.com
>>
>> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
>> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
>> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
>> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
>> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
>> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
>> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
>> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
>> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
>> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
>> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
>> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
>> EcOB
>> =f1X7
>> -----END PGP SIGNATURE-----
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-04-03 17:57 ` Logan Gunthorpe
@ 2016-04-03 22:38 ` Robert LeBlanc
2016-04-04 16:22 ` Logan Gunthorpe
2016-04-03 22:39 ` Robert LeBlanc
1 sibling, 1 reply; 9+ messages in thread
From: Robert LeBlanc @ 2016-04-03 22:38 UTC (permalink / raw)
To: Logan Gunthorpe; +Cc: fio, Jens Axboe
[-- Attachment #1: Type: text/plain, Size: 3261 bytes --]
I'm able to do ibstat and everything is up. I can do ibping between the
nodes and iSER is working properly. I have not done ib_write_bw but I can
try it tomorrow if you think it would help.
I included the entire message from fio. Can you give me some more
information about additional error messages you may needed.
Sent from a mobile device, please excuse any typos.
On Apr 3, 2016 11:57 AM, "Logan Gunthorpe" <logang@deltatee.com> wrote:
> Hi Robert,
>
> It looks like rdma_create_event_channel has failed which is a pretty basic
> part of the RDMA initialization. To me, this likely indicates your RDMA
> setup is broken. Do commands like ibstatus and ibv_devices report active
> interfaces? Do other RDMA test programs like ib_write_bw, etc (from the
> perftest package) work?
>
> It may be worth printing the errno with that error message, I may have
> time to make a patch to that effect tomorrow.
>
> Logan
>
> On 03/04/16 10:00 AM, Jens Axboe wrote:
>
>> CC'ing Logan, who might have an idea.
>>
>>
>> On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> When trying to use the RDMA engine and the example job for the server
>>> with the port set to an arbitrary value I get:
>>>
>>> # /home/rleblanc/fio/fio /home/rleblanc/fio/examples/rdmaio-server.fio
>>> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
>>> fio-2.8-14-g23a8
>>> Starting 1 process
>>> fio: rdma_create_event_channel fail
>>> fio: io engine init failed. Perhaps try reducing io depth?
>>> fio: pid=18588, err=1/
>>>
>>> Setting I/O depth=1 only removed the corresponding message from the
>>> output.
>>> # uname -r
>>> 4.1.15.bs.ufd
>>>
>>> # rpm -qa | grep -E "ibverbs|rdmacm"
>>> libibverbs-devel-1.1.8-8.el7.x86_64
>>> librdmacm-1.0.21-1.el7.x86_64
>>> libibverbs-1.1.8-8.el7.x86_64
>>> librdmacm-devel-1.0.21-1.el7.x86_64
>>>
>>> I'm not sure what is wrong as this is the first time I've tried using
>>> the RDMA engine.
>>> - ----------------
>>> Robert LeBlanc
>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: Mailvelope v1.3.6
>>> Comment: https://www.mailvelope.com
>>>
>>> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
>>> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
>>> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
>>> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
>>> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
>>> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
>>> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
>>> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
>>> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
>>> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
>>> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
>>> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
>>> EcOB
>>> =f1X7
>>> -----END PGP SIGNATURE-----
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>>
[-- Attachment #2: Type: text/html, Size: 4133 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-04-03 17:57 ` Logan Gunthorpe
2016-04-03 22:38 ` Robert LeBlanc
@ 2016-04-03 22:39 ` Robert LeBlanc
1 sibling, 0 replies; 9+ messages in thread
From: Robert LeBlanc @ 2016-04-03 22:39 UTC (permalink / raw)
To: Logan Gunthorpe; +Cc: Jens Axboe, fio
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
I'm able to do ibstat and everything is up. I can do ibping between the
nodes and iSER is working properly. I have not done ib_write_bw but I can
try it tomorrow if you think it would help.
I included the entire message from fio. Can you give me some more
information about additional error messages you may needed.
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJXAZu2CRDmVDuy+mK58QAALRUP/3T/kzvTwGjc04bKdD6z
xV1dqWDd2dIzqwpIgMtZjVXOo3KdQ6RWkwprQa/0G+qNK9GCBP7+bs5S9yqu
rMRRcwZRoKZQApkbm58HhyIkMbHG9HhaArtiFXERcYx8nfwU2VKVLirYuEXS
8dEM56jZoaEYJTCoodDByCFrUZ5w/FI+TfRRfbrxq3eHLscuVQdMgrRPTAWG
7r5tvRFggRxsQ37deW51ZCBbsXU/4AY4bNNku3E5yGY3X4Si/GHPD1XqBEFT
oz/Uorf0cXG40JfqDDV0edxzncQBsmQNUvY2JtDTKfPrF2nDFcYIJ01VahdB
UZqyI4Hi9/0fwNiQQX9KHt5Y2XsirQ6FpZF4cjAQrEkObHiqqL1RZD95neld
BnOBlNr5lIyiHaDaeuxTsP2bU9SpLdPz0LGR/IiDGBIL6uOxt793pcX1V1Sf
cpVEadxkoy5F0ktDGFXS3sZZe7eBOwJpVzxNdFt1k+8xMWWFQo3YRhVQwzcU
We9pjw8joHiFFHiKuV4d73t8VtD3kPH2+ECFh8U11Yp5ymM1k1VvkFZURexd
M/MiJPh0+d7lkO1tCXEfUvc6C8+rU7GDRoGvEa/VupfvMMiKhywAV7HY2lOs
YQ0XBApxdCMfySrb9isvXtqiawuJhOekb3J2ZPm4m1vRbAaGwuimML/odzu6
lfGg
=eFSE
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Sun, Apr 3, 2016 at 11:57 AM, Logan Gunthorpe <logang@deltatee.com> wrote:
> Hi Robert,
>
> It looks like rdma_create_event_channel has failed which is a pretty basic
> part of the RDMA initialization. To me, this likely indicates your RDMA
> setup is broken. Do commands like ibstatus and ibv_devices report active
> interfaces? Do other RDMA test programs like ib_write_bw, etc (from the
> perftest package) work?
>
> It may be worth printing the errno with that error message, I may have time
> to make a patch to that effect tomorrow.
>
> Logan
>
>
> On 03/04/16 10:00 AM, Jens Axboe wrote:
>>
>> CC'ing Logan, who might have an idea.
>>
>>
>> On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> When trying to use the RDMA engine and the example job for the server
>>> with the port set to an arbitrary value I get:
>>>
>>> # /home/rleblanc/fio/fio /home/rleblanc/fio/examples/rdmaio-server.fio
>>> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
>>> fio-2.8-14-g23a8
>>> Starting 1 process
>>> fio: rdma_create_event_channel fail
>>> fio: io engine init failed. Perhaps try reducing io depth?
>>> fio: pid=18588, err=1/
>>>
>>> Setting I/O depth=1 only removed the corresponding message from the
>>> output.
>>> # uname -r
>>> 4.1.15.bs.ufd
>>>
>>> # rpm -qa | grep -E "ibverbs|rdmacm"
>>> libibverbs-devel-1.1.8-8.el7.x86_64
>>> librdmacm-1.0.21-1.el7.x86_64
>>> libibverbs-1.1.8-8.el7.x86_64
>>> librdmacm-devel-1.0.21-1.el7.x86_64
>>>
>>> I'm not sure what is wrong as this is the first time I've tried using
>>> the RDMA engine.
>>> - ----------------
>>> Robert LeBlanc
>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: Mailvelope v1.3.6
>>> Comment: https://www.mailvelope.com
>>>
>>> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
>>> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
>>> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
>>> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
>>> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
>>> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
>>> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
>>> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
>>> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
>>> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
>>> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
>>> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
>>> EcOB
>>> =f1X7
>>> -----END PGP SIGNATURE-----
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-04-03 22:38 ` Robert LeBlanc
@ 2016-04-04 16:22 ` Logan Gunthorpe
2016-04-04 17:43 ` Robert LeBlanc
0 siblings, 1 reply; 9+ messages in thread
From: Logan Gunthorpe @ 2016-04-04 16:22 UTC (permalink / raw)
To: Robert LeBlanc; +Cc: fio, Jens Axboe
Hey,
I've created a patch on github that will print slightly more information
in the error message. If you could try it and post the output it may
shine some more light on the situation:
https://github.com/axboe/fio/pull/160
https://github.com/lsgunth/fio/tree/rdma_err
It may also be worth sending the output of ibstat and maybe lsmod as well.
Logan
On 03/04/16 04:38 PM, Robert LeBlanc wrote:
> I'm able to do ibstat and everything is up. I can do ibping between the
> nodes and iSER is working properly. I have not done ib_write_bw but I
> can try it tomorrow if you think it would help.
>
> I included the entire message from fio. Can you give me some more
> information about additional error messages you may needed.
>
> Sent from a mobile device, please excuse any typos.
>
> On Apr 3, 2016 11:57 AM, "Logan Gunthorpe" <logang@deltatee.com
> <mailto:logang@deltatee.com>> wrote:
>
> Hi Robert,
>
> It looks like rdma_create_event_channel has failed which is a pretty
> basic part of the RDMA initialization. To me, this likely indicates
> your RDMA setup is broken. Do commands like ibstatus and ibv_devices
> report active interfaces? Do other RDMA test programs like
> ib_write_bw, etc (from the perftest package) work?
>
> It may be worth printing the errno with that error message, I may
> have time to make a patch to that effect tomorrow.
>
> Logan
>
> On 03/04/16 10:00 AM, Jens Axboe wrote:
>
> CC'ing Logan, who might have an idea.
>
>
> On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> When trying to use the RDMA engine and the example job for
> the server
> with the port set to an arbitrary value I get:
>
> # /home/rleblanc/fio/fio
> /home/rleblanc/fio/examples/rdmaio-server.fio
> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M,
> ioengine=rdma, iodepth=16
> fio-2.8-14-g23a8
> Starting 1 process
> fio: rdma_create_event_channel fail
> fio: io engine init failed. Perhaps try reducing io depth?
> fio: pid=18588, err=1/
>
> Setting I/O depth=1 only removed the corresponding message
> from the
> output.
> # uname -r
> 4.1.15.bs.ufd
>
> # rpm -qa | grep -E "ibverbs|rdmacm"
> libibverbs-devel-1.1.8-8.el7.x86_64
> librdmacm-1.0.21-1.el7.x86_64
> libibverbs-1.1.8-8.el7.x86_64
> librdmacm-devel-1.0.21-1.el7.x86_64
>
> I'm not sure what is wrong as this is the first time I've
> tried using
> the RDMA engine.
> - ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2
> FA62 B9F1
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
> EcOB
> =f1X7
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe
> fio" in
> the body of a message to majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>
> More majordomo info at
> http://vger.kernel.org/majordomo-info.html
>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-04-04 16:22 ` Logan Gunthorpe
@ 2016-04-04 17:43 ` Robert LeBlanc
2016-04-04 18:17 ` Logan Gunthorpe
0 siblings, 1 reply; 9+ messages in thread
From: Robert LeBlanc @ 2016-04-04 17:43 UTC (permalink / raw)
To: Logan Gunthorpe; +Cc: fio, Jens Axboe
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Logan,
Here is the output from your branch. Thanks for helping with this.
Could the problem be due to us having multiple IB cards?
# cat examples/rdmaio-server.fio
# Example rdma server job
[global]
ioengine=rdma
port=5557
bs=1m
size=100g
[receiver]
rw=read
iodepth=16
$ ./fio examples/rdmaio-server.fio
receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
fio-2.2.8-381-g2a44
Starting 1 process
fio: rdma_create_event_channel fail: No such device
fio: io engine init failed. Perhaps try reducing io depth?
fio: pid=5518, err=1/
Run status group 0 (all jobs):
[rleblanc@localhost fio-logan]$ ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.35.5100
Hardware version: 0
Node GUID: 0x0cc47affff4fe8bc
System image GUID: 0x0cc47affff4fe8bf
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 58
LMC: 0
SM lid: 1
Capability mask: 0x02594868
Port GUID: 0x0cc47affff4fe8bd
Link layer: InfiniBand
CA 'mlx5_0'
CA type: MT4113
Number of ports: 2
Firmware version: 10.14.1100
Hardware version: 0
Node GUID: 0xe41d2d030006d0d0
System image GUID: 0xe41d2d030006d0d0
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 56
LMC: 0
SM lid: 1
Capability mask: 0x26596848
Port GUID: 0xe41d2d030006d0d0
Link layer: InfiniBand
Port 2:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 57
LMC: 0
SM lid: 1
Capability mask: 0x26596848
Port GUID: 0xe41d2d030006d0d8
Link layer: InfiniBand
# lsmod
Module Size Used by
ebtable_filter 16384 0
ebtables 36864 1 ebtable_filter
ip6table_filter 16384 0
ip6_tables 28672 1 ip6table_filter
iptable_filter 16384 0
iptable_raw 16384 0
xprtrdma 53248 0
ib_isert 57344 0
iscsi_target_mod 294912 1 ib_isert
ib_iser 53248 0
ib_srpt 53248 0
target_core_mod 372736 3 iscsi_target_mod,ib_srpt,ib_isert
ib_srp 49152 0
scsi_transport_srp 24576 1 ib_srp
ib_ipoib 94208 0
rdma_ucm 24576 0
ib_ucm 24576 0
ib_uverbs 49152 2 ib_ucm,rdma_ucm
ib_umad 24576 0
rdma_cm 45056 4 xprtrdma,ib_iser,rdma_ucm,ib_isert
ib_cm 49152 5 rdma_cm,ib_srp,ib_ucm,ib_srpt,ib_ipoib
iw_cxgb4 172032 0
iw_cm 45056 2 iw_cxgb4,rdma_cm
iw_cxgb3 126976 0
mlx5_ib 110592 0
mlx4_ib 151552 0
ib_sa 36864 6 rdma_cm,ib_cm,mlx4_ib,ib_srp,rdma_ucm,ib_ipoib
ib_mad 49152 5 ib_cm,ib_sa,mlx4_ib,ib_srpt,ib_umad
ib_core 102400 19
iw_cxgb3,iw_cxgb4,rdma_cm,ib_cm,ib_sa,iw_cm,xprtrdma,mlx4_ib,mlx5_ib,ib_mad,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert
ib_addr 20480 3 rdma_cm,ib_core,rdma_ucm
ipmi_devintf 20480 2
kvm_intel 155648 0
kvm 495616 1 kvm_intel
coretemp 16384 0
iTCO_wdt 16384 0
intel_powerclamp 16384 0
iTCO_vendor_support 16384 1 iTCO_wdt
sb_edac 28672 0
x86_pkg_temp_thermal 16384 0
joydev 20480 0
sg 40960 0
pcspkr 16384 0
lpc_ich 24576 0
edac_core 61440 1 sb_edac
i2c_i801 20480 0
mfd_core 16384 1 lpc_ich
ipmi_si 57344 1
8250_fintek 16384 0
acpi_power_meter 20480 0
ipmi_msghandler 49152 2 ipmi_devintf,ipmi_si
acpi_pad 180224 0
mei_me 24576 0
mei 90112 1 mei_me
ioatdma 69632 0
shpchp 40960 0
ip_tables 28672 2 iptable_filter,iptable_raw
xfs 929792 1
libcrc32c 16384 1 xfs
raid1 40960 1
sd_mod 40960 2
mlx4_en 110592 0
vxlan 45056 1 mlx4_en
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 16384 1 vxlan
crc32_pclmul 16384 0
ast 61440 1
syscopyarea 16384 1 ast
sysfillrect 16384 1 ast
aesni_intel 172032 0
sysimgblt 16384 1 ast
drm_kms_helper 126976 1 ast
lrw 16384 1 aesni_intel
gf128mul 16384 1 lrw
glue_helper 16384 1 aesni_intel
ablk_helper 16384 1 aesni_intel
ttm 94208 1 ast
igb 196608 0
ahci 36864 0
cryptd 20480 2 aesni_intel,ablk_helper
ptp 20480 2 igb,mlx4_en
crct10dif_pclmul 16384 0
libahci 32768 1 ahci
drm 352256 4 ast,ttm,drm_kms_helper
crc32c_intel 24576 9
mlx5_core 102400 1 mlx5_ib
pps_core 20480 1 ptp
mlx4_core 286720 2 mlx4_en,mlx4_ib
libata 245760 2 ahci,libahci
i2c_algo_bit 16384 2 ast,igb
dca 16384 2 igb,ioatdma
wmi 20480 0
sunrpc 327680 2 xprtrdma
dm_mirror 24576 0
dm_region_hash 24576 1 dm_mirror
dm_log 20480 2 dm_region_hash,dm_mirror
dm_mod 110592 2 dm_log,dm_mirror
iscsi_tcp 20480 4
zfs 2826240 0
be2iscsi 114688 0
bnx2i 57344 0
cnic 65536 1 bnx2i
uio 20480 1 cnic
cxgb4 204800 1 iw_cxgb4
cxgb3 159744 1 iw_cxgb3
libcxgbi 65536 0
libiscsi_tcp 28672 2 iscsi_tcp,libcxgbi
mdio 16384 1 cxgb3
qla4xxx 286720 0
iscsi_boot_sysfs 16384 2 qla4xxx,be2iscsi
libiscsi 57344 7
qla4xxx,libiscsi_tcp,bnx2i,be2iscsi,iscsi_tcp,ib_iser,libcxgbi
scsi_transport_iscsi 102400 8
qla4xxx,bnx2i,be2iscsi,iscsi_tcp,ib_iser,libcxgbi,libiscsi
zunicode 331776 1 zfs
zcommon 57344 1 zfs
znvpair 94208 2 zfs,zcommon
spl 98304 3 zfs,zcommon,znvpair
zavl 16384 1 zfs
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJXAqeiCRDmVDuy+mK58QAAeNoP/1pFkBV3piTGx9WPtXLA
cK+SLIVirFcX1QcxJOSjXjYkTvHNwFgg5YzWbpQqN7ZzzEmt8baUiDeolTrD
SwIKxE+SBS8qHGiFLndY2fO//poYevFAdq1JgYVGokv2eSvsrAJDR8snEJ/a
qjA38VIIA3UoHE3ABDGgKO/vYE6dZZBfYHQRFIhp5tvDyFaCZzw8RGZWY7FR
9iTyQhONJxYV3oStTtmeM2B9txl+8HfdTTPpZXZsNZ0g8DKyF78gksBNfv4v
HJYwLUhZBQNPlHhR82a9mxMvecaNBlyA33k4+uWjUieSX66YGwt+tn5vXl/k
RY8QwchYI8v4642wPKCqXy+SAjJDn45wp4z/Z9Gx2cJfQFNw3c1rpHugINBF
Ri+CS2IWG3ucCrn3K+Nqmu+SvH051j1xoyozZzBmLeMXILVLq2Fd93cjpN7r
XVlmyZJ8kVyMoUwNkX/hRHVL4QRFP2vNKI2dOA0AfXgYOaMKrMUaAVmBWAMO
XnyInJGR1ReUTZKMITdlNTn4lAoaYV3lug/1Uxk31T+hNhvS2uEBnK/JKv12
QAO1Scl3dnD7uiMgtmoXp8jtCisd1N2jB6qblG4IVU22U8ROkfHxwmaHoHVr
5n4F/0WcY2SZVmwhKskX0bl0/bW8CWbeSb1LDEx67uPcTJk31Io9NE9rJQU6
m4nq
=pwG4
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Mon, Apr 4, 2016 at 10:22 AM, Logan Gunthorpe <logang@deltatee.com> wrote:
> Hey,
>
> I've created a patch on github that will print slightly more information
> in the error message. If you could try it and post the output it may
> shine some more light on the situation:
>
> https://github.com/axboe/fio/pull/160
> https://github.com/lsgunth/fio/tree/rdma_err
>
> It may also be worth sending the output of ibstat and maybe lsmod as well.
>
> Logan
>
>
> On 03/04/16 04:38 PM, Robert LeBlanc wrote:
>> I'm able to do ibstat and everything is up. I can do ibping between the
>> nodes and iSER is working properly. I have not done ib_write_bw but I
>> can try it tomorrow if you think it would help.
>>
>> I included the entire message from fio. Can you give me some more
>> information about additional error messages you may needed.
>>
>> Sent from a mobile device, please excuse any typos.
>>
>> On Apr 3, 2016 11:57 AM, "Logan Gunthorpe" <logang@deltatee.com
>> <mailto:logang@deltatee.com>> wrote:
>>
>> Hi Robert,
>>
>> It looks like rdma_create_event_channel has failed which is a pretty
>> basic part of the RDMA initialization. To me, this likely indicates
>> your RDMA setup is broken. Do commands like ibstatus and ibv_devices
>> report active interfaces? Do other RDMA test programs like
>> ib_write_bw, etc (from the perftest package) work?
>>
>> It may be worth printing the errno with that error message, I may
>> have time to make a patch to that effect tomorrow.
>>
>> Logan
>>
>> On 03/04/16 10:00 AM, Jens Axboe wrote:
>>
>> CC'ing Logan, who might have an idea.
>>
>>
>> On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> When trying to use the RDMA engine and the example job for
>> the server
>> with the port set to an arbitrary value I get:
>>
>> # /home/rleblanc/fio/fio
>> /home/rleblanc/fio/examples/rdmaio-server.fio
>> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M,
>> ioengine=rdma, iodepth=16
>> fio-2.8-14-g23a8
>> Starting 1 process
>> fio: rdma_create_event_channel fail
>> fio: io engine init failed. Perhaps try reducing io depth?
>> fio: pid=18588, err=1/
>>
>> Setting I/O depth=1 only removed the corresponding message
>> from the
>> output.
>> # uname -r
>> 4.1.15.bs.ufd
>>
>> # rpm -qa | grep -E "ibverbs|rdmacm"
>> libibverbs-devel-1.1.8-8.el7.x86_64
>> librdmacm-1.0.21-1.el7.x86_64
>> libibverbs-1.1.8-8.el7.x86_64
>> librdmacm-devel-1.0.21-1.el7.x86_64
>>
>> I'm not sure what is wrong as this is the first time I've
>> tried using
>> the RDMA engine.
>> - ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2
>> FA62 B9F1
>> -----BEGIN PGP SIGNATURE-----
>> Version: Mailvelope v1.3.6
>> Comment: https://www.mailvelope.com
>>
>> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
>> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
>> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
>> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
>> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
>> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
>> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
>> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
>> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
>> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
>> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
>> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
>> EcOB
>> =f1X7
>> -----END PGP SIGNATURE-----
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> fio" in
>> the body of a message to majordomo@vger.kernel.org
>> <mailto:majordomo@vger.kernel.org>
>> More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>>
>>
>>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-04-04 17:43 ` Robert LeBlanc
@ 2016-04-04 18:17 ` Logan Gunthorpe
2016-04-04 18:48 ` Robert LeBlanc
0 siblings, 1 reply; 9+ messages in thread
From: Logan Gunthorpe @ 2016-04-04 18:17 UTC (permalink / raw)
To: Robert LeBlanc; +Cc: fio, Jens Axboe
Hey,
It looks like the ib verbs stuff isn't detecting your devices. Though
I'm not sure why. What you've sent so far looks good. Can you also send
the output of 'ibv_devices'? (It's from the ibverbs-utils package if you
don't have it installed.) You may have a missing or improperly installed
userspace driver and that command should inform you if that's the case.
It's still also worth trying some of the perftest tools (ib_write_bw) to
see if they have similar problems.
Logan
On 04/04/16 11:43 AM, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Logan,
>
> Here is the output from your branch. Thanks for helping with this.
> Could the problem be due to us having multiple IB cards?
>
> # cat examples/rdmaio-server.fio
> # Example rdma server job
> [global]
> ioengine=rdma
> port=5557
> bs=1m
> size=100g
>
> [receiver]
> rw=read
> iodepth=16
>
> $ ./fio examples/rdmaio-server.fio
> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
> fio-2.2.8-381-g2a44
> Starting 1 process
> fio: rdma_create_event_channel fail: No such device
> fio: io engine init failed. Perhaps try reducing io depth?
> fio: pid=5518, err=1/
>
>
> Run status group 0 (all jobs):
> [rleblanc@localhost fio-logan]$ ibstat
> CA 'mlx4_0'
> CA type: MT4099
> Number of ports: 1
> Firmware version: 2.35.5100
> Hardware version: 0
> Node GUID: 0x0cc47affff4fe8bc
> System image GUID: 0x0cc47affff4fe8bf
> Port 1:
> State: Active
> Physical state: LinkUp
> Rate: 56
> Base lid: 58
> LMC: 0
> SM lid: 1
> Capability mask: 0x02594868
> Port GUID: 0x0cc47affff4fe8bd
> Link layer: InfiniBand
> CA 'mlx5_0'
> CA type: MT4113
> Number of ports: 2
> Firmware version: 10.14.1100
> Hardware version: 0
> Node GUID: 0xe41d2d030006d0d0
> System image GUID: 0xe41d2d030006d0d0
> Port 1:
> State: Active
> Physical state: LinkUp
> Rate: 56
> Base lid: 56
> LMC: 0
> SM lid: 1
> Capability mask: 0x26596848
> Port GUID: 0xe41d2d030006d0d0
> Link layer: InfiniBand
> Port 2:
> State: Active
> Physical state: LinkUp
> Rate: 56
> Base lid: 57
> LMC: 0
> SM lid: 1
> Capability mask: 0x26596848
> Port GUID: 0xe41d2d030006d0d8
> Link layer: InfiniBand
>
> # lsmod
> Module Size Used by
> ebtable_filter 16384 0
> ebtables 36864 1 ebtable_filter
> ip6table_filter 16384 0
> ip6_tables 28672 1 ip6table_filter
> iptable_filter 16384 0
> iptable_raw 16384 0
> xprtrdma 53248 0
> ib_isert 57344 0
> iscsi_target_mod 294912 1 ib_isert
> ib_iser 53248 0
> ib_srpt 53248 0
> target_core_mod 372736 3 iscsi_target_mod,ib_srpt,ib_isert
> ib_srp 49152 0
> scsi_transport_srp 24576 1 ib_srp
> ib_ipoib 94208 0
> rdma_ucm 24576 0
> ib_ucm 24576 0
> ib_uverbs 49152 2 ib_ucm,rdma_ucm
> ib_umad 24576 0
> rdma_cm 45056 4 xprtrdma,ib_iser,rdma_ucm,ib_isert
> ib_cm 49152 5 rdma_cm,ib_srp,ib_ucm,ib_srpt,ib_ipoib
> iw_cxgb4 172032 0
> iw_cm 45056 2 iw_cxgb4,rdma_cm
> iw_cxgb3 126976 0
> mlx5_ib 110592 0
> mlx4_ib 151552 0
> ib_sa 36864 6 rdma_cm,ib_cm,mlx4_ib,ib_srp,rdma_ucm,ib_ipoib
> ib_mad 49152 5 ib_cm,ib_sa,mlx4_ib,ib_srpt,ib_umad
> ib_core 102400 19
> iw_cxgb3,iw_cxgb4,rdma_cm,ib_cm,ib_sa,iw_cm,xprtrdma,mlx4_ib,mlx5_ib,ib_mad,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert
> ib_addr 20480 3 rdma_cm,ib_core,rdma_ucm
> ipmi_devintf 20480 2
> kvm_intel 155648 0
> kvm 495616 1 kvm_intel
> coretemp 16384 0
> iTCO_wdt 16384 0
> intel_powerclamp 16384 0
> iTCO_vendor_support 16384 1 iTCO_wdt
> sb_edac 28672 0
> x86_pkg_temp_thermal 16384 0
> joydev 20480 0
> sg 40960 0
> pcspkr 16384 0
> lpc_ich 24576 0
> edac_core 61440 1 sb_edac
> i2c_i801 20480 0
> mfd_core 16384 1 lpc_ich
> ipmi_si 57344 1
> 8250_fintek 16384 0
> acpi_power_meter 20480 0
> ipmi_msghandler 49152 2 ipmi_devintf,ipmi_si
> acpi_pad 180224 0
> mei_me 24576 0
> mei 90112 1 mei_me
> ioatdma 69632 0
> shpchp 40960 0
> ip_tables 28672 2 iptable_filter,iptable_raw
> xfs 929792 1
> libcrc32c 16384 1 xfs
> raid1 40960 1
> sd_mod 40960 2
> mlx4_en 110592 0
> vxlan 45056 1 mlx4_en
> ip6_udp_tunnel 16384 1 vxlan
> udp_tunnel 16384 1 vxlan
> crc32_pclmul 16384 0
> ast 61440 1
> syscopyarea 16384 1 ast
> sysfillrect 16384 1 ast
> aesni_intel 172032 0
> sysimgblt 16384 1 ast
> drm_kms_helper 126976 1 ast
> lrw 16384 1 aesni_intel
> gf128mul 16384 1 lrw
> glue_helper 16384 1 aesni_intel
> ablk_helper 16384 1 aesni_intel
> ttm 94208 1 ast
> igb 196608 0
> ahci 36864 0
> cryptd 20480 2 aesni_intel,ablk_helper
> ptp 20480 2 igb,mlx4_en
> crct10dif_pclmul 16384 0
> libahci 32768 1 ahci
> drm 352256 4 ast,ttm,drm_kms_helper
> crc32c_intel 24576 9
> mlx5_core 102400 1 mlx5_ib
> pps_core 20480 1 ptp
> mlx4_core 286720 2 mlx4_en,mlx4_ib
> libata 245760 2 ahci,libahci
> i2c_algo_bit 16384 2 ast,igb
> dca 16384 2 igb,ioatdma
> wmi 20480 0
> sunrpc 327680 2 xprtrdma
> dm_mirror 24576 0
> dm_region_hash 24576 1 dm_mirror
> dm_log 20480 2 dm_region_hash,dm_mirror
> dm_mod 110592 2 dm_log,dm_mirror
> iscsi_tcp 20480 4
> zfs 2826240 0
> be2iscsi 114688 0
> bnx2i 57344 0
> cnic 65536 1 bnx2i
> uio 20480 1 cnic
> cxgb4 204800 1 iw_cxgb4
> cxgb3 159744 1 iw_cxgb3
> libcxgbi 65536 0
> libiscsi_tcp 28672 2 iscsi_tcp,libcxgbi
> mdio 16384 1 cxgb3
> qla4xxx 286720 0
> iscsi_boot_sysfs 16384 2 qla4xxx,be2iscsi
> libiscsi 57344 7
> qla4xxx,libiscsi_tcp,bnx2i,be2iscsi,iscsi_tcp,ib_iser,libcxgbi
> scsi_transport_iscsi 102400 8
> qla4xxx,bnx2i,be2iscsi,iscsi_tcp,ib_iser,libcxgbi,libiscsi
> zunicode 331776 1 zfs
> zcommon 57344 1 zfs
> znvpair 94208 2 zfs,zcommon
> spl 98304 3 zfs,zcommon,znvpair
> zavl 16384 1 zfs
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJXAqeiCRDmVDuy+mK58QAAeNoP/1pFkBV3piTGx9WPtXLA
> cK+SLIVirFcX1QcxJOSjXjYkTvHNwFgg5YzWbpQqN7ZzzEmt8baUiDeolTrD
> SwIKxE+SBS8qHGiFLndY2fO//poYevFAdq1JgYVGokv2eSvsrAJDR8snEJ/a
> qjA38VIIA3UoHE3ABDGgKO/vYE6dZZBfYHQRFIhp5tvDyFaCZzw8RGZWY7FR
> 9iTyQhONJxYV3oStTtmeM2B9txl+8HfdTTPpZXZsNZ0g8DKyF78gksBNfv4v
> HJYwLUhZBQNPlHhR82a9mxMvecaNBlyA33k4+uWjUieSX66YGwt+tn5vXl/k
> RY8QwchYI8v4642wPKCqXy+SAjJDn45wp4z/Z9Gx2cJfQFNw3c1rpHugINBF
> Ri+CS2IWG3ucCrn3K+Nqmu+SvH051j1xoyozZzBmLeMXILVLq2Fd93cjpN7r
> XVlmyZJ8kVyMoUwNkX/hRHVL4QRFP2vNKI2dOA0AfXgYOaMKrMUaAVmBWAMO
> XnyInJGR1ReUTZKMITdlNTn4lAoaYV3lug/1Uxk31T+hNhvS2uEBnK/JKv12
> QAO1Scl3dnD7uiMgtmoXp8jtCisd1N2jB6qblG4IVU22U8ROkfHxwmaHoHVr
> 5n4F/0WcY2SZVmwhKskX0bl0/bW8CWbeSb1LDEx67uPcTJk31Io9NE9rJQU6
> m4nq
> =pwG4
> -----END PGP SIGNATURE-----
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>
>
> On Mon, Apr 4, 2016 at 10:22 AM, Logan Gunthorpe <logang@deltatee.com> wrote:
>> Hey,
>>
>> I've created a patch on github that will print slightly more information
>> in the error message. If you could try it and post the output it may
>> shine some more light on the situation:
>>
>> https://github.com/axboe/fio/pull/160
>> https://github.com/lsgunth/fio/tree/rdma_err
>>
>> It may also be worth sending the output of ibstat and maybe lsmod as well.
>>
>> Logan
>>
>>
>> On 03/04/16 04:38 PM, Robert LeBlanc wrote:
>>> I'm able to do ibstat and everything is up. I can do ibping between the
>>> nodes and iSER is working properly. I have not done ib_write_bw but I
>>> can try it tomorrow if you think it would help.
>>>
>>> I included the entire message from fio. Can you give me some more
>>> information about additional error messages you may needed.
>>>
>>> Sent from a mobile device, please excuse any typos.
>>>
>>> On Apr 3, 2016 11:57 AM, "Logan Gunthorpe" <logang@deltatee.com
>>> <mailto:logang@deltatee.com>> wrote:
>>>
>>> Hi Robert,
>>>
>>> It looks like rdma_create_event_channel has failed which is a pretty
>>> basic part of the RDMA initialization. To me, this likely indicates
>>> your RDMA setup is broken. Do commands like ibstatus and ibv_devices
>>> report active interfaces? Do other RDMA test programs like
>>> ib_write_bw, etc (from the perftest package) work?
>>>
>>> It may be worth printing the errno with that error message, I may
>>> have time to make a patch to that effect tomorrow.
>>>
>>> Logan
>>>
>>> On 03/04/16 10:00 AM, Jens Axboe wrote:
>>>
>>> CC'ing Logan, who might have an idea.
>>>
>>>
>>> On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> When trying to use the RDMA engine and the example job for
>>> the server
>>> with the port set to an arbitrary value I get:
>>>
>>> # /home/rleblanc/fio/fio
>>> /home/rleblanc/fio/examples/rdmaio-server.fio
>>> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M,
>>> ioengine=rdma, iodepth=16
>>> fio-2.8-14-g23a8
>>> Starting 1 process
>>> fio: rdma_create_event_channel fail
>>> fio: io engine init failed. Perhaps try reducing io depth?
>>> fio: pid=18588, err=1/
>>>
>>> Setting I/O depth=1 only removed the corresponding message
>>> from the
>>> output.
>>> # uname -r
>>> 4.1.15.bs.ufd
>>>
>>> # rpm -qa | grep -E "ibverbs|rdmacm"
>>> libibverbs-devel-1.1.8-8.el7.x86_64
>>> librdmacm-1.0.21-1.el7.x86_64
>>> libibverbs-1.1.8-8.el7.x86_64
>>> librdmacm-devel-1.0.21-1.el7.x86_64
>>>
>>> I'm not sure what is wrong as this is the first time I've
>>> tried using
>>> the RDMA engine.
>>> - ----------------
>>> Robert LeBlanc
>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2
>>> FA62 B9F1
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: Mailvelope v1.3.6
>>> Comment: https://www.mailvelope.com
>>>
>>> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
>>> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
>>> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
>>> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
>>> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
>>> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
>>> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
>>> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
>>> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
>>> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
>>> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
>>> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
>>> EcOB
>>> =f1X7
>>> -----END PGP SIGNATURE-----
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> <mailto:majordomo@vger.kernel.org>
>>> More majordomo info at
>>> http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: fio: rdma_create_event_channel fail
2016-04-04 18:17 ` Logan Gunthorpe
@ 2016-04-04 18:48 ` Robert LeBlanc
0 siblings, 0 replies; 9+ messages in thread
From: Robert LeBlanc @ 2016-04-04 18:48 UTC (permalink / raw)
To: Logan Gunthorpe; +Cc: fio, Jens Axboe
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Logan,
Thanks for all your help. I was lacking the libmlx4 and libmklx5
packages. After installing those, fio started without issue. Since we
have been using all kernel space programs, we didn't run into this
dependency issue until now. We had other userspace RDMA libraries, but
was missing this piece.
Thanks again.
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJXArbyCRDmVDuy+mK58QAAPdUP/2Q3ZVtF39Tb9Y6CBcXo
HOvnKD/ds5pxygQ5HsXNVlpd2rYabC8TdlEJVWwaLSbJ8rl2WzKeXU3rLqRB
0XK7UWzZeyg0kZm762LwwkMLcmSoV2KN89ra6VWWbGCzPy51tPmTmcYwNbHR
hjmFGWWUNVJKD02fXNP4qjbTS+eCfWNMxvxeGv+/kkiNGHB1DuBkyykCY74N
bKCmEE3MsMznGfZT0g60ZknbbBuNdyNkw4aGYV2TZ4QxRGMnBfb16jGdmJma
ZoyfzxVUQiN2qGYnNhsPYs0/I+Qtmy8KCslp7r6bPRSJ4NmoI9HwktU8U2cS
gbGYYo4jPxVO9UMHWBJ7cnNeUYnmi+yzAWa7zGlQL3ko352W5Zef8T8AvSfz
leaZ3BFlmQZB6OideOpuLY9RGsRMP5TbFaKmysyLruaPxFoMXB8Yu/0XgBeG
3m6aLBx0zbcJkfezwZ9/RxAZm2Nd0nrxCUvEokp9kylufDQCfApKprB6CImr
cjaxIYVHN1BEu3PKqy2WdpYehLGDNCiXsAlTQTmh869XpsF3WyUW7gCePa0k
ba8qoVUeOIWZWA4iw9AiEKmBFfqNmV6PWq75NYC0KOkDTXePSCCSA6LCM7rC
RKMjQsDf0PyVxVTOeHkw7eaGp8AQ/NjuUTMjlGXQpk9cXW9py0bzeMinjYvn
hjDN
=5tGZ
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Mon, Apr 4, 2016 at 12:17 PM, Logan Gunthorpe <logang@deltatee.com> wrote:
> Hey,
>
> It looks like the ib verbs stuff isn't detecting your devices. Though
> I'm not sure why. What you've sent so far looks good. Can you also send
> the output of 'ibv_devices'? (It's from the ibverbs-utils package if you
> don't have it installed.) You may have a missing or improperly installed
> userspace driver and that command should inform you if that's the case.
>
> It's still also worth trying some of the perftest tools (ib_write_bw) to
> see if they have similar problems.
>
> Logan
>
> On 04/04/16 11:43 AM, Robert LeBlanc wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Logan,
>>
>> Here is the output from your branch. Thanks for helping with this.
>> Could the problem be due to us having multiple IB cards?
>>
>> # cat examples/rdmaio-server.fio
>> # Example rdma server job
>> [global]
>> ioengine=rdma
>> port=5557
>> bs=1m
>> size=100g
>>
>> [receiver]
>> rw=read
>> iodepth=16
>>
>> $ ./fio examples/rdmaio-server.fio
>> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=rdma, iodepth=16
>> fio-2.2.8-381-g2a44
>> Starting 1 process
>> fio: rdma_create_event_channel fail: No such device
>> fio: io engine init failed. Perhaps try reducing io depth?
>> fio: pid=5518, err=1/
>>
>>
>> Run status group 0 (all jobs):
>> [rleblanc@localhost fio-logan]$ ibstat
>> CA 'mlx4_0'
>> CA type: MT4099
>> Number of ports: 1
>> Firmware version: 2.35.5100
>> Hardware version: 0
>> Node GUID: 0x0cc47affff4fe8bc
>> System image GUID: 0x0cc47affff4fe8bf
>> Port 1:
>> State: Active
>> Physical state: LinkUp
>> Rate: 56
>> Base lid: 58
>> LMC: 0
>> SM lid: 1
>> Capability mask: 0x02594868
>> Port GUID: 0x0cc47affff4fe8bd
>> Link layer: InfiniBand
>> CA 'mlx5_0'
>> CA type: MT4113
>> Number of ports: 2
>> Firmware version: 10.14.1100
>> Hardware version: 0
>> Node GUID: 0xe41d2d030006d0d0
>> System image GUID: 0xe41d2d030006d0d0
>> Port 1:
>> State: Active
>> Physical state: LinkUp
>> Rate: 56
>> Base lid: 56
>> LMC: 0
>> SM lid: 1
>> Capability mask: 0x26596848
>> Port GUID: 0xe41d2d030006d0d0
>> Link layer: InfiniBand
>> Port 2:
>> State: Active
>> Physical state: LinkUp
>> Rate: 56
>> Base lid: 57
>> LMC: 0
>> SM lid: 1
>> Capability mask: 0x26596848
>> Port GUID: 0xe41d2d030006d0d8
>> Link layer: InfiniBand
>>
>> # lsmod
>> Module Size Used by
>> ebtable_filter 16384 0
>> ebtables 36864 1 ebtable_filter
>> ip6table_filter 16384 0
>> ip6_tables 28672 1 ip6table_filter
>> iptable_filter 16384 0
>> iptable_raw 16384 0
>> xprtrdma 53248 0
>> ib_isert 57344 0
>> iscsi_target_mod 294912 1 ib_isert
>> ib_iser 53248 0
>> ib_srpt 53248 0
>> target_core_mod 372736 3 iscsi_target_mod,ib_srpt,ib_isert
>> ib_srp 49152 0
>> scsi_transport_srp 24576 1 ib_srp
>> ib_ipoib 94208 0
>> rdma_ucm 24576 0
>> ib_ucm 24576 0
>> ib_uverbs 49152 2 ib_ucm,rdma_ucm
>> ib_umad 24576 0
>> rdma_cm 45056 4 xprtrdma,ib_iser,rdma_ucm,ib_isert
>> ib_cm 49152 5 rdma_cm,ib_srp,ib_ucm,ib_srpt,ib_ipoib
>> iw_cxgb4 172032 0
>> iw_cm 45056 2 iw_cxgb4,rdma_cm
>> iw_cxgb3 126976 0
>> mlx5_ib 110592 0
>> mlx4_ib 151552 0
>> ib_sa 36864 6 rdma_cm,ib_cm,mlx4_ib,ib_srp,rdma_ucm,ib_ipoib
>> ib_mad 49152 5 ib_cm,ib_sa,mlx4_ib,ib_srpt,ib_umad
>> ib_core 102400 19
>> iw_cxgb3,iw_cxgb4,rdma_cm,ib_cm,ib_sa,iw_cm,xprtrdma,mlx4_ib,mlx5_ib,ib_mad,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert
>> ib_addr 20480 3 rdma_cm,ib_core,rdma_ucm
>> ipmi_devintf 20480 2
>> kvm_intel 155648 0
>> kvm 495616 1 kvm_intel
>> coretemp 16384 0
>> iTCO_wdt 16384 0
>> intel_powerclamp 16384 0
>> iTCO_vendor_support 16384 1 iTCO_wdt
>> sb_edac 28672 0
>> x86_pkg_temp_thermal 16384 0
>> joydev 20480 0
>> sg 40960 0
>> pcspkr 16384 0
>> lpc_ich 24576 0
>> edac_core 61440 1 sb_edac
>> i2c_i801 20480 0
>> mfd_core 16384 1 lpc_ich
>> ipmi_si 57344 1
>> 8250_fintek 16384 0
>> acpi_power_meter 20480 0
>> ipmi_msghandler 49152 2 ipmi_devintf,ipmi_si
>> acpi_pad 180224 0
>> mei_me 24576 0
>> mei 90112 1 mei_me
>> ioatdma 69632 0
>> shpchp 40960 0
>> ip_tables 28672 2 iptable_filter,iptable_raw
>> xfs 929792 1
>> libcrc32c 16384 1 xfs
>> raid1 40960 1
>> sd_mod 40960 2
>> mlx4_en 110592 0
>> vxlan 45056 1 mlx4_en
>> ip6_udp_tunnel 16384 1 vxlan
>> udp_tunnel 16384 1 vxlan
>> crc32_pclmul 16384 0
>> ast 61440 1
>> syscopyarea 16384 1 ast
>> sysfillrect 16384 1 ast
>> aesni_intel 172032 0
>> sysimgblt 16384 1 ast
>> drm_kms_helper 126976 1 ast
>> lrw 16384 1 aesni_intel
>> gf128mul 16384 1 lrw
>> glue_helper 16384 1 aesni_intel
>> ablk_helper 16384 1 aesni_intel
>> ttm 94208 1 ast
>> igb 196608 0
>> ahci 36864 0
>> cryptd 20480 2 aesni_intel,ablk_helper
>> ptp 20480 2 igb,mlx4_en
>> crct10dif_pclmul 16384 0
>> libahci 32768 1 ahci
>> drm 352256 4 ast,ttm,drm_kms_helper
>> crc32c_intel 24576 9
>> mlx5_core 102400 1 mlx5_ib
>> pps_core 20480 1 ptp
>> mlx4_core 286720 2 mlx4_en,mlx4_ib
>> libata 245760 2 ahci,libahci
>> i2c_algo_bit 16384 2 ast,igb
>> dca 16384 2 igb,ioatdma
>> wmi 20480 0
>> sunrpc 327680 2 xprtrdma
>> dm_mirror 24576 0
>> dm_region_hash 24576 1 dm_mirror
>> dm_log 20480 2 dm_region_hash,dm_mirror
>> dm_mod 110592 2 dm_log,dm_mirror
>> iscsi_tcp 20480 4
>> zfs 2826240 0
>> be2iscsi 114688 0
>> bnx2i 57344 0
>> cnic 65536 1 bnx2i
>> uio 20480 1 cnic
>> cxgb4 204800 1 iw_cxgb4
>> cxgb3 159744 1 iw_cxgb3
>> libcxgbi 65536 0
>> libiscsi_tcp 28672 2 iscsi_tcp,libcxgbi
>> mdio 16384 1 cxgb3
>> qla4xxx 286720 0
>> iscsi_boot_sysfs 16384 2 qla4xxx,be2iscsi
>> libiscsi 57344 7
>> qla4xxx,libiscsi_tcp,bnx2i,be2iscsi,iscsi_tcp,ib_iser,libcxgbi
>> scsi_transport_iscsi 102400 8
>> qla4xxx,bnx2i,be2iscsi,iscsi_tcp,ib_iser,libcxgbi,libiscsi
>> zunicode 331776 1 zfs
>> zcommon 57344 1 zfs
>> znvpair 94208 2 zfs,zcommon
>> spl 98304 3 zfs,zcommon,znvpair
>> zavl 16384 1 zfs
>> -----BEGIN PGP SIGNATURE-----
>> Version: Mailvelope v1.3.6
>> Comment: https://www.mailvelope.com
>>
>> wsFcBAEBCAAQBQJXAqeiCRDmVDuy+mK58QAAeNoP/1pFkBV3piTGx9WPtXLA
>> cK+SLIVirFcX1QcxJOSjXjYkTvHNwFgg5YzWbpQqN7ZzzEmt8baUiDeolTrD
>> SwIKxE+SBS8qHGiFLndY2fO//poYevFAdq1JgYVGokv2eSvsrAJDR8snEJ/a
>> qjA38VIIA3UoHE3ABDGgKO/vYE6dZZBfYHQRFIhp5tvDyFaCZzw8RGZWY7FR
>> 9iTyQhONJxYV3oStTtmeM2B9txl+8HfdTTPpZXZsNZ0g8DKyF78gksBNfv4v
>> HJYwLUhZBQNPlHhR82a9mxMvecaNBlyA33k4+uWjUieSX66YGwt+tn5vXl/k
>> RY8QwchYI8v4642wPKCqXy+SAjJDn45wp4z/Z9Gx2cJfQFNw3c1rpHugINBF
>> Ri+CS2IWG3ucCrn3K+Nqmu+SvH051j1xoyozZzBmLeMXILVLq2Fd93cjpN7r
>> XVlmyZJ8kVyMoUwNkX/hRHVL4QRFP2vNKI2dOA0AfXgYOaMKrMUaAVmBWAMO
>> XnyInJGR1ReUTZKMITdlNTn4lAoaYV3lug/1Uxk31T+hNhvS2uEBnK/JKv12
>> QAO1Scl3dnD7uiMgtmoXp8jtCisd1N2jB6qblG4IVU22U8ROkfHxwmaHoHVr
>> 5n4F/0WcY2SZVmwhKskX0bl0/bW8CWbeSb1LDEx67uPcTJk31Io9NE9rJQU6
>> m4nq
>> =pwG4
>> -----END PGP SIGNATURE-----
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>>
>>
>> On Mon, Apr 4, 2016 at 10:22 AM, Logan Gunthorpe <logang@deltatee.com> wrote:
>>> Hey,
>>>
>>> I've created a patch on github that will print slightly more information
>>> in the error message. If you could try it and post the output it may
>>> shine some more light on the situation:
>>>
>>> https://github.com/axboe/fio/pull/160
>>> https://github.com/lsgunth/fio/tree/rdma_err
>>>
>>> It may also be worth sending the output of ibstat and maybe lsmod as well.
>>>
>>> Logan
>>>
>>>
>>> On 03/04/16 04:38 PM, Robert LeBlanc wrote:
>>>> I'm able to do ibstat and everything is up. I can do ibping between the
>>>> nodes and iSER is working properly. I have not done ib_write_bw but I
>>>> can try it tomorrow if you think it would help.
>>>>
>>>> I included the entire message from fio. Can you give me some more
>>>> information about additional error messages you may needed.
>>>>
>>>> Sent from a mobile device, please excuse any typos.
>>>>
>>>> On Apr 3, 2016 11:57 AM, "Logan Gunthorpe" <logang@deltatee.com
>>>> <mailto:logang@deltatee.com>> wrote:
>>>>
>>>> Hi Robert,
>>>>
>>>> It looks like rdma_create_event_channel has failed which is a pretty
>>>> basic part of the RDMA initialization. To me, this likely indicates
>>>> your RDMA setup is broken. Do commands like ibstatus and ibv_devices
>>>> report active interfaces? Do other RDMA test programs like
>>>> ib_write_bw, etc (from the perftest package) work?
>>>>
>>>> It may be worth printing the errno with that error message, I may
>>>> have time to make a patch to that effect tomorrow.
>>>>
>>>> Logan
>>>>
>>>> On 03/04/16 10:00 AM, Jens Axboe wrote:
>>>>
>>>> CC'ing Logan, who might have an idea.
>>>>
>>>>
>>>> On 03/31/2016 10:44 AM, Robert LeBlanc wrote:
>>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA256
>>>>
>>>> When trying to use the RDMA engine and the example job for
>>>> the server
>>>> with the port set to an arbitrary value I get:
>>>>
>>>> # /home/rleblanc/fio/fio
>>>> /home/rleblanc/fio/examples/rdmaio-server.fio
>>>> receiver: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M,
>>>> ioengine=rdma, iodepth=16
>>>> fio-2.8-14-g23a8
>>>> Starting 1 process
>>>> fio: rdma_create_event_channel fail
>>>> fio: io engine init failed. Perhaps try reducing io depth?
>>>> fio: pid=18588, err=1/
>>>>
>>>> Setting I/O depth=1 only removed the corresponding message
>>>> from the
>>>> output.
>>>> # uname -r
>>>> 4.1.15.bs.ufd
>>>>
>>>> # rpm -qa | grep -E "ibverbs|rdmacm"
>>>> libibverbs-devel-1.1.8-8.el7.x86_64
>>>> librdmacm-1.0.21-1.el7.x86_64
>>>> libibverbs-1.1.8-8.el7.x86_64
>>>> librdmacm-devel-1.0.21-1.el7.x86_64
>>>>
>>>> I'm not sure what is wrong as this is the first time I've
>>>> tried using
>>>> the RDMA engine.
>>>> - ----------------
>>>> Robert LeBlanc
>>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2
>>>> FA62 B9F1
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: Mailvelope v1.3.6
>>>> Comment: https://www.mailvelope.com
>>>>
>>>> wsFcBAEBCAAQBQJW/VPPCRDmVDuy+mK58QAAPKMQAJF8B3y8pk9l2emtsrGm
>>>> 2Rt7ufstv6c4XtuCk2wsc6ocZe8yNfAM1BNkW6pTF96orHZuLTt/QDvbDlnN
>>>> q6N0vPkGJbDVbm7YNDzFc4qOU1pbrn8a66eck5BKuHPPogXCsJJTu+rdfAd9
>>>> TNUGD4b9MzogTCzI8Zs6YRdWLIeaJRsPaHqJGYsD5G83rxGFagjx0qoOPuF+
>>>> CNcFVYXZeU3+/YzsTDfuvNtiSDDJTUe3Shjw6fSu8ZFNabucAbbGOflovIIL
>>>> kGjFmprrFgqOLiTnw7muF6tSXcc205YMGbCgOiEye4i9Ajd/ITiEQ3QlbQZ1
>>>> WDz5WPSukDR8KqJoREKcksWVL7zVciulE5/+ZlJajD02JfOTz7j9QydLAPJ/
>>>> sQM1g7Ft5HZK8TB9IgVKBernHCpahNQ5dU2OadDgpe0rxjzjrVcxegYOqLPd
>>>> iUVFT2/UUFwzxaVnxHXTDNGO5A4JSyctvPTQ4uKLFFox9p6L5pFgrz9o86Fs
>>>> lbW/72IjJD/8AEC64cqJp6JuC/sSEmz2hPpOvKdbpWPlVijzPB0OnpkMB4cC
>>>> DATo4afT5uDRDe7IS8Ypi/WcriVLA+O9jRsigARri1F4FFc1QR/FDtXQnKYE
>>>> SKB8sOE6sAv8qsyNLDqyD3rAzjRJ267zweNcWficcrHD3pYljXCLK1oaZgHD
>>>> EcOB
>>>> =f1X7
>>>> -----END PGP SIGNATURE-----
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> fio" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> <mailto:majordomo@vger.kernel.org>
>>>> More majordomo info at
>>>> http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-04-04 18:48 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-03-31 16:44 fio: rdma_create_event_channel fail Robert LeBlanc
2016-04-03 16:00 ` Jens Axboe
2016-04-03 17:57 ` Logan Gunthorpe
2016-04-03 22:38 ` Robert LeBlanc
2016-04-04 16:22 ` Logan Gunthorpe
2016-04-04 17:43 ` Robert LeBlanc
2016-04-04 18:17 ` Logan Gunthorpe
2016-04-04 18:48 ` Robert LeBlanc
2016-04-03 22:39 ` Robert LeBlanc
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.