* Help needed: nvmet configuration
@ 2018-10-12 18:08 Sudheendra Sampath
2018-10-12 18:37 ` Keith Busch
2018-10-15 7:38 ` Sagi Grimberg
0 siblings, 2 replies; 6+ messages in thread
From: Sudheendra Sampath @ 2018-10-12 18:08 UTC (permalink / raw)
Hello,
I am using nvmetcli to configure the target.
Below are the steps that I am following :
[root at test38 config]# modprobe -v iw_cxgb4
insmod /lib/modules/4.15.0/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko
insmod /lib/modules/4.15.0/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko
insmod /lib/modules/4.15.0/kernel/drivers/infiniband/core/ib_core.ko
insmod /lib/modules/4.15.0/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko
[root at test38 config]# modprobe -v rdma_ucm
insmod /lib/modules/4.15.0/kernel/drivers/infiniband/core/ib_cm.ko
insmod /lib/modules/4.15.0/kernel/drivers/infiniband/core/iw_cm.ko
insmod /lib/modules/4.15.0/kernel/drivers/infiniband/core/rdma_cm.ko
insmod /lib/modules/4.15.0/kernel/drivers/infiniband/core/ib_uverbs.ko
insmod /lib/modules/4.15.0/kernel/drivers/infiniband/core/rdma_ucm.ko
[root at test38 config]# ifup enp59s0
Connection successfully activated (D-Bus active path:
/org/freedesktop/NetworkManager/ActiveConnection/4)
[root at test38 config]# modprobe -v null_blk
insmod /lib/modules/4.15.0/kernel/drivers/block/null_blk.ko
[root at test38 config]# modprobe -v nvmet
insmod /lib/modules/4.15.0/kernel/drivers/nvme/target/nvmet.ko
[root at test38 config]# modprobe -v nvmet-rdma
insmod /lib/modules/4.15.0/kernel/drivers/nvme/target/nvmet-rdma.ko
[root at test38 config]# nvme list
Node SN Model
Namespace Usage Format FW Rev
---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1 S3VJNY0K500089 SAMSUNG MZQLB3T8HALS-000AZ
1 0.00 B / 3.84 TB 512 B + 0 B ETA50K18
/dev/nvme1n1 S3VJNY0K500104 SAMSUNG MZQLB3T8HALS-000AZ
1 0.00 B / 3.84 TB 512 B + 0 B ETA50K18
[root at test38 config]# nvmetcli
/> cd subsystems
/subsystems> create testvm
/subsystems> cd testvm/namespaces
/subsystems/testvm/namespaces> create nsid=1
/subsystems/testvm/namespaces> cd 1
/subsystems/t.../namespaces/1> set device path=/dev/nvme0n1
Parameter path is now '/dev/nvme0n1'.
/subsystems/t.../namespaces/1> cd ../../
/subsystems/testvm> set attr allow_any_host=1
Parameter allow_any_host is now '1'.
/subsystems/testvm> cd namespaces/1
/subsystems/t.../namespaces/1> enable
The Namespace has been enabled.
/subsystems/t.../namespaces/1> cd ../../../..
/> cd ports
/ports> create 1
/ports> cd 1
/ports/1> set addr adrfam=ipv4
Parameter adrfam is now 'ipv4'.
/ports/1> set addr trtype=rdma
Parameter trtype is now 'rdma'.
/ports/1> set addr trsvcid=4420
Parameter trsvcid is now '4420'.
/ports/1> set addr traddr=12.12.12.38
Parameter traddr is now '12.12.12.38'.
/ports/1> cd subsystems
/ports/1/subsystems> create testvm
Could not symlink testvm in configFS: [Errno 19] No such device
/ports/1/subsystems> exit
When I look at ?dmesg?, I see :
[Fri Oct 12 10:42:16 2018] null: module loaded
[Fri Oct 12 10:42:54 2018] nvmet: adding nsid 1 to subsystem testvm
[Fri Oct 12 10:44:53 2018] nvmet_rdma: binding CM ID to
12.12.12.38:4420 failed (-19)
[root at test38 config]#
enp216s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 12.12.14.38 netmask 255.255.255.0 broadcast 12.12.14.255
inet6 fe80::ee0d:9aff:fe9c:93a2 prefixlen 64 scopeid 0x20<link>
ether ec:0d:9a:9c:93:a2 txqueuelen 1000 (Ethernet)
RX packets 2 bytes 120 (120.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16 bytes 1196 (1.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root at test38 config]# lsmod | grep nvme
nvme_rdma 32768 0
nvme_fabrics 24576 1 nvme_rdma
nvmet_rdma 28672 0
rdma_cm 69632 3 nvme_rdma,rdma_ucm,nvmet_rdma
ib_core 253952 9
ib_cm,rdma_cm,ib_umad,nvme_rdma,ib_uverbs,iw_cm,ib_ucm,rdma_ucm,nvmet_rdma
nvmet 61440 7 nvmet_rdma
nvme 45056 0
nvme_core 77824 4 nvme_fabrics,nvme_rdma,nvme
[root at test38 config]# uname -r
4.15.0
[root at test38 config]#
Below is my target setup configuration :
{
"hosts": [],
"ports": [
{
"addr": {
"adrfam": "ipv4",
"traddr": "12.12.12.38",
"treq": "not specified",
"trsvcid": "4420",
"trtype": "rdma"
},
"portid": 1,
"referrals": [],
"subsystems": []
}
],
"subsystems": [
{
"allowed_hosts": [],
"attr": {
"allow_any_host": "1",
"serial": "472997b23fb70a38",
"version": "1.3"
},
"namespaces": [
{
"device": {
"nguid": "00000000-0000-0000-0000-000000000000",
"path": "/dev/nvme0n1",
"uuid": "1e6b9ac5-1623-4349-bd43-5843642b6c91"
},
"enable": 1,
"nsid": 1
}
],
"nqn": "test"
}
]
}
Here are the drives:
[root at xxxxx ~]# nvme list
Node SN Model
Namespace Usage Format FW Rev
---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1 xxxxxxxx SAMSUNG xxxxxxxx-000AZ 1
0.00 B / 3.84 TB 512 B + 0 B ABCDEF
/dev/nvme1n1 xxxxxxxx SAMSUNG xxxxxxxx-000AZ 1
0.00 B / 3.84 TB 512 B + 0 B ABCDEF
[root at xxxxx ~]#
[root at test38 config]# pwd
/sys/kernel/config
[root at test38 config]# tree nvmet/
nvmet/
??? hosts
??? ports
? ??? 1
? ??? addr_adrfam
? ??? addr_traddr
? ??? addr_treq
? ??? addr_trsvcid
? ??? addr_trtype
? ??? referrals
? ??? subsystems
??? subsystems
??? test
??? allowed_hosts
??? attr_allow_any_host
??? attr_serial
??? attr_version
??? namespaces
??? 1
??? device_nguid
??? device_path
??? device_uuid
??? enable
10 directories, 12 files
Any help appreciated.
Thanks,
-Sudheendra G Sampath
--
Regards
Sudheendra Sampath
^ permalink raw reply [flat|nested] 6+ messages in thread
* Help needed: nvmet configuration
2018-10-12 18:08 Help needed: nvmet configuration Sudheendra Sampath
@ 2018-10-12 18:37 ` Keith Busch
2018-10-12 18:45 ` Sudheendra Sampath
2018-10-15 7:38 ` Sagi Grimberg
1 sibling, 1 reply; 6+ messages in thread
From: Keith Busch @ 2018-10-12 18:37 UTC (permalink / raw)
On Fri, Oct 12, 2018@11:08:21AM -0700, Sudheendra Sampath wrote:
> Any help appreciated.
I didn't find a single question or problem description in this information
dump, so I've no idea what help you're seeking. Also, please turn off
word wrap next time for console output.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Help needed: nvmet configuration
2018-10-12 18:37 ` Keith Busch
@ 2018-10-12 18:45 ` Sudheendra Sampath
0 siblings, 0 replies; 6+ messages in thread
From: Sudheendra Sampath @ 2018-10-12 18:45 UTC (permalink / raw)
Hi,
Sorry for the confusion.
I am running into the following error and need help I am doing
something incorrect.
/ports/1/subsystems> create testvm
=====> Could not symlink testvm in configFS: [Errno 19] No such device <======
/ports/1/subsystems> exit
When I look at ?dmesg?, I see :
[Fri Oct 12 10:42:16 2018] null: module loaded
[Fri Oct 12 10:42:54 2018] nvmet: adding nsid 1 to subsystem testvm
====> [Fri Oct 12 10:44:53 2018] nvmet_rdma: binding CM ID to
12.12.12.38:4420 failed (-19) <=======
On Fri, Oct 12, 2018 at 11:40 AM Keith Busch
<keith.busch@linux.intel.com> wrote:
>
> On Fri, Oct 12, 2018@11:08:21AM -0700, Sudheendra Sampath wrote:
> > Any help appreciated.
>
> I didn't find a single question or problem description in this information
> dump, so I've no idea what help you're seeking. Also, please turn off
> word wrap next time for console output.
--
Regards
Sudheendra Sampath
^ permalink raw reply [flat|nested] 6+ messages in thread
* Help needed: nvmet configuration
2018-10-12 18:08 Help needed: nvmet configuration Sudheendra Sampath
2018-10-12 18:37 ` Keith Busch
@ 2018-10-15 7:38 ` Sagi Grimberg
[not found] ` <CABnqofwo5xQGnuNTOzC+jsWqUmB-awS6nGWGK4SPPTDp=cFfeA@mail.gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Sagi Grimberg @ 2018-10-15 7:38 UTC (permalink / raw)
> /ports/1> set addr adrfam=ipv4
>
> Parameter adrfam is now 'ipv4'.
>
> /ports/1> set addr trtype=rdma
>
> Parameter trtype is now 'rdma'.
>
> /ports/1> set addr trsvcid=4420
>
> Parameter trsvcid is now '4420'.
>
> /ports/1> set addr traddr=12.12.12.38
>
> Parameter traddr is now '12.12.12.38'.
So you set this but
....
> enp216s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
>
> inet 12.12.14.38 netmask 255.255.255.0 broadcast 12.12.14.255
This is the device IP address. You are trying to have an rdma listener
on an IP addressed that is not resolved to any device, which explains
the ENODEV error you are seeing...
^ permalink raw reply [flat|nested] 6+ messages in thread
* Help needed: nvmet configuration
[not found] ` <CABnqofwo5xQGnuNTOzC+jsWqUmB-awS6nGWGK4SPPTDp=cFfeA@mail.gmail.com>
@ 2018-10-15 21:19 ` Sagi Grimberg
[not found] ` <CABnqofzSqFaNxaU=byG1NGgBXXB_-qX-Zhq7EimwZQT4EN_KBg@mail.gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Sagi Grimberg @ 2018-10-15 21:19 UTC (permalink / raw)
> Thanks Sagi for your reply.
>
> I do have 2 100 Gig mellanox NICs (with 12.12.12.38 and 12.12.14.38).
>
> I copy pasted the incorrect interface.
Still looks like there is an address resolution listening to this IP.
Are you able to run any rdma_cm application over this device? Would
be good to run rdma_server/client on the same setup.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Help needed: nvmet configuration
[not found] ` <CABnqofy7bHCUu5n_n+j8bPrCrd7wnfrP3XSapA_UwQWsja6UAg@mail.gmail.com>
@ 2018-10-19 1:14 ` Sagi Grimberg
0 siblings, 0 replies; 6+ messages in thread
From: Sagi Grimberg @ 2018-10-19 1:14 UTC (permalink / raw)
> Hi Sagi,
>
> Just wanted to know if you got my earlier message.
>
> Like I mentioned,? I am able to do "rping" using the interface.
>
> Do you want me to try out any other thing?
>
Something is not clear to me, you mention that you are using Mellanox
NIC, but in your original message you probe iw_cxgb4. You should be
loading mlx5_ib.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-10-19 1:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-12 18:08 Help needed: nvmet configuration Sudheendra Sampath
2018-10-12 18:37 ` Keith Busch
2018-10-12 18:45 ` Sudheendra Sampath
2018-10-15 7:38 ` Sagi Grimberg
[not found] ` <CABnqofwo5xQGnuNTOzC+jsWqUmB-awS6nGWGK4SPPTDp=cFfeA@mail.gmail.com>
2018-10-15 21:19 ` Sagi Grimberg
[not found] ` <CABnqofzSqFaNxaU=byG1NGgBXXB_-qX-Zhq7EimwZQT4EN_KBg@mail.gmail.com>
[not found] ` <CABnqofy7bHCUu5n_n+j8bPrCrd7wnfrP3XSapA_UwQWsja6UAg@mail.gmail.com>
2018-10-19 1:14 ` Sagi Grimberg
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.