iSER with policy based routing error

iSER with policy based routing error

[Robert LeBlanc]

We are trying to leverage multiple cards/ports for iSER for
performance and resiliency reasons. The ports are configured with only
IPv6 addresses and each port is on a separate VLAN/subnet that is
routable to each other subnet. We are using rules with tables to set a
default gateway for each adapter/subnet based on the source IPv6
address (policy based routing). Using TCP for iSCSI, everything works
fine and traffic ingresses/egresses the right ports. However, when we
try using iSER, we get connection errors.

May 12 13:39:27 prv-0-14-roberttest kernel: iser: iser_connect:
rdma_resolve_addr failed: -101
May 12 13:39:27 prv-0-14-roberttest iscsid: Received iferror -101:
Network is unreachable.
May 12 13:39:27 prv-0-14-roberttest iscsid: cannot make a connection
to 2604:3140:40:300:0:580:d0:0:3260 (-101,0)

If we put a default gateway for IPv6 in the 'default' table, then iSER
is able to make a connection, but we can only use one port. It looks
as if iSER is not following the rules in the default routing table to
find the appropriate default gateway in a different table.

In iscsiadm, we are setting up an iface with iface.net_ifacename,
iface.ipaddress, and iface.transport_name appropriately and using that
iface with a default gateway in the 'default' routing table works, so
we feel confident that our problem is not with the iface
configuration.

Any ideas how to fix this? We are running CentOS 7.2 with 4.9.13 Linux
kernel, in-box drivers and ConnectX-4 LX 25 Gb cards.

Thanks.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Or Gerlitz]

On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
[...]
> If we put a default gateway for IPv6 in the 'default' table, then iSER
> is able to make a connection, but we can only use one port. It looks
> as if iSER is not following the rules in the default routing table to
> find the appropriate default gateway in a different table.

can you just try to make rdma connection with the rping utility? do
you see the same problem/s?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Robert LeBlanc]

On Sat, May 13, 2017 at 11:55 AM, Or Gerlitz <gerlitz.or-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> [...]
>> If we put a default gateway for IPv6 in the 'default' table, then iSER
>> is able to make a connection, but we can only use one port. It looks
>> as if iSER is not following the rules in the default routing table to
>> find the appropriate default gateway in a different table.
>
> can you just try to make rdma connection with the rping utility? do
> you see the same problem/s?

Using mlx4 cards with IB/RoCE, I am seeing the same connection problem
with rping. There seems to be a bug that prevents rping from working
with RoCE on mlx5.

I also found a thread between you and Mike Christie
https://groups.google.com/forum/#!topic/open-iscsi/IywifztV7Xs. It
doesn't say that things were all worked out about binding to dev/ip
address. Should iSER be able to bind to ip address (my preference), or
by device name (workable) to provide multipathing? Should this binding
be able to work with additional tables for finding the proper gateway?
We are intending to route all RoCE traffic over L3 boundaries as it
greatly simplifies the network configuration.

Here is the test that I ran to show the problem (it is also in the
"rdma_cm unable to find route on sub-table" thread):


----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Robert LeBlanc]

On Thu, May 18, 2017 at 12:51 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> On Sat, May 13, 2017 at 11:55 AM, Or Gerlitz <gerlitz.or-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>> [...]
>>> If we put a default gateway for IPv6 in the 'default' table, then iSER
>>> is able to make a connection, but we can only use one port. It looks
>>> as if iSER is not following the rules in the default routing table to
>>> find the appropriate default gateway in a different table.
>>
>> can you just try to make rdma connection with the rping utility? do
>> you see the same problem/s?
>
> Using mlx4 cards with IB/RoCE, I am seeing the same connection problem
> with rping. There seems to be a bug that prevents rping from working
> with RoCE on mlx5.
>
> I also found a thread between you and Mike Christie
> https://groups.google.com/forum/#!topic/open-iscsi/IywifztV7Xs. It
> doesn't say that things were all worked out about binding to dev/ip
> address. Should iSER be able to bind to ip address (my preference), or
> by device name (workable) to provide multipathing? Should this binding
> be able to work with additional tables for finding the proper gateway?
> We are intending to route all RoCE traffic over L3 boundaries as it
> greatly simplifies the network configuration.
>
> Here is the test that I ran to show the problem (it is also in the
> "rdma_cm unable to find route on sub-table" thread):

Test #1
-------
Using ConnectX-3 and Infiniband with two hosts back to back.

Server
------

# systemctl start opensm

# ip -6 addr add dev ib0 fd00::13/64

# ip link set ib0 up

# rping -s -a ::0
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10


Client
------

# ip -6 addr add dev ib0 fd00::14/64

# ip link set ib0 up

# ping6 fd00::13
PING fd00::13(fd00::13) 56 data bytes
64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.927 ms
64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.134 ms
^C
--- fd00::13 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.134/0.530/0.927/0.397 ms

# rping -c -a fd00::13
^C

Results #1
----------
Tests complete successfully


Test #2
-------
Built upon the configuration of Test #1

Server
------
# rping -s -a ::0

Client
------
# echo "100 r0" >> /etc/iproute2/rt_tables

# ip -6 route del fd00::/64 dev ib0

# ip -6 rule add from fd00::14 table r0

# ip -6 route add fd00::/64 dev ib0 table r0

# ping6 fd00::13
PING fd00::13(fd00::13) 56 data bytes
64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.158 ms
64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.136 ms
^C
--- fd00::13 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.136/0.147/0.158/0.011 ms

# rping -c -a fd00::13
rdma_resolve_addr: Network is unreachable

Results #2
----------
rping is unable to find a route to use

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Robert LeBlanc]

I could not replicate ping6 working today without passing the -I
parameter. I created https://github.com/linux-rdma/rdma-core/pull/136
and now you can pass -I to rping to have it bind to a source address
just like ping and the rules and additional routing table works for
rping.

However, iSER is not able to find a network route. Where do you think
I should look next? It seems that RDMA_CM is able to work properly if
given the right information.

Thanks,
Robert
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, May 18, 2017 at 12:52 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> On Thu, May 18, 2017 at 12:51 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>> On Sat, May 13, 2017 at 11:55 AM, Or Gerlitz <gerlitz.or-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>>> [...]
>>>> If we put a default gateway for IPv6 in the 'default' table, then iSER
>>>> is able to make a connection, but we can only use one port. It looks
>>>> as if iSER is not following the rules in the default routing table to
>>>> find the appropriate default gateway in a different table.
>>>
>>> can you just try to make rdma connection with the rping utility? do
>>> you see the same problem/s?
>>
>> Using mlx4 cards with IB/RoCE, I am seeing the same connection problem
>> with rping. There seems to be a bug that prevents rping from working
>> with RoCE on mlx5.
>>
>> I also found a thread between you and Mike Christie
>> https://groups.google.com/forum/#!topic/open-iscsi/IywifztV7Xs. It
>> doesn't say that things were all worked out about binding to dev/ip
>> address. Should iSER be able to bind to ip address (my preference), or
>> by device name (workable) to provide multipathing? Should this binding
>> be able to work with additional tables for finding the proper gateway?
>> We are intending to route all RoCE traffic over L3 boundaries as it
>> greatly simplifies the network configuration.
>>
>> Here is the test that I ran to show the problem (it is also in the
>> "rdma_cm unable to find route on sub-table" thread):
>
> Test #1
> -------
> Using ConnectX-3 and Infiniband with two hosts back to back.
>
> Server
> ------
>
> # systemctl start opensm
>
> # ip -6 addr add dev ib0 fd00::13/64
>
> # ip link set ib0 up
>
> # rping -s -a ::0
> server DISCONNECT EVENT...
> wait for RDMA_READ_ADV state 10
>
>
> Client
> ------
>
> # ip -6 addr add dev ib0 fd00::14/64
>
> # ip link set ib0 up
>
> # ping6 fd00::13
> PING fd00::13(fd00::13) 56 data bytes
> 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.927 ms
> 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.134 ms
> ^C
> --- fd00::13 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1002ms
> rtt min/avg/max/mdev = 0.134/0.530/0.927/0.397 ms
>
> # rping -c -a fd00::13
> ^C
>
> Results #1
> ----------
> Tests complete successfully
>
>
> Test #2
> -------
> Built upon the configuration of Test #1
>
> Server
> ------
> # rping -s -a ::0
>
> Client
> ------
> # echo "100 r0" >> /etc/iproute2/rt_tables
>
> # ip -6 route del fd00::/64 dev ib0
>
> # ip -6 rule add from fd00::14 table r0
>
> # ip -6 route add fd00::/64 dev ib0 table r0
>
> # ping6 fd00::13
> PING fd00::13(fd00::13) 56 data bytes
> 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.158 ms
> 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.136 ms
> ^C
> --- fd00::13 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1020ms
> rtt min/avg/max/mdev = 0.136/0.147/0.158/0.011 ms
>
> # rping -c -a fd00::13
> rdma_resolve_addr: Network is unreachable
>
> Results #2
> ----------
> rping is unable to find a route to use
>
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Robert LeBlanc]

On Tue, May 23, 2017 at 5:17 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> I could not replicate ping6 working today without passing the -I
> parameter. I created https://github.com/linux-rdma/rdma-core/pull/136
> and now you can pass -I to rping to have it bind to a source address
> just like ping and the rules and additional routing table works for
> rping.
>
> However, iSER is not able to find a network route. Where do you think
> I should look next? It seems that RDMA_CM is able to work properly if
> given the right information.
>
> Thanks,
> Robert
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>

Looks like the bug is in iSER, it is always passing NULL for the source address.

diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c
b/drivers/infiniband/ulp/iser/iscsi_iser.c
index 5a887ef..e9741df 100644
--- a/drivers/infiniband/ulp/iser/iscsi_iser.c
+++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
@@ -59,6 +59,7 @@
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/module.h>
+#include <linux/inet.h>

#include <net/sock.h>

@@ -814,6 +815,10 @@ static int iscsi_iser_get_ep_param(struct
iscsi_endpoint *ep,
       int err;
       struct iser_conn *iser_conn;
       struct iscsi_endpoint *ep;
+       struct sockaddr_in s_addr;
+       memset(&s_addr, 0, sizeof(s_addr));
+       s_addr.sin_family = AF_INET;
+       s_addr.sin_addr.s_addr = in_aton("192.168.13.14");

       ep = iscsi_create_endpoint(0);
       if (!ep)
@@ -829,7 +834,8 @@ static int iscsi_iser_get_ep_param(struct
iscsi_endpoint *ep,
       iser_conn->ep = ep;
       iser_conn_init(iser_conn);

-       err = iser_connect(iser_conn, NULL, dst_addr, non_blocking);
+       //err = iser_connect(iser_conn, NULL, dst_addr, non_blocking);
+       err = iser_connect(iser_conn, (struct sockaddr *) &s_addr,
dst_addr, non_blocking);
       if (err)
               goto failure;

diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c
b/drivers/infiniband/ulp/iser/iser_verbs.c
index c538a38..6ce1845 100644
--- a/drivers/infiniband/ulp/iser/iser_verbs.c
+++ b/drivers/infiniband/ulp/iser/iser_verbs.c
@@ -953,6 +953,7 @@ int iser_connect(struct iser_conn   *iser_conn,
       sprintf(iser_conn->name, "%pISp", dst_addr);

       iser_info("connecting to: %s\n", iser_conn->name);
+       iser_err("connecting from: %p\n", src_addr);

       /* the device is known only --after-- address resolution */
       ib_conn->device = NULL;

Hard coding the source IP address allows the connection to work fine.
It seems for some reason, it is not using the information from the
iface that we configured. I can try to create a patch, but I'm not
sure where to query the ifaces for the required info. If you can point
me in the right place, I can give it a shot.

Thanks,
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Jason Gunthorpe]

On Wed, May 24, 2017 at 12:14:53PM -0600, Robert LeBlanc wrote:
> On Tue, May 23, 2017 at 5:17 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> > I could not replicate ping6 working today without passing the -I
> > parameter. I created https://github.com/linux-rdma/rdma-core/pull/136

It shouldn't have worked, the route cache probably needed flushing
after configuring policy routing.

> Hard coding the source IP address allows the connection to work fine.
> It seems for some reason, it is not using the information from the
> iface that we configured.

Only information provided by the user can be used for route lookup.

Do not try an extract a source address from some other information..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Robert LeBlanc]

On Wed, May 24, 2017 at 1:25 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, May 24, 2017 at 12:14:53PM -0600, Robert LeBlanc wrote:
>> On Tue, May 23, 2017 at 5:17 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>> > I could not replicate ping6 working today without passing the -I
>> > parameter. I created https://github.com/linux-rdma/rdma-core/pull/136
>
> It shouldn't have worked, the route cache probably needed flushing
> after configuring policy routing.

What I'm thinking happened is I tried the test too quickly after
modifing the route tables, so something was cached that made it work.

>> Hard coding the source IP address allows the connection to work fine.
>> It seems for some reason, it is not using the information from the
>> iface that we configured.
>
> Only information provided by the user can be used for route lookup.

I'm trying to provide this information through iscsiadm ifaces, but
iSER completely ignores it.

# iscsiadm -m iface -I eth0iser -o new
# iscsiadm -m iface -I eth0iser -o update -n iface.ipaddress -v 192.168.13.14
# iscsiadm -m iface -I eth0iser -o update -n iface.transport_name -v iser
# iscsiadm -m discovery -t st -p 192.168.13.13 -I eth0iser

My routing information:

# ip route
default via 10.64.0.1 dev eno1 proto static metric 100
10.64.0.0/10 dev eno1 proto kernel scope link src 10.91.0.14 metric 100

# ip rule
0:      from all lookup local
32765:  from 192.168.13.14 lookup r0
32766:  from all lookup main
32767:  from all lookup default

# ip route show table r0
192.168.13.0/24 dev eth0 scope link

The target has the IP 192.168.13.13 and the route in the default
routing table (automatic entries not modified by me for simplicity).

If I take out the hardcoded 'NULL' in iser_connect() and put in a
hardcoded source IP address, just to test, then the connection is set
up properly. That tells me that iser_connect() knows what to do and
how to do it, it just never tries to use a source address if supplied
through the iface configuration.

> Do not try an extract a source address from some other information..

I want to get the source address (iface.ipaddress) from the iface
(eth0iser in this case) and pass that into iser_connect(). I'm just
not sure how to do that and where the best place to do that is. That
is what I need some help understanding. Any push in the right
direction is appreciated. Is the iface info parsed by the base iscsi
code and then put into a struct that I can query? I sure hope it is
that easy, but I'm having difficulty locating that code.

Thanks,
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Jason Gunthorpe]

On Wed, May 24, 2017 at 01:41:25PM -0600, Robert LeBlanc wrote:

> > Do not try an extract a source address from some other information..
> 
> I want to get the source address (iface.ipaddress) from the iface
> (eth0iser in this case) and pass that into iser_connect().

That makes sense, I can't help you figure out where iface.ipaddress
appears in the kernel though :)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Or Gerlitz]

On Wed, May 24, 2017 at 10:41 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
[...]
> I want to get the source address (iface.ipaddress) from the iface
> (eth0iser in this case) and pass that into iser_connect(). I'm just
> not sure how to do that and where the best place to do that is. That
> is what I need some help understanding. Any push in the right
> direction is appreciated. Is the iface info parsed by the base iscsi
> code and then put into a struct that I can query? I sure hope it is
> that easy, but I'm having difficulty locating that code.

If you get the source address into the iscsi initiator UAPI it would
be very simple kernel
code change to get that to iser_connect --> rdma_resolve_address as
you indicated.

Currently only the dst address is passed from user-space, to see it
clone [1] and look

$ vim usr/netlink.c +/ktransport_ep_connect

see  there

memcpy(setparam_buf + sizeof(*ev), dst_addr, addrlen);

for the kernel side of things look here

$ vim drivers/scsi/scsi_transport_iscsi.c +/ep_connect

see there

dst_addr = (struct sockaddr *)((char*)ev + sizeof(*ev));
ep = transport->ep_connect(shost, dst_addr, non_blocking);


Or.


[1] https://github.com/open-iscsi/open-iscsi.git
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Sagi Grimberg]

Hi Robert,

> We are trying to leverage multiple cards/ports for iSER for
> performance and resiliency reasons. The ports are configured with only
> IPv6 addresses and each port is on a separate VLAN/subnet that is
> routable to each other subnet. We are using rules with tables to set a
> default gateway for each adapter/subnet based on the source IPv6
> address (policy based routing). Using TCP for iSCSI, everything works
> fine and traffic ingresses/egresses the right ports. However, when we
> try using iSER, we get connection errors.
>
> May 12 13:39:27 prv-0-14-roberttest kernel: iser: iser_connect:
> rdma_resolve_addr failed: -101
> May 12 13:39:27 prv-0-14-roberttest iscsid: Received iferror -101:
> Network is unreachable.
> May 12 13:39:27 prv-0-14-roberttest iscsid: cannot make a connection
> to 2604:3140:40:300:0:580:d0:0:3260 (-101,0)

This looks 100% rdma_cm to me. iser is completely agnostic to address
families and routes.

> If we put a default gateway for IPv6 in the 'default' table, then iSER
> is able to make a connection, but we can only use one port. It looks
> as if iSER is not following the rules in the default routing table to
> find the appropriate default gateway in a different table.

As I said, iser relies on rdma_cm for routing decisions.
I would suspect that all rdma_cm based protocols to be
affected as well (nfs, nvmf).

Did you check plain rping like Or suggested?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Robert LeBlanc]

We are getting errors with rping too, even on the same IPv4 subnet.
All userland RDMA programs seem to be failing. It is like there is a
missing library or something. We installed "Infiniband Support" group
install, so I'm not sure what could be missing.

# rping -s -a 0.0.0.0
Segmentation fault

>From dmesg:
[Mon May 15 14:28:36 2017] rping[3289]: segfault at 18 ip
00007f0142ca9a34 sp 00007ffd7f0a8cc0 error 4 in
libibverbs.so.1.0.0[7f0142c9e000+11000]

# gdb rping core.3289
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/rping...Reading symbols from
/usr/lib/debug/usr/bin/rping.debug...done.
done.
[New LWP 3289]
[New LWP 3292]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rping -s -a 0.0.0.0'.
Program terminated with signal 11, Segmentation fault.
#0 __ibv_alloc_pd (context=0x0) at src/verbs.c:196
196 pd = context->ops.alloc_pd(context);
Missing separate debuginfos, use: debuginfo-install
libcxgb3-1.3.1-8.el7.x86_64 libcxgb4-1.3.5-3.el7.x86_64
libipathverbs-1.3-2.el7.x86_64 libmlx4-1.0.6-5.el7.x86_64
libmlx5-1.0.2-1.el7.x86_64 libmthca-1.0.6-13.el7.x86_64 libne
s-1.1.4-2.el7.x86_64 libnl3-3.2.21-10.el7.x86_64
(gdb) bt
#0 __ibv_alloc_pd (context=0x0) at src/verbs.c:196
#1 0x0000563bc5c1f5c6 in rping_setup_qp (cb=cb@entry=0x563bc61f9780,
cm_id=<optimized out>)
at examples/rping.c:519
#2 0x0000563bc5c1de5a in rping_run_server (cb=0x563bc61f9780) at
examples/rping.c:890
#3 main (argc=4, argv=0x7ffd7f0a8ee8) at examples/rping.c:1268
(gdb) f 0
#0 __ibv_alloc_pd (context=0x0) at src/verbs.c:196
196 pd = context->ops.alloc_pd(context);
(gdb) list
191
192 struct ibv_pd *__ibv_alloc_pd(struct ibv_context *context)
193 {
194 struct ibv_pd *pd;
195
196 pd = context->ops.alloc_pd(context);
197 if (pd)
198 pd->context = context;
199
200 return pd;
(gdb) p context
$1 = (struct ibv_context *) 0x0
(gdb)



# rping -c -a 192.168.0.13
cma event RDMA_CM_EVENT_REJECTED, error 28
wait for CONNECTED state 4
connect error -1
Segmentation fault

>From dmesg
[Mon May 15 14:27:24 2017] rping[3075]: segfault at 7f2386c800a8 ip
00007f2386672adf sp 00007ffd40e5df60 error 4 in
libibverbs.so.1.0.0[7f2386667000+11000]

# gdb rping core.3075
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/rping...Reading symbols from
/usr/lib/debug/usr/bin/rping.debug...done.
done.
[New LWP 3075]
[New LWP 3078]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rping -c -a 192.168.0.13'.
Program terminated with signal 11, Segmentation fault.
#0 __ibv_dereg_mr (mr=0x5624b478c740) at src/verbs.c:237
237 ret = mr->context->ops.dereg_mr(mr);
Missing separate debuginfos, use: debuginfo-install
libcxgb3-1.3.1-8.el7.x86_64 libcxgb4-1.3.5-3.el7.x86_64
libgcc-4.8.5-4.el7.x86_64 libipathverbs-1.3-2.el7.x86_64
libmlx4-1.0.6-5.el7.x86_64 libmlx5-1.0.2-1.el7.x86_64 libmthca
-1.0.6-13.el7.x86_64 libnes-1.1.4-2.el7.x86_64 libnl3-3.2.21-10.el7.x86_64
(gdb) bt
#0 __ibv_dereg_mr (mr=0x5624b478c740) at src/verbs.c:237
#1 0x00005624b2b618a7 in rping_free_buffers (cb=0x5624b4786780) at
examples/rping.c:470
#2 0x00005624b2b5fef3 in rping_run_client (cb=<optimized out>) at
examples/rping.c:1111
#3 main (argc=<optimized out>, argv=<optimized out>) at examples/rping.c:1270
(gdb) f 0
#0 __ibv_dereg_mr (mr=0x5624b478c740) at src/verbs.c:237
237 ret = mr->context->ops.dereg_mr(mr);
(gdb) list
232 {
233 int ret;
234 void *addr = mr->addr;
235 size_t length = mr->length;
236
237 ret = mr->context->ops.dereg_mr(mr);
238 if (!ret)
239 ibv_dofork_range(addr, length);
240
241 return ret;
(gdb) p *mr
$1 = {context = 0x7f2386c80070, pd = 0x5624b4789f30, addr =
0x5624b47867e8, length = 16, handle = 0,
lkey = 162608, rkey = 162608}
(gdb) p *mr->context
Cannot access memory at address 0x7f2386c80070
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Mon, May 15, 2017 at 4:39 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> Hi Robert,
>
>> We are trying to leverage multiple cards/ports for iSER for
>> performance and resiliency reasons. The ports are configured with only
>> IPv6 addresses and each port is on a separate VLAN/subnet that is
>> routable to each other subnet. We are using rules with tables to set a
>> default gateway for each adapter/subnet based on the source IPv6
>> address (policy based routing). Using TCP for iSCSI, everything works
>> fine and traffic ingresses/egresses the right ports. However, when we
>> try using iSER, we get connection errors.
>>
>> May 12 13:39:27 prv-0-14-roberttest kernel: iser: iser_connect:
>> rdma_resolve_addr failed: -101
>> May 12 13:39:27 prv-0-14-roberttest iscsid: Received iferror -101:
>> Network is unreachable.
>> May 12 13:39:27 prv-0-14-roberttest iscsid: cannot make a connection
>> to 2604:3140:40:300:0:580:d0:0:3260 (-101,0)
>
>
> This looks 100% rdma_cm to me. iser is completely agnostic to address
> families and routes.
>
>> If we put a default gateway for IPv6 in the 'default' table, then iSER
>> is able to make a connection, but we can only use one port. It looks
>> as if iSER is not following the rules in the default routing table to
>> find the appropriate default gateway in a different table.
>
>
> As I said, iser relies on rdma_cm for routing decisions.
> I would suspect that all rdma_cm based protocols to be
> affected as well (nfs, nvmf).
>
> Did you check plain rping like Or suggested?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Sagi Grimberg]

> We are getting errors with rping too, even on the same IPv4 subnet. All
> RDMA programs seem to be failing. It is like there is a missing library
> or something. We installed "Infiniband Support" group install, so I'm
> not sure what could be missing.

I assume you are running a custom kernel? This looks like a ABI
breakage.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER with policy based routing error

[Robert LeBlanc]

On Tue, May 16, 2017 at 1:19 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
>
>> We are getting errors with rping too, even on the same IPv4 subnet. All
>> RDMA programs seem to be failing. It is like there is a missing library
>> or something. We installed "Infiniband Support" group install, so I'm
>> not sure what could be missing.
>
>
> I assume you are running a custom kernel? This looks like a ABI
> breakage.

This is a custom kernel. I just built 4.9.28 this morning and made
sure to install the uAPI headers (make headers_install
INSTALL_HDR_PATH=/usr). Is there something that I'm missing? Do I need
to rebuild rping/libibverbs/etc?

Thanks

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html