* iSER with policy based routing error @ 2017-05-12 21:02 Robert LeBlanc [not found] ` <CAANLjFoB3V6_exH8TnQQobaHkz2K1Lxj0sVOVa2Q+MKAsCqCKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Robert LeBlanc @ 2017-05-12 21:02 UTC (permalink / raw) To: linux-rdma We are trying to leverage multiple cards/ports for iSER for performance and resiliency reasons. The ports are configured with only IPv6 addresses and each port is on a separate VLAN/subnet that is routable to each other subnet. We are using rules with tables to set a default gateway for each adapter/subnet based on the source IPv6 address (policy based routing). Using TCP for iSCSI, everything works fine and traffic ingresses/egresses the right ports. However, when we try using iSER, we get connection errors. May 12 13:39:27 prv-0-14-roberttest kernel: iser: iser_connect: rdma_resolve_addr failed: -101 May 12 13:39:27 prv-0-14-roberttest iscsid: Received iferror -101: Network is unreachable. May 12 13:39:27 prv-0-14-roberttest iscsid: cannot make a connection to 2604:3140:40:300:0:580:d0:0:3260 (-101,0) If we put a default gateway for IPv6 in the 'default' table, then iSER is able to make a connection, but we can only use one port. It looks as if iSER is not following the rules in the default routing table to find the appropriate default gateway in a different table. In iscsiadm, we are setting up an iface with iface.net_ifacename, iface.ipaddress, and iface.transport_name appropriately and using that iface with a default gateway in the 'default' routing table works, so we feel confident that our problem is not with the iface configuration. Any ideas how to fix this? We are running CentOS 7.2 with 4.9.13 Linux kernel, in-box drivers and ConnectX-4 LX 25 Gb cards. Thanks. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAANLjFoB3V6_exH8TnQQobaHkz2K1Lxj0sVOVa2Q+MKAsCqCKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAANLjFoB3V6_exH8TnQQobaHkz2K1Lxj0sVOVa2Q+MKAsCqCKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-13 17:55 ` Or Gerlitz [not found] ` <CAJ3xEMiYwGDott_gLrh7e4tG9Ti71vftMPuvyT3+czu7jWwqFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-15 10:39 ` Sagi Grimberg 1 sibling, 1 reply; 14+ messages in thread From: Or Gerlitz @ 2017-05-13 17:55 UTC (permalink / raw) To: Robert LeBlanc; +Cc: linux-rdma On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: [...] > If we put a default gateway for IPv6 in the 'default' table, then iSER > is able to make a connection, but we can only use one port. It looks > as if iSER is not following the rules in the default routing table to > find the appropriate default gateway in a different table. can you just try to make rdma connection with the rping utility? do you see the same problem/s? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAJ3xEMiYwGDott_gLrh7e4tG9Ti71vftMPuvyT3+czu7jWwqFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAJ3xEMiYwGDott_gLrh7e4tG9Ti71vftMPuvyT3+czu7jWwqFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-18 18:51 ` Robert LeBlanc [not found] ` <CAANLjFqNkNVDiGr-yW9y6XCiWE_O7+OewmxWM+xmWHQ39n4wkw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Robert LeBlanc @ 2017-05-18 18:51 UTC (permalink / raw) To: Or Gerlitz; +Cc: linux-rdma On Sat, May 13, 2017 at 11:55 AM, Or Gerlitz <gerlitz.or-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: > [...] >> If we put a default gateway for IPv6 in the 'default' table, then iSER >> is able to make a connection, but we can only use one port. It looks >> as if iSER is not following the rules in the default routing table to >> find the appropriate default gateway in a different table. > > can you just try to make rdma connection with the rping utility? do > you see the same problem/s? Using mlx4 cards with IB/RoCE, I am seeing the same connection problem with rping. There seems to be a bug that prevents rping from working with RoCE on mlx5. I also found a thread between you and Mike Christie https://groups.google.com/forum/#!topic/open-iscsi/IywifztV7Xs. It doesn't say that things were all worked out about binding to dev/ip address. Should iSER be able to bind to ip address (my preference), or by device name (workable) to provide multipathing? Should this binding be able to work with additional tables for finding the proper gateway? We are intending to route all RoCE traffic over L3 boundaries as it greatly simplifies the network configuration. Here is the test that I ran to show the problem (it is also in the "rdma_cm unable to find route on sub-table" thread): ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAANLjFqNkNVDiGr-yW9y6XCiWE_O7+OewmxWM+xmWHQ39n4wkw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAANLjFqNkNVDiGr-yW9y6XCiWE_O7+OewmxWM+xmWHQ39n4wkw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-18 18:52 ` Robert LeBlanc [not found] ` <CAANLjFpKGoruxvwhtHwF-fBH+dLT9RKqkd3POYwLVK856WEfFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Robert LeBlanc @ 2017-05-18 18:52 UTC (permalink / raw) To: Or Gerlitz; +Cc: linux-rdma On Thu, May 18, 2017 at 12:51 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: > On Sat, May 13, 2017 at 11:55 AM, Or Gerlitz <gerlitz.or-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: >> [...] >>> If we put a default gateway for IPv6 in the 'default' table, then iSER >>> is able to make a connection, but we can only use one port. It looks >>> as if iSER is not following the rules in the default routing table to >>> find the appropriate default gateway in a different table. >> >> can you just try to make rdma connection with the rping utility? do >> you see the same problem/s? > > Using mlx4 cards with IB/RoCE, I am seeing the same connection problem > with rping. There seems to be a bug that prevents rping from working > with RoCE on mlx5. > > I also found a thread between you and Mike Christie > https://groups.google.com/forum/#!topic/open-iscsi/IywifztV7Xs. It > doesn't say that things were all worked out about binding to dev/ip > address. Should iSER be able to bind to ip address (my preference), or > by device name (workable) to provide multipathing? Should this binding > be able to work with additional tables for finding the proper gateway? > We are intending to route all RoCE traffic over L3 boundaries as it > greatly simplifies the network configuration. > > Here is the test that I ran to show the problem (it is also in the > "rdma_cm unable to find route on sub-table" thread): Test #1 ------- Using ConnectX-3 and Infiniband with two hosts back to back. Server ------ # systemctl start opensm # ip -6 addr add dev ib0 fd00::13/64 # ip link set ib0 up # rping -s -a ::0 server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10 Client ------ # ip -6 addr add dev ib0 fd00::14/64 # ip link set ib0 up # ping6 fd00::13 PING fd00::13(fd00::13) 56 data bytes 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.927 ms 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.134 ms ^C --- fd00::13 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 0.134/0.530/0.927/0.397 ms # rping -c -a fd00::13 ^C Results #1 ---------- Tests complete successfully Test #2 ------- Built upon the configuration of Test #1 Server ------ # rping -s -a ::0 Client ------ # echo "100 r0" >> /etc/iproute2/rt_tables # ip -6 route del fd00::/64 dev ib0 # ip -6 rule add from fd00::14 table r0 # ip -6 route add fd00::/64 dev ib0 table r0 # ping6 fd00::13 PING fd00::13(fd00::13) 56 data bytes 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.158 ms 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.136 ms ^C --- fd00::13 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1020ms rtt min/avg/max/mdev = 0.136/0.147/0.158/0.011 ms # rping -c -a fd00::13 rdma_resolve_addr: Network is unreachable Results #2 ---------- rping is unable to find a route to use ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAANLjFpKGoruxvwhtHwF-fBH+dLT9RKqkd3POYwLVK856WEfFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAANLjFpKGoruxvwhtHwF-fBH+dLT9RKqkd3POYwLVK856WEfFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-23 23:17 ` Robert LeBlanc [not found] ` <CAANLjFqhPUd=kyuv0CBR69KuAwGZzne8ZY3-1J6YiNHuWqZBZA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Robert LeBlanc @ 2017-05-23 23:17 UTC (permalink / raw) To: Or Gerlitz, Sagi Grimberg; +Cc: linux-rdma I could not replicate ping6 working today without passing the -I parameter. I created https://github.com/linux-rdma/rdma-core/pull/136 and now you can pass -I to rping to have it bind to a source address just like ping and the rules and additional routing table works for rping. However, iSER is not able to find a network route. Where do you think I should look next? It seems that RDMA_CM is able to work properly if given the right information. Thanks, Robert ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, May 18, 2017 at 12:52 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: > On Thu, May 18, 2017 at 12:51 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: >> On Sat, May 13, 2017 at 11:55 AM, Or Gerlitz <gerlitz.or-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >>> On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: >>> [...] >>>> If we put a default gateway for IPv6 in the 'default' table, then iSER >>>> is able to make a connection, but we can only use one port. It looks >>>> as if iSER is not following the rules in the default routing table to >>>> find the appropriate default gateway in a different table. >>> >>> can you just try to make rdma connection with the rping utility? do >>> you see the same problem/s? >> >> Using mlx4 cards with IB/RoCE, I am seeing the same connection problem >> with rping. There seems to be a bug that prevents rping from working >> with RoCE on mlx5. >> >> I also found a thread between you and Mike Christie >> https://groups.google.com/forum/#!topic/open-iscsi/IywifztV7Xs. It >> doesn't say that things were all worked out about binding to dev/ip >> address. Should iSER be able to bind to ip address (my preference), or >> by device name (workable) to provide multipathing? Should this binding >> be able to work with additional tables for finding the proper gateway? >> We are intending to route all RoCE traffic over L3 boundaries as it >> greatly simplifies the network configuration. >> >> Here is the test that I ran to show the problem (it is also in the >> "rdma_cm unable to find route on sub-table" thread): > > Test #1 > ------- > Using ConnectX-3 and Infiniband with two hosts back to back. > > Server > ------ > > # systemctl start opensm > > # ip -6 addr add dev ib0 fd00::13/64 > > # ip link set ib0 up > > # rping -s -a ::0 > server DISCONNECT EVENT... > wait for RDMA_READ_ADV state 10 > > > Client > ------ > > # ip -6 addr add dev ib0 fd00::14/64 > > # ip link set ib0 up > > # ping6 fd00::13 > PING fd00::13(fd00::13) 56 data bytes > 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.927 ms > 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.134 ms > ^C > --- fd00::13 ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 1002ms > rtt min/avg/max/mdev = 0.134/0.530/0.927/0.397 ms > > # rping -c -a fd00::13 > ^C > > Results #1 > ---------- > Tests complete successfully > > > Test #2 > ------- > Built upon the configuration of Test #1 > > Server > ------ > # rping -s -a ::0 > > Client > ------ > # echo "100 r0" >> /etc/iproute2/rt_tables > > # ip -6 route del fd00::/64 dev ib0 > > # ip -6 rule add from fd00::14 table r0 > > # ip -6 route add fd00::/64 dev ib0 table r0 > > # ping6 fd00::13 > PING fd00::13(fd00::13) 56 data bytes > 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.158 ms > 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.136 ms > ^C > --- fd00::13 ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 1020ms > rtt min/avg/max/mdev = 0.136/0.147/0.158/0.011 ms > > # rping -c -a fd00::13 > rdma_resolve_addr: Network is unreachable > > Results #2 > ---------- > rping is unable to find a route to use > > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAANLjFqhPUd=kyuv0CBR69KuAwGZzne8ZY3-1J6YiNHuWqZBZA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAANLjFqhPUd=kyuv0CBR69KuAwGZzne8ZY3-1J6YiNHuWqZBZA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-24 18:14 ` Robert LeBlanc [not found] ` <CAANLjFpHbMXjMnxO0B7sn9rRz2Sn3CEx-bEWKW5BrYj3itQ0HQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Robert LeBlanc @ 2017-05-24 18:14 UTC (permalink / raw) To: Or Gerlitz, Sagi Grimberg; +Cc: linux-rdma On Tue, May 23, 2017 at 5:17 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: > I could not replicate ping6 working today without passing the -I > parameter. I created https://github.com/linux-rdma/rdma-core/pull/136 > and now you can pass -I to rping to have it bind to a source address > just like ping and the rules and additional routing table works for > rping. > > However, iSER is not able to find a network route. Where do you think > I should look next? It seems that RDMA_CM is able to work properly if > given the right information. > > Thanks, > Robert > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > Looks like the bug is in iSER, it is always passing NULL for the source address. diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c index 5a887ef..e9741df 100644 --- a/drivers/infiniband/ulp/iser/iscsi_iser.c +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c @@ -59,6 +59,7 @@ #include <linux/delay.h> #include <linux/slab.h> #include <linux/module.h> +#include <linux/inet.h> #include <net/sock.h> @@ -814,6 +815,10 @@ static int iscsi_iser_get_ep_param(struct iscsi_endpoint *ep, int err; struct iser_conn *iser_conn; struct iscsi_endpoint *ep; + struct sockaddr_in s_addr; + memset(&s_addr, 0, sizeof(s_addr)); + s_addr.sin_family = AF_INET; + s_addr.sin_addr.s_addr = in_aton("192.168.13.14"); ep = iscsi_create_endpoint(0); if (!ep) @@ -829,7 +834,8 @@ static int iscsi_iser_get_ep_param(struct iscsi_endpoint *ep, iser_conn->ep = ep; iser_conn_init(iser_conn); - err = iser_connect(iser_conn, NULL, dst_addr, non_blocking); + //err = iser_connect(iser_conn, NULL, dst_addr, non_blocking); + err = iser_connect(iser_conn, (struct sockaddr *) &s_addr, dst_addr, non_blocking); if (err) goto failure; diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c index c538a38..6ce1845 100644 --- a/drivers/infiniband/ulp/iser/iser_verbs.c +++ b/drivers/infiniband/ulp/iser/iser_verbs.c @@ -953,6 +953,7 @@ int iser_connect(struct iser_conn *iser_conn, sprintf(iser_conn->name, "%pISp", dst_addr); iser_info("connecting to: %s\n", iser_conn->name); + iser_err("connecting from: %p\n", src_addr); /* the device is known only --after-- address resolution */ ib_conn->device = NULL; Hard coding the source IP address allows the connection to work fine. It seems for some reason, it is not using the information from the iface that we configured. I can try to create a patch, but I'm not sure where to query the ifaces for the required info. If you can point me in the right place, I can give it a shot. Thanks, ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 14+ messages in thread
[parent not found: <CAANLjFpHbMXjMnxO0B7sn9rRz2Sn3CEx-bEWKW5BrYj3itQ0HQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAANLjFpHbMXjMnxO0B7sn9rRz2Sn3CEx-bEWKW5BrYj3itQ0HQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-24 19:25 ` Jason Gunthorpe [not found] ` <20170524192550.GB25200-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Jason Gunthorpe @ 2017-05-24 19:25 UTC (permalink / raw) To: Robert LeBlanc; +Cc: Or Gerlitz, Sagi Grimberg, linux-rdma On Wed, May 24, 2017 at 12:14:53PM -0600, Robert LeBlanc wrote: > On Tue, May 23, 2017 at 5:17 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: > > I could not replicate ping6 working today without passing the -I > > parameter. I created https://github.com/linux-rdma/rdma-core/pull/136 It shouldn't have worked, the route cache probably needed flushing after configuring policy routing. > Hard coding the source IP address allows the connection to work fine. > It seems for some reason, it is not using the information from the > iface that we configured. Only information provided by the user can be used for route lookup. Do not try an extract a source address from some other information.. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <20170524192550.GB25200-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <20170524192550.GB25200-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-05-24 19:41 ` Robert LeBlanc [not found] ` <CAANLjFpZU6GodD-HAaTtive+dQZs4Gfgpw=u-1=UgC4zDTryKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Robert LeBlanc @ 2017-05-24 19:41 UTC (permalink / raw) To: Jason Gunthorpe; +Cc: Or Gerlitz, Sagi Grimberg, linux-rdma On Wed, May 24, 2017 at 1:25 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote: > On Wed, May 24, 2017 at 12:14:53PM -0600, Robert LeBlanc wrote: >> On Tue, May 23, 2017 at 5:17 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: >> > I could not replicate ping6 working today without passing the -I >> > parameter. I created https://github.com/linux-rdma/rdma-core/pull/136 > > It shouldn't have worked, the route cache probably needed flushing > after configuring policy routing. What I'm thinking happened is I tried the test too quickly after modifing the route tables, so something was cached that made it work. >> Hard coding the source IP address allows the connection to work fine. >> It seems for some reason, it is not using the information from the >> iface that we configured. > > Only information provided by the user can be used for route lookup. I'm trying to provide this information through iscsiadm ifaces, but iSER completely ignores it. # iscsiadm -m iface -I eth0iser -o new # iscsiadm -m iface -I eth0iser -o update -n iface.ipaddress -v 192.168.13.14 # iscsiadm -m iface -I eth0iser -o update -n iface.transport_name -v iser # iscsiadm -m discovery -t st -p 192.168.13.13 -I eth0iser My routing information: # ip route default via 10.64.0.1 dev eno1 proto static metric 100 10.64.0.0/10 dev eno1 proto kernel scope link src 10.91.0.14 metric 100 # ip rule 0: from all lookup local 32765: from 192.168.13.14 lookup r0 32766: from all lookup main 32767: from all lookup default # ip route show table r0 192.168.13.0/24 dev eth0 scope link The target has the IP 192.168.13.13 and the route in the default routing table (automatic entries not modified by me for simplicity). If I take out the hardcoded 'NULL' in iser_connect() and put in a hardcoded source IP address, just to test, then the connection is set up properly. That tells me that iser_connect() knows what to do and how to do it, it just never tries to use a source address if supplied through the iface configuration. > Do not try an extract a source address from some other information.. I want to get the source address (iface.ipaddress) from the iface (eth0iser in this case) and pass that into iser_connect(). I'm just not sure how to do that and where the best place to do that is. That is what I need some help understanding. Any push in the right direction is appreciated. Is the iface info parsed by the base iscsi code and then put into a struct that I can query? I sure hope it is that easy, but I'm having difficulty locating that code. Thanks, ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAANLjFpZU6GodD-HAaTtive+dQZs4Gfgpw=u-1=UgC4zDTryKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAANLjFpZU6GodD-HAaTtive+dQZs4Gfgpw=u-1=UgC4zDTryKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-24 21:13 ` Jason Gunthorpe 2017-05-25 16:03 ` Or Gerlitz 1 sibling, 0 replies; 14+ messages in thread From: Jason Gunthorpe @ 2017-05-24 21:13 UTC (permalink / raw) To: Robert LeBlanc; +Cc: Or Gerlitz, Sagi Grimberg, linux-rdma On Wed, May 24, 2017 at 01:41:25PM -0600, Robert LeBlanc wrote: > > Do not try an extract a source address from some other information.. > > I want to get the source address (iface.ipaddress) from the iface > (eth0iser in this case) and pass that into iser_connect(). That makes sense, I can't help you figure out where iface.ipaddress appears in the kernel though :) Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: iSER with policy based routing error [not found] ` <CAANLjFpZU6GodD-HAaTtive+dQZs4Gfgpw=u-1=UgC4zDTryKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-24 21:13 ` Jason Gunthorpe @ 2017-05-25 16:03 ` Or Gerlitz 1 sibling, 0 replies; 14+ messages in thread From: Or Gerlitz @ 2017-05-25 16:03 UTC (permalink / raw) To: Robert LeBlanc; +Cc: Jason Gunthorpe, Sagi Grimberg, linux-rdma On Wed, May 24, 2017 at 10:41 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: [...] > I want to get the source address (iface.ipaddress) from the iface > (eth0iser in this case) and pass that into iser_connect(). I'm just > not sure how to do that and where the best place to do that is. That > is what I need some help understanding. Any push in the right > direction is appreciated. Is the iface info parsed by the base iscsi > code and then put into a struct that I can query? I sure hope it is > that easy, but I'm having difficulty locating that code. If you get the source address into the iscsi initiator UAPI it would be very simple kernel code change to get that to iser_connect --> rdma_resolve_address as you indicated. Currently only the dst address is passed from user-space, to see it clone [1] and look $ vim usr/netlink.c +/ktransport_ep_connect see there memcpy(setparam_buf + sizeof(*ev), dst_addr, addrlen); for the kernel side of things look here $ vim drivers/scsi/scsi_transport_iscsi.c +/ep_connect see there dst_addr = (struct sockaddr *)((char*)ev + sizeof(*ev)); ep = transport->ep_connect(shost, dst_addr, non_blocking); Or. [1] https://github.com/open-iscsi/open-iscsi.git -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: iSER with policy based routing error [not found] ` <CAANLjFoB3V6_exH8TnQQobaHkz2K1Lxj0sVOVa2Q+MKAsCqCKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-13 17:55 ` Or Gerlitz @ 2017-05-15 10:39 ` Sagi Grimberg [not found] ` <71232433-a3d1-1bc0-a995-ae32fc05913f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> [not found] ` <CAANLjFogRE3ShHoUtGG1DPt5VzxBe8SBOAL86xc+AebQLBomJg@mail.gmail.com> 1 sibling, 2 replies; 14+ messages in thread From: Sagi Grimberg @ 2017-05-15 10:39 UTC (permalink / raw) To: Robert LeBlanc, linux-rdma Hi Robert, > We are trying to leverage multiple cards/ports for iSER for > performance and resiliency reasons. The ports are configured with only > IPv6 addresses and each port is on a separate VLAN/subnet that is > routable to each other subnet. We are using rules with tables to set a > default gateway for each adapter/subnet based on the source IPv6 > address (policy based routing). Using TCP for iSCSI, everything works > fine and traffic ingresses/egresses the right ports. However, when we > try using iSER, we get connection errors. > > May 12 13:39:27 prv-0-14-roberttest kernel: iser: iser_connect: > rdma_resolve_addr failed: -101 > May 12 13:39:27 prv-0-14-roberttest iscsid: Received iferror -101: > Network is unreachable. > May 12 13:39:27 prv-0-14-roberttest iscsid: cannot make a connection > to 2604:3140:40:300:0:580:d0:0:3260 (-101,0) This looks 100% rdma_cm to me. iser is completely agnostic to address families and routes. > If we put a default gateway for IPv6 in the 'default' table, then iSER > is able to make a connection, but we can only use one port. It looks > as if iSER is not following the rules in the default routing table to > find the appropriate default gateway in a different table. As I said, iser relies on rdma_cm for routing decisions. I would suspect that all rdma_cm based protocols to be affected as well (nfs, nvmf). Did you check plain rping like Or suggested? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <71232433-a3d1-1bc0-a995-ae32fc05913f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <71232433-a3d1-1bc0-a995-ae32fc05913f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2017-05-16 3:33 ` Robert LeBlanc 0 siblings, 0 replies; 14+ messages in thread From: Robert LeBlanc @ 2017-05-16 3:33 UTC (permalink / raw) To: Sagi Grimberg; +Cc: linux-rdma We are getting errors with rping too, even on the same IPv4 subnet. All userland RDMA programs seem to be failing. It is like there is a missing library or something. We installed "Infiniband Support" group install, so I'm not sure what could be missing. # rping -s -a 0.0.0.0 Segmentation fault >From dmesg: [Mon May 15 14:28:36 2017] rping[3289]: segfault at 18 ip 00007f0142ca9a34 sp 00007ffd7f0a8cc0 error 4 in libibverbs.so.1.0.0[7f0142c9e000+11000] # gdb rping core.3289 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/rping...Reading symbols from /usr/lib/debug/usr/bin/rping.debug...done. done. [New LWP 3289] [New LWP 3292] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `rping -s -a 0.0.0.0'. Program terminated with signal 11, Segmentation fault. #0 __ibv_alloc_pd (context=0x0) at src/verbs.c:196 196 pd = context->ops.alloc_pd(context); Missing separate debuginfos, use: debuginfo-install libcxgb3-1.3.1-8.el7.x86_64 libcxgb4-1.3.5-3.el7.x86_64 libipathverbs-1.3-2.el7.x86_64 libmlx4-1.0.6-5.el7.x86_64 libmlx5-1.0.2-1.el7.x86_64 libmthca-1.0.6-13.el7.x86_64 libne s-1.1.4-2.el7.x86_64 libnl3-3.2.21-10.el7.x86_64 (gdb) bt #0 __ibv_alloc_pd (context=0x0) at src/verbs.c:196 #1 0x0000563bc5c1f5c6 in rping_setup_qp (cb=cb@entry=0x563bc61f9780, cm_id=<optimized out>) at examples/rping.c:519 #2 0x0000563bc5c1de5a in rping_run_server (cb=0x563bc61f9780) at examples/rping.c:890 #3 main (argc=4, argv=0x7ffd7f0a8ee8) at examples/rping.c:1268 (gdb) f 0 #0 __ibv_alloc_pd (context=0x0) at src/verbs.c:196 196 pd = context->ops.alloc_pd(context); (gdb) list 191 192 struct ibv_pd *__ibv_alloc_pd(struct ibv_context *context) 193 { 194 struct ibv_pd *pd; 195 196 pd = context->ops.alloc_pd(context); 197 if (pd) 198 pd->context = context; 199 200 return pd; (gdb) p context $1 = (struct ibv_context *) 0x0 (gdb) # rping -c -a 192.168.0.13 cma event RDMA_CM_EVENT_REJECTED, error 28 wait for CONNECTED state 4 connect error -1 Segmentation fault >From dmesg [Mon May 15 14:27:24 2017] rping[3075]: segfault at 7f2386c800a8 ip 00007f2386672adf sp 00007ffd40e5df60 error 4 in libibverbs.so.1.0.0[7f2386667000+11000] # gdb rping core.3075 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/rping...Reading symbols from /usr/lib/debug/usr/bin/rping.debug...done. done. [New LWP 3075] [New LWP 3078] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `rping -c -a 192.168.0.13'. Program terminated with signal 11, Segmentation fault. #0 __ibv_dereg_mr (mr=0x5624b478c740) at src/verbs.c:237 237 ret = mr->context->ops.dereg_mr(mr); Missing separate debuginfos, use: debuginfo-install libcxgb3-1.3.1-8.el7.x86_64 libcxgb4-1.3.5-3.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libipathverbs-1.3-2.el7.x86_64 libmlx4-1.0.6-5.el7.x86_64 libmlx5-1.0.2-1.el7.x86_64 libmthca -1.0.6-13.el7.x86_64 libnes-1.1.4-2.el7.x86_64 libnl3-3.2.21-10.el7.x86_64 (gdb) bt #0 __ibv_dereg_mr (mr=0x5624b478c740) at src/verbs.c:237 #1 0x00005624b2b618a7 in rping_free_buffers (cb=0x5624b4786780) at examples/rping.c:470 #2 0x00005624b2b5fef3 in rping_run_client (cb=<optimized out>) at examples/rping.c:1111 #3 main (argc=<optimized out>, argv=<optimized out>) at examples/rping.c:1270 (gdb) f 0 #0 __ibv_dereg_mr (mr=0x5624b478c740) at src/verbs.c:237 237 ret = mr->context->ops.dereg_mr(mr); (gdb) list 232 { 233 int ret; 234 void *addr = mr->addr; 235 size_t length = mr->length; 236 237 ret = mr->context->ops.dereg_mr(mr); 238 if (!ret) 239 ibv_dofork_range(addr, length); 240 241 return ret; (gdb) p *mr $1 = {context = 0x7f2386c80070, pd = 0x5624b4789f30, addr = 0x5624b47867e8, length = 16, handle = 0, lkey = 162608, rkey = 162608} (gdb) p *mr->context Cannot access memory at address 0x7f2386c80070 ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, May 15, 2017 at 4:39 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote: > Hi Robert, > >> We are trying to leverage multiple cards/ports for iSER for >> performance and resiliency reasons. The ports are configured with only >> IPv6 addresses and each port is on a separate VLAN/subnet that is >> routable to each other subnet. We are using rules with tables to set a >> default gateway for each adapter/subnet based on the source IPv6 >> address (policy based routing). Using TCP for iSCSI, everything works >> fine and traffic ingresses/egresses the right ports. However, when we >> try using iSER, we get connection errors. >> >> May 12 13:39:27 prv-0-14-roberttest kernel: iser: iser_connect: >> rdma_resolve_addr failed: -101 >> May 12 13:39:27 prv-0-14-roberttest iscsid: Received iferror -101: >> Network is unreachable. >> May 12 13:39:27 prv-0-14-roberttest iscsid: cannot make a connection >> to 2604:3140:40:300:0:580:d0:0:3260 (-101,0) > > > This looks 100% rdma_cm to me. iser is completely agnostic to address > families and routes. > >> If we put a default gateway for IPv6 in the 'default' table, then iSER >> is able to make a connection, but we can only use one port. It looks >> as if iSER is not following the rules in the default routing table to >> find the appropriate default gateway in a different table. > > > As I said, iser relies on rdma_cm for routing decisions. > I would suspect that all rdma_cm based protocols to be > affected as well (nfs, nvmf). > > Did you check plain rping like Or suggested? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAANLjFogRE3ShHoUtGG1DPt5VzxBe8SBOAL86xc+AebQLBomJg@mail.gmail.com>]
[parent not found: <CAANLjFogRE3ShHoUtGG1DPt5VzxBe8SBOAL86xc+AebQLBomJg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <CAANLjFogRE3ShHoUtGG1DPt5VzxBe8SBOAL86xc+AebQLBomJg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-05-16 7:19 ` Sagi Grimberg [not found] ` <5441f53c-f81f-b5d6-3c05-0bbe141f152e-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Sagi Grimberg @ 2017-05-16 7:19 UTC (permalink / raw) To: Robert LeBlanc; +Cc: linux-rdma > We are getting errors with rping too, even on the same IPv4 subnet. All > RDMA programs seem to be failing. It is like there is a missing library > or something. We installed "Infiniband Support" group install, so I'm > not sure what could be missing. I assume you are running a custom kernel? This looks like a ABI breakage. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <5441f53c-f81f-b5d6-3c05-0bbe141f152e-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER with policy based routing error [not found] ` <5441f53c-f81f-b5d6-3c05-0bbe141f152e-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2017-05-16 17:39 ` Robert LeBlanc 0 siblings, 0 replies; 14+ messages in thread From: Robert LeBlanc @ 2017-05-16 17:39 UTC (permalink / raw) To: Sagi Grimberg; +Cc: linux-rdma On Tue, May 16, 2017 at 1:19 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote: > >> We are getting errors with rping too, even on the same IPv4 subnet. All >> RDMA programs seem to be failing. It is like there is a missing library >> or something. We installed "Infiniband Support" group install, so I'm >> not sure what could be missing. > > > I assume you are running a custom kernel? This looks like a ABI > breakage. This is a custom kernel. I just built 4.9.28 this morning and made sure to install the uAPI headers (make headers_install INSTALL_HDR_PATH=/usr). Is there something that I'm missing? Do I need to rebuild rping/libibverbs/etc? Thanks ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2017-05-25 16:03 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-05-12 21:02 iSER with policy based routing error Robert LeBlanc [not found] ` <CAANLjFoB3V6_exH8TnQQobaHkz2K1Lxj0sVOVa2Q+MKAsCqCKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-13 17:55 ` Or Gerlitz [not found] ` <CAJ3xEMiYwGDott_gLrh7e4tG9Ti71vftMPuvyT3+czu7jWwqFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-18 18:51 ` Robert LeBlanc [not found] ` <CAANLjFqNkNVDiGr-yW9y6XCiWE_O7+OewmxWM+xmWHQ39n4wkw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-18 18:52 ` Robert LeBlanc [not found] ` <CAANLjFpKGoruxvwhtHwF-fBH+dLT9RKqkd3POYwLVK856WEfFA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-23 23:17 ` Robert LeBlanc [not found] ` <CAANLjFqhPUd=kyuv0CBR69KuAwGZzne8ZY3-1J6YiNHuWqZBZA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-24 18:14 ` Robert LeBlanc [not found] ` <CAANLjFpHbMXjMnxO0B7sn9rRz2Sn3CEx-bEWKW5BrYj3itQ0HQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-24 19:25 ` Jason Gunthorpe [not found] ` <20170524192550.GB25200-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-05-24 19:41 ` Robert LeBlanc [not found] ` <CAANLjFpZU6GodD-HAaTtive+dQZs4Gfgpw=u-1=UgC4zDTryKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-24 21:13 ` Jason Gunthorpe 2017-05-25 16:03 ` Or Gerlitz 2017-05-15 10:39 ` Sagi Grimberg [not found] ` <71232433-a3d1-1bc0-a995-ae32fc05913f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-16 3:33 ` Robert LeBlanc [not found] ` <CAANLjFogRE3ShHoUtGG1DPt5VzxBe8SBOAL86xc+AebQLBomJg@mail.gmail.com> [not found] ` <CAANLjFogRE3ShHoUtGG1DPt5VzxBe8SBOAL86xc+AebQLBomJg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-05-16 7:19 ` Sagi Grimberg [not found] ` <5441f53c-f81f-b5d6-3c05-0bbe141f152e-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-16 17:39 ` Robert LeBlanc
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.