* vrf and multicast is broken in some cases @ 2020-01-15 17:57 Ben Greear 2020-01-15 18:45 ` David Ahern 0 siblings, 1 reply; 10+ messages in thread From: Ben Greear @ 2020-01-15 17:57 UTC (permalink / raw) To: netdev Hello, We put two different ports into their own VRF, and then tried to run a multicast sender on one and receiver on the other. The receiver does not receive anything. Is this a known problem? If we do a similar setup with policy based routing rules instead of VRF, then the multicast test works. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 17:57 vrf and multicast is broken in some cases Ben Greear @ 2020-01-15 18:45 ` David Ahern 2020-01-15 19:02 ` Ben Greear 0 siblings, 1 reply; 10+ messages in thread From: David Ahern @ 2020-01-15 18:45 UTC (permalink / raw) To: Ben Greear, netdev On 1/15/20 10:57 AM, Ben Greear wrote: > Hello, > > We put two different ports into their own VRF, and then tried to run a > multicast > sender on one and receiver on the other. The receiver does not receive > anything. > > Is this a known problem? > > If we do a similar setup with policy based routing rules instead of VRF, > then the multicast > test works. > It works for OSPF for example. I have lost track of FRR features that use it, so you will need to specify more details. Are the sender / receiver on the same host? ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 18:45 ` David Ahern @ 2020-01-15 19:02 ` Ben Greear 2020-01-15 19:19 ` Ido Schimmel 0 siblings, 1 reply; 10+ messages in thread From: Ben Greear @ 2020-01-15 19:02 UTC (permalink / raw) To: David Ahern, netdev On 01/15/2020 10:45 AM, David Ahern wrote: > On 1/15/20 10:57 AM, Ben Greear wrote: >> Hello, >> >> We put two different ports into their own VRF, and then tried to run a >> multicast >> sender on one and receiver on the other. The receiver does not receive >> anything. >> >> Is this a known problem? >> >> If we do a similar setup with policy based routing rules instead of VRF, >> then the multicast >> test works. >> > > It works for OSPF for example. I have lost track of FRR features that > use it, so you will need to specify more details. > > Are the sender / receiver on the same host? Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3. I'll go poking at the code. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 19:02 ` Ben Greear @ 2020-01-15 19:19 ` Ido Schimmel 2020-01-15 20:00 ` Ben Greear 0 siblings, 1 reply; 10+ messages in thread From: Ido Schimmel @ 2020-01-15 19:19 UTC (permalink / raw) To: Ben Greear; +Cc: David Ahern, netdev On Wed, Jan 15, 2020 at 11:02:26AM -0800, Ben Greear wrote: > > > On 01/15/2020 10:45 AM, David Ahern wrote: > > On 1/15/20 10:57 AM, Ben Greear wrote: > > > Hello, > > > > > > We put two different ports into their own VRF, and then tried to run a > > > multicast > > > sender on one and receiver on the other. The receiver does not receive > > > anything. > > > > > > Is this a known problem? > > > > > > If we do a similar setup with policy based routing rules instead of VRF, > > > then the multicast > > > test works. > > > > > > > It works for OSPF for example. I have lost track of FRR features that > > use it, so you will need to specify more details. > > > > Are the sender / receiver on the same host? > > Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3. Two questions: 1. Did you re-order the FIB rules so that l3mdev rule is before the main table? 2. Did you configure a default unreachable route in the VRF? IIRC, locally generated multicast packets are forwarded according to the unicast FIB rules, so if you don't have the unreachable route, it is possible the packet is forwarded according to the default route in the main table. > > I'll go poking at the code. > > Thanks, > Ben > > -- > Ben Greear <greearb@candelatech.com> > Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 19:19 ` Ido Schimmel @ 2020-01-15 20:00 ` Ben Greear 2020-01-15 20:23 ` Ido Schimmel 2020-01-15 20:24 ` David Ahern 0 siblings, 2 replies; 10+ messages in thread From: Ben Greear @ 2020-01-15 20:00 UTC (permalink / raw) To: Ido Schimmel; +Cc: David Ahern, netdev On 01/15/2020 11:19 AM, Ido Schimmel wrote: > On Wed, Jan 15, 2020 at 11:02:26AM -0800, Ben Greear wrote: >> >> >> On 01/15/2020 10:45 AM, David Ahern wrote: >>> On 1/15/20 10:57 AM, Ben Greear wrote: >>>> Hello, >>>> >>>> We put two different ports into their own VRF, and then tried to run a >>>> multicast >>>> sender on one and receiver on the other. The receiver does not receive >>>> anything. >>>> >>>> Is this a known problem? >>>> >>>> If we do a similar setup with policy based routing rules instead of VRF, >>>> then the multicast >>>> test works. >>>> >>> >>> It works for OSPF for example. I have lost track of FRR features that >>> use it, so you will need to specify more details. >>> >>> Are the sender / receiver on the same host? >> >> Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3. > > Two questions: > > 1. Did you re-order the FIB rules so that l3mdev rule is before the main > table? That seems OK: [root@lf0313-6477 lanforge]# ip ru show 1000: from all lookup [l3mdev-table] 1512: from all lookup local 32766: from all lookup main 32767: from all lookup default > 2. Did you configure a default unreachable route in the VRF? I did not have this, so maybe that is the issue. This is my mcast transmitter table. [root@lf0313-6477 lanforge]# ip route show table 10 broadcast 7.7.1.0 dev rddVR0 proto kernel scope link src 7.7.1.2 7.7.1.0/24 dev rddVR0 scope link src 7.7.1.2 local 7.7.1.2 dev rddVR0 proto kernel scope host src 7.7.1.2 broadcast 7.7.1.255 dev rddVR0 proto kernel scope link src 7.7.1.2 When sniffing, I see IGMP group add/delete messages sent from the receiver towards the sender, but transmitted mcast frames are not seen on the rddVR0 (veth mcast sender port). > > IIRC, locally generated multicast packets are forwarded according to the > unicast FIB rules, so if you don't have the unreachable route, it is > possible the packet is forwarded according to the default route in the > main table. And now that is interesting. When I sniff on eth0, which holds the default route outside of the VRFs, then I do see the mcast frames sent there. I tried adding default routes, and now sure enough it starts working! [root@lf0313-6477 lanforge]# ip route show table 10 default via 7.7.1.1 dev rddVR0 broadcast 7.7.1.0 dev rddVR0 proto kernel scope link src 7.7.1.2 7.7.1.0/24 dev rddVR0 scope link src 7.7.1.2 local 7.7.1.2 dev rddVR0 proto kernel scope host src 7.7.1.2 broadcast 7.7.1.255 dev rddVR0 proto kernel scope link src 7.7.1.2 I'll work on adding an un-reachable route when a real gateway is not configured... Thanks for the hint, saved me lots of work! --Ben > >> >> I'll go poking at the code. >> >> Thanks, >> Ben >> >> -- >> Ben Greear <greearb@candelatech.com> >> Candela Technologies Inc http://www.candelatech.com > -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 20:00 ` Ben Greear @ 2020-01-15 20:23 ` Ido Schimmel 2020-01-15 20:33 ` David Ahern 2020-01-15 20:24 ` David Ahern 1 sibling, 1 reply; 10+ messages in thread From: Ido Schimmel @ 2020-01-15 20:23 UTC (permalink / raw) To: Ben Greear; +Cc: David Ahern, netdev On Wed, Jan 15, 2020 at 12:00:08PM -0800, Ben Greear wrote: > > > On 01/15/2020 11:19 AM, Ido Schimmel wrote: > > On Wed, Jan 15, 2020 at 11:02:26AM -0800, Ben Greear wrote: > > > > > > > > > On 01/15/2020 10:45 AM, David Ahern wrote: > > > > On 1/15/20 10:57 AM, Ben Greear wrote: > > > > > Hello, > > > > > > > > > > We put two different ports into their own VRF, and then tried to run a > > > > > multicast > > > > > sender on one and receiver on the other. The receiver does not receive > > > > > anything. > > > > > > > > > > Is this a known problem? > > > > > > > > > > If we do a similar setup with policy based routing rules instead of VRF, > > > > > then the multicast > > > > > test works. > > > > > > > > > > > > > It works for OSPF for example. I have lost track of FRR features that > > > > use it, so you will need to specify more details. > > > > > > > > Are the sender / receiver on the same host? > > > > > > Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3. > > > > Two questions: > > > > 1. Did you re-order the FIB rules so that l3mdev rule is before the main > > table? > > That seems OK: > > [root@lf0313-6477 lanforge]# ip ru show > 1000: from all lookup [l3mdev-table] > 1512: from all lookup local > 32766: from all lookup main > 32767: from all lookup default > > > > 2. Did you configure a default unreachable route in the VRF? > > I did not have this, so maybe that is the issue. This is my mcast > transmitter table. > > [root@lf0313-6477 lanforge]# ip route show table 10 > broadcast 7.7.1.0 dev rddVR0 proto kernel scope link src 7.7.1.2 > 7.7.1.0/24 dev rddVR0 scope link src 7.7.1.2 > local 7.7.1.2 dev rddVR0 proto kernel scope host src 7.7.1.2 > broadcast 7.7.1.255 dev rddVR0 proto kernel scope link src 7.7.1.2 > > When sniffing, I see IGMP group add/delete messages sent from the receiver > towards the sender, but transmitted mcast frames are not seen on the rddVR0 > (veth mcast sender port). > > > > > IIRC, locally generated multicast packets are forwarded according to the > > unicast FIB rules, so if you don't have the unreachable route, it is > > possible the packet is forwarded according to the default route in the > > main table. > > And now that is interesting. When I sniff on eth0, which holds the default > route outside of the VRFs, then I do see the mcast frames sent there. > > I tried adding default routes, and now sure enough it starts working! I'm not sure this is the correct way (David?). Can you try to delete this default route and instead add a default unreachable route with an high metric according to step 3 in Documentation/networking/vrf.txt: " 3. Set the default route for the table (and hence default route for the VRF). ip route add table 10 unreachable default metric 4278198272 This high metric value ensures that the default unreachable route can be overridden by a routing protocol suite. FRRouting interprets kernel metrics as a combined admin distance (upper byte) and priority (lower 3 bytes). Thus the above metric translates to [255/8192]. " If I'm reading ip_route_output_key_hash_rcu() correctly, then the error returned from fib_lookup() because of the unreachable route should allow you to route the packet via the requested interface. > > [root@lf0313-6477 lanforge]# ip route show table 10 > default via 7.7.1.1 dev rddVR0 > broadcast 7.7.1.0 dev rddVR0 proto kernel scope link src 7.7.1.2 > 7.7.1.0/24 dev rddVR0 scope link src 7.7.1.2 > local 7.7.1.2 dev rddVR0 proto kernel scope host src 7.7.1.2 > broadcast 7.7.1.255 dev rddVR0 proto kernel scope link src 7.7.1.2 > > > I'll work on adding an un-reachable route when a real gateway is not > configured... > > Thanks for the hint, saved me lots of work! > > --Ben > > > > > > > > > I'll go poking at the code. > > > > > > Thanks, > > > Ben > > > > > > -- > > > Ben Greear <greearb@candelatech.com> > > > Candela Technologies Inc http://www.candelatech.com > > > > -- > Ben Greear <greearb@candelatech.com> > Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 20:23 ` Ido Schimmel @ 2020-01-15 20:33 ` David Ahern 2020-01-15 21:09 ` Ben Greear 0 siblings, 1 reply; 10+ messages in thread From: David Ahern @ 2020-01-15 20:33 UTC (permalink / raw) To: Ido Schimmel, Ben Greear; +Cc: netdev On 1/15/20 1:23 PM, Ido Schimmel wrote: > > I'm not sure this is the correct way (David?). Can you try to delete > this default route and instead add a default unreachable route with an > high metric according to step 3 in Documentation/networking/vrf.txt: > > " > 3. Set the default route for the table (and hence default route for the VRF). > ip route add table 10 unreachable default metric 4278198272 > > This high metric value ensures that the default unreachable route can > be overridden by a routing protocol suite. FRRouting interprets > kernel metrics as a combined admin distance (upper byte) and priority > (lower 3 bytes). Thus the above metric translates to [255/8192]. > " > > If I'm reading ip_route_output_key_hash_rcu() correctly, then the error > returned from fib_lookup() because of the unreachable route should allow > you to route the packet via the requested interface. > yes, IPv4 is a bit goofy with multicast (at least to me, but then I have not done much with mcast). ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 20:33 ` David Ahern @ 2020-01-15 21:09 ` Ben Greear 2020-01-15 22:53 ` David Ahern 0 siblings, 1 reply; 10+ messages in thread From: Ben Greear @ 2020-01-15 21:09 UTC (permalink / raw) To: David Ahern, Ido Schimmel; +Cc: netdev On 01/15/2020 12:33 PM, David Ahern wrote: > On 1/15/20 1:23 PM, Ido Schimmel wrote: >> >> I'm not sure this is the correct way (David?). Can you try to delete >> this default route and instead add a default unreachable route with an >> high metric according to step 3 in Documentation/networking/vrf.txt: >> >> " >> 3. Set the default route for the table (and hence default route for the VRF). >> ip route add table 10 unreachable default metric 4278198272 >> >> This high metric value ensures that the default unreachable route can >> be overridden by a routing protocol suite. FRRouting interprets >> kernel metrics as a combined admin distance (upper byte) and priority >> (lower 3 bytes). Thus the above metric translates to [255/8192]. >> " >> >> If I'm reading ip_route_output_key_hash_rcu() correctly, then the error >> returned from fib_lookup() because of the unreachable route should allow >> you to route the packet via the requested interface. >> > > yes, IPv4 is a bit goofy with multicast (at least to me, but then I have > not done much with mcast). For the vrf.txt referenced above, it would be nice if it mentioned that if you do NOT add the default route, it will skip to the main table and use it's gateway. I was visualizing VRF as a self contained thing that would not do such a thing. When we are using xorp (which is our mcast router), then it automatically takes care of dealing with replacing an unreachable route and/or metrics, so in my case, I think the default metric on the unreachable route will be fine. That said, I am not sure xorp is completely working with our vrf setup yet, so I'll keep this in mind next time I am poking at that. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 21:09 ` Ben Greear @ 2020-01-15 22:53 ` David Ahern 0 siblings, 0 replies; 10+ messages in thread From: David Ahern @ 2020-01-15 22:53 UTC (permalink / raw) To: Ben Greear, Ido Schimmel; +Cc: netdev On 1/15/20 2:09 PM, Ben Greear wrote: > For the vrf.txt referenced above, it would be nice if it mentioned that > if you do NOT add the default route, it will skip to the main table > and use it's gateway. I was visualizing VRF as a self contained thing > that would not do such a thing. The in-tree documentation needs an update. Every tutorial I have given on VRF mentions that it is implemented as policy routing meaning FIB lookups follow the FIB rules. You say 'main' table but really it leaves the l3mdev rule and drops to the next rule which in your case is the main table. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: vrf and multicast is broken in some cases 2020-01-15 20:00 ` Ben Greear 2020-01-15 20:23 ` Ido Schimmel @ 2020-01-15 20:24 ` David Ahern 1 sibling, 0 replies; 10+ messages in thread From: David Ahern @ 2020-01-15 20:24 UTC (permalink / raw) To: Ben Greear, Ido Schimmel; +Cc: netdev On 1/15/20 1:00 PM, Ben Greear wrote: > I'll work on adding an un-reachable route when a real gateway is not > configured... I recommend a high metric unreachable default route every time you create a VRF for just the reason Ido mentioned. You can always add a lower priority default as needed. I think we need an update to the VRF documetnation. ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-01-15 22:53 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-15 17:57 vrf and multicast is broken in some cases Ben Greear 2020-01-15 18:45 ` David Ahern 2020-01-15 19:02 ` Ben Greear 2020-01-15 19:19 ` Ido Schimmel 2020-01-15 20:00 ` Ben Greear 2020-01-15 20:23 ` Ido Schimmel 2020-01-15 20:33 ` David Ahern 2020-01-15 21:09 ` Ben Greear 2020-01-15 22:53 ` David Ahern 2020-01-15 20:24 ` David Ahern
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).