vrf and multicast is broken in some cases

vrf and multicast is broken in some cases

[Ben Greear]

Hello,

We put two different ports into their own VRF, and then tried to run a multicast
sender on one and receiver on the other.  The receiver does not receive anything.

Is this a known problem?

If we do a similar setup with policy based routing rules instead of VRF, then the multicast
test works.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: vrf and multicast is broken in some cases

[David Ahern]

On 1/15/20 10:57 AM, Ben Greear wrote:
> Hello,
> 
> We put two different ports into their own VRF, and then tried to run a
> multicast
> sender on one and receiver on the other.  The receiver does not receive
> anything.
> 
> Is this a known problem?
> 
> If we do a similar setup with policy based routing rules instead of VRF,
> then the multicast
> test works.
> 

It works for OSPF for example. I have lost track of FRR features that
use it, so you will need to specify more details.

Are the sender / receiver on the same host?

Re: vrf and multicast is broken in some cases

[Ben Greear]

On 01/15/2020 10:45 AM, David Ahern wrote:
> On 1/15/20 10:57 AM, Ben Greear wrote:
>> Hello,
>>
>> We put two different ports into their own VRF, and then tried to run a
>> multicast
>> sender on one and receiver on the other.  The receiver does not receive
>> anything.
>>
>> Is this a known problem?
>>
>> If we do a similar setup with policy based routing rules instead of VRF,
>> then the multicast
>> test works.
>>
>
> It works for OSPF for example. I have lost track of FRR features that
> use it, so you will need to specify more details.
>
> Are the sender / receiver on the same host?

Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3.

I'll go poking at the code.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: vrf and multicast is broken in some cases

[Ido Schimmel]

On Wed, Jan 15, 2020 at 11:02:26AM -0800, Ben Greear wrote:
> 
> 
> On 01/15/2020 10:45 AM, David Ahern wrote:
> > On 1/15/20 10:57 AM, Ben Greear wrote:
> > > Hello,
> > > 
> > > We put two different ports into their own VRF, and then tried to run a
> > > multicast
> > > sender on one and receiver on the other.  The receiver does not receive
> > > anything.
> > > 
> > > Is this a known problem?
> > > 
> > > If we do a similar setup with policy based routing rules instead of VRF,
> > > then the multicast
> > > test works.
> > > 
> > 
> > It works for OSPF for example. I have lost track of FRR features that
> > use it, so you will need to specify more details.
> > 
> > Are the sender / receiver on the same host?
> 
> Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3.

Two questions:

1. Did you re-order the FIB rules so that l3mdev rule is before the main
table?
2. Did you configure a default unreachable route in the VRF?

IIRC, locally generated multicast packets are forwarded according to the
unicast FIB rules, so if you don't have the unreachable route, it is
possible the packet is forwarded according to the default route in the
main table.

> 
> I'll go poking at the code.
> 
> Thanks,
> Ben
> 
> -- 
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com

Re: vrf and multicast is broken in some cases

[Ben Greear]

On 01/15/2020 11:19 AM, Ido Schimmel wrote:
> On Wed, Jan 15, 2020 at 11:02:26AM -0800, Ben Greear wrote:
>>
>>
>> On 01/15/2020 10:45 AM, David Ahern wrote:
>>> On 1/15/20 10:57 AM, Ben Greear wrote:
>>>> Hello,
>>>>
>>>> We put two different ports into their own VRF, and then tried to run a
>>>> multicast
>>>> sender on one and receiver on the other.  The receiver does not receive
>>>> anything.
>>>>
>>>> Is this a known problem?
>>>>
>>>> If we do a similar setup with policy based routing rules instead of VRF,
>>>> then the multicast
>>>> test works.
>>>>
>>>
>>> It works for OSPF for example. I have lost track of FRR features that
>>> use it, so you will need to specify more details.
>>>
>>> Are the sender / receiver on the same host?
>>
>> Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3.
>
> Two questions:
>
> 1. Did you re-order the FIB rules so that l3mdev rule is before the main
> table?

That seems OK:

[root@lf0313-6477 lanforge]# ip ru show
1000:	from all lookup [l3mdev-table]
1512:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default


> 2. Did you configure a default unreachable route in the VRF?

I did not have this, so maybe that is the issue.  This is my mcast
transmitter table.

[root@lf0313-6477 lanforge]# ip route show table 10
broadcast 7.7.1.0 dev rddVR0  proto kernel  scope link  src 7.7.1.2
7.7.1.0/24 dev rddVR0  scope link  src 7.7.1.2
local 7.7.1.2 dev rddVR0  proto kernel  scope host  src 7.7.1.2
broadcast 7.7.1.255 dev rddVR0  proto kernel  scope link  src 7.7.1.2

When sniffing, I see IGMP group add/delete messages sent from the receiver
towards the sender, but transmitted mcast frames are not seen on the rddVR0
(veth mcast sender port).

>
> IIRC, locally generated multicast packets are forwarded according to the
> unicast FIB rules, so if you don't have the unreachable route, it is
> possible the packet is forwarded according to the default route in the
> main table.

And now that is interesting.  When I sniff on eth0, which holds the default
route outside of the VRFs, then I do see the mcast frames sent there.

I tried adding default routes, and now sure enough it starts working!

[root@lf0313-6477 lanforge]# ip route show table 10
default via 7.7.1.1 dev rddVR0
broadcast 7.7.1.0 dev rddVR0  proto kernel  scope link  src 7.7.1.2
7.7.1.0/24 dev rddVR0  scope link  src 7.7.1.2
local 7.7.1.2 dev rddVR0  proto kernel  scope host  src 7.7.1.2
broadcast 7.7.1.255 dev rddVR0  proto kernel  scope link  src 7.7.1.2


I'll work on adding an un-reachable route when a real gateway is not
configured...

Thanks for the hint, saved me lots of work!

--Ben

>
>>
>> I'll go poking at the code.
>>
>> Thanks,
>> Ben
>>
>> --
>> Ben Greear <greearb@candelatech.com>
>> Candela Technologies Inc  http://www.candelatech.com
>

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: vrf and multicast is broken in some cases

[Ido Schimmel]

On Wed, Jan 15, 2020 at 12:00:08PM -0800, Ben Greear wrote:
> 
> 
> On 01/15/2020 11:19 AM, Ido Schimmel wrote:
> > On Wed, Jan 15, 2020 at 11:02:26AM -0800, Ben Greear wrote:
> > > 
> > > 
> > > On 01/15/2020 10:45 AM, David Ahern wrote:
> > > > On 1/15/20 10:57 AM, Ben Greear wrote:
> > > > > Hello,
> > > > > 
> > > > > We put two different ports into their own VRF, and then tried to run a
> > > > > multicast
> > > > > sender on one and receiver on the other.  The receiver does not receive
> > > > > anything.
> > > > > 
> > > > > Is this a known problem?
> > > > > 
> > > > > If we do a similar setup with policy based routing rules instead of VRF,
> > > > > then the multicast
> > > > > test works.
> > > > > 
> > > > 
> > > > It works for OSPF for example. I have lost track of FRR features that
> > > > use it, so you will need to specify more details.
> > > > 
> > > > Are the sender / receiver on the same host?
> > > 
> > > Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3.
> > 
> > Two questions:
> > 
> > 1. Did you re-order the FIB rules so that l3mdev rule is before the main
> > table?
> 
> That seems OK:
> 
> [root@lf0313-6477 lanforge]# ip ru show
> 1000:	from all lookup [l3mdev-table]
> 1512:	from all lookup local
> 32766:	from all lookup main
> 32767:	from all lookup default
> 
> 
> > 2. Did you configure a default unreachable route in the VRF?
> 
> I did not have this, so maybe that is the issue.  This is my mcast
> transmitter table.
> 
> [root@lf0313-6477 lanforge]# ip route show table 10
> broadcast 7.7.1.0 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 7.7.1.0/24 dev rddVR0  scope link  src 7.7.1.2
> local 7.7.1.2 dev rddVR0  proto kernel  scope host  src 7.7.1.2
> broadcast 7.7.1.255 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 
> When sniffing, I see IGMP group add/delete messages sent from the receiver
> towards the sender, but transmitted mcast frames are not seen on the rddVR0
> (veth mcast sender port).
> 
> > 
> > IIRC, locally generated multicast packets are forwarded according to the
> > unicast FIB rules, so if you don't have the unreachable route, it is
> > possible the packet is forwarded according to the default route in the
> > main table.
> 
> And now that is interesting.  When I sniff on eth0, which holds the default
> route outside of the VRFs, then I do see the mcast frames sent there.
> 
> I tried adding default routes, and now sure enough it starts working!

I'm not sure this is the correct way (David?). Can you try to delete
this default route and instead add a default unreachable route with an
high metric according to step 3 in Documentation/networking/vrf.txt:

"
3. Set the default route for the table (and hence default route for the VRF).
       ip route add table 10 unreachable default metric 4278198272

   This high metric value ensures that the default unreachable route can
   be overridden by a routing protocol suite.  FRRouting interprets
   kernel metrics as a combined admin distance (upper byte) and priority
   (lower 3 bytes).  Thus the above metric translates to [255/8192].
"

If I'm reading ip_route_output_key_hash_rcu() correctly, then the error
returned from fib_lookup() because of the unreachable route should allow
you to route the packet via the requested interface.

> 
> [root@lf0313-6477 lanforge]# ip route show table 10
> default via 7.7.1.1 dev rddVR0
> broadcast 7.7.1.0 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 7.7.1.0/24 dev rddVR0  scope link  src 7.7.1.2
> local 7.7.1.2 dev rddVR0  proto kernel  scope host  src 7.7.1.2
> broadcast 7.7.1.255 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 
> 
> I'll work on adding an un-reachable route when a real gateway is not
> configured...
> 
> Thanks for the hint, saved me lots of work!
> 
> --Ben
> 
> > 
> > > 
> > > I'll go poking at the code.
> > > 
> > > Thanks,
> > > Ben
> > > 
> > > --
> > > Ben Greear <greearb@candelatech.com>
> > > Candela Technologies Inc  http://www.candelatech.com
> > 
> 
> -- 
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com

Re: vrf and multicast is broken in some cases

[David Ahern]

On 1/15/20 1:00 PM, Ben Greear wrote:
> I'll work on adding an un-reachable route when a real gateway is not
> configured...

I recommend a high metric unreachable default route every time you
create a VRF for just the reason Ido mentioned. You can always add a
lower priority default as needed.

I think we need an update to the VRF documetnation.

Re: vrf and multicast is broken in some cases

[David Ahern]

On 1/15/20 1:23 PM, Ido Schimmel wrote:
> 
> I'm not sure this is the correct way (David?). Can you try to delete
> this default route and instead add a default unreachable route with an
> high metric according to step 3 in Documentation/networking/vrf.txt:
> 
> "
> 3. Set the default route for the table (and hence default route for the VRF).
>        ip route add table 10 unreachable default metric 4278198272
> 
>    This high metric value ensures that the default unreachable route can
>    be overridden by a routing protocol suite.  FRRouting interprets
>    kernel metrics as a combined admin distance (upper byte) and priority
>    (lower 3 bytes).  Thus the above metric translates to [255/8192].
> "
> 
> If I'm reading ip_route_output_key_hash_rcu() correctly, then the error
> returned from fib_lookup() because of the unreachable route should allow
> you to route the packet via the requested interface.
> 

yes, IPv4 is a bit goofy with multicast (at least to me, but then I have
not done much with mcast).

Re: vrf and multicast is broken in some cases

[Ben Greear]

On 01/15/2020 12:33 PM, David Ahern wrote:
> On 1/15/20 1:23 PM, Ido Schimmel wrote:
>>
>> I'm not sure this is the correct way (David?). Can you try to delete
>> this default route and instead add a default unreachable route with an
>> high metric according to step 3 in Documentation/networking/vrf.txt:
>>
>> "
>> 3. Set the default route for the table (and hence default route for the VRF).
>>        ip route add table 10 unreachable default metric 4278198272
>>
>>    This high metric value ensures that the default unreachable route can
>>    be overridden by a routing protocol suite.  FRRouting interprets
>>    kernel metrics as a combined admin distance (upper byte) and priority
>>    (lower 3 bytes).  Thus the above metric translates to [255/8192].
>> "
>>
>> If I'm reading ip_route_output_key_hash_rcu() correctly, then the error
>> returned from fib_lookup() because of the unreachable route should allow
>> you to route the packet via the requested interface.
>>
>
> yes, IPv4 is a bit goofy with multicast (at least to me, but then I have
> not done much with mcast).

For the vrf.txt referenced above, it would be nice if it mentioned that
if you do NOT add the default route, it will skip to the main table
and use it's gateway.  I was visualizing VRF as a self contained thing
that would not do such a thing.

When we are using xorp (which is our mcast router), then it automatically
takes care of dealing with replacing an unreachable route and/or metrics,
so in my case, I think the default metric on the unreachable route will
be fine.

That said, I am not sure xorp is completely working with our vrf setup
yet, so I'll keep this in mind next time I am poking at that.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: vrf and multicast is broken in some cases

[David Ahern]

On 1/15/20 2:09 PM, Ben Greear wrote:
> For the vrf.txt referenced above, it would be nice if it mentioned that
> if you do NOT add the default route, it will skip to the main table
> and use it's gateway.  I was visualizing VRF as a self contained thing
> that would not do such a thing.

The in-tree documentation needs an update.

Every tutorial I have given on VRF mentions that it is implemented as
policy routing meaning FIB lookups follow the FIB rules. You say 'main'
table but really it leaves the l3mdev rule and drops to the next rule
which in your case is the main table.