[MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Erik Hugne]

If macvlan's are configured on a device, IPv6 multicast
traffic received on the underlying ethernet device is not
delivered properly to higher layers. Large amounts of packets are lost, and
the problem seems to get worse the more macvlans are defined
on top of the device, up to the point where no traffic is delivered
at all...


Steps to reproduce (2 hosts required)
On host 1, set up some macvlans:
ip link add link eth0 name macvlan0 type macvlan mode bridge
ip link add link eth0 name macvlan2 type macvlan mode bridge
ip link add link eth0 name macvlan3 type macvlan mode bridge
ip link set link dev macvlan0 up
ip link set link dev macvlan2 up
ip link set link dev macvlan3 up

Start a server that binds to ff02::1 scoped to eth0
./server6 -sff02::1 -eeth0


On host 2:
./client6 -dff02::1 -eeth0


This seems to only occur for large (fragmented) IPv6 multicast traffic.
If you bind to :: instead (-s:: on the server) and send to
the linklocal address of Host1/eth0, packets are delivered just fine.

Source for client6/server6 on github:
git clone git@github.com:Hugne/udp_mc.git

//E

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Eric Dumazet]

On Fri, 2013-02-15 at 18:00 +0100, Erik Hugne wrote:
> If macvlan's are configured on a device, IPv6 multicast
> traffic received on the underlying ethernet device is not
> delivered properly to higher layers. Large amounts of packets are lost, and
> the problem seems to get worse the more macvlans are defined
> on top of the device, up to the point where no traffic is delivered
> at all...
> 
> 
> Steps to reproduce (2 hosts required)
> On host 1, set up some macvlans:
> ip link add link eth0 name macvlan0 type macvlan mode bridge
> ip link add link eth0 name macvlan2 type macvlan mode bridge
> ip link add link eth0 name macvlan3 type macvlan mode bridge
> ip link set link dev macvlan0 up
> ip link set link dev macvlan2 up
> ip link set link dev macvlan3 up
> 
> Start a server that binds to ff02::1 scoped to eth0
> ./server6 -sff02::1 -eeth0
> 
> 
> On host 2:
> ./client6 -dff02::1 -eeth0
> 
> 
> This seems to only occur for large (fragmented) IPv6 multicast traffic.
> If you bind to :: instead (-s:: on the server) and send to
> the linklocal address of Host1/eth0, packets are delivered just fine.
> 
> Source for client6/server6 on github:
> git clone git@github.com:Hugne/udp_mc.git

I guess we need to defrag before doing the broadcast,
as we did for IPv4 in commit bc416d9768aa9a2e4
(macvlan: handle fragmented multicast frames)

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Ben Greear]

On 02/15/2013 09:00 AM, Erik Hugne wrote:
> If macvlan's are configured on a device, IPv6 multicast
> traffic received on the underlying ethernet device is not
> delivered properly to higher layers. Large amounts of packets are lost, and
> the problem seems to get worse the more macvlans are defined
> on top of the device, up to the point where no traffic is delivered
> at all...
>
>
> Steps to reproduce (2 hosts required)
> On host 1, set up some macvlans:
> ip link add link eth0 name macvlan0 type macvlan mode bridge
> ip link add link eth0 name macvlan2 type macvlan mode bridge
> ip link add link eth0 name macvlan3 type macvlan mode bridge
> ip link set link dev macvlan0 up
> ip link set link dev macvlan2 up
> ip link set link dev macvlan3 up
>
> Start a server that binds to ff02::1 scoped to eth0
> ./server6 -sff02::1 -eeth0
>
>
> On host 2:
> ./client6 -dff02::1 -eeth0
>
>
> This seems to only occur for large (fragmented) IPv6 multicast traffic.
> If you bind to :: instead (-s:: on the server) and send to
> the linklocal address of Host1/eth0, packets are delivered just fine.

This has been around for a long time, maybe forever.  Last time
we looked at this it did not appear there was an easy fix, something
to do with how UDP packets are re-assembled....

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Eric Dumazet]

On Fri, 2013-02-15 at 11:07 -0800, Ben Greear wrote:

> This has been around for a long time, maybe forever.  Last time
> we looked at this it did not appear there was an easy fix, something
> to do with how UDP packets are re-assembled....

This can be fixed the same way we did for IPv4

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Ben Greear]

On 02/15/2013 11:18 AM, Eric Dumazet wrote:
> On Fri, 2013-02-15 at 11:07 -0800, Ben Greear wrote:
>
>> This has been around for a long time, maybe forever.  Last time
>> we looked at this it did not appear there was an easy fix, something
>> to do with how UDP packets are re-assembled....
>
> This can be fixed the same way we did for IPv4

For some reason I was thinking this wasn't fully fixed for IPv4, but
maybe it is..the bug in our internal tracker only mentions IPv6
as having issues...

We'll do some testing on IPv4 sometime soon to make sure, and can test
IPv6 patches as well...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Eric Dumazet]

On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:

> For some reason I was thinking this wasn't fully fixed for IPv4, but
> maybe it is..the bug in our internal tracker only mentions IPv6
> as having issues...
> 
> We'll do some testing on IPv4 sometime soon to make sure, and can test
> IPv6 patches as well...

This worries me a bit, as I wrote this patch because you reported the
issue.

commit bc416d9768aa9a2e46eb11354a9c58399dafeb01
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Thu Oct 6 10:28:31 2011 +0000

    macvlan: handle fragmented multicast frames
    
    Fragmented multicast frames are delivered to a single macvlan port,
    because ip defrag logic considers other samples are redundant.
    
    Implement a defrag step before trying to send the multicast frame.
    
    Reported-by: Ben Greear <greearb@candelatech.com>
    Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Vlad Yasevich]

On 02/15/2013 02:35 PM, Eric Dumazet wrote:
> On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:
>
>> For some reason I was thinking this wasn't fully fixed for IPv4, but
>> maybe it is..the bug in our internal tracker only mentions IPv6
>> as having issues...
>>
>> We'll do some testing on IPv4 sometime soon to make sure, and can test
>> IPv6 patches as well...
>
> This worries me a bit, as I wrote this patch because you reported the
> issue.
>
> commit bc416d9768aa9a2e46eb11354a9c58399dafeb01
> Author: Eric Dumazet <eric.dumazet@gmail.com>
> Date:   Thu Oct 6 10:28:31 2011 +0000
>
>      macvlan: handle fragmented multicast frames
>
>      Fragmented multicast frames are delivered to a single macvlan port,
>      because ip defrag logic considers other samples are redundant.
>
>      Implement a defrag step before trying to send the multicast frame.
>
>      Reported-by: Ben Greear <greearb@candelatech.com>
>      Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
>      Signed-off-by: David S. Miller <davem@davemloft.net>
>
>

Yep, IPv4 is there.  IPv6 is not and is a lot harder/more interesting 
since IPv6 may be disable, but reassembly may still need to work to
service the taps connected to VMs.  :(

The only reason I say this is because I've ran into too many people who
turn IPv6 off for "security reasons".

-vlad

>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Vlad Yasevich]

On 02/15/2013 09:25 PM, Vlad Yasevich wrote:
> On 02/15/2013 02:35 PM, Eric Dumazet wrote:
>> On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:
>>
>>> For some reason I was thinking this wasn't fully fixed for IPv4, but
>>> maybe it is..the bug in our internal tracker only mentions IPv6
>>> as having issues...
>>>
>>> We'll do some testing on IPv4 sometime soon to make sure, and can test
>>> IPv6 patches as well...
>>
>> This worries me a bit, as I wrote this patch because you reported the
>> issue.
>>
>> commit bc416d9768aa9a2e46eb11354a9c58399dafeb01
>> Author: Eric Dumazet <eric.dumazet@gmail.com>
>> Date:   Thu Oct 6 10:28:31 2011 +0000
>>
>>      macvlan: handle fragmented multicast frames
>>
>>      Fragmented multicast frames are delivered to a single macvlan port,
>>      because ip defrag logic considers other samples are redundant.
>>
>>      Implement a defrag step before trying to send the multicast frame.
>>
>>      Reported-by: Ben Greear <greearb@candelatech.com>
>>      Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
>>      Signed-off-by: David S. Miller <davem@davemloft.net>
>>
>>
>
> Yep, IPv4 is there.  IPv6 is not and is a lot harder/more interesting
> since IPv6 may be disable, but reassembly may still need to work to
> service the taps connected to VMs.  :(
>
> The only reason I say this is because I've ran into too many people who
> turn IPv6 off for "security reasons".

Looking a bit more, I don't think this is going to be a problem with 
macvtaps since the root cause of the drops here is most likely inside
IPv6 reassembly, since the same frags may get queued multiple times to 
the same queue.  That would not be an issue with macvtaps.

-vlad

>
> -vlad
>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Erik Hugne]

On Fri, Feb 15, 2013 at 09:57:30PM -0500, Vlad Yasevich wrote:
> On 02/15/2013 09:25 PM, Vlad Yasevich wrote:
> >On 02/15/2013 02:35 PM, Eric Dumazet wrote:
> >>On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:
> >>
> >>>For some reason I was thinking this wasn't fully fixed for IPv4, but
> >>>maybe it is..the bug in our internal tracker only mentions IPv6
> >>>as having issues...
> >>>
> >>>We'll do some testing on IPv4 sometime soon to make sure, and can test
> >>>IPv6 patches as well...
> >>
> >>This worries me a bit, as I wrote this patch because you reported the
> >>issue.
> >>
> >>commit bc416d9768aa9a2e46eb11354a9c58399dafeb01
> >>Author: Eric Dumazet <eric.dumazet@gmail.com>
> >>Date:   Thu Oct 6 10:28:31 2011 +0000
> >>
> >>     macvlan: handle fragmented multicast frames
> >>
> >>     Fragmented multicast frames are delivered to a single macvlan port,
> >>     because ip defrag logic considers other samples are redundant.
> >>
> >>     Implement a defrag step before trying to send the multicast frame.
> >>
> >>     Reported-by: Ben Greear <greearb@candelatech.com>
> >>     Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> >>     Signed-off-by: David S. Miller <davem@davemloft.net>
> >>
> >>

Here's the correct link to the test program:
git://github.com/Hugne/udp_mc.git

My tree was just a few days old so it did contain this patch.
Anyway, I retested with latest now and can confirm that it's still there.

I set up some ip6tables logging, and it is clear that all interfaces are 
receiving all fragments, but fail to reassemble them.
netstat -6 -s |grep reass
    8880 reassemblies required
    490 packets reassembled ok
    2898 packet reassembles failed

ip6tables -N LOGME
ip6tables -A LOGME -j LOG
ip6tables -A INPUT -d ff02::/16 -i macvlan0 -j LOGME
ip6tables -A INPUT -d ff02::/16 -i eth0 -j LOGME
ip6tables -A LOGME -j ACCEPT

//E

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Ben Greear]

On 02/15/2013 11:35 AM, Eric Dumazet wrote:
> On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:
>
>> For some reason I was thinking this wasn't fully fixed for IPv4, but
>> maybe it is..the bug in our internal tracker only mentions IPv6
>> as having issues...
>>
>> We'll do some testing on IPv4 sometime soon to make sure, and can test
>> IPv6 patches as well...
>
> This worries me a bit, as I wrote this patch because you reported the
> issue.

We just did some testing on this on the 3.7.6+ kernel.  IPv4 multicast to
mac-vlans works OK with large packets if there is a single multicast receiver,
but does NOT work if there are multiple multicast receivers (ie, several
mac-vlans with an mcast receive bound to each mvlan).

Our setup also involves routing rules and send-to-self rules,
with sender on the same machine, but I'm not sure that matters
too much or not.

Multiple receivers work fine with smaller packets that do not
need to be segmented.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Ben Greear]

On 02/19/2013 02:22 PM, Ben Greear wrote:
> On 02/15/2013 11:35 AM, Eric Dumazet wrote:
>> On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:
>>
>>> For some reason I was thinking this wasn't fully fixed for IPv4, but
>>> maybe it is..the bug in our internal tracker only mentions IPv6
>>> as having issues...
>>>
>>> We'll do some testing on IPv4 sometime soon to make sure, and can test
>>> IPv6 patches as well...
>>
>> This worries me a bit, as I wrote this patch because you reported the
>> issue.
>
> We just did some testing on this on the 3.7.6+ kernel.  IPv4 multicast to
> mac-vlans works OK with large packets if there is a single multicast receiver,
> but does NOT work if there are multiple multicast receivers (ie, several
> mac-vlans with an mcast receive bound to each mvlan).

I should clarify this a bit:  This kernel has some additional
patches...probably nothing that makes this issue worse, but
I guess you never know.  We have not tested this on standard
upstream kernels.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Eric Dumazet]

On Tue, 2013-02-19 at 14:22 -0800, Ben Greear wrote:
> On 02/15/2013 11:35 AM, Eric Dumazet wrote:
> > On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:
> >
> >> For some reason I was thinking this wasn't fully fixed for IPv4, but
> >> maybe it is..the bug in our internal tracker only mentions IPv6
> >> as having issues...
> >>
> >> We'll do some testing on IPv4 sometime soon to make sure, and can test
> >> IPv6 patches as well...
> >
> > This worries me a bit, as I wrote this patch because you reported the
> > issue.
> 
> We just did some testing on this on the 3.7.6+ kernel.  IPv4 multicast to
> mac-vlans works OK with large packets if there is a single multicast receiver,
> but does NOT work if there are multiple multicast receivers (ie, several
> mac-vlans with an mcast receive bound to each mvlan).
> 
> Our setup also involves routing rules and send-to-self rules,
> with sender on the same machine, but I'm not sure that matters
> too much or not.

It might matter, because before commit 6d1ccff627806829c (net: reset mac
header in dev_start_xmit() ), we were accessing wrong ethernet header in
macvlan_broadcast() in the send-to-self path

struct ethhdr *eth = eth_hdr(skb);

was simply wrong.

You should try net-next and if it still fails, maybe share your test
programs.

Re: [MacVLAN] failure to deliver reassembled IPv6 multicast traffic

[Ben Greear]

On 02/19/2013 02:38 PM, Eric Dumazet wrote:
> On Tue, 2013-02-19 at 14:22 -0800, Ben Greear wrote:
>> On 02/15/2013 11:35 AM, Eric Dumazet wrote:
>>> On Fri, 2013-02-15 at 11:27 -0800, Ben Greear wrote:
>>>
>>>> For some reason I was thinking this wasn't fully fixed for IPv4, but
>>>> maybe it is..the bug in our internal tracker only mentions IPv6
>>>> as having issues...
>>>>
>>>> We'll do some testing on IPv4 sometime soon to make sure, and can test
>>>> IPv6 patches as well...
>>>
>>> This worries me a bit, as I wrote this patch because you reported the
>>> issue.
>>
>> We just did some testing on this on the 3.7.6+ kernel.  IPv4 multicast to
>> mac-vlans works OK with large packets if there is a single multicast receiver,
>> but does NOT work if there are multiple multicast receivers (ie, several
>> mac-vlans with an mcast receive bound to each mvlan).
>>
>> Our setup also involves routing rules and send-to-self rules,
>> with sender on the same machine, but I'm not sure that matters
>> too much or not.
>
> It might matter, because before commit 6d1ccff627806829c (net: reset mac
> header in dev_start_xmit() ), we were accessing wrong ethernet header in
> macvlan_broadcast() in the send-to-self path

I have that patch in my tree already, so it does not fix the
problem...  Also, I mos-spoke slightly...the test was on 3.7.9+
kernel...

Our test setup is not simple, and is driven by a proprietary
user-space application.  I can set it up on a system and
give you access if you want (you could investigate the
various routing rules, etc, to see how our config actually
happens).

Or, can help you install our app on your system and configure
it to cause this bug.  All these features should run fine on
standard upstream kernels.

There's no sane way for me to reproduce our test setup
with standard tools and mcast traffic generator/receivers,
however...


Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com