MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Mark Smith]

Hi,

Ben said,
>I wouldn't deny sending with wrong source mac..ethernet interfaces can 
>do this,
>and mac-vlan should look as much like ethernet is possible.

I agree, however there's further things that mac-vlans aren't
currently doing as virtual ethernet interfaces that real ones do.
Unicast ethernet traffic sent out one mac-vlan interface with a
destination address of another mac-vlan interface on the same host
isn't delivered. mac-vlan interfaces, even though they're conceptually
located on the same ethernet segment, are currently isolated from each
other for unicast traffic.

Thinking about the few scenarios where I've been using mac-vlans, it
seems to me that the fundamental goal of mac-vlans is to be
functionaly equivalent to having multiple physical ethernet network
cards attached to the same LAN segment, with those physical
interfaces attached together via a switch or a hub.

Virtualising that scenario using mac-vlans and a single physical
interface means that the outbound mac-vlan code on a host needs to
replicate the functionality of a switch or a hub. Outbound unicast
traffic from one mac-vlan interface needs to be either seen by the
destination address matching code of all other mac-vlan interfaces (i.e.
replicating hub functionality), or selectively forwarded to the
intended mac-vlan interface (replicating switch functionality). If the
unicast destination doesn't match one of the other mac-vlan interfaces,
then it should be forwarded out the mac-vlan interfaces' parent physical
interface, as the destination is likely to be on the physical ethernet.
Blindly forwarding the outbound mac-vlan origin traffic out the physical
interface doesn't solve the problem - switches (and I'm pretty sure at
least functionaly, hubs) don't forward traffic back out the interface
they receive it on. So the hub/switch forwarding functionality between
mac-vlan interfaces and their parent physical interface needs to happen
within the kernel.

I've been thinking of having a go at adding this functionality. But
then I realised that that would just be more duplication of the
functionality of the code that is aleady in the kernel - the bridging
code. It does all those things already.

It seems to me that the only reason the bridging code can't be used
today for "mac-vlan" functionality is that it only allows creation of a
single bridge virtual interface. If the bridging code and associated
utilities were extended to support multiple virtual interfaces
(possibly with the first interface being the "handle" interface for the
bridge instance), then the mac-vlan code would be redundant.
Conceptually, it'd also be more consistent with some of the other
mechanisms that can be added to a bridge, such as adding tap, dummy or
other virtual interfaces to a bridge in conjunction with multiple bridge
virtual interfaces - a scenario I tried to do a while back and asked
about in December in a post titled, "Is it valid to add a macvlan
virtual interface to a bridge? If so, there seems to be a bug with it."

Using the bridge method wouldn't allow the real physical
interface to be used as a normal e.g. IP interface, as is allowed when
using mac-vlans. But I don't think is much of an issue - another bridge
virtual interface could be created to be used instead.

Regards,
Mark.

(please CC me, as I'm not subscribed, although I do read the list via
gmane. Thanks.)

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Ben Greear]

Mark Smith wrote:
> Hi,
>
> Ben said,
>   
>> I wouldn't deny sending with wrong source mac..ethernet interfaces can 
>> do this,
>> and mac-vlan should look as much like ethernet is possible.
>>     
>
> I agree, however there's further things that mac-vlans aren't
> currently doing as virtual ethernet interfaces that real ones do.
> Unicast ethernet traffic sent out one mac-vlan interface with a
> destination address of another mac-vlan interface on the same host
> isn't delivered. mac-vlan interfaces, even though they're conceptually
> located on the same ethernet segment, are currently isolated from each
> other for unicast traffic.
>   
At least for my use, having them all blindly TX is fine.  For thousands
of interfaces, if you did this right and also delivered all broadcast 
packets locally
(ie, ARP), you will cause a lot of overhead, and unless you are running 
a patched
kernel (or namespaces perhaps), you can't really communicate with 
yourself over the
network anyway using IP.

For the behaviour you want, try adding pairs of VETH interfaces and add 
one end
of the veth's to the bridge.  Add a physical port to the bridge for 
egress.  Since this
can be done, I don't really see any reason to change mac-vlan 
significantly...

If the veth/bridge thing doesn't work, then let us know, as I think that 
would be
a bug.  I use a similar-to-veth virtual-device pair in this way and it 
works fine.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Eric W. Biederman]

Ben Greear <greearb@candelatech.com> writes:

> Mark Smith wrote:
>> Hi,
>>
>> Ben said,
>>   
>>> I wouldn't deny sending with wrong source mac..ethernet interfaces can do
>>> this,
>>> and mac-vlan should look as much like ethernet is possible.
>>>     
>>
>> I agree, however there's further things that mac-vlans aren't
>> currently doing as virtual ethernet interfaces that real ones do.
>> Unicast ethernet traffic sent out one mac-vlan interface with a
>> destination address of another mac-vlan interface on the same host
>> isn't delivered. mac-vlan interfaces, even though they're conceptually
>> located on the same ethernet segment, are currently isolated from each
>> other for unicast traffic.
>>   
> At least for my use, having them all blindly TX is fine.  For thousands
> of interfaces, if you did this right and also delivered all broadcast packets
> locally
> (ie, ARP), you will cause a lot of overhead, and unless you are running a
> patched
> kernel (or namespaces perhaps), you can't really communicate with yourself over
> the
> network anyway using IP.
>
> For the behaviour you want, try adding pairs of VETH interfaces and add one end
> of the veth's to the bridge.  Add a physical port to the bridge for egress.
> Since this
> can be done, I don't really see any reason to change mac-vlan significantly...
>
> If the veth/bridge thing doesn't work, then let us know, as I think that would
> be
> a bug.  I use a similar-to-veth virtual-device pair in this way and it works
> fine.

There is one scenario in which macvlans totally beat bridging veth
devices.  macvlans support the full set of stateless hardware
offloads that the hardware supports.  Whereas veth device support none
of them.

I don't think changing macvlans makes a lot of sense.  Beyond the
pain of making it work, there are the semantic differences of local
broadcast working.

Doing something so that bridges have roughly the same performance 
as macvlans would be very nice.  I think it requires advertising
most if not all stateless hardware offloads, and then implementing
them in software on the endpoints that don't support them.

I did get as far as implementing a first draft at looping packets back
locally and behaviour difference for broadcasts and multicast
differences made macvlans a bad fit.  For clean code something like
the bridge code where you don't use the original interface directly
for sending and receiving traffic seems required.

Eric

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Mark Smith]

On Sat, 07 Mar 2009 10:13:16 -0800
ebiederm@xmission.com (Eric W. Biederman) wrote:

> Ben Greear <greearb@candelatech.com> writes:
> 
> > Mark Smith wrote:
> >> Hi,
> >>
> >> Ben said,
> >>   
> >>> I wouldn't deny sending with wrong source mac..ethernet interfaces can do
> >>> this,
> >>> and mac-vlan should look as much like ethernet is possible.
> >>>     
> >>
> >> I agree, however there's further things that mac-vlans aren't
> >> currently doing as virtual ethernet interfaces that real ones do.
> >> Unicast ethernet traffic sent out one mac-vlan interface with a
> >> destination address of another mac-vlan interface on the same host
> >> isn't delivered. mac-vlan interfaces, even though they're conceptually
> >> located on the same ethernet segment, are currently isolated from each
> >> other for unicast traffic.
> >>   
> > At least for my use, having them all blindly TX is fine.  For thousands
> > of interfaces, if you did this right and also delivered all broadcast packets
> > locally
> > (ie, ARP), you will cause a lot of overhead, and unless you are running a
> > patched
> > kernel (or namespaces perhaps), you can't really communicate with yourself over
> > the
> > network anyway using IP.
> >
> > For the behaviour you want, try adding pairs of VETH interfaces and add one end
> > of the veth's to the bridge.  Add a physical port to the bridge for egress.
> > Since this
> > can be done, I don't really see any reason to change mac-vlan significantly...
> >
> > If the veth/bridge thing doesn't work, then let us know, as I think that would
> > be
> > a bug.  I use a similar-to-veth virtual-device pair in this way and it works
> > fine.
> 
> There is one scenario in which macvlans totally beat bridging veth
> devices.  macvlans support the full set of stateless hardware
> offloads that the hardware supports.  Whereas veth device support none
> of them.
> 
> I don't think changing macvlans makes a lot of sense.  Beyond the
> pain of making it work, there are the semantic differences of local
> broadcast working.
> 
> Doing something so that bridges have roughly the same performance 
> as macvlans would be very nice.  I think it requires advertising
> most if not all stateless hardware offloads, and then implementing
> them in software on the endpoints that don't support them.
> 
> I did get as far as implementing a first draft at looping packets back
> locally and behaviour difference for broadcasts and multicast
> differences made macvlans a bad fit.  For clean code something like
> the bridge code where you don't use the original interface directly
> for sending and receiving traffic seems required.
> 

So then, my question is, what are mac-vlans for i.e. what is their
common use case?

The problem I was trying to solve was to run up an arbitrary
number of PPPoE servers on a single LAN segment. I could do that
with physical interfaces, however I only had a maximum of 4 ethernet
interfaces in the host. Using mac-vlans seemed to be the obvious way to
eliminate the physical constraints of the host. I did expect though that
the mac-vlan virtual interfaces would work the same real interfaces, so
I was expecting that I could bridge them and that unicast traffic
between them would work.

If bridged veth pairs is a solution to my problem, what would I use
mac-vlans for? Ben seems to have a use case, however I know that he
does network testing, so some of his use cases could be quite uncommon.
I do some odd things with networking occasionally, so the mac-vlan
behaviour / limitations might be useful, but I'm curious if there would
be some more common uses?

Regards,
mark.

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Ben Greear]

Mark Smith wrote:
> On Sat, 07 Mar 2009 10:13:16 -0800
> ebiederm@xmission.com (Eric W. Biederman) wrote:
>
>   
>> Ben Greear <greearb@candelatech.com> writes:
>>
>>     
>>> Mark Smith wrote:
>>>       
>>>> Hi,
>>>>
>>>> Ben said,
>>>>   
>>>>         
>>>>> I wouldn't deny sending with wrong source mac..ethernet interfaces can do
>>>>> this,
>>>>> and mac-vlan should look as much like ethernet is possible.
>>>>>     
>>>>>           
>>>> I agree, however there's further things that mac-vlans aren't
>>>> currently doing as virtual ethernet interfaces that real ones do.
>>>> Unicast ethernet traffic sent out one mac-vlan interface with a
>>>> destination address of another mac-vlan interface on the same host
>>>> isn't delivered. mac-vlan interfaces, even though they're conceptually
>>>> located on the same ethernet segment, are currently isolated from each
>>>> other for unicast traffic.
>>>>   
>>>>         
>>> At least for my use, having them all blindly TX is fine.  For thousands
>>> of interfaces, if you did this right and also delivered all broadcast packets
>>> locally
>>> (ie, ARP), you will cause a lot of overhead, and unless you are running a
>>> patched
>>> kernel (or namespaces perhaps), you can't really communicate with yourself over
>>> the
>>> network anyway using IP.
>>>
>>> For the behaviour you want, try adding pairs of VETH interfaces and add one end
>>> of the veth's to the bridge.  Add a physical port to the bridge for egress.
>>> Since this
>>> can be done, I don't really see any reason to change mac-vlan significantly...
>>>
>>> If the veth/bridge thing doesn't work, then let us know, as I think that would
>>> be
>>> a bug.  I use a similar-to-veth virtual-device pair in this way and it works
>>> fine.
>>>       
>> There is one scenario in which macvlans totally beat bridging veth
>> devices.  macvlans support the full set of stateless hardware
>> offloads that the hardware supports.  Whereas veth device support none
>> of them.
>>
>> I don't think changing macvlans makes a lot of sense.  Beyond the
>> pain of making it work, there are the semantic differences of local
>> broadcast working.
>>
>> Doing something so that bridges have roughly the same performance 
>> as macvlans would be very nice.  I think it requires advertising
>> most if not all stateless hardware offloads, and then implementing
>> them in software on the endpoints that don't support them.
>>
>> I did get as far as implementing a first draft at looping packets back
>> locally and behaviour difference for broadcasts and multicast
>> differences made macvlans a bad fit.  For clean code something like
>> the bridge code where you don't use the original interface directly
>> for sending and receiving traffic seems required.
>>
>>     
>
> So then, my question is, what are mac-vlans for i.e. what is their
> common use case?
>
> The problem I was trying to solve was to run up an arbitrary
> number of PPPoE servers on a single LAN segment. I could do that
> with physical interfaces, however I only had a maximum of 4 ethernet
> interfaces in the host. Using mac-vlans seemed to be the obvious way to
> eliminate the physical constraints of the host. I did expect though that
> the mac-vlan virtual interfaces would work the same real interfaces, so
> I was expecting that I could bridge them and that unicast traffic
> between them would work.
>   
Doesn't pppoe always talk to an upstream box (the pppoe-server)?  If 
that is so,
why would the local mac-vlans ever need to communicate directly to 
eachother?

We've used pppoe on mac-vlans, and it *seemed* to work, but perhaps we 
were missing
something...

I think they might also be useful for adding a more realistic 'virtual 
ip' to an interface, perhaps
for interesting routing setups.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Mark Smith]

Hi Ben,

On Sun, 08 Mar 2009 09:54:02 -0700
Ben Greear <greearb@candelatech.com> wrote:

> Mark Smith wrote:
> > On Sat, 07 Mar 2009 10:13:16 -0800
> > ebiederm@xmission.com (Eric W. Biederman) wrote:
> >
> >   
> >> Ben Greear <greearb@candelatech.com> writes:
> >>
> >>     
> >>> Mark Smith wrote:
> >>>       
> >>>> Hi,
> >>>>
> >>>> Ben said,
> >>>>   
> >>>>         
> >>>>> I wouldn't deny sending with wrong source mac..ethernet interfaces can do
> >>>>> this,
> >>>>> and mac-vlan should look as much like ethernet is possible.
> >>>>>     
> >>>>>           
> >>>> I agree, however there's further things that mac-vlans aren't
> >>>> currently doing as virtual ethernet interfaces that real ones do.
> >>>> Unicast ethernet traffic sent out one mac-vlan interface with a
> >>>> destination address of another mac-vlan interface on the same host
> >>>> isn't delivered. mac-vlan interfaces, even though they're conceptually
> >>>> located on the same ethernet segment, are currently isolated from each
> >>>> other for unicast traffic.
> >>>>   

<snip for brevity>

> > So then, my question is, what are mac-vlans for i.e. what is their
> > common use case?
> >
> > The problem I was trying to solve was to run up an arbitrary
> > number of PPPoE servers on a single LAN segment. I could do that
> > with physical interfaces, however I only had a maximum of 4 ethernet
> > interfaces in the host. Using mac-vlans seemed to be the obvious way to
> > eliminate the physical constraints of the host. I did expect though that
> > the mac-vlan virtual interfaces would work the same real interfaces, so
> > I was expecting that I could bridge them and that unicast traffic
> > between them would work.
> >   
> Doesn't pppoe always talk to an upstream box (the pppoe-server)?  If 
> that is so,
> why would the local mac-vlans ever need to communicate directly to 
> eachother?
> 

That's true, and in that sense, I was 'lucky' that I didn't encounter
this limitation in my test environment, but it was only because of my
test environment (i.e. protocol), didn't require unicast communication
between the mac-vlan interfaces on the same host. If I'd been testing
some other protocol that did require unicast communications between
macvlan interfaces, I would have been scratching my head, wondering
why things weren't working correctly, and may have spent a lot of time
thinking that my test setup was the thing that had failed, rather than
being caused by missing functionality that I assumed was there.

I think the "truth in advertising" or "principle of least surprise"
should hold - if mac-vlans are to be seen by their users as virtual
ethernet network cards, then they should function no differently to real
network cards, or alternatively, if they aren't going to function that
way, there needs to be quite obvious documentation somewhere (other
than this thread) about their limitations, and a corresponding
recommendation that you use bridged veth interfaces if you can't afford
those limitations. 

> We've used pppoe on mac-vlans, and it *seemed* to work, but perhaps we 
> were missing
> something...
> 
> I think they might also be useful for adding a more realistic 'virtual 
> ip' to an interface, perhaps
> for interesting routing setups.
> 
> Thanks,
> Ben
> 
> -- 
> Ben Greear <greearb@candelatech.com> 
> Candela Technologies Inc  http://www.candelatech.com
> 
> 

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Patrick McHardy]

Mark Smith wrote:
> Hi,
> 
> Ben said,
>> I wouldn't deny sending with wrong source mac..ethernet interfaces can 
>> do this,
>> and mac-vlan should look as much like ethernet is possible.
> 
> I agree, however there's further things that mac-vlans aren't
> currently doing as virtual ethernet interfaces that real ones do.
> Unicast ethernet traffic sent out one mac-vlan interface with a
> destination address of another mac-vlan interface on the same host
> isn't delivered. mac-vlan interfaces, even though they're conceptually
> located on the same ethernet segment, are currently isolated from each
> other for unicast traffic.
> 
> Thinking about the few scenarios where I've been using mac-vlans, it
> seems to me that the fundamental goal of mac-vlans is to be
> functionaly equivalent to having multiple physical ethernet network
> cards attached to the same LAN segment, with those physical
> interfaces attached together via a switch or a hub.
> 
> Virtualising that scenario using mac-vlans and a single physical
> interface means that the outbound mac-vlan code on a host needs to
> replicate the functionality of a switch or a hub. Outbound unicast
> traffic from one mac-vlan interface needs to be either seen by the
> destination address matching code of all other mac-vlan interfaces (i.e.
> replicating hub functionality), or selectively forwarded to the
> intended mac-vlan interface (replicating switch functionality). If the
> unicast destination doesn't match one of the other mac-vlan interfaces,
> then it should be forwarded out the mac-vlan interfaces' parent physical
> interface, as the destination is likely to be on the physical ethernet.
> Blindly forwarding the outbound mac-vlan origin traffic out the physical
> interface doesn't solve the problem - switches (and I'm pretty sure at
> least functionaly, hubs) don't forward traffic back out the interface
> they receive it on. So the hub/switch forwarding functionality between
> mac-vlan interfaces and their parent physical interface needs to happen
> within the kernel.
> 
> I've been thinking of having a go at adding this functionality. But
> then I realised that that would just be more duplication of the
> functionality of the code that is aleady in the kernel - the bridging
> code. It does all those things already.
> 
> It seems to me that the only reason the bridging code can't be used
> today for "mac-vlan" functionality is that it only allows creation of a
> single bridge virtual interface. If the bridging code and associated
> utilities were extended to support multiple virtual interfaces
> (possibly with the first interface being the "handle" interface for the
> bridge instance), then the mac-vlan code would be redundant.
> Conceptually, it'd also be more consistent with some of the other
> mechanisms that can be added to a bridge, such as adding tap, dummy or
> other virtual interfaces to a bridge in conjunction with multiple bridge
> virtual interfaces - a scenario I tried to do a while back and asked
> about in December in a post titled, "Is it valid to add a macvlan
> virtual interface to a bridge? If so, there seems to be a bug with it."
> 
> Using the bridge method wouldn't allow the real physical
> interface to be used as a normal e.g. IP interface, as is allowed when
> using mac-vlans. But I don't think is much of an issue - another bridge
> virtual interface could be created to be used instead.

I agree on most points. There is one fundamental operational difference
however. With macvlan, all MAC addresses are known are therefore can be
programmed as secondary unicast addresses, while a bridge always uses
promiscous mode and for unknown addresses needs to flood forward them.

This could be changed in the bridging code of course for bridges
consisting purely of local devices. Most of the bridging stuff isn't
needed for macvlans though, so its probably easier to simply perform
a lookup for local devices in macvlan on transmit, similar to what
is done on reception.

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Eric W. Biederman]

Patrick McHardy <kaber@trash.net> writes:

> I agree on most points. There is one fundamental operational difference
> however. With macvlan, all MAC addresses are known are therefore can be
> programmed as secondary unicast addresses, while a bridge always uses
> promiscous mode and for unknown addresses needs to flood forward them.
>
> This could be changed in the bridging code of course for bridges
> consisting purely of local devices. Most of the bridging stuff isn't
> needed for macvlans though, so its probably easier to simply perform
> a lookup for local devices in macvlan on transmit, similar to what
> is done on reception.

What I haven't figured out is how you handle the transmit path for
broadcast and multicast ethernet traffic.  How do you test to see if
you have already preformed local transmission?

For discussion but not for application because it is incomplete:
This is what I came up with when I played with getting the local
transmission case working the other day.


>From 15e4a58ae0cea86338ef9d73ae14ba32e4819f5a Mon Sep 17 00:00:00 2001
From: Eric Biederman <ebiederm@xmission.com>
Date: Thu, 5 Mar 2009 07:46:10 -0800
Subject: [PATCH] macvlan: Reflect macvlan packets meant for other macvlan devices

Switch ports do not send packets back out the same port they came
in on.  This causes problems when using a macvlan device inside
of a network namespace as it becomes impossible to talk to
other macvlan devices.

Signed-off-by: Eric Biederman <ebiederm@aristanetworks.com>
---
 drivers/net/macvlan.c |   92 ++++++++++++++++++++++++++++++++++++-------------
 1 files changed, 68 insertions(+), 24 deletions(-)

diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index b5241fc..eb2539f 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -29,6 +29,7 @@
 #include <linux/if_link.h>
 #include <linux/if_macvlan.h>
 #include <net/rtnetlink.h>
+#include <net/xfrm.h>
 
 #define MACVLAN_HASH_SIZE	(1 << BITS_PER_BYTE)
 
@@ -61,7 +62,8 @@ static struct macvlan_dev *macvlan_hash_lookup(const struct macvlan_port *port,
 }
 
 static void macvlan_broadcast(struct sk_buff *skb,
-			      const struct macvlan_port *port)
+			      const struct macvlan_port *port,
+			      struct net_device *src)
 {
 	const struct ethhdr *eth = eth_hdr(skb);
 	const struct macvlan_dev *vlan;
@@ -77,6 +79,9 @@ static void macvlan_broadcast(struct sk_buff *skb,
 		hlist_for_each_entry_rcu(vlan, n, &port->vlan_hash[i], hlist) {
 			dev = vlan->dev;
 
+			if (dev == src)
+				continue;
+
 			nskb = skb_clone(skb, GFP_ATOMIC);
 			if (nskb == NULL) {
 				dev->stats.rx_errors++;
@@ -99,20 +104,45 @@ static void macvlan_broadcast(struct sk_buff *skb,
 	}
 }
 
+static int macvlan_unicast(struct sk_buff *skb, const struct macvlan_dev *dest)
+{
+	struct net_device *dev = dest->dev;
+
+	if (unlikely(!dev->flags & IFF_UP)) {
+		kfree_skb(skb);
+		return NET_XMIT_DROP;
+	}
+
+	skb = skb_share_check(skb, GFP_ATOMIC);
+	if (!skb) {
+		dev->stats.rx_errors++;
+		dev->stats.rx_dropped++;
+		return NET_XMIT_DROP;
+	}
+
+	dev->stats.rx_bytes += skb->len + ETH_HLEN;
+	dev->stats.rx_packets++;
+
+	skb->dev = dev;
+	skb->pkt_type = PACKET_HOST;
+	netif_rx(skb);
+	return NET_XMIT_SUCCESS;
+}
+
+
 /* called under rcu_read_lock() from netif_receive_skb */
 static struct sk_buff *macvlan_handle_frame(struct sk_buff *skb)
 {
 	const struct ethhdr *eth = eth_hdr(skb);
 	const struct macvlan_port *port;
 	const struct macvlan_dev *vlan;
-	struct net_device *dev;
 
 	port = rcu_dereference(skb->dev->macvlan_port);
 	if (port == NULL)
 		return skb;
 
 	if (is_multicast_ether_addr(eth->h_dest)) {
-		macvlan_broadcast(skb, port);
+		macvlan_broadcast(skb, port, NULL);
 		return skb;
 	}
 
@@ -120,38 +150,52 @@ static struct sk_buff *macvlan_handle_frame(struct sk_buff *skb)
 	if (vlan == NULL)
 		return skb;
 
-	dev = vlan->dev;
-	if (unlikely(!(dev->flags & IFF_UP))) {
-		kfree_skb(skb);
-		return NULL;
-	}
+	macvlan_unicast(skb, vlan);
+	return NULL;
+}
 
-	skb = skb_share_check(skb, GFP_ATOMIC);
-	if (skb == NULL) {
-		dev->stats.rx_errors++;
-		dev->stats.rx_dropped++;
-		return NULL;
-	}
+static int macvlan_xmit_world(struct sk_buff *skb, struct net_device *dev)
+{
+	const struct macvlan_dev *vlan = netdev_priv(dev);
+	__skb_push(skb, skb->data - skb_mac_header(skb));
+	skb->dev = vlan->lowerdev;
+	return dev_queue_xmit(skb);
+}
 
-	dev->stats.rx_bytes += skb->len + ETH_HLEN;
-	dev->stats.rx_packets++;
+static int macvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+	const struct macvlan_dev *vlan = netdev_priv(dev);
+	const struct macvlan_port *port = vlan->port;
+	const struct macvlan_dev *dest;
+	const struct ethhdr *eth;
 
-	skb->dev = dev;
-	skb->pkt_type = PACKET_HOST;
+	skb->protocol = eth_type_trans(skb, dev);
+	eth = eth_hdr(skb);
 
-	netif_rx(skb);
-	return NULL;
+	dst_release(skb->dst);
+	skb->dst = NULL;
+	skb->mark = 0;
+	secpath_reset(skb);
+	nf_reset(skb);
+
+	if (is_multicast_ether_addr(eth->h_dest)) {
+		macvlan_broadcast(skb, port, dev);
+		return macvlan_xmit_world(skb, dev);
+	}
+
+	dest = macvlan_hash_lookup(port, eth->h_dest);
+	if (dest)
+		return macvlan_unicast(skb, dest);
+			
+	return macvlan_xmit_world(skb, dev);
 }
 
 static int macvlan_start_xmit(struct sk_buff *skb, struct net_device *dev)
 {
-	const struct macvlan_dev *vlan = netdev_priv(dev);
 	unsigned int len = skb->len;
 	int ret;
 
-	skb->dev = vlan->lowerdev;
-	ret = dev_queue_xmit(skb);
-
+	ret = macvlan_queue_xmit(skb, dev);
 	if (likely(ret == NET_XMIT_SUCCESS)) {
 		dev->stats.tx_packets++;
 		dev->stats.tx_bytes += len;
-- 
1.6.1.2.350.g88cc

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces? (was Re: [PATCH] macvlan: Support creating macvlans from macvlans)

[Patrick McHardy]

Eric W. Biederman wrote:
> Patrick McHardy <kaber@trash.net> writes:
> 
>> I agree on most points. There is one fundamental operational difference
>> however. With macvlan, all MAC addresses are known are therefore can be
>> programmed as secondary unicast addresses, while a bridge always uses
>> promiscous mode and for unknown addresses needs to flood forward them.
>>
>> This could be changed in the bridging code of course for bridges
>> consisting purely of local devices. Most of the bridging stuff isn't
>> needed for macvlans though, so its probably easier to simply perform
>> a lookup for local devices in macvlan on transmit, similar to what
>> is done on reception.
> 
> What I haven't figured out is how you handle the transmit path for
> broadcast and multicast ethernet traffic.  How do you test to see if
> you have already preformed local transmission?

I'm not sure I understand the problem. Whats wrong with doing
the same as on transmit, i.e.:

- for multicast/broadcast, deliver everywhere (except self)

- for unicast, deliver to matching local macvlan device or
   underlying device

> +static int macvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev)
> +{
> +	const struct macvlan_dev *vlan = netdev_priv(dev);
> +	const struct macvlan_port *port = vlan->port;
> +	const struct macvlan_dev *dest;
> +	const struct ethhdr *eth;
>  
> -	skb->dev = dev;
> -	skb->pkt_type = PACKET_HOST;
> +	skb->protocol = eth_type_trans(skb, dev);
> +	eth = eth_hdr(skb);
>  
> -	netif_rx(skb);
> -	return NULL;
> +	dst_release(skb->dst);
> +	skb->dst = NULL;
> +	skb->mark = 0;
> +	secpath_reset(skb);
> +	nf_reset(skb);
> +
> +	if (is_multicast_ether_addr(eth->h_dest)) {
> +		macvlan_broadcast(skb, port, dev);
> +		return macvlan_xmit_world(skb, dev);
> +	}
> +
> +	dest = macvlan_hash_lookup(port, eth->h_dest);
> +	if (dest)
> +		return macvlan_unicast(skb, dest);
> +			
> +	return macvlan_xmit_world(skb, dev);
>  }

Pretty much like this :)

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Eric W. Biederman]

Patrick McHardy <kaber@trash.net> writes:

> Eric W. Biederman wrote:
>> Patrick McHardy <kaber@trash.net> writes:
>>
>>> I agree on most points. There is one fundamental operational difference
>>> however. With macvlan, all MAC addresses are known are therefore can be
>>> programmed as secondary unicast addresses, while a bridge always uses
>>> promiscous mode and for unknown addresses needs to flood forward them.
>>>
>>> This could be changed in the bridging code of course for bridges
>>> consisting purely of local devices. Most of the bridging stuff isn't
>>> needed for macvlans though, so its probably easier to simply perform
>>> a lookup for local devices in macvlan on transmit, similar to what
>>> is done on reception.
>>
>> What I haven't figured out is how you handle the transmit path for
>> broadcast and multicast ethernet traffic.  How do you test to see if
>> you have already preformed local transmission?
>
> I'm not sure I understand the problem. Whats wrong with doing
> the same as on transmit, i.e.:
>
> - for multicast/broadcast, deliver everywhere (except self)
>
> - for unicast, deliver to matching local macvlan device or
>   underlying device
>
>> +static int macvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev)
>> +{
>> +	const struct macvlan_dev *vlan = netdev_priv(dev);
>> +	const struct macvlan_port *port = vlan->port;
>> +	const struct macvlan_dev *dest;
>> +	const struct ethhdr *eth;
>>  -	skb->dev = dev;
>> -	skb->pkt_type = PACKET_HOST;
>> +	skb->protocol = eth_type_trans(skb, dev);
>> +	eth = eth_hdr(skb);
>>  -	netif_rx(skb);
>> -	return NULL;
>> +	dst_release(skb->dst);
>> +	skb->dst = NULL;
>> +	skb->mark = 0;
>> +	secpath_reset(skb);
>> +	nf_reset(skb);
>> +
>> +	if (is_multicast_ether_addr(eth->h_dest)) {
>> +		macvlan_broadcast(skb, port, dev);
>> +		return macvlan_xmit_world(skb, dev);
>> +	}
>> +
>> +	dest = macvlan_hash_lookup(port, eth->h_dest);
>> +	if (dest)
>> +		return macvlan_unicast(skb, dest);
>> +			
>> +	return macvlan_xmit_world(skb, dev);
>>  }
>
> Pretty much like this :)

Yes.

There are two tricky parts.

One problem is that macvlans and the primary hardware device share the
same transmit queue.  So when I have a broadcast packet on the primary
devices queue I don't know if I have already sent it out to the
macvlan devices or not.

The second problem is that when I transmit a multicast packet and I
have a local listener.  I believe replicating the packet both at the
ip layer and at the ethernet layer will result in receiving the packet
locally twice.

I'm not certain we need to solve the second problem as having two physical
interfaces plugged into a switch will have the same problem.

The first problem is all about how do we deliver packets everywhere except self.

Eric

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Patrick McHardy]

Eric W. Biederman wrote:
> There are two tricky parts.
> 
> One problem is that macvlans and the primary hardware device share the
> same transmit queue.  So when I have a broadcast packet on the primary
> devices queue I don't know if I have already sent it out to the
> macvlan devices or not.

So its about receiving packets on macvlan when transmitting on the
real device? That sounds like a really hard problem that would probably
indeed be better solved by a bridge.

If its just about whether the packet should be sent out by macvlan
to the wire as well, I'd say yes since thats what two real devices
would have done.

> The second problem is that when I transmit a multicast packet and I
> have a local listener.  I believe replicating the packet both at the
> ip layer and at the ethernet layer will result in receiving the packet
> locally twice.
> 
> I'm not certain we need to solve the second problem as having two physical
> interfaces plugged into a switch will have the same problem.

Agreed.

> The first problem is all about how do we deliver packets everywhere except self.

I think "except self should just mean "not to the originating virtual
device".

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Eric W. Biederman]

Patrick McHardy <kaber@trash.net> writes:

> Eric W. Biederman wrote:
>> There are two tricky parts.
>>
>> One problem is that macvlans and the primary hardware device share the
>> same transmit queue.  So when I have a broadcast packet on the primary
>> devices queue I don't know if I have already sent it out to the
>> macvlan devices or not.
>
> So its about receiving packets on macvlan when transmitting on the
> real device? That sounds like a really hard problem that would probably
> indeed be better solved by a bridge.

Yes.  My concern is that if we hook the real device we will
software broadcast packets twice.

Now that I think about it we could call ndo_start_xmit directly
from the macvlan code, and bypass whatever hook we use to
intercept packets going out the normal device it should not
be too difficult.

Operationally it would be very nice if arp worked between a macvlan
and the real device.

Eric

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Patrick McHardy]

Eric W. Biederman wrote:
> Patrick McHardy <kaber@trash.net> writes:
> 
>> Eric W. Biederman wrote:
>>> There are two tricky parts.
>>>
>>> One problem is that macvlans and the primary hardware device share the
>>> same transmit queue.  So when I have a broadcast packet on the primary
>>> devices queue I don't know if I have already sent it out to the
>>> macvlan devices or not.
>> So its about receiving packets on macvlan when transmitting on the
>> real device? That sounds like a really hard problem that would probably
>> indeed be better solved by a bridge.
> 
> Yes.  My concern is that if we hook the real device we will
> software broadcast packets twice.

It would also require an additional hook in the networking core,
right? Which doesn't seem appropriate for a corner case like this.

> Now that I think about it we could call ndo_start_xmit directly
> from the macvlan code, and bypass whatever hook we use to
> intercept packets going out the normal device it should not
> be too difficult.

We don't intercept packets on TX, they have to be explicitly delivered
to macvlan.

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Brian Haley]

Eric W. Biederman wrote:
> There are two tricky parts.
> 
> One problem is that macvlans and the primary hardware device share the
> same transmit queue.  So when I have a broadcast packet on the primary
> devices queue I don't know if I have already sent it out to the
> macvlan devices or not.
> 
> The second problem is that when I transmit a multicast packet and I
> have a local listener.  I believe replicating the packet both at the
> ip layer and at the ethernet layer will result in receiving the packet
> locally twice.

If you want a local listener to see the packet you have to set
IP_MULTICAST_LOOP.  A packet from a local source address coming off the "wire"
will be dropped in fib_validate_source(), it would have to come over lo.  I'm
not sure how that relates to how macvlan works, just something I've run into
before (2 nics in same subnet, listener on one, sender on other,
IP_MULTICAST_LOOP=0, no packets).

-Brian

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Ben Greear]

Eric W. Biederman wrote:
> Patrick McHardy <kaber@trash.net> writes:
> 
>> Eric W. Biederman wrote:
>>> Patrick McHardy <kaber@trash.net> writes:
>>>
>>>> I agree on most points. There is one fundamental operational difference
>>>> however. With macvlan, all MAC addresses are known are therefore can be
>>>> programmed as secondary unicast addresses, while a bridge always uses
>>>> promiscous mode and for unknown addresses needs to flood forward them.
>>>>
>>>> This could be changed in the bridging code of course for bridges
>>>> consisting purely of local devices. Most of the bridging stuff isn't
>>>> needed for macvlans though, so its probably easier to simply perform
>>>> a lookup for local devices in macvlan on transmit, similar to what
>>>> is done on reception.
>>> What I haven't figured out is how you handle the transmit path for
>>> broadcast and multicast ethernet traffic.  How do you test to see if
>>> you have already preformed local transmission?
>> I'm not sure I understand the problem. Whats wrong with doing
>> the same as on transmit, i.e.:
>>
>> - for multicast/broadcast, deliver everywhere (except self)
>>
>> - for unicast, deliver to matching local macvlan device or
>>   underlying device
>>
>>> +static int macvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev)
>>> +{
>>> +	const struct macvlan_dev *vlan = netdev_priv(dev);
>>> +	const struct macvlan_port *port = vlan->port;
>>> +	const struct macvlan_dev *dest;
>>> +	const struct ethhdr *eth;
>>>  -	skb->dev = dev;
>>> -	skb->pkt_type = PACKET_HOST;
>>> +	skb->protocol = eth_type_trans(skb, dev);
>>> +	eth = eth_hdr(skb);
>>>  -	netif_rx(skb);
>>> -	return NULL;
>>> +	dst_release(skb->dst);
>>> +	skb->dst = NULL;
>>> +	skb->mark = 0;
>>> +	secpath_reset(skb);
>>> +	nf_reset(skb);
>>> +
>>> +	if (is_multicast_ether_addr(eth->h_dest)) {
>>> +		macvlan_broadcast(skb, port, dev);
>>> +		return macvlan_xmit_world(skb, dev);
>>> +	}
>>> +
>>> +	dest = macvlan_hash_lookup(port, eth->h_dest);
>>> +	if (dest)
>>> +		return macvlan_unicast(skb, dest);
>>> +			
>>> +	return macvlan_xmit_world(skb, dev);
>>>  }
>> Pretty much like this :)
> 
> Yes.
> 
> There are two tricky parts.
> 
> One problem is that macvlans and the primary hardware device share the
> same transmit queue.  So when I have a broadcast packet on the primary
> devices queue I don't know if I have already sent it out to the
> macvlan devices or not.

A flag could be added to the skb so that we know it originated from
a mac-vlan.  That shouldn't require any extra hooks but just an extra
check in the mac-vlan rx code to drop any pkt received from the
underlying NIC with this flag set.

For broadcasting (or unicasting) to other mac-vlans or the underlying physical
device, the mac-vlan tx logic could check for local delivery before telling the
lower-level NIC to transmit the pkt.

Since we already have a mac hash, we could probably key off of the dest MAC
fairly easily.  For broadcast, it would be a flood.  We could also add a
flag to mac-vlan tx logic to be clever and only send ARP to the mac-vlans
likely to care.  This might not be a good filter for all possible cases, but
for general cases, and thousands of mac-vlans, it would save a lot of work
cheaply.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Ben Greear]

Patrick McHardy wrote:

>> Now that I think about it we could call ndo_start_xmit directly
>> from the macvlan code, and bypass whatever hook we use to
>> intercept packets going out the normal device it should not
>> be too difficult.
> 
> We don't intercept packets on TX, they have to be explicitly delivered
> to macvlan.

It might suck for performance, but mac-vlan could register an 'ALL' protocol
on the physical dev, similar to tcp-dump, to grab pkts on tx and pass the
ones it cares about back up to the vlans?

I'd want run-time control to disable any of these costly options for those that
don't need it, however.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Eric W. Biederman]

Ben Greear <greearb@candelatech.com> writes:

> Patrick McHardy wrote:
>
>>> Now that I think about it we could call ndo_start_xmit directly
>>> from the macvlan code, and bypass whatever hook we use to
>>> intercept packets going out the normal device it should not
>>> be too difficult.
>>
>> We don't intercept packets on TX, they have to be explicitly delivered
>> to macvlan.
>
> It might suck for performance, but mac-vlan could register an 'ALL' protocol
> on the physical dev, similar to tcp-dump, to grab pkts on tx and pass the
> ones it cares about back up to the vlans?

I like that idea.  At least for prototyping.

I wonder if pkt_type all could be have a per interface optimized variant.

> I'd want run-time control to disable any of these costly options for those that
> don't need it, however.

If well implemented it should not be more expensive than the ingress path where
we already have, and where we already do that.  Unless your traffic is highly
assymmetric.

Eric

Re: MACVLANs really best solution? How about a bridge with multiple bridge virtual interfaces?

[Ben Greear]

Eric W. Biederman wrote:
> Ben Greear <greearb@candelatech.com> writes:
> 
>> Patrick McHardy wrote:
>>
>>>> Now that I think about it we could call ndo_start_xmit directly
>>>> from the macvlan code, and bypass whatever hook we use to
>>>> intercept packets going out the normal device it should not
>>>> be too difficult.
>>> We don't intercept packets on TX, they have to be explicitly delivered
>>> to macvlan.
>> It might suck for performance, but mac-vlan could register an 'ALL' protocol
>> on the physical dev, similar to tcp-dump, to grab pkts on tx and pass the
>> ones it cares about back up to the vlans?
> 
> I like that idea.  At least for prototyping.
> 
> I wonder if pkt_type all could be have a per interface optimized variant.
> 
>> I'd want run-time control to disable any of these costly options for those that
>> don't need it, however.
> 
> If well implemented it should not be more expensive than the ingress path where
> we already have, and where we already do that.  Unless your traffic is highly
> assymmetric.

Well, the ingress path isn't free, and especially for broadcast pkts it is quite
expensive with large numbers of devices.

In a single namespace implementation, there are very few uses for having two
NICs on the same system able to send to each other since an un-patched kernel
will not do IPv4 traffic between two external ports, and multicast loops back
in software already.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com