[RFC 0/3] VLANs, DSA switches and multiple bridges

[RFC 0/3] VLANs, DSA switches and multiple bridges

[Russell King - ARM Linux admin]

Hi,

I've been trying to configure DSA for VLANs and not having much success.
The setup is quite simple:

- The main network is untagged
- The wifi network is a vlan tagged with id $VN running over the main
  network.

I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to
setup to provide wifi access to the vlan $VN network, while the switch
is also part of the main network.

However, I'm encountering problems:

1) vlan support in DSA has a different behaviour from the Linux
   software bridge implementation.

    # bridge vlan
    port    vlan ids
    lan1     1 PVID Egress Untagged
    ...

   shows the default setup - the bridge ports are all configured for
   vlan 1, untagged egress, and vlan 1 as the port vid.  Issuing:

    # ip li set dev br0 type bridge vlan_filtering 1

   with no other vlan configuration commands on a Linux software bridge
   continues to allow untagged traffic to flow across the bridge.
   
   This difference in behaviour is because the MV88E6xxx VTU is
   completely empty - because net/dsa ignores all vlan settings for
   a port if br_vlan_enabled(dp->bridge_dev) is false - this reflects
   the vlan filtering state of the bridge, not whether the bridge is
   vlan aware.
   
   What this means is that attempting to configure the bridge port
   vlans before enabling vlan filtering works for Linux software
   bridges, but fails for DSA bridges.

2) Assuming the above is sorted, we move on to the next issue, which
   is altogether more weird.  Let's take a setup where we have a
   DSA bridge with lan1..6 in a bridge device, br0, with vlan
   filtering enabled.  lan1 is the upstream port, lan2 is a downstream
   port that also wants to see traffic on vlan id $VN.

   Both lan1 and lan2 are configured for that:

     # bridge vlan add vid $VN dev lan1
     # bridge vlan add vid $VN dev lan2
     # ip li set br0 type bridge vlan_filtering 1

   Untagged traffic can now pass between all the six lan ports, and
   vlan $VN between lan1 and lan2 only.  The MV88E6xxx 8021q_mode
   debugfs file shows all lan ports are in mode "secure" - this is
   important!  /sys/class/net/br0/bridge/vlan_filtering contains 1.

   tcpdumping from another machine on lan4 shows that no $VN traffic
   reaches it.  Everything seems to be working correctly...
   
   In order to further bridge vlan $VN traffic to hostapd's wifi
   interface, things get a little more complex - we can't add hostapd's
   wifi interface to br0 directly, because hostapd will bring up the
   wifi interface and leak the main, untagged traffic onto the wifi.
   (hostapd does have vlan support, but only as a dynamic per-client
   thing, and there's no hooks I can see to allow script-based config
   of the network setup before hostapd up's the wifi interface.)

   So, what I tried was:

     # ip li add link br0 name br0.$VN type vlan id $VN
     # bridge vlan add vid $VN dev br0 self
     # ip li set dev br0.$VN up

   So far so good, we get a vlan interface on top of the bridge, and
   tcpdumping it shows we get traffic.  The 8021q_mode file has not
   changed state.  Everything still seems to be correct.

     # bridge addbr br1

   Still nothing has changed.

     # bridge addif br1 br0.$VN

   And now the 8021q_mode debugfs file shows that all ports are now in
   "disabled" mode, but /sys/class/net/br0/bridge/vlan_filtering still
   contains '1'.  In other words, br0 still thinks vlan filtering is
   enabled, but the hardware has had vlan filtering disabled.

   Adding some stack traces to an appropriate point indicates that this
   is because __switchdev_handle_port_attr_set() recurses down through
   the tree of interfaces, skipping over the vlan interface, applying
   br1's configuration to br0's ports.

   This surely can not be right - surely
   __switchdev_handle_port_attr_set() and similar should stop recursing
   down through another master bridge device?  There are probably other
   network device classes that switchdev shouldn't recurse down too.

   I've considered whether switchdev is the right level to do it, and
   I think it is - as we want the check/set callbacks to be called for
   the top level device even if it is a master bridge device, but we
   don't want to recurse through a lower master bridge device.

Some proposed patches are attached.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

[RFC 1/3] net: switchdev: do not propagate bridge updates across bridges

[Russell King]

When configuring a tree of independent bridges, propagating changes
from the upper bridge across a bridge master to the lower bridge
ports brings surprises.

For example, a lower bridge may have vlan filtering enabled.  It
may have a vlan interface attached to the bridge master, which may
then be incorporated into another bridge.  As soon as the lower
bridge vlan interface is attached to the upper bridge, the lower
bridge has vlan filtering disabled.

This occurs because switchdev recursively applies its changes to
all lower devices no matter what.

Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
---
 net/switchdev/switchdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/net/switchdev/switchdev.c b/net/switchdev/switchdev.c
index 3a1d428c1336..d881e5e4a889 100644
--- a/net/switchdev/switchdev.c
+++ b/net/switchdev/switchdev.c
@@ -475,6 +475,9 @@ static int __switchdev_handle_port_obj_add(struct net_device *dev,
 	 * necessary to go through this helper.
 	 */
 	netdev_for_each_lower_dev(dev, lower_dev, iter) {
+		if (netif_is_bridge_master(lower_dev))
+			continue;
+
 		err = __switchdev_handle_port_obj_add(lower_dev, port_obj_info,
 						      check_cb, add_cb);
 		if (err && err != -EOPNOTSUPP)
@@ -526,6 +529,9 @@ static int __switchdev_handle_port_obj_del(struct net_device *dev,
 	 * necessary to go through this helper.
 	 */
 	netdev_for_each_lower_dev(dev, lower_dev, iter) {
+		if (netif_is_bridge_master(lower_dev))
+			continue;
+
 		err = __switchdev_handle_port_obj_del(lower_dev, port_obj_info,
 						      check_cb, del_cb);
 		if (err && err != -EOPNOTSUPP)
@@ -576,6 +582,9 @@ static int __switchdev_handle_port_attr_set(struct net_device *dev,
 	 * necessary to go through this helper.
 	 */
 	netdev_for_each_lower_dev(dev, lower_dev, iter) {
+		if (netif_is_bridge_master(lower_dev))
+			continue;
+
 		err = __switchdev_handle_port_attr_set(lower_dev, port_attr_info,
 						       check_cb, set_cb);
 		if (err && err != -EOPNOTSUPP)
-- 
2.20.1

[RFC 2/3] net: dsa: mv88e6xxx: fix duplicate vlan warning

[Russell King]

When setting VLANs on DSA switches, the VLAN is added to both the port
concerned as well as the CPU port by dsa_slave_vlan_add().  If multiple
ports are configured with the same VLAN ID, this triggers a warning on
the CPU port.

Avoid this warning for CPU ports.

Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
---
 drivers/net/dsa/mv88e6xxx/chip.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/dsa/mv88e6xxx/chip.c b/drivers/net/dsa/mv88e6xxx/chip.c
index 0aad22f99fb5..cd79ee14ea5f 100644
--- a/drivers/net/dsa/mv88e6xxx/chip.c
+++ b/drivers/net/dsa/mv88e6xxx/chip.c
@@ -1760,7 +1760,7 @@ static int mv88e6xxx_broadcast_setup(struct mv88e6xxx_chip *chip, u16 vid)
 }
 
 static int mv88e6xxx_port_vlan_join(struct mv88e6xxx_chip *chip, int port,
-				    u16 vid, u8 member)
+				    u16 vid, u8 member, bool warn)
 {
 	const u8 non_member = MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_NON_MEMBER;
 	struct mv88e6xxx_vtu_entry vlan;
@@ -1805,7 +1805,7 @@ static int mv88e6xxx_port_vlan_join(struct mv88e6xxx_chip *chip, int port,
 		err = mv88e6xxx_vtu_loadpurge(chip, &vlan);
 		if (err)
 			return err;
-	} else {
+	} else if (warn) {
 		dev_info(chip->dev, "p%d: already a member of VLAN %d\n",
 			 port, vid);
 	}
@@ -1819,6 +1819,7 @@ static void mv88e6xxx_port_vlan_add(struct dsa_switch *ds, int port,
 	struct mv88e6xxx_chip *chip = ds->priv;
 	bool untagged = vlan->flags & BRIDGE_VLAN_INFO_UNTAGGED;
 	bool pvid = vlan->flags & BRIDGE_VLAN_INFO_PVID;
+	bool warn;
 	u8 member;
 	u16 vid;
 
@@ -1832,10 +1833,15 @@ static void mv88e6xxx_port_vlan_add(struct dsa_switch *ds, int port,
 	else
 		member = MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_TAGGED;
 
+	/* net/dsa/slave.c will call dsa_port_vlan_add() for the affected port
+	 * and then the CPU port. Do not warn for duplicates for the CPU port.
+	 */
+	warn = !dsa_is_cpu_port(ds, port);
+
 	mv88e6xxx_reg_lock(chip);
 
 	for (vid = vlan->vid_begin; vid <= vlan->vid_end; ++vid)
-		if (mv88e6xxx_port_vlan_join(chip, port, vid, member))
+		if (mv88e6xxx_port_vlan_join(chip, port, vid, member, warn))
 			dev_err(ds->dev, "p%d: failed to add VLAN %d%c\n", port,
 				vid, untagged ? 'u' : 't');
 
-- 
2.20.1

[RFC 3/3] net: dsa: mv88e6xxx: fix vlan setup

[Russell King]

DSA assumes that a bridge which has vlan filtering disabled is not
vlan aware, and ignores all vlan configuration. However, the kernel
software bridge code allows configuration in this state.

This causes the kernel's idea of the bridge vlan state and the
hardware state to disagree, so "bridge vlan show" indicates a correct
configuration but the hardware lacks all configuration. Even worse,
enabling vlan filtering on a DSA bridge immediately blocks all traffic
which, given the output of "bridge vlan show", is very confusing.

Provide an option that drivers can set to indicate they want to receive
vlan configuration even when vlan filtering is disabled. This is safe
for Marvell DSA bridges, which do not look up ingress traffic in the
VTU if the port is in 8021Q disabled state. Whether this change is
suitable for all DSA bridges is not known.

Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
---
 drivers/net/dsa/mv88e6xxx/chip.c |  1 +
 include/net/dsa.h                |  1 +
 net/dsa/slave.c                  | 12 ++++++++----
 3 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/net/dsa/mv88e6xxx/chip.c b/drivers/net/dsa/mv88e6xxx/chip.c
index cd79ee14ea5f..650b101caad7 100644
--- a/drivers/net/dsa/mv88e6xxx/chip.c
+++ b/drivers/net/dsa/mv88e6xxx/chip.c
@@ -2680,6 +2680,7 @@ static int mv88e6xxx_setup(struct dsa_switch *ds)
 
 	chip->ds = ds;
 	ds->slave_mii_bus = mv88e6xxx_default_mdio_bus(chip);
+	ds->vlan_bridge_vtu = true;
 
 	mv88e6xxx_reg_lock(chip);
 
diff --git a/include/net/dsa.h b/include/net/dsa.h
index 99f71be8cfdb..5e5c1a95af1a 100644
--- a/include/net/dsa.h
+++ b/include/net/dsa.h
@@ -269,6 +269,7 @@ struct dsa_switch {
 	 * settings on ports if not hardware-supported
 	 */
 	bool			vlan_filtering_is_global;
+	bool			vlan_bridge_vtu;
 
 	/* In case vlan_filtering_is_global is set, the VLAN awareness state
 	 * should be retrieved from here and not from the per-port settings.
diff --git a/net/dsa/slave.c b/net/dsa/slave.c
index a7c1e7ded8ca..6d167713da83 100644
--- a/net/dsa/slave.c
+++ b/net/dsa/slave.c
@@ -323,7 +323,8 @@ static int dsa_slave_vlan_add(struct net_device *dev,
 	if (obj->orig_dev != dev)
 		return -EOPNOTSUPP;
 
-	if (dp->bridge_dev && !br_vlan_enabled(dp->bridge_dev))
+	if (dp->bridge_dev && !dp->ds->vlan_bridge_vtu &&
+	    !br_vlan_enabled(dp->bridge_dev))
 		return 0;
 
 	vlan = *SWITCHDEV_OBJ_PORT_VLAN(obj);
@@ -390,7 +391,8 @@ static int dsa_slave_vlan_del(struct net_device *dev,
 	if (obj->orig_dev != dev)
 		return -EOPNOTSUPP;
 
-	if (dp->bridge_dev && !br_vlan_enabled(dp->bridge_dev))
+	if (dp->bridge_dev && !dp->ds->vlan_bridge_vtu &&
+	    !br_vlan_enabled(dp->bridge_dev))
 		return 0;
 
 	/* Do not deprogram the CPU port as it may be shared with other user
@@ -1122,7 +1124,8 @@ static int dsa_slave_vlan_rx_add_vid(struct net_device *dev, __be16 proto,
 	 * need to emulate the switchdev prepare + commit phase.
 	 */
 	if (dp->bridge_dev) {
-		if (!br_vlan_enabled(dp->bridge_dev))
+		if (!dp->ds->vlan_bridge_vtu &&
+		    !br_vlan_enabled(dp->bridge_dev))
 			return 0;
 
 		/* br_vlan_get_info() returns -EINVAL or -ENOENT if the
@@ -1156,7 +1159,8 @@ static int dsa_slave_vlan_rx_kill_vid(struct net_device *dev, __be16 proto,
 	 * need to emulate the switchdev prepare + commit phase.
 	 */
 	if (dp->bridge_dev) {
-		if (!br_vlan_enabled(dp->bridge_dev))
+		if (!dp->ds->vlan_bridge_vtu &&
+		    !br_vlan_enabled(dp->bridge_dev))
 			return 0;
 
 		/* br_vlan_get_info() returns -EINVAL or -ENOENT if the
-- 
2.20.1

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Andrew Lunn]

On Sun, Dec 22, 2019 at 07:22:35PM +0000, Russell King - ARM Linux admin wrote:
> Hi,
> 
> I've been trying to configure DSA for VLANs and not having much success.

Hi Russell

I'm hoping Vivien will review these patches. He understands all the
VLAN things better than i.

     Andrew

Re: [RFC 2/3] net: dsa: mv88e6xxx: fix duplicate vlan warning

[Florian Fainelli]

On 12/22/2019 11:24 AM, Russell King wrote:
> When setting VLANs on DSA switches, the VLAN is added to both the port
> concerned as well as the CPU port by dsa_slave_vlan_add().  If multiple
> ports are configured with the same VLAN ID, this triggers a warning on
> the CPU port.
> 
> Avoid this warning for CPU ports.
> 
> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>

Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
-- 
Florian

Re: [RFC 3/3] net: dsa: mv88e6xxx: fix vlan setup

[Florian Fainelli]

+Ido,

On 12/22/2019 11:24 AM, Russell King wrote:
> DSA assumes that a bridge which has vlan filtering disabled is not
> vlan aware, and ignores all vlan configuration. However, the kernel
> software bridge code allows configuration in this state.
> 
> This causes the kernel's idea of the bridge vlan state and the
> hardware state to disagree, so "bridge vlan show" indicates a correct
> configuration but the hardware lacks all configuration. Even worse,
> enabling vlan filtering on a DSA bridge immediately blocks all traffic
> which, given the output of "bridge vlan show", is very confusing.
> 
> Provide an option that drivers can set to indicate they want to receive
> vlan configuration even when vlan filtering is disabled. This is safe
> for Marvell DSA bridges, which do not look up ingress traffic in the
> VTU if the port is in 8021Q disabled state. Whether this change is
> suitable for all DSA bridges is not known.

s/DSA bridges/DSA switches/ ?

this is also safe to do with b53 switches in fact this is even desirable
because VLAN filtering is a global attribute so if you have at least one
bridge that spans one of your switch ports and that bridge requests
vlan_filtering=1, you *must* have a valid VID entry for the non-bridged
ports, or the bridged ports with vlan_filtering=0 otherwise there is no
default VID entry programmed to ingress the frame. Today this is
achieved by making sure that the default untagged VID 0 for non bridged
ports is always programmed at start-up and the switch is always
configured with VLAN awareness.

> 
> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>

This ties in with this commit:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2ea7a679ca2abd251c1ec03f20508619707e1749

which I believe is still correct in the sense that with Linux a bridge
with vlan_filtering=0 also means that the bridge is not VLAN aware. Ido,
Jiri, do you disagree?

This seems to be coming every now and then, so maybe it is time to
revisit this documentation patch:

https://github.com/ffainelli/linux/commit/3fe61b1722a3b79d2e317a812c54f3afc902e5b0
-- 
Florian

Re: [RFC 3/3] net: dsa: mv88e6xxx: fix vlan setup

[Ido Schimmel]

On Mon, Dec 23, 2019 at 10:02:12AM -0800, Florian Fainelli wrote:
> +Ido,
> 
> On 12/22/2019 11:24 AM, Russell King wrote:
> > DSA assumes that a bridge which has vlan filtering disabled is not
> > vlan aware, and ignores all vlan configuration. However, the kernel
> > software bridge code allows configuration in this state.
> > 
> > This causes the kernel's idea of the bridge vlan state and the
> > hardware state to disagree, so "bridge vlan show" indicates a correct
> > configuration but the hardware lacks all configuration. Even worse,
> > enabling vlan filtering on a DSA bridge immediately blocks all traffic
> > which, given the output of "bridge vlan show", is very confusing.
> > 
> > Provide an option that drivers can set to indicate they want to receive
> > vlan configuration even when vlan filtering is disabled. This is safe
> > for Marvell DSA bridges, which do not look up ingress traffic in the
> > VTU if the port is in 8021Q disabled state. Whether this change is
> > suitable for all DSA bridges is not known.
> 
> s/DSA bridges/DSA switches/ ?
> 
> this is also safe to do with b53 switches in fact this is even desirable
> because VLAN filtering is a global attribute so if you have at least one
> bridge that spans one of your switch ports and that bridge requests
> vlan_filtering=1, you *must* have a valid VID entry for the non-bridged
> ports, or the bridged ports with vlan_filtering=0 otherwise there is no
> default VID entry programmed to ingress the frame. Today this is
> achieved by making sure that the default untagged VID 0 for non bridged
> ports is always programmed at start-up and the switch is always
> configured with VLAN awareness.
> 
> > 
> > Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
> 
> This ties in with this commit:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2ea7a679ca2abd251c1ec03f20508619707e1749
> 
> which I believe is still correct in the sense that with Linux a bridge
> with vlan_filtering=0 also means that the bridge is not VLAN aware. Ido,
> Jiri, do you disagree?

Hi Florian,

Yes, vlan_filtering=0 means that the bridge is not VLAN aware. It's not
only about filtering at ingress / egress. The man page says "When
disabled, the bridge will not consider the VLAN tag when handling
packets." It also affects the VLAN with which FDB entries are learned
and the VLAN used for FDB lookup.

It is really problematic for switch drivers to properly support the
dynamic toggling of this option. This is why mlxsw forbids this
toggling. Either you create the bridge with VLAN filtering disabled or
enabled. Assuming I'm reading Cumulus documentation correctly, it seems
they enforce the same behavior:

https://docs.cumulusnetworks.com/cumulus-linux/Layer-2/Ethernet-Bridging-VLANs/VLAN-aware-Bridge-Mode/#convert-bridges-between-supported-modes

> This seems to be coming every now and then, so maybe it is time to
> revisit this documentation patch:
> 
> https://github.com/ffainelli/linux/commit/3fe61b1722a3b79d2e317a812c54f3afc902e5b0

Yes, I remember reviewing it. Not sure why you didn't send another
version :)

Re: [RFC 1/3] net: switchdev: do not propagate bridge updates across bridges

[Ido Schimmel]

On Sun, Dec 22, 2019 at 07:24:03PM +0000, Russell King wrote:
> When configuring a tree of independent bridges, propagating changes
> from the upper bridge across a bridge master to the lower bridge
> ports brings surprises.
> 
> For example, a lower bridge may have vlan filtering enabled.  It
> may have a vlan interface attached to the bridge master, which may
> then be incorporated into another bridge.  As soon as the lower
> bridge vlan interface is attached to the upper bridge, the lower
> bridge has vlan filtering disabled.

Interesting topology :) The change looks OK to me. I'll add the patch to
our internal tree and let it go through regression to make sure I didn't
miss anything. Will report the results tomorrow.

> 
> This occurs because switchdev recursively applies its changes to
> all lower devices no matter what.
> 
> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
> ---
>  net/switchdev/switchdev.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/net/switchdev/switchdev.c b/net/switchdev/switchdev.c
> index 3a1d428c1336..d881e5e4a889 100644
> --- a/net/switchdev/switchdev.c
> +++ b/net/switchdev/switchdev.c
> @@ -475,6 +475,9 @@ static int __switchdev_handle_port_obj_add(struct net_device *dev,
>  	 * necessary to go through this helper.
>  	 */
>  	netdev_for_each_lower_dev(dev, lower_dev, iter) {
> +		if (netif_is_bridge_master(lower_dev))
> +			continue;
> +
>  		err = __switchdev_handle_port_obj_add(lower_dev, port_obj_info,
>  						      check_cb, add_cb);
>  		if (err && err != -EOPNOTSUPP)
> @@ -526,6 +529,9 @@ static int __switchdev_handle_port_obj_del(struct net_device *dev,
>  	 * necessary to go through this helper.
>  	 */
>  	netdev_for_each_lower_dev(dev, lower_dev, iter) {
> +		if (netif_is_bridge_master(lower_dev))
> +			continue;
> +
>  		err = __switchdev_handle_port_obj_del(lower_dev, port_obj_info,
>  						      check_cb, del_cb);
>  		if (err && err != -EOPNOTSUPP)
> @@ -576,6 +582,9 @@ static int __switchdev_handle_port_attr_set(struct net_device *dev,
>  	 * necessary to go through this helper.
>  	 */
>  	netdev_for_each_lower_dev(dev, lower_dev, iter) {
> +		if (netif_is_bridge_master(lower_dev))
> +			continue;
> +
>  		err = __switchdev_handle_port_attr_set(lower_dev, port_attr_info,
>  						       check_cb, set_cb);
>  		if (err && err != -EOPNOTSUPP)
> -- 
> 2.20.1
> 

Re: [RFC 1/3] net: switchdev: do not propagate bridge updates across bridges

[Ido Schimmel]

On Tue, Dec 24, 2019 at 10:39:33AM +0200, Ido Schimmel wrote:
> On Sun, Dec 22, 2019 at 07:24:03PM +0000, Russell King wrote:
> > When configuring a tree of independent bridges, propagating changes
> > from the upper bridge across a bridge master to the lower bridge
> > ports brings surprises.
> > 
> > For example, a lower bridge may have vlan filtering enabled.  It
> > may have a vlan interface attached to the bridge master, which may
> > then be incorporated into another bridge.  As soon as the lower
> > bridge vlan interface is attached to the upper bridge, the lower
> > bridge has vlan filtering disabled.
> 
> Interesting topology :) The change looks OK to me. I'll add the patch to
> our internal tree and let it go through regression to make sure I didn't
> miss anything. Will report the results tomorrow.

Reviewed-by: Ido Schimmel <idosch@mellanox.com>
Tested-by: Ido Schimmel <idosch@mellanox.com>

Thanks!

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 2695 bytes --]

On Sunday 22 December 2019 19:22:35 Russell King - ARM Linux admin wrote:
> Hi,
> 
> I've been trying to configure DSA for VLANs and not having much success.
> The setup is quite simple:
> 
> - The main network is untagged
> - The wifi network is a vlan tagged with id $VN running over the main
>   network.
> 
> I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to
> setup to provide wifi access to the vlan $VN network, while the switch
> is also part of the main network.

Hello, I do not know if it is related, but I have a problem with DSa,
VLAN and mv88e6085 on Espressobin board (armada-3720).

My setup/topology is similar:

eth0 --> main interface for mv88e6085 switch
wan --> first RJ45 port from eth0
lan0 --> second RJ45 port from eth0
wan.10 --> unpacked VLAN with id 10 packets from wan

Just one note, wan and wan.10 uses different MAC addresses. Also lan0
has another MAC address.

Basically on upstream wan are two different/separated networks. First
one is untagged, second one is tagged with vlan id 10 and tagged packets
should come on wan interface (linux kernel then pass untagged packets to
wan and vlan id 10 tagged as "untagged" to wan.10). lan0 is downstream
network and in this configuration Espressobin works as router. So there
is no switching between RJ45 ports, all packets should come to CPU and
Linux's iptables do all stuff.

And now the problem. All (untagged) traffic for first network on wan
works fine (incoming + outgoing). Also all outgoing packets from wan.10
interface are correctly transmitted (other side see on first RJ45 port
that packets are properly tagged by id 10). But for unknown reason all
incoming packets with vlan id 10 on first RJ45 port are dropped. Even
tcpdump on eth0 does not see them.

Could be this problem related to one which Russel described? I tried to
debug this problem but I give up 2 days before Russel send this email
with patches, so I have not had a chance to test it.

One very strange behavior is that sometimes mv88e6085 starts accepting
those vlan id 10 packets and kernel them properly send to wan.10
interface and userspacee applications see them. And once they start
appearing it works for 5 minutes, exactly 300s. After 300s they are
again silently somehow dropped (tcpdump again does not see them). I was
not able to detect anything which could cause that kernel started seeing
them. Looks for me it was really random. But exact time 300s is really
strange.

I used default Debian Buster kernel (without any custom patches). Also
one from Debian Buster backports, but behavior was still same.

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Ido Schimmel]

On Tue, Dec 31, 2019 at 05:10:20PM +0100, Pali Rohár wrote:
> On Sunday 22 December 2019 19:22:35 Russell King - ARM Linux admin wrote:
> > Hi,
> > 
> > I've been trying to configure DSA for VLANs and not having much success.
> > The setup is quite simple:
> > 
> > - The main network is untagged
> > - The wifi network is a vlan tagged with id $VN running over the main
> >   network.
> > 
> > I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to
> > setup to provide wifi access to the vlan $VN network, while the switch
> > is also part of the main network.
> 
> Hello, I do not know if it is related, but I have a problem with DSa,
> VLAN and mv88e6085 on Espressobin board (armada-3720).
> 
> My setup/topology is similar:
> 
> eth0 --> main interface for mv88e6085 switch

This is the DSA master interface?

> wan --> first RJ45 port from eth0
> lan0 --> second RJ45 port from eth0
> wan.10 --> unpacked VLAN with id 10 packets from wan
> 
> Just one note, wan and wan.10 uses different MAC addresses.

Is this something you configured? By default the MAC address of the VLAN
device should be inherited from the real device. What happens if they
have the same MAC address?

> Also lan0 has another MAC address.
> 
> Basically on upstream wan are two different/separated networks. First
> one is untagged, second one is tagged with vlan id 10 and tagged packets
> should come on wan interface (linux kernel then pass untagged packets to
> wan and vlan id 10 tagged as "untagged" to wan.10). lan0 is downstream
> network and in this configuration Espressobin works as router. So there
> is no switching between RJ45 ports, all packets should come to CPU and
> Linux's iptables do all stuff.
> 
> And now the problem. All (untagged) traffic for first network on wan
> works fine (incoming + outgoing). Also all outgoing packets from wan.10
> interface are correctly transmitted (other side see on first RJ45 port
> that packets are properly tagged by id 10). But for unknown reason all
> incoming packets with vlan id 10 on first RJ45 port are dropped. Even
> tcpdump on eth0 does not see them.
> 
> Could be this problem related to one which Russel described? I tried to
> debug this problem but I give up 2 days before Russel send this email
> with patches, so I have not had a chance to test it.

I'm not sure. I believe Russel was not able to receive tagged packets at
all and he was using a bridged setup, unlike you (IIUC). Also, below you
write that sometimes you're able to receive packets with VLAN 10.

> One very strange behavior is that sometimes mv88e6085 starts accepting
> those vlan id 10 packets and kernel them properly send to wan.10
> interface and userspacee applications see them. And once they start
> appearing it works for 5 minutes, exactly 300s. After 300s they are
> again silently somehow dropped (tcpdump again does not see them). I was
> not able to detect anything which could cause that kernel started seeing
> them. Looks for me it was really random. But exact time 300s is really
> strange.

300 seconds is the default ageing time the Linux bridge uses, so maybe
the problem is the hardware FDB table and not the VLAN filter.

Andrew / Vivien, how routed traffic is handled on this platform? Is it
possible that the FDB table is consulted first and the expectation is
that it would direct the packet towards the CPU port? If so, I guess
that the FDB entry with {VID 10, wan.10's MAC} is aged-out and flooding
towards the CPU is disabled?

> 
> I used default Debian Buster kernel (without any custom patches). Also
> one from Debian Buster backports, but behavior was still same.
> 
> -- 
> Pali Rohár
> pali.rohar@gmail.com

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 5371 bytes --]

On Tuesday 31 December 2019 20:06:14 Ido Schimmel wrote:
> On Tue, Dec 31, 2019 at 05:10:20PM +0100, Pali Rohár wrote:
> > On Sunday 22 December 2019 19:22:35 Russell King - ARM Linux admin wrote:
> > > Hi,
> > > 
> > > I've been trying to configure DSA for VLANs and not having much success.
> > > The setup is quite simple:
> > > 
> > > - The main network is untagged
> > > - The wifi network is a vlan tagged with id $VN running over the main
> > >   network.
> > > 
> > > I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to
> > > setup to provide wifi access to the vlan $VN network, while the switch
> > > is also part of the main network.
> > 
> > Hello, I do not know if it is related, but I have a problem with DSa,
> > VLAN and mv88e6085 on Espressobin board (armada-3720).
> > 
> > My setup/topology is similar:
> > 
> > eth0 --> main interface for mv88e6085 switch
> 
> This is the DSA master interface?

Yes.

> > wan --> first RJ45 port from eth0
> > lan0 --> second RJ45 port from eth0
> > wan.10 --> unpacked VLAN with id 10 packets from wan
> > 
> > Just one note, wan and wan.10 uses different MAC addresses.
> 
> Is this something you configured?

Yes. My configuration is:

  ip link set dev wan address <address1>
  ip link add link wan name wan.10 type vlan id 10
  ip link set dev wan.10 address <address2>

Just to note that same setup on Raspberry PI is working fine. Here on
Raspberry PI for second RJ45 port I'm using external USB based network
adapter, so DSA on RPI is not used at all. Which means that Linux kernel
does not have any problem with setting / using two different MAC
addresses on one physical network adapter.

> By default the MAC address of the VLAN
> device should be inherited from the real device. What happens if they
> have the same MAC address?

I do not know. I have not tested this scenario. Currently I have my
Espressobin with mv88e6085 out of my hands and is disconnected from
internet, so I cannot do this test right now.

> > Also lan0 has another MAC address.
> > 
> > Basically on upstream wan are two different/separated networks. First
> > one is untagged, second one is tagged with vlan id 10 and tagged packets
> > should come on wan interface (linux kernel then pass untagged packets to
> > wan and vlan id 10 tagged as "untagged" to wan.10). lan0 is downstream
> > network and in this configuration Espressobin works as router. So there
> > is no switching between RJ45 ports, all packets should come to CPU and
> > Linux's iptables do all stuff.
> > 
> > And now the problem. All (untagged) traffic for first network on wan
> > works fine (incoming + outgoing). Also all outgoing packets from wan.10
> > interface are correctly transmitted (other side see on first RJ45 port
> > that packets are properly tagged by id 10). But for unknown reason all
> > incoming packets with vlan id 10 on first RJ45 port are dropped. Even
> > tcpdump on eth0 does not see them.
> > 
> > Could be this problem related to one which Russel described? I tried to
> > debug this problem but I give up 2 days before Russel send this email
> > with patches, so I have not had a chance to test it.
> 
> I'm not sure. I believe Russel was not able to receive tagged packets at
> all and he was using a bridged setup, unlike you (IIUC). Also, below you
> write that sometimes you're able to receive packets with VLAN 10.

Yes, sometimes I was able to receive packets, but I run this setup for
about 4 hours (or more) and only few times (for 5 minutes) it worked.

> > One very strange behavior is that sometimes mv88e6085 starts accepting
> > those vlan id 10 packets and kernel them properly send to wan.10
> > interface and userspacee applications see them. And once they start
> > appearing it works for 5 minutes, exactly 300s. After 300s they are
> > again silently somehow dropped (tcpdump again does not see them). I was
> > not able to detect anything which could cause that kernel started seeing
> > them. Looks for me it was really random. But exact time 300s is really
> > strange.
> 
> 300 seconds is the default ageing time the Linux bridge uses, so maybe
> the problem is the hardware FDB table and not the VLAN filter.
> 
> Andrew / Vivien, how routed traffic is handled on this platform? Is it
> possible that the FDB table is consulted first and the expectation is
> that it would direct the packet towards the CPU port? If so, I guess
> that the FDB entry with {VID 10, wan.10's MAC} is aged-out and flooding
> towards the CPU is disabled?

Hm... that is interesting.

When run was running tcpdump I did not passed -p flag, so tcpdump
automatically was listening in promiscuous mode.

And I was trying those tests also with setting "promisc" mode via ip
link and nothing was changed.

So if above is truth then also promiscuous mode is broken and filtered.

(I expect that network adapter in promiscuous mode receive also packets
which target MAC address is not same as MAC address configured for
network interface.)

> > 
> > I used default Debian Buster kernel (without any custom patches). Also
> > one from Debian Buster backports, but behavior was still same.
> > 
> > -- 
> > Pali Rohár
> > pali.rohar@gmail.com
> 
> 

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Russell King - ARM Linux admin]

Hi Pali,

I've just tried this setup here:

# ip ad add 192.168.4.1/24 dev lan1
# ip li set dev lan1 up
# ip li add li lan1 name lan1.128 type vlan id 128
# ip ad add 192.168.5.1/24 dev lan1.128
# ip li set dev lan1.128 up

without lan1 being part of at Linux software bridge.  lan2..6 are
part of a software bridge that has vlan filtering disabled.

On the other end of lan1, I have another machine setup with addresses
192.168.4.2/24 on the raw network interface and 192.168.5.2/24 on vlan
128 - and it works perfectly, without issue.

This is my 5.4.0 kernel, which has a lot of patches on top of 5.4.0,
including the patch set that started this thread. The hardware is a
SolidRun Clearfog (MV88E6176 DSA switch with mvneta host ethernet).

I think the most important thing to do if you're suffering problems
like this is to monitor and analyse packets being received from the
DSA switch on the host interface:

# tcpdump -enXXi $host_dsa_interface

Here's an example ping packet received over the vlan with the above
configuration, captured from the host DSA interface (ether mac
addresses obfuscated):

        0x0000:  DDDD DDDD DDDD SSSS SSSS SSSS dada 0000  .PC.....[h:.....
                                               ^^^^^^^^^
        0x0010:  c020 0000 8100 0080 0800 4500 0054 ec40  ..........E..T.@
                 ^^^^^^^^^ ^^^^^^^^^ ^^^^
        0x0020:  4000 4001 c314 c0a8 0502 c0a8 0501 0800  @.@.............
        0x0030:  8784 0c85 0001 32c8 0c5e 0000 0000 59fc  ......2..^....Y.
        0x0040:  0c00 0000 0000 1011 1213 1415 1617 1819  ................
        0x0050:  1a1b 1c1d 1e1f 2021 2223 2425 2627 2829  .......!"#$%&'()
        0x0060:  2a2b 2c2d 2e2f 3031 3233 3435 3637       *+,-./01234567

dada 0000 c020 0000	- EDSA tag
8100 0080		- VLAN ethertype, vlan id
0800			- IPv4 ethertype, and what follows is the ipv4
			  packet.

That way it would be possible to know whether the DSA switch is
forwarding the packets, and in what manner it's forwarding them.

Another tool that I've found useful is Vivien's debugfs patch,
which seems to be way superior for understanding the Marvell DSA
switch state than any other tool out there. It's my understanding
that DaveM doesn't want that in the mainline kernel, but it's
really useful for understanding what's going on. It was key to me
discovering why vlan stuff wasn't working for me.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Wednesday 01 January 2020 17:30:14 Russell King - ARM Linux admin wrote:
> I think the most important thing to do if you're suffering problems
> like this is to monitor and analyse packets being received from the
> DSA switch on the host interface:
> 
> # tcpdump -enXXi $host_dsa_interface

Hello Russell! Main dsa interface for me is eth0 and it does not see any
incoming vlan tagged packets. (Except that sometimes for those 5 minutes
periods it sometimes see them. And when tcpdump saw them also they
arrived to userspace.)

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Russell King - ARM Linux admin]

On Wed, Jan 01, 2020 at 07:07:27PM +0100, Pali Rohár wrote:
> On Wednesday 01 January 2020 17:30:14 Russell King - ARM Linux admin wrote:
> > I think the most important thing to do if you're suffering problems
> > like this is to monitor and analyse packets being received from the
> > DSA switch on the host interface:
> > 
> > # tcpdump -enXXi $host_dsa_interface
> 
> Hello Russell! Main dsa interface for me is eth0 and it does not see any
> incoming vlan tagged packets. (Except that sometimes for those 5 minutes
> periods it sometimes see them. And when tcpdump saw them also they
> arrived to userspace.)

I think having Vivien's debugfs patch would be really useful to take
this further forward. This patch provides direct access to the atu
(address translation unit) entries, vtu (vlan translation unit)
entries and all the device registers. The atu takes a long time to
dump.

It also allows for some experimentation, by writing these files,
entries can be added to or removed from the translation units, and
registers written.

I think dumping the on-chip ATU contents with it in the "working"
state and the "non-working" state may be revealing.

I'm afraid that I couldn't tell you where to get Vivien's debugfs
patch from; I'm using an old version that I've ported forward to
subsequent kernels - probably much like anyone else who gets their
hands dirty with Marvell DSA hacking.  Vivien's copied on this
thread already, so might chime in...

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Florian Fainelli]

On 1/1/2020 10:07 AM, Pali Rohár wrote:
> On Wednesday 01 January 2020 17:30:14 Russell King - ARM Linux admin wrote:
>> I think the most important thing to do if you're suffering problems
>> like this is to monitor and analyse packets being received from the
>> DSA switch on the host interface:
>>
>> # tcpdump -enXXi $host_dsa_interface
> 
> Hello Russell! Main dsa interface for me is eth0 and it does not see any
> incoming vlan tagged packets. (Except that sometimes for those 5 minutes
> periods it sometimes see them. And when tcpdump saw them also they
> arrived to userspace.)

Based on your description it sounds like there is possibly an address
learning issue, which is why the ATU patches that Russell mentioned
(latest I know are available at [1]) could come in handy.

[1]: https://github.com/vivien/linux/tree/dsa/debugfs

Since this is a DSA set-up, you could try a few things to isolate the
problems further:

- if you keep the same MAC address for wan and wan.10 does the problem
go away? If so, this could suggest that the DSA master network device
driver potentially has a problem with programming its receive filter for
other unicast MAC addresses than the default one (more on that below)

- if your DSA master supports receive VLAN filter offload
(NETIF_F_HW_VLAN_CTAG_FILTER) does it work better if you turn it off?

- does it help if you go back to a kernel before and not including v5.1
which does not have commit 061f6a505ac33659eab007731c0f6374df39ab55
("net: dsa: Add ndo_vlan_rx_{add, kill}_vid implementation") or if you
can change your kernel, try something similar to [2] for mv88e6xxx and
see if it helps

[2]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e9bf96943b408e6c99dd13fb01cb907335787c61

Regarding the programing of MAC addresses happening through
ndo_set_rx_mode(), we should program static unicast and multicast
addresses when dev_{mc,uc}_sync() gets called similar to [3] because
those addresses are static in nature and the host stack can add/remove
them accordingly. This would help with aging as much as unknown/known
multicast traffic. I will try to cook something later this week.

[3]:
https://github.com/ffainelli/linux/commit/f91b53449e0346a8d7157e5a9225faaa810cbeab
--
Florian

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Andrew Lunn]

> # tcpdump -enXXi $host_dsa_interface
> 
> Here's an example ping packet received over the vlan with the above
> configuration, captured from the host DSA interface (ether mac
> addresses obfuscated):
> 
>         0x0000:  DDDD DDDD DDDD SSSS SSSS SSSS dada 0000  .PC.....[h:.....
>                                                ^^^^^^^^^
>         0x0010:  c020 0000 8100 0080 0800 4500 0054 ec40  ..........E..T.@
>                  ^^^^^^^^^ ^^^^^^^^^ ^^^^
>         0x0020:  4000 4001 c314 c0a8 0502 c0a8 0501 0800  @.@.............
>         0x0030:  8784 0c85 0001 32c8 0c5e 0000 0000 59fc  ......2..^....Y.
>         0x0040:  0c00 0000 0000 1011 1213 1415 1617 1819  ................
>         0x0050:  1a1b 1c1d 1e1f 2021 2223 2425 2627 2829  .......!"#$%&'()
>         0x0060:  2a2b 2c2d 2e2f 3031 3233 3435 3637       *+,-./01234567
> 
> dada 0000 c020 0000	- EDSA tag
> 8100 0080		- VLAN ethertype, vlan id
> 0800			- IPv4 ethertype, and what follows is the ipv4
> 			  packet.
> 
> That way it would be possible to know whether the DSA switch is
> forwarding the packets, and in what manner it's forwarding them.

Hi Russell

Try upgrading your tcpdump and pcap library. It now understands DSA
headers, will dump them in a semi human readable way, and also the
rest of the frame.

     Andrew

Re: [RFC 0/3] VLANs, DSA switches and multiple bridges

[Pali Rohár]

Hello, thank you for steps! For sure I will try it, but probably next
month. Currently I do not have my board in hands, so I cannot do tests.
So I will just comment this part:

On Wednesday 01 January 2020 20:53:57 Florian Fainelli wrote:
> - does it help if you go back to a kernel before and not including v5.1
> which does not have commit 061f6a505ac33659eab007731c0f6374df39ab55
> ("net: dsa: Add ndo_vlan_rx_{add, kill}_vid implementation") or if you
> can change your kernel, try something similar to [2] for mv88e6xxx and
> see if it helps
> 
> [2]:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e9bf96943b408e6c99dd13fb01cb907335787c61

Debian Buster has kernel version 4.19 and in backports is 5.3. I tested
both versions and there was no difference.

Was above mentioned commit propagated to stable trees (and there is
possibility that was backported to stable 4.19 version)?

I can use any kernel version since 4.17 (or 4.18?) as board is supported
by mainline kernel.

-- 
Pali Rohár
pali.rohar@gmail.com