[PATCH v3 net-next 0/4] Cross-chip bridging for disjoint DSA trees

[PATCH v3 net-next 0/4] Cross-chip bridging for disjoint DSA trees

[Vladimir Oltean]

From: Vladimir Oltean <vladimir.oltean@nxp.com>

This series adds support for boards where DSA switches of multiple types
are cascaded together. Actually this type of setup was brought up before
on netdev, and it looks like utilizing disjoint trees is the way to go:

https://lkml.org/lkml/2019/7/7/225

The trouble with disjoint trees (prior to this patch series) is that only
bridging of ports within the same hardware switch can be offloaded.
After scratching my head for a while, it looks like the easiest way to
support hardware bridging between different DSA trees is to bridge their
DSA masters and extend the crosschip bridging operations.

I have given some thought to bridging the DSA masters with the slaves
themselves, but given the hardware topology described in the commit
message of patch 4/4, virtually any number (and combination) of bridges
(forwarding domains) can be created on top of those 3x4-port front-panel
switches. So it becomes a lot less obvious, when the front-panel ports
are enslaved to more than 1 bridge, which bridge should the DSA masters
be enslaved to.

So the least awkward approach was to just create a completely separate
bridge for the DSA masters, whose entire purpose is to permit hardware
forwarding between the discrete switches beneath it.

v1 was submitted here:
https://patchwork.ozlabs.org/project/netdev/cover/20200429161952.17769-1-olteanv@gmail.com/

v2 was submitted here:
https://patchwork.ozlabs.org/project/netdev/cover/20200430202542.11797-1-olteanv@gmail.com/

Vladimir Oltean (4):
  net: bridge: allow enslaving some DSA master network devices
  net: dsa: permit cross-chip bridging between all trees in the system
  net: dsa: introduce a dsa_switch_find function
  net: dsa: sja1105: implement cross-chip bridging operations

 drivers/net/dsa/mv88e6xxx/chip.c       |  16 ++-
 drivers/net/dsa/sja1105/sja1105.h      |   2 +
 drivers/net/dsa/sja1105/sja1105_main.c |  90 +++++++++++++++
 include/linux/dsa/8021q.h              |  45 ++++++++
 include/net/dsa.h                      |  13 ++-
 net/bridge/br_if.c                     |  32 ++++--
 net/bridge/br_input.c                  |  23 +++-
 net/bridge/br_private.h                |   6 +-
 net/dsa/dsa2.c                         |  21 ++++
 net/dsa/dsa_priv.h                     |   1 +
 net/dsa/port.c                         |  23 +++-
 net/dsa/switch.c                       |  21 +++-
 net/dsa/tag_8021q.c                    | 151 +++++++++++++++++++++++++
 13 files changed, 414 insertions(+), 30 deletions(-)

-- 
2.17.1

[PATCH v3 net-next 1/4] net: bridge: allow enslaving some DSA master network devices

[Vladimir Oltean]

From: Vladimir Oltean <vladimir.oltean@nxp.com>

Commit 8db0a2ee2c63 ("net: bridge: reject DSA-enabled master netdevices
as bridge members") added a special check in br_if.c in order to check
for a DSA master network device with a tagging protocol configured. This
was done because back then, such devices, once enslaved in a bridge
would become inoperative and would not pass DSA tagged traffic anymore
due to br_handle_frame returning RX_HANDLER_CONSUMED.

But right now we have valid use cases which do require bridging of DSA
masters. One such example is when the DSA master ports are DSA switch
ports themselves (in a disjoint tree setup). This should be completely
equivalent, functionally speaking, from having multiple DSA switches
hanging off of the ports of a switchdev driver. So we should allow the
enslaving of DSA tagged master network devices.

Instead of the regular br_handle_frame(), install a new function
br_handle_frame_dummy() on these DSA masters, which returns
RX_HANDLER_PASS in order to call into the DSA specific tagging protocol
handlers, and lift the restriction from br_add_if.

Suggested-by: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
Suggested-by: Florian Fainelli <f.fainelli@gmail.com>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Acked-by: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
---
Changes in v3:
None.

Changes in v2:
* Removed the hotpath netdev_uses_dsa check and installed a dummy
  rx_handler for such net devices.
* Improved the check of which DSA master net devices are able to be
  bridged and which aren't.
* At this stage, the patch is different enough from where I took it from
  (aka https://github.com/ffainelli/linux/commit/75618cea75ada8d9eef7936c002b5ec3dd3e4eac)
  that I just added my authorship to it).

 include/net/dsa.h       |  2 +-
 net/bridge/br_if.c      | 32 +++++++++++++++++++++++---------
 net/bridge/br_input.c   | 23 ++++++++++++++++++++++-
 net/bridge/br_private.h |  6 +++---
 4 files changed, 49 insertions(+), 14 deletions(-)

diff --git a/include/net/dsa.h b/include/net/dsa.h
index fb3f9222f2a1..3c0ae84156c8 100644
--- a/include/net/dsa.h
+++ b/include/net/dsa.h
@@ -650,7 +650,7 @@ struct dsa_switch_driver {
 struct net_device *dsa_dev_to_net_device(struct device *dev);
 
 /* Keep inline for faster access in hot path */
-static inline bool netdev_uses_dsa(struct net_device *dev)
+static inline bool netdev_uses_dsa(const struct net_device *dev)
 {
 #if IS_ENABLED(CONFIG_NET_DSA)
 	return dev->dsa_ptr && dev->dsa_ptr->rcv;
diff --git a/net/bridge/br_if.c b/net/bridge/br_if.c
index ca685c0cdf95..a0e9a7937412 100644
--- a/net/bridge/br_if.c
+++ b/net/bridge/br_if.c
@@ -563,18 +563,32 @@ int br_add_if(struct net_bridge *br, struct net_device *dev,
 	unsigned br_hr, dev_hr;
 	bool changed_addr;
 
-	/* Don't allow bridging non-ethernet like devices, or DSA-enabled
-	 * master network devices since the bridge layer rx_handler prevents
-	 * the DSA fake ethertype handler to be invoked, so we do not strip off
-	 * the DSA switch tag protocol header and the bridge layer just return
-	 * RX_HANDLER_CONSUMED, stopping RX processing for these frames.
-	 */
+	/* Don't allow bridging non-ethernet like devices. */
 	if ((dev->flags & IFF_LOOPBACK) ||
 	    dev->type != ARPHRD_ETHER || dev->addr_len != ETH_ALEN ||
-	    !is_valid_ether_addr(dev->dev_addr) ||
-	    netdev_uses_dsa(dev))
+	    !is_valid_ether_addr(dev->dev_addr))
 		return -EINVAL;
 
+	/* Also don't allow bridging of net devices that are DSA masters, since
+	 * the bridge layer rx_handler prevents the DSA fake ethertype handler
+	 * to be invoked, so we don't get the chance to strip off and parse the
+	 * DSA switch tag protocol header (the bridge layer just returns
+	 * RX_HANDLER_CONSUMED, stopping RX processing for these frames).
+	 * The only case where that would not be an issue is when bridging can
+	 * already be offloaded, such as when the DSA master is itself a DSA
+	 * or plain switchdev port, and is bridged only with other ports from
+	 * the same hardware device.
+	 */
+	if (netdev_uses_dsa(dev)) {
+		list_for_each_entry(p, &br->port_list, list) {
+			if (!netdev_port_same_parent_id(dev, p->dev)) {
+				NL_SET_ERR_MSG(extack,
+					       "Cannot do software bridging with a DSA master");
+				return -EINVAL;
+			}
+		}
+	}
+
 	/* No bridging of bridges */
 	if (dev->netdev_ops->ndo_start_xmit == br_dev_xmit) {
 		NL_SET_ERR_MSG(extack,
@@ -618,7 +632,7 @@ int br_add_if(struct net_bridge *br, struct net_device *dev,
 	if (err)
 		goto err3;
 
-	err = netdev_rx_handler_register(dev, br_handle_frame, p);
+	err = netdev_rx_handler_register(dev, br_get_rx_handler(dev), p);
 	if (err)
 		goto err4;
 
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index d5c34f36f0f4..59a318b9f646 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -17,6 +17,7 @@
 #endif
 #include <linux/neighbour.h>
 #include <net/arp.h>
+#include <net/dsa.h>
 #include <linux/export.h>
 #include <linux/rculist.h>
 #include "br_private.h"
@@ -257,7 +258,7 @@ static int nf_hook_bridge_pre(struct sk_buff *skb, struct sk_buff **pskb)
  * Return NULL if skb is handled
  * note: already called with rcu_read_lock
  */
-rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
+static rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
 {
 	struct net_bridge_port *p;
 	struct sk_buff *skb = *pskb;
@@ -359,3 +360,23 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
 	}
 	return RX_HANDLER_CONSUMED;
 }
+
+/* This function has no purpose other than to appease the br_port_get_rcu/rtnl
+ * helpers which identify bridged ports according to the rx_handler installed
+ * on them (so there _needs_ to be a bridge rx_handler even if we don't need it
+ * to do anything useful). This bridge won't support traffic to/from the stack,
+ * but only hardware bridging. So return RX_HANDLER_PASS so we don't steal
+ * frames from the ETH_P_XDSA packet_type handler.
+ */
+static rx_handler_result_t br_handle_frame_dummy(struct sk_buff **pskb)
+{
+	return RX_HANDLER_PASS;
+}
+
+rx_handler_func_t *br_get_rx_handler(const struct net_device *dev)
+{
+	if (netdev_uses_dsa(dev))
+		return br_handle_frame_dummy;
+
+	return br_handle_frame;
+}
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index c35647cb138a..0defc2ff365f 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -705,16 +705,16 @@ int nbp_backup_change(struct net_bridge_port *p, struct net_device *backup_dev);
 
 /* br_input.c */
 int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb);
-rx_handler_result_t br_handle_frame(struct sk_buff **pskb);
+rx_handler_func_t *br_get_rx_handler(const struct net_device *dev);
 
 static inline bool br_rx_handler_check_rcu(const struct net_device *dev)
 {
-	return rcu_dereference(dev->rx_handler) == br_handle_frame;
+	return rcu_dereference(dev->rx_handler) == br_get_rx_handler(dev);
 }
 
 static inline bool br_rx_handler_check_rtnl(const struct net_device *dev)
 {
-	return rcu_dereference_rtnl(dev->rx_handler) == br_handle_frame;
+	return rcu_dereference_rtnl(dev->rx_handler) == br_get_rx_handler(dev);
 }
 
 static inline struct net_bridge_port *br_port_get_check_rcu(const struct net_device *dev)
-- 
2.17.1

[PATCH v3 net-next 2/4] net: dsa: permit cross-chip bridging between all trees in the system

[Vladimir Oltean]

From: Vladimir Oltean <vladimir.oltean@nxp.com>

One way of utilizing DSA is by cascading switches which do not all have
compatible taggers. Consider the following real-life topology:

      +---------------------------------------------------------------+
      | LS1028A                                                       |
      |               +------------------------------+                |
      |               |      DSA master for Felix    |                |
      |               |(internal ENETC port 2: eno2))|                |
      |  +------------+------------------------------+-------------+  |
      |  | Felix embedded L2 switch                                |  |
      |  |                                                         |  |
      |  | +--------------+   +--------------+   +--------------+  |  |
      |  | |DSA master for|   |DSA master for|   |DSA master for|  |  |
      |  | |  SJA1105 1   |   |  SJA1105 2   |   |  SJA1105 3   |  |  |
      |  | |(Felix port 1)|   |(Felix port 2)|   |(Felix port 3)|  |  |
      +--+-+--------------+---+--------------+---+--------------+--+--+

+-----------------------+ +-----------------------+ +-----------------------+
|   SJA1105 switch 1    | |   SJA1105 switch 2    | |   SJA1105 switch 3    |
+-----+-----+-----+-----+ +-----+-----+-----+-----+ +-----+-----+-----+-----+
|sw1p0|sw1p1|sw1p2|sw1p3| |sw2p0|sw2p1|sw2p2|sw2p3| |sw3p0|sw3p1|sw3p2|sw3p3|
+-----+-----+-----+-----+ +-----+-----+-----+-----+ +-----+-----+-----+-----+

The above can be described in the device tree as follows (obviously not
complete):

mscc_felix {
	dsa,member = <0 0>;
	ports {
		port@4 {
			ethernet = <&enetc_port2>;
		};
	};
};

sja1105_switch1 {
	dsa,member = <1 1>;
	ports {
		port@4 {
			ethernet = <&mscc_felix_port1>;
		};
	};
};

sja1105_switch2 {
	dsa,member = <2 2>;
	ports {
		port@4 {
			ethernet = <&mscc_felix_port2>;
		};
	};
};

sja1105_switch3 {
	dsa,member = <3 3>;
	ports {
		port@4 {
			ethernet = <&mscc_felix_port3>;
		};
	};
};

Basically we instantiate one DSA switch tree for every hardware switch
in the system, but we still give them globally unique switch IDs (will
come back to that later). Having 3 disjoint switch trees makes the
tagger drivers "just work", because net devices are registered for the
3 Felix DSA master ports, and they are also DSA slave ports to the ENETC
port. So packets received on the ENETC port are stripped of their
stacked DSA tags one by one.

Currently, hardware bridging between ports on the same sja1105 chip is
possible, but switching between sja1105 ports on different chips is
handled by the software bridge. This is fine, but we can do better.

In fact, the dsa_8021q tag used by sja1105 is compatible with cascading.
In other words, a sja1105 switch can correctly parse and route a packet
containing a dsa_8021q tag. So if we could enable hardware bridging on
the Felix DSA master ports, cross-chip bridging could be completely
offloaded.

Such as system would be used as follows:

ip link add dev br0 type bridge && ip link set dev br0 up
for port in sw0p0 sw0p1 sw0p2 sw0p3 \
	    sw1p0 sw1p1 sw1p2 sw1p3 \
	    sw2p0 sw2p1 sw2p2 sw2p3; do
	ip link set dev $port master br0
done

The above makes switching between ports on the same row be performed in
hardware, and between ports on different rows in software. Now assume
the Felix switch ports are called swp0, swp1, swp2. By running the
following extra commands:

ip link add dev br1 type bridge && ip link set dev br1 up
for port in swp0 swp1 swp2; do
	ip link set dev $port master br1
done

the CPU no longer sees packets which traverse sja1105 switch boundaries
and can be forwarded directly by Felix. The br1 bridge would not be used
for any sort of traffic termination.

For this to work, we need to give drivers an opportunity to listen for
bridging events on DSA trees other than their own, and pass that other
tree index as argument. I have made the assumption, for the moment, that
the other existing DSA notifiers don't need to be broadcast to other
trees. That assumption might turn out to be incorrect. But in the
meantime, introduce a dsa_broadcast function, similar in purpose to
dsa_port_notify, which is used only by the bridging notifiers.

Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
Changes in v3:
None.

Changes in v2:
None.

 drivers/net/dsa/mv88e6xxx/chip.c | 16 ++++++++++++----
 include/net/dsa.h                | 10 ++++++----
 net/dsa/dsa_priv.h               |  1 +
 net/dsa/port.c                   | 23 +++++++++++++++++++++--
 net/dsa/switch.c                 | 21 +++++++++++++++------
 5 files changed, 55 insertions(+), 16 deletions(-)

diff --git a/drivers/net/dsa/mv88e6xxx/chip.c b/drivers/net/dsa/mv88e6xxx/chip.c
index dd8a5666a584..9ecfbe0c4f6c 100644
--- a/drivers/net/dsa/mv88e6xxx/chip.c
+++ b/drivers/net/dsa/mv88e6xxx/chip.c
@@ -2233,26 +2233,34 @@ static void mv88e6xxx_port_bridge_leave(struct dsa_switch *ds, int port,
 	mv88e6xxx_reg_unlock(chip);
 }
 
-static int mv88e6xxx_crosschip_bridge_join(struct dsa_switch *ds, int dev,
+static int mv88e6xxx_crosschip_bridge_join(struct dsa_switch *ds,
+					   int tree_index, int sw_index,
 					   int port, struct net_device *br)
 {
 	struct mv88e6xxx_chip *chip = ds->priv;
 	int err;
 
+	if (tree_index != ds->dst->index)
+		return 0;
+
 	mv88e6xxx_reg_lock(chip);
-	err = mv88e6xxx_pvt_map(chip, dev, port);
+	err = mv88e6xxx_pvt_map(chip, sw_index, port);
 	mv88e6xxx_reg_unlock(chip);
 
 	return err;
 }
 
-static void mv88e6xxx_crosschip_bridge_leave(struct dsa_switch *ds, int dev,
+static void mv88e6xxx_crosschip_bridge_leave(struct dsa_switch *ds,
+					     int tree_index, int sw_index,
 					     int port, struct net_device *br)
 {
 	struct mv88e6xxx_chip *chip = ds->priv;
 
+	if (tree_index != ds->dst->index)
+		return;
+
 	mv88e6xxx_reg_lock(chip);
-	if (mv88e6xxx_pvt_map(chip, dev, port))
+	if (mv88e6xxx_pvt_map(chip, sw_index, port))
 		dev_err(ds->dev, "failed to remap cross-chip Port VLAN\n");
 	mv88e6xxx_reg_unlock(chip);
 }
diff --git a/include/net/dsa.h b/include/net/dsa.h
index 3c0ae84156c8..bd5561dacb13 100644
--- a/include/net/dsa.h
+++ b/include/net/dsa.h
@@ -574,10 +574,12 @@ struct dsa_switch_ops {
 	/*
 	 * Cross-chip operations
 	 */
-	int	(*crosschip_bridge_join)(struct dsa_switch *ds, int sw_index,
-					 int port, struct net_device *br);
-	void	(*crosschip_bridge_leave)(struct dsa_switch *ds, int sw_index,
-					  int port, struct net_device *br);
+	int	(*crosschip_bridge_join)(struct dsa_switch *ds, int tree_index,
+					 int sw_index, int port,
+					 struct net_device *br);
+	void	(*crosschip_bridge_leave)(struct dsa_switch *ds, int tree_index,
+					  int sw_index, int port,
+					  struct net_device *br);
 
 	/*
 	 * PTP functionality
diff --git a/net/dsa/dsa_priv.h b/net/dsa/dsa_priv.h
index 6d9a1ef65fa0..a1a0ae242012 100644
--- a/net/dsa/dsa_priv.h
+++ b/net/dsa/dsa_priv.h
@@ -35,6 +35,7 @@ struct dsa_notifier_ageing_time_info {
 /* DSA_NOTIFIER_BRIDGE_* */
 struct dsa_notifier_bridge_info {
 	struct net_device *br;
+	int tree_index;
 	int sw_index;
 	int port;
 };
diff --git a/net/dsa/port.c b/net/dsa/port.c
index a58fdd362574..ebc8d6cbd1d4 100644
--- a/net/dsa/port.c
+++ b/net/dsa/port.c
@@ -13,6 +13,23 @@
 
 #include "dsa_priv.h"
 
+static int dsa_broadcast(unsigned long e, void *v)
+{
+	struct dsa_switch_tree *dst;
+	int err = 0;
+
+	list_for_each_entry(dst, &dsa_tree_list, list) {
+		struct raw_notifier_head *nh = &dst->nh;
+
+		err = raw_notifier_call_chain(nh, e, v);
+		err = notifier_to_errno(err);
+		if (err)
+			break;
+	}
+
+	return err;
+}
+
 static int dsa_port_notify(const struct dsa_port *dp, unsigned long e, void *v)
 {
 	struct raw_notifier_head *nh = &dp->ds->dst->nh;
@@ -120,6 +137,7 @@ void dsa_port_disable(struct dsa_port *dp)
 int dsa_port_bridge_join(struct dsa_port *dp, struct net_device *br)
 {
 	struct dsa_notifier_bridge_info info = {
+		.tree_index = dp->ds->dst->index,
 		.sw_index = dp->ds->index,
 		.port = dp->index,
 		.br = br,
@@ -136,7 +154,7 @@ int dsa_port_bridge_join(struct dsa_port *dp, struct net_device *br)
 	 */
 	dp->bridge_dev = br;
 
-	err = dsa_port_notify(dp, DSA_NOTIFIER_BRIDGE_JOIN, &info);
+	err = dsa_broadcast(DSA_NOTIFIER_BRIDGE_JOIN, &info);
 
 	/* The bridging is rolled back on error */
 	if (err) {
@@ -150,6 +168,7 @@ int dsa_port_bridge_join(struct dsa_port *dp, struct net_device *br)
 void dsa_port_bridge_leave(struct dsa_port *dp, struct net_device *br)
 {
 	struct dsa_notifier_bridge_info info = {
+		.tree_index = dp->ds->dst->index,
 		.sw_index = dp->ds->index,
 		.port = dp->index,
 		.br = br,
@@ -161,7 +180,7 @@ void dsa_port_bridge_leave(struct dsa_port *dp, struct net_device *br)
 	 */
 	dp->bridge_dev = NULL;
 
-	err = dsa_port_notify(dp, DSA_NOTIFIER_BRIDGE_LEAVE, &info);
+	err = dsa_broadcast(DSA_NOTIFIER_BRIDGE_LEAVE, &info);
 	if (err)
 		pr_err("DSA: failed to notify DSA_NOTIFIER_BRIDGE_LEAVE\n");
 
diff --git a/net/dsa/switch.c b/net/dsa/switch.c
index f3c32ff552b3..86c8dc5c32a0 100644
--- a/net/dsa/switch.c
+++ b/net/dsa/switch.c
@@ -89,11 +89,16 @@ static int dsa_switch_mtu(struct dsa_switch *ds,
 static int dsa_switch_bridge_join(struct dsa_switch *ds,
 				  struct dsa_notifier_bridge_info *info)
 {
-	if (ds->index == info->sw_index && ds->ops->port_bridge_join)
+	struct dsa_switch_tree *dst = ds->dst;
+
+	if (dst->index == info->tree_index && ds->index == info->sw_index &&
+	    ds->ops->port_bridge_join)
 		return ds->ops->port_bridge_join(ds, info->port, info->br);
 
-	if (ds->index != info->sw_index && ds->ops->crosschip_bridge_join)
-		return ds->ops->crosschip_bridge_join(ds, info->sw_index,
+	if ((dst->index != info->tree_index || ds->index != info->sw_index) &&
+	    ds->ops->crosschip_bridge_join)
+		return ds->ops->crosschip_bridge_join(ds, info->tree_index,
+						      info->sw_index,
 						      info->port, info->br);
 
 	return 0;
@@ -103,13 +108,17 @@ static int dsa_switch_bridge_leave(struct dsa_switch *ds,
 				   struct dsa_notifier_bridge_info *info)
 {
 	bool unset_vlan_filtering = br_vlan_enabled(info->br);
+	struct dsa_switch_tree *dst = ds->dst;
 	int err, i;
 
-	if (ds->index == info->sw_index && ds->ops->port_bridge_leave)
+	if (dst->index == info->tree_index && ds->index == info->sw_index &&
+	    ds->ops->port_bridge_join)
 		ds->ops->port_bridge_leave(ds, info->port, info->br);
 
-	if (ds->index != info->sw_index && ds->ops->crosschip_bridge_leave)
-		ds->ops->crosschip_bridge_leave(ds, info->sw_index, info->port,
+	if ((dst->index != info->tree_index || ds->index != info->sw_index) &&
+	    ds->ops->crosschip_bridge_join)
+		ds->ops->crosschip_bridge_leave(ds, info->tree_index,
+						info->sw_index, info->port,
 						info->br);
 
 	/* If the bridge was vlan_filtering, the bridge core doesn't trigger an
-- 
2.17.1

[PATCH v3 net-next 3/4] net: dsa: introduce a dsa_switch_find function

[Vladimir Oltean]

From: Vladimir Oltean <vladimir.oltean@nxp.com>

Somewhat similar to dsa_tree_find, dsa_switch_find returns a dsa_switch
structure pointer by searching for its tree index and switch index (the
parameters from dsa,member). To be used, for example, by drivers who
implement .crosschip_bridge_join and need a reference to the other
switch indicated to by the tree_index and sw_index arguments.

Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
Changes in v3:
None.

Changes in v2:
None.

 include/net/dsa.h |  1 +
 net/dsa/dsa2.c    | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/include/net/dsa.h b/include/net/dsa.h
index bd5561dacb13..c5e2530f54c9 100644
--- a/include/net/dsa.h
+++ b/include/net/dsa.h
@@ -671,6 +671,7 @@ static inline bool dsa_can_decode(const struct sk_buff *skb,
 
 void dsa_unregister_switch(struct dsa_switch *ds);
 int dsa_register_switch(struct dsa_switch *ds);
+struct dsa_switch *dsa_switch_find(int tree_index, int sw_index);
 #ifdef CONFIG_PM_SLEEP
 int dsa_switch_suspend(struct dsa_switch *ds);
 int dsa_switch_resume(struct dsa_switch *ds);
diff --git a/net/dsa/dsa2.c b/net/dsa/dsa2.c
index 9a271a58a41d..07e01b195975 100644
--- a/net/dsa/dsa2.c
+++ b/net/dsa/dsa2.c
@@ -24,6 +24,27 @@ LIST_HEAD(dsa_tree_list);
 static const struct devlink_ops dsa_devlink_ops = {
 };
 
+struct dsa_switch *dsa_switch_find(int tree_index, int sw_index)
+{
+	struct dsa_switch_tree *dst;
+	struct dsa_port *dp;
+
+	list_for_each_entry(dst, &dsa_tree_list, list) {
+		if (dst->index != tree_index)
+			continue;
+
+		list_for_each_entry(dp, &dst->ports, list) {
+			if (dp->ds->index != sw_index)
+				continue;
+
+			return dp->ds;
+		}
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(dsa_switch_find);
+
 static struct dsa_switch_tree *dsa_tree_find(int index)
 {
 	struct dsa_switch_tree *dst;
-- 
2.17.1

[PATCH v3 net-next 4/4] net: dsa: sja1105: implement cross-chip bridging operations

[Vladimir Oltean]

From: Vladimir Oltean <vladimir.oltean@nxp.com>

sja1105 uses dsa_8021q for DSA tagging, a format which is VLAN at heart
and which is compatible with cascading. A complete description of this
tagging format is in net/dsa/tag_8021q.c, but a quick summary is that
each external-facing port tags incoming frames with a unique pvid, and
this special VLAN is transmitted as tagged towards the inside of the
system, and as untagged towards the exterior. The tag encodes the switch
id and the source port index.

This means that cross-chip bridging for dsa_8021q only entails adding
the dsa_8021q pvids of one switch to the RX filter of the other
switches. Everything else falls naturally into place, as long as the
bottom-end of ports (the leaves in the tree) is comprised exclusively of
dsa_8021q-compatible (i.e. sja1105 switches). Otherwise, there would be
a chance that a front-panel switch transmits a packet tagged with a
dsa_8021q header, header which it wouldn't be able to remove, and which
would hence "leak" out.

The only use case I tested (due to lack of board availability) was when
the sja1105 switches are part of disjoint trees (however, this doesn't
change the fact that multiple sja1105 switches still need unique switch
identifiers in such a system). But in principle, even "true" single-tree
setups (with DSA links) should work just as fine, except for a small
change which I can't test: dsa_towards_port should be used instead of
dsa_upstream_port (I made the assumption that the routing port that any
sja1105 should use towards its neighbours is the CPU port. That might
not hold true in other setups).

Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
Changes in v3:
* Moved the implementation in tag_8021q.c, because 1. it is generic
  enough that is can be reused by anyone employing dsa_8021q tagging,
  and 2. sja1105_main.c could use some de-cluttering.
* Provided reference counting to the crosschip links to avoid
  workarounds such as never deleting VLANs from the CPU port.

Changes in v2:
Now replaying the crosschip operations when toggling the VLAN filtering
state (aka when we need to hide the dsa_8021q VLANs, as the VLAN
filtering table becomes visible to the user so it needs to be cleared -
and perhaps restored afterwards).

 drivers/net/dsa/sja1105/sja1105.h      |   2 +
 drivers/net/dsa/sja1105/sja1105_main.c |  90 +++++++++++++++
 include/linux/dsa/8021q.h              |  45 ++++++++
 net/dsa/tag_8021q.c                    | 151 +++++++++++++++++++++++++
 4 files changed, 288 insertions(+)

diff --git a/drivers/net/dsa/sja1105/sja1105.h b/drivers/net/dsa/sja1105/sja1105.h
index 2f62942692ec..a12779c9fa19 100644
--- a/drivers/net/dsa/sja1105/sja1105.h
+++ b/drivers/net/dsa/sja1105/sja1105.h
@@ -8,6 +8,7 @@
 #include <linux/ptp_clock_kernel.h>
 #include <linux/timecounter.h>
 #include <linux/dsa/sja1105.h>
+#include <linux/dsa/8021q.h>
 #include <net/dsa.h>
 #include <linux/mutex.h>
 #include "sja1105_static_config.h"
@@ -135,6 +136,7 @@ struct sja1105_private {
 	struct gpio_desc *reset_gpio;
 	struct spi_device *spidev;
 	struct dsa_switch *ds;
+	struct list_head crosschip_links;
 	struct sja1105_flow_block flow_block;
 	struct sja1105_port ports[SJA1105_NUM_PORTS];
 	/* Serializes transmission of management frames so that
diff --git a/drivers/net/dsa/sja1105/sja1105_main.c b/drivers/net/dsa/sja1105/sja1105_main.c
index 472f4eb20c49..f75ceabb4bf9 100644
--- a/drivers/net/dsa/sja1105/sja1105_main.c
+++ b/drivers/net/dsa/sja1105/sja1105_main.c
@@ -25,6 +25,8 @@
 #include "sja1105_sgmii.h"
 #include "sja1105_tas.h"
 
+static const struct dsa_switch_ops sja1105_switch_ops;
+
 static void sja1105_hw_reset(struct gpio_desc *gpio, unsigned int pulse_len,
 			     unsigned int startup_delay)
 {
@@ -1790,6 +1792,84 @@ static int sja1105_vlan_apply(struct sja1105_private *priv, int port, u16 vid,
 	return 0;
 }
 
+static int sja1105_crosschip_bridge_join(struct dsa_switch *ds,
+					 int tree_index, int sw_index,
+					 int other_port, struct net_device *br)
+{
+	struct dsa_switch *other_ds = dsa_switch_find(tree_index, sw_index);
+	struct sja1105_private *other_priv = other_ds->priv;
+	struct sja1105_private *priv = ds->priv;
+	int port, rc;
+
+	if (other_ds->ops != &sja1105_switch_ops)
+		return 0;
+
+	for (port = 0; port < ds->num_ports; port++) {
+		if (!dsa_is_user_port(ds, port))
+			continue;
+		if (dsa_to_port(ds, port)->bridge_dev != br)
+			continue;
+
+		rc = dsa_8021q_crosschip_bridge_join(ds, port, other_ds,
+						     other_port, br,
+						     &priv->crosschip_links);
+		if (rc)
+			return rc;
+
+		rc = dsa_8021q_crosschip_bridge_join(other_ds, other_port, ds,
+						     port, br,
+						     &other_priv->crosschip_links);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
+
+static void sja1105_crosschip_bridge_leave(struct dsa_switch *ds,
+					   int tree_index, int sw_index,
+					   int other_port,
+					   struct net_device *br)
+{
+	struct dsa_switch *other_ds = dsa_switch_find(tree_index, sw_index);
+	struct sja1105_private *other_priv = other_ds->priv;
+	struct sja1105_private *priv = ds->priv;
+	int port;
+
+	if (other_ds->ops != &sja1105_switch_ops)
+		return;
+
+	for (port = 0; port < ds->num_ports; port++) {
+		if (!dsa_is_user_port(ds, port))
+			continue;
+		if (dsa_to_port(ds, port)->bridge_dev != br)
+			continue;
+
+		dsa_8021q_crosschip_bridge_leave(ds, port, other_ds, other_port,
+						 br, &priv->crosschip_links);
+
+		dsa_8021q_crosschip_bridge_leave(other_ds, other_port, ds,
+						 port, br,
+						 &other_priv->crosschip_links);
+	}
+}
+
+static int sja1105_replay_crosschip_vlans(struct dsa_switch *ds, bool enabled)
+{
+	struct sja1105_private *priv = ds->priv;
+	struct dsa_8021q_crosschip_link *c;
+	int rc;
+
+	list_for_each_entry(c, &priv->crosschip_links, list) {
+		rc = dsa_8021q_crosschip_link_apply(ds, c->port, c->other_ds,
+						    c->other_port, enabled);
+		if (rc)
+			break;
+	}
+
+	return rc;
+}
+
 static int sja1105_setup_8021q_tagging(struct dsa_switch *ds, bool enabled)
 {
 	int rc, i;
@@ -1802,6 +1882,12 @@ static int sja1105_setup_8021q_tagging(struct dsa_switch *ds, bool enabled)
 			return rc;
 		}
 	}
+	rc = sja1105_replay_crosschip_vlans(ds, enabled);
+	if (rc) {
+		dev_err(ds->dev, "Failed to replay crosschip VLANs: %d\n", rc);
+		return rc;
+	}
+
 	dev_info(ds->dev, "%s switch tagging\n",
 		 enabled ? "Enabled" : "Disabled");
 	return 0;
@@ -2359,6 +2445,8 @@ static const struct dsa_switch_ops sja1105_switch_ops = {
 	.port_policer_del	= sja1105_port_policer_del,
 	.cls_flower_add		= sja1105_cls_flower_add,
 	.cls_flower_del		= sja1105_cls_flower_del,
+	.crosschip_bridge_join	= sja1105_crosschip_bridge_join,
+	.crosschip_bridge_leave	= sja1105_crosschip_bridge_leave,
 };
 
 static int sja1105_check_device_id(struct sja1105_private *priv)
@@ -2461,6 +2549,8 @@ static int sja1105_probe(struct spi_device *spi)
 	mutex_init(&priv->ptp_data.lock);
 	mutex_init(&priv->mgmt_lock);
 
+	INIT_LIST_HEAD(&priv->crosschip_links);
+
 	sja1105_tas_setup(ds);
 	sja1105_flower_setup(ds);
 
diff --git a/include/linux/dsa/8021q.h b/include/linux/dsa/8021q.h
index c620d9139c28..b8daaec0896e 100644
--- a/include/linux/dsa/8021q.h
+++ b/include/linux/dsa/8021q.h
@@ -12,11 +12,33 @@ struct sk_buff;
 struct net_device;
 struct packet_type;
 
+struct dsa_8021q_crosschip_link {
+	struct list_head list;
+	int port;
+	struct dsa_switch *other_ds;
+	int other_port;
+	refcount_t refcount;
+};
+
 #if IS_ENABLED(CONFIG_NET_DSA_TAG_8021Q)
 
 int dsa_port_setup_8021q_tagging(struct dsa_switch *ds, int index,
 				 bool enabled);
 
+int dsa_8021q_crosschip_link_apply(struct dsa_switch *ds, int port,
+				   struct dsa_switch *other_ds,
+				   int other_port, bool enabled);
+
+int dsa_8021q_crosschip_bridge_join(struct dsa_switch *ds, int port,
+				    struct dsa_switch *other_ds,
+				    int other_port, struct net_device *br,
+				    struct list_head *crosschip_links);
+
+int dsa_8021q_crosschip_bridge_leave(struct dsa_switch *ds, int port,
+				     struct dsa_switch *other_ds,
+				     int other_port, struct net_device *br,
+				     struct list_head *crosschip_links);
+
 struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev,
 			       u16 tpid, u16 tci);
 
@@ -36,6 +58,29 @@ int dsa_port_setup_8021q_tagging(struct dsa_switch *ds, int index,
 	return 0;
 }
 
+int dsa_8021q_crosschip_link_apply(struct dsa_switch *ds, int port,
+				   struct dsa_switch *other_ds,
+				   int other_port, bool enabled)
+{
+	return 0;
+}
+
+int dsa_8021q_crosschip_bridge_join(struct dsa_switch *ds, int port,
+				    struct dsa_switch *other_ds,
+				    int other_port, struct net_device *br,
+				    struct list_head *crosschip_links)
+{
+	return 0;
+}
+
+int dsa_8021q_crosschip_bridge_leave(struct dsa_switch *ds, int port,
+				     struct dsa_switch *other_ds,
+				     int other_port, struct net_device *br,
+				     struct list_head *crosschip_links)
+{
+	return 0;
+}
+
 struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev,
 			       u16 tpid, u16 tci)
 {
diff --git a/net/dsa/tag_8021q.c b/net/dsa/tag_8021q.c
index b97ad93d1c1a..ff9c5bf64bda 100644
--- a/net/dsa/tag_8021q.c
+++ b/net/dsa/tag_8021q.c
@@ -8,6 +8,7 @@
  */
 #include <linux/if_bridge.h>
 #include <linux/if_vlan.h>
+#include <linux/dsa/8021q.h>
 
 #include "dsa_priv.h"
 
@@ -288,6 +289,156 @@ int dsa_port_setup_8021q_tagging(struct dsa_switch *ds, int port, bool enabled)
 }
 EXPORT_SYMBOL_GPL(dsa_port_setup_8021q_tagging);
 
+int dsa_8021q_crosschip_link_apply(struct dsa_switch *ds, int port,
+				   struct dsa_switch *other_ds,
+				   int other_port, bool enabled)
+{
+	u16 rx_vid = dsa_8021q_rx_vid(ds, port);
+
+	/* @rx_vid of local @ds port @port goes to @other_port of
+	 * @other_ds
+	 */
+	return dsa_8021q_vid_apply(other_ds, other_port, rx_vid,
+				   BRIDGE_VLAN_INFO_UNTAGGED, enabled);
+}
+EXPORT_SYMBOL_GPL(dsa_8021q_crosschip_link_apply);
+
+static int dsa_8021q_crosschip_link_add(struct dsa_switch *ds, int port,
+					struct dsa_switch *other_ds,
+					int other_port,
+					struct list_head *crosschip_links)
+{
+	struct dsa_8021q_crosschip_link *c;
+
+	list_for_each_entry(c, crosschip_links, list) {
+		if (c->port == port && c->other_ds == other_ds &&
+		    c->other_port == other_port) {
+			refcount_inc(&c->refcount);
+			return 0;
+		}
+	}
+
+	dev_dbg(ds->dev, "adding crosschip link from port %d to %s port %d\n",
+		port, dev_name(other_ds->dev), other_port);
+
+	c = kzalloc(sizeof(*c), GFP_KERNEL);
+	if (!c)
+		return -ENOMEM;
+
+	c->port = port;
+	c->other_ds = other_ds;
+	c->other_port = other_port;
+	refcount_set(&c->refcount, 1);
+
+	list_add(&c->list, crosschip_links);
+
+	return 0;
+}
+
+static void dsa_8021q_crosschip_link_del(struct dsa_switch *ds,
+					 struct dsa_8021q_crosschip_link *c,
+					 struct list_head *crosschip_links,
+					 bool *keep)
+{
+	*keep = !refcount_dec_and_test(&c->refcount);
+
+	if (*keep)
+		return;
+
+	dev_dbg(ds->dev,
+		"deleting crosschip link from port %d to %s port %d\n",
+		c->port, dev_name(c->other_ds->dev), c->other_port);
+
+	list_del(&c->list);
+	kfree(c);
+}
+
+/* Make traffic from local port @port be received by remote port @other_port.
+ * This means that our @rx_vid needs to be installed on @other_ds's upstream
+ * and user ports. The user ports should be egress-untagged so that they can
+ * pop the dsa_8021q VLAN. But the @other_upstream can be either egress-tagged
+ * or untagged: it doesn't matter, since it should never egress a frame having
+ * our @rx_vid.
+ */
+int dsa_8021q_crosschip_bridge_join(struct dsa_switch *ds, int port,
+				    struct dsa_switch *other_ds,
+				    int other_port, struct net_device *br,
+				    struct list_head *crosschip_links)
+{
+	/* @other_upstream is how @other_ds reaches us. If we are part
+	 * of disjoint trees, then we are probably connected through
+	 * our CPU ports. If we're part of the same tree though, we should
+	 * probably use dsa_towards_port.
+	 */
+	int other_upstream = dsa_upstream_port(other_ds, other_port);
+	int rc;
+
+	rc = dsa_8021q_crosschip_link_add(ds, port, other_ds,
+					  other_port, crosschip_links);
+	if (rc)
+		return rc;
+
+	if (!br_vlan_enabled(br)) {
+		rc = dsa_8021q_crosschip_link_apply(ds, port, other_ds,
+						    other_port, true);
+		if (rc)
+			return rc;
+	}
+
+	rc = dsa_8021q_crosschip_link_add(ds, port, other_ds,
+					  other_upstream,
+					  crosschip_links);
+	if (rc)
+		return rc;
+
+	if (!br_vlan_enabled(br)) {
+		rc = dsa_8021q_crosschip_link_apply(ds, port, other_ds,
+						    other_upstream, true);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dsa_8021q_crosschip_bridge_join);
+
+int dsa_8021q_crosschip_bridge_leave(struct dsa_switch *ds, int port,
+				     struct dsa_switch *other_ds,
+				     int other_port, struct net_device *br,
+				     struct list_head *crosschip_links)
+{
+	int other_upstream = dsa_upstream_port(other_ds, other_port);
+	struct dsa_8021q_crosschip_link *c, *n;
+
+	list_for_each_entry_safe(c, n, crosschip_links, list) {
+		if (c->port == port && c->other_ds == other_ds &&
+		    (c->other_port == other_port ||
+		     c->other_port == other_upstream)) {
+			struct dsa_switch *other_ds = c->other_ds;
+			int other_port = c->other_port;
+			bool keep;
+			int rc;
+
+			dsa_8021q_crosschip_link_del(ds, c, crosschip_links,
+						     &keep);
+			if (keep)
+				continue;
+
+			if (!br_vlan_enabled(br)) {
+				rc = dsa_8021q_crosschip_link_apply(ds, port,
+								    other_ds,
+								    other_port,
+								    false);
+				if (rc)
+					return rc;
+			}
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dsa_8021q_crosschip_bridge_leave);
+
 struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev,
 			       u16 tpid, u16 tci)
 {
-- 
2.17.1

Re: [PATCH v3 net-next 0/4] Cross-chip bridging for disjoint DSA trees

[Vladimir Oltean]

Hi David,

On Mon, 4 May 2020 at 01:12, Vladimir Oltean <olteanv@gmail.com> wrote:
>
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
>
> This series adds support for boards where DSA switches of multiple types
> are cascaded together. Actually this type of setup was brought up before
> on netdev, and it looks like utilizing disjoint trees is the way to go:
>
> https://lkml.org/lkml/2019/7/7/225
>
> The trouble with disjoint trees (prior to this patch series) is that only
> bridging of ports within the same hardware switch can be offloaded.
> After scratching my head for a while, it looks like the easiest way to
> support hardware bridging between different DSA trees is to bridge their
> DSA masters and extend the crosschip bridging operations.
>
> I have given some thought to bridging the DSA masters with the slaves
> themselves, but given the hardware topology described in the commit
> message of patch 4/4, virtually any number (and combination) of bridges
> (forwarding domains) can be created on top of those 3x4-port front-panel
> switches. So it becomes a lot less obvious, when the front-panel ports
> are enslaved to more than 1 bridge, which bridge should the DSA masters
> be enslaved to.
>
> So the least awkward approach was to just create a completely separate
> bridge for the DSA masters, whose entire purpose is to permit hardware
> forwarding between the discrete switches beneath it.
>
> v1 was submitted here:
> https://patchwork.ozlabs.org/project/netdev/cover/20200429161952.17769-1-olteanv@gmail.com/
>
> v2 was submitted here:
> https://patchwork.ozlabs.org/project/netdev/cover/20200430202542.11797-1-olteanv@gmail.com/
>
> Vladimir Oltean (4):
>   net: bridge: allow enslaving some DSA master network devices
>   net: dsa: permit cross-chip bridging between all trees in the system
>   net: dsa: introduce a dsa_switch_find function
>   net: dsa: sja1105: implement cross-chip bridging operations
>
>  drivers/net/dsa/mv88e6xxx/chip.c       |  16 ++-
>  drivers/net/dsa/sja1105/sja1105.h      |   2 +
>  drivers/net/dsa/sja1105/sja1105_main.c |  90 +++++++++++++++
>  include/linux/dsa/8021q.h              |  45 ++++++++
>  include/net/dsa.h                      |  13 ++-
>  net/bridge/br_if.c                     |  32 ++++--
>  net/bridge/br_input.c                  |  23 +++-
>  net/bridge/br_private.h                |   6 +-
>  net/dsa/dsa2.c                         |  21 ++++
>  net/dsa/dsa_priv.h                     |   1 +
>  net/dsa/port.c                         |  23 +++-
>  net/dsa/switch.c                       |  21 +++-
>  net/dsa/tag_8021q.c                    | 151 +++++++++++++++++++++++++
>  13 files changed, 414 insertions(+), 30 deletions(-)
>
> --
> 2.17.1
>

What does it mean that this series is "deferred" in patchwork?

Thanks,
-Vladimir

Re: [PATCH v3 net-next 0/4] Cross-chip bridging for disjoint DSA trees

[David Miller]

From: Vladimir Oltean <olteanv@gmail.com>
Date: Thu, 7 May 2020 19:07:32 +0300

> What does it mean that this series is "deferred" in patchwork?

I need it to be reviewed, nobody reviewed it for days so I just toss
it in the deferred state.

I don't feel comfortable applying this without Andrew/Florian's
review, but if that doesn't happen I don't want this series clogging
up my backlog so I toss it because you can always repost after
some time.

Re: [PATCH v3 net-next 0/4] Cross-chip bridging for disjoint DSA trees

[Florian Fainelli]

On 5/7/2020 3:15 PM, David Miller wrote:
> From: Vladimir Oltean <olteanv@gmail.com>
> Date: Thu, 7 May 2020 19:07:32 +0300
> 
>> What does it mean that this series is "deferred" in patchwork?
> 
> I need it to be reviewed, nobody reviewed it for days so I just toss
> it in the deferred state.
> 
> I don't feel comfortable applying this without Andrew/Florian's
> review, but if that doesn't happen I don't want this series clogging
> up my backlog so I toss it because you can always repost after
> some time.

I should be able to review those patches later today and give them a
spin too.
-- 
Florian

Re: [PATCH v3 net-next 1/4] net: bridge: allow enslaving some DSA master network devices

[Florian Fainelli]

On 5/3/2020 3:12 PM, Vladimir Oltean wrote:
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
> 
> Commit 8db0a2ee2c63 ("net: bridge: reject DSA-enabled master netdevices
> as bridge members") added a special check in br_if.c in order to check
> for a DSA master network device with a tagging protocol configured. This
> was done because back then, such devices, once enslaved in a bridge
> would become inoperative and would not pass DSA tagged traffic anymore
> due to br_handle_frame returning RX_HANDLER_CONSUMED.
> 
> But right now we have valid use cases which do require bridging of DSA
> masters. One such example is when the DSA master ports are DSA switch
> ports themselves (in a disjoint tree setup). This should be completely
> equivalent, functionally speaking, from having multiple DSA switches
> hanging off of the ports of a switchdev driver. So we should allow the
> enslaving of DSA tagged master network devices.
> 
> Instead of the regular br_handle_frame(), install a new function
> br_handle_frame_dummy() on these DSA masters, which returns
> RX_HANDLER_PASS in order to call into the DSA specific tagging protocol
> handlers, and lift the restriction from br_add_if.
> 
> Suggested-by: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
> Suggested-by: Florian Fainelli <f.fainelli@gmail.com>
> Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
> Acked-by: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>

Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
Tested-by: Florian Fainelli <f.fainelli@gmail.com>
-- 
Florian

Re: [PATCH v3 net-next 3/4] net: dsa: introduce a dsa_switch_find function

[Florian Fainelli]

On 5/3/2020 3:12 PM, Vladimir Oltean wrote:
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
> 
> Somewhat similar to dsa_tree_find, dsa_switch_find returns a dsa_switch
> structure pointer by searching for its tree index and switch index (the
> parameters from dsa,member). To be used, for example, by drivers who
> implement .crosschip_bridge_join and need a reference to the other
> switch indicated to by the tree_index and sw_index arguments.
> 
> Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>

Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
-- 
Florian

Re: [PATCH v3 net-next 2/4] net: dsa: permit cross-chip bridging between all trees in the system

[Florian Fainelli]

On 5/3/2020 3:12 PM, Vladimir Oltean wrote:
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
> 
> One way of utilizing DSA is by cascading switches which do not all have
> compatible taggers. Consider the following real-life topology:
> 
>       +---------------------------------------------------------------+
>       | LS1028A                                                       |
>       |               +------------------------------+                |
>       |               |      DSA master for Felix    |                |
>       |               |(internal ENETC port 2: eno2))|                |
>       |  +------------+------------------------------+-------------+  |
>       |  | Felix embedded L2 switch                                |  |
>       |  |                                                         |  |
>       |  | +--------------+   +--------------+   +--------------+  |  |
>       |  | |DSA master for|   |DSA master for|   |DSA master for|  |  |
>       |  | |  SJA1105 1   |   |  SJA1105 2   |   |  SJA1105 3   |  |  |
>       |  | |(Felix port 1)|   |(Felix port 2)|   |(Felix port 3)|  |  |
>       +--+-+--------------+---+--------------+---+--------------+--+--+
> 
> +-----------------------+ +-----------------------+ +-----------------------+
> |   SJA1105 switch 1    | |   SJA1105 switch 2    | |   SJA1105 switch 3    |
> +-----+-----+-----+-----+ +-----+-----+-----+-----+ +-----+-----+-----+-----+
> |sw1p0|sw1p1|sw1p2|sw1p3| |sw2p0|sw2p1|sw2p2|sw2p3| |sw3p0|sw3p1|sw3p2|sw3p3|
> +-----+-----+-----+-----+ +-----+-----+-----+-----+ +-----+-----+-----+-----+
> 
> The above can be described in the device tree as follows (obviously not
> complete):
> 
> mscc_felix {
> 	dsa,member = <0 0>;
> 	ports {
> 		port@4 {
> 			ethernet = <&enetc_port2>;
> 		};
> 	};
> };
> 
> sja1105_switch1 {
> 	dsa,member = <1 1>;
> 	ports {
> 		port@4 {
> 			ethernet = <&mscc_felix_port1>;
> 		};
> 	};
> };
> 
> sja1105_switch2 {
> 	dsa,member = <2 2>;
> 	ports {
> 		port@4 {
> 			ethernet = <&mscc_felix_port2>;
> 		};
> 	};
> };
> 
> sja1105_switch3 {
> 	dsa,member = <3 3>;
> 	ports {
> 		port@4 {
> 			ethernet = <&mscc_felix_port3>;
> 		};
> 	};
> };
> 
> Basically we instantiate one DSA switch tree for every hardware switch
> in the system, but we still give them globally unique switch IDs (will
> come back to that later). Having 3 disjoint switch trees makes the
> tagger drivers "just work", because net devices are registered for the
> 3 Felix DSA master ports, and they are also DSA slave ports to the ENETC
> port. So packets received on the ENETC port are stripped of their
> stacked DSA tags one by one.
> 
> Currently, hardware bridging between ports on the same sja1105 chip is
> possible, but switching between sja1105 ports on different chips is
> handled by the software bridge. This is fine, but we can do better.
> 
> In fact, the dsa_8021q tag used by sja1105 is compatible with cascading.
> In other words, a sja1105 switch can correctly parse and route a packet
> containing a dsa_8021q tag. So if we could enable hardware bridging on
> the Felix DSA master ports, cross-chip bridging could be completely
> offloaded.
> 
> Such as system would be used as follows:
> 
> ip link add dev br0 type bridge && ip link set dev br0 up
> for port in sw0p0 sw0p1 sw0p2 sw0p3 \
> 	    sw1p0 sw1p1 sw1p2 sw1p3 \
> 	    sw2p0 sw2p1 sw2p2 sw2p3; do
> 	ip link set dev $port master br0
> done
> 
> The above makes switching between ports on the same row be performed in
> hardware, and between ports on different rows in software. Now assume
> the Felix switch ports are called swp0, swp1, swp2. By running the
> following extra commands:
> 
> ip link add dev br1 type bridge && ip link set dev br1 up
> for port in swp0 swp1 swp2; do
> 	ip link set dev $port master br1
> done
> 
> the CPU no longer sees packets which traverse sja1105 switch boundaries
> and can be forwarded directly by Felix. The br1 bridge would not be used
> for any sort of traffic termination.

Is there anything that prevents br1 from terminating traffic though
(just curious)?

> 
> For this to work, we need to give drivers an opportunity to listen for
> bridging events on DSA trees other than their own, and pass that other
> tree index as argument. I have made the assumption, for the moment, that
> the other existing DSA notifiers don't need to be broadcast to other
> trees. That assumption might turn out to be incorrect. But in the
> meantime, introduce a dsa_broadcast function, similar in purpose to
> dsa_port_notify, which is used only by the bridging notifiers.
> 
> Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
> ---
Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
-- 
Florian

Re: [PATCH v3 net-next 4/4] net: dsa: sja1105: implement cross-chip bridging operations

[Florian Fainelli]

On 5/3/2020 3:12 PM, Vladimir Oltean wrote:
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
> 
> sja1105 uses dsa_8021q for DSA tagging, a format which is VLAN at heart
> and which is compatible with cascading. A complete description of this
> tagging format is in net/dsa/tag_8021q.c, but a quick summary is that
> each external-facing port tags incoming frames with a unique pvid, and
> this special VLAN is transmitted as tagged towards the inside of the
> system, and as untagged towards the exterior. The tag encodes the switch
> id and the source port index.
> 
> This means that cross-chip bridging for dsa_8021q only entails adding
> the dsa_8021q pvids of one switch to the RX filter of the other
> switches. Everything else falls naturally into place, as long as the
> bottom-end of ports (the leaves in the tree) is comprised exclusively of
> dsa_8021q-compatible (i.e. sja1105 switches). Otherwise, there would be
> a chance that a front-panel switch transmits a packet tagged with a
> dsa_8021q header, header which it wouldn't be able to remove, and which
> would hence "leak" out.
> 
> The only use case I tested (due to lack of board availability) was when
> the sja1105 switches are part of disjoint trees (however, this doesn't
> change the fact that multiple sja1105 switches still need unique switch
> identifiers in such a system). But in principle, even "true" single-tree
> setups (with DSA links) should work just as fine, except for a small
> change which I can't test: dsa_towards_port should be used instead of
> dsa_upstream_port (I made the assumption that the routing port that any
> sja1105 should use towards its neighbours is the CPU port. That might
> not hold true in other setups).
> 
> Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>

Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
-- 
Florian

Re: [PATCH v3 net-next 2/4] net: dsa: permit cross-chip bridging between all trees in the system

[Vladimir Oltean]

Hi Florian,

Thank you so much for the review!

On Fri, 8 May 2020 at 06:16, Florian Fainelli <f.fainelli@gmail.com> wrote:
>
>
>
> On 5/3/2020 3:12 PM, Vladimir Oltean wrote:
> > From: Vladimir Oltean <vladimir.oltean@nxp.com>
> >
> > One way of utilizing DSA is by cascading switches which do not all have
> > compatible taggers. Consider the following real-life topology:
> >
> >       +---------------------------------------------------------------+
> >       | LS1028A                                                       |
> >       |               +------------------------------+                |
> >       |               |      DSA master for Felix    |                |
> >       |               |(internal ENETC port 2: eno2))|                |
> >       |  +------------+------------------------------+-------------+  |
> >       |  | Felix embedded L2 switch                                |  |
> >       |  |                                                         |  |
> >       |  | +--------------+   +--------------+   +--------------+  |  |
> >       |  | |DSA master for|   |DSA master for|   |DSA master for|  |  |
> >       |  | |  SJA1105 1   |   |  SJA1105 2   |   |  SJA1105 3   |  |  |
> >       |  | |(Felix port 1)|   |(Felix port 2)|   |(Felix port 3)|  |  |
> >       +--+-+--------------+---+--------------+---+--------------+--+--+
> >
> > +-----------------------+ +-----------------------+ +-----------------------+
> > |   SJA1105 switch 1    | |   SJA1105 switch 2    | |   SJA1105 switch 3    |
> > +-----+-----+-----+-----+ +-----+-----+-----+-----+ +-----+-----+-----+-----+
> > |sw1p0|sw1p1|sw1p2|sw1p3| |sw2p0|sw2p1|sw2p2|sw2p3| |sw3p0|sw3p1|sw3p2|sw3p3|
> > +-----+-----+-----+-----+ +-----+-----+-----+-----+ +-----+-----+-----+-----+
> >
> > The above can be described in the device tree as follows (obviously not
> > complete):
> >
> > mscc_felix {
> >       dsa,member = <0 0>;
> >       ports {
> >               port@4 {
> >                       ethernet = <&enetc_port2>;
> >               };
> >       };
> > };
> >
> > sja1105_switch1 {
> >       dsa,member = <1 1>;
> >       ports {
> >               port@4 {
> >                       ethernet = <&mscc_felix_port1>;
> >               };
> >       };
> > };
> >
> > sja1105_switch2 {
> >       dsa,member = <2 2>;
> >       ports {
> >               port@4 {
> >                       ethernet = <&mscc_felix_port2>;
> >               };
> >       };
> > };
> >
> > sja1105_switch3 {
> >       dsa,member = <3 3>;
> >       ports {
> >               port@4 {
> >                       ethernet = <&mscc_felix_port3>;
> >               };
> >       };
> > };
> >
> > Basically we instantiate one DSA switch tree for every hardware switch
> > in the system, but we still give them globally unique switch IDs (will
> > come back to that later). Having 3 disjoint switch trees makes the
> > tagger drivers "just work", because net devices are registered for the
> > 3 Felix DSA master ports, and they are also DSA slave ports to the ENETC
> > port. So packets received on the ENETC port are stripped of their
> > stacked DSA tags one by one.
> >
> > Currently, hardware bridging between ports on the same sja1105 chip is
> > possible, but switching between sja1105 ports on different chips is
> > handled by the software bridge. This is fine, but we can do better.
> >
> > In fact, the dsa_8021q tag used by sja1105 is compatible with cascading.
> > In other words, a sja1105 switch can correctly parse and route a packet
> > containing a dsa_8021q tag. So if we could enable hardware bridging on
> > the Felix DSA master ports, cross-chip bridging could be completely
> > offloaded.
> >
> > Such as system would be used as follows:
> >
> > ip link add dev br0 type bridge && ip link set dev br0 up
> > for port in sw0p0 sw0p1 sw0p2 sw0p3 \
> >           sw1p0 sw1p1 sw1p2 sw1p3 \
> >           sw2p0 sw2p1 sw2p2 sw2p3; do
> >       ip link set dev $port master br0
> > done
> >
> > The above makes switching between ports on the same row be performed in
> > hardware, and between ports on different rows in software. Now assume
> > the Felix switch ports are called swp0, swp1, swp2. By running the
> > following extra commands:
> >
> > ip link add dev br1 type bridge && ip link set dev br1 up
> > for port in swp0 swp1 swp2; do
> >       ip link set dev $port master br1
> > done
> >
> > the CPU no longer sees packets which traverse sja1105 switch boundaries
> > and can be forwarded directly by Felix. The br1 bridge would not be used
> > for any sort of traffic termination.
>
> Is there anything that prevents br1 from terminating traffic though
> (just curious)?
>

Well, one obvious limitation is the fact that to support termination
on br1, the bridge rx_handler would have to steal packets from DSA
software RX processing path. We just need the upstream switch to
forward packets in hardware between ports that are DSA masters, so the
choice was to at least permit that.
So given the fact that now we have a dummy rx_handler on br1, it _can_
not terminate any traffic.

For the particular hardware layout presented above, the choice was to
let the user bridge the Felix ports. Functionally it is optional
(sja1105 ports are still bridged both ways), but the data paths are
different:
- if br1 doesn't exist, then a packet that needs to go from sw1p0 to
sw2p0 is bridged in software by br0 (because Felix is not bridged, all
of its traffic goes to the CPU, then the rules on br0 kick in, and
this reinjects the packet to sw2p0, which calls dev_queue_xmit to
Felix port 2, which calls dev_queue_xmit to the one and only ENETC
master).
- If br1 exists and we want to forward packets along the same route
(sw1p0 -> sw2p0), then br0 only defines the forwarding domain to which
packets are allowed to go to. There would initially be one duplicate,
when Felix floods the first packet to the CPU _and_ to its other port
(the DSA master of sw2p0), because the packet sent to the stack will
still get software-bridged and re-enqueued just as in the case above.
But for further packets, Felix will no longer flood packets to the
CPU, but just to the other switch. On that end, the other switch will
look at the dsa_8021q tag and decide which ports are allowed to see
the packet and which aren't (these are the "crosschip links" that
depend on which ports are part of br0).

So to answer your question, we never need to terminate traffic on br1
because it only serves as double duty for br0 (accelerating its
forwarding path).
The alternative would have been to build some sja1105 awareness in
Felix of some sorts. The question, of course, is when can the Felix
driver automatically decide that its DSA masters can be bridged
together? And if we take an "automatic decision" route, is it sane
that Felix ports 1 and 2 are forwarding packets autonomously between
them, even though there is no Linux bridge that asked for that?

On the other hand, we may imagine a few situations where things might
look differently.
Let's say Felix had 4 ports, but sja1105 switches were hanging off of
only 3 of its ports. The 4th interface goes straight to a copper port.
If sw1p0 wants to talk to the copper port of Felix, how can we model
that, and what are the chances of it working in hardware?
Spoiler alert, it won't work purely in hardware, because the copper
port would see the unpopped dsa_8021q headers coming from sw1p0.
But we can still put sw1p0 and Felix port 4 in the same br0 interface,
and packets from sw1p0 would go to the CPU, where a new packet would
be forwarded to Felix port 4 without the dsa_8021q tag of sw1p0.
So bridging a Felix standalone (not DSA master) interface with a
sja1105 interface could work under some circumstances (through
software bridging), but that is non-ideal, so as long as the DSA
master switch doesn't have any understanding of the DSA headers it's
transporting, it's simply easier to not do that :) and design boards
where there's a sja1105 switch hanging off of every used Felix port.

But what if we build a super-Felix switch in the future, that
understands the DSA tags of the switches cascaded beneath it? Let's
treat this "super-Felix" in the generic case where it's not a DSA
device. Currently DSA only means that it has an Ethernet connection
towards the system, so its I/O is performed indirectly. But
"super-Felix" can be a pure switchdev device just as well, we need to
think about this situation in a generic way.
The point is just that "super-Felix" has awareness of the DSA tags of
switches beneath it. It can listen for "change upper" events for
bridging, and it can detect when its standalone copper port 4 gets
added to the same bridge as one such downstream switch that it can
understand.
So in that case, the "super-Felix" switch can do some magic in the
background: it can permit hardware bridging of its copper port 4 with
a downstream sja1105 hanging off of its port 0. Based on the topology
described in the device tree, packets sent to the sja1105 would
contain a DSA tag, and packets sent to the copper port wouldn't. From
a user perspective, things would "just work".

I know the data flow sja1105 <-> super-Felix copper port 4 that I just
described is different than what this patch set is providing. With
current Felix, this data flow is not even possible in hardware. But I
would like to look forward and imagine, with that super-Felix, if br1
would still be necessary for the simple case where we're bridging
sw1p0 with sw2p0. I think it would still be necessary, because there's
still no "natural" place for super-Felix to listen on "change upper"
events of the DSA net devices below it. That's the dilemma I'm having,
but it looks like br1 between masters is still the way to go, and that
model won't change regardless of whether the parent switchdev driver
is DSA-aware or not.

I would like to get some more feedback on this.

> >
> > For this to work, we need to give drivers an opportunity to listen for
> > bridging events on DSA trees other than their own, and pass that other
> > tree index as argument. I have made the assumption, for the moment, that
> > the other existing DSA notifiers don't need to be broadcast to other
> > trees. That assumption might turn out to be incorrect. But in the
> > meantime, introduce a dsa_broadcast function, similar in purpose to
> > dsa_port_notify, which is used only by the bridging notifiers.
> >
> > Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
> > ---
> Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
> --
> Florian

Thanks,
-Vladimir