[PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Vladimir Oltean]

It seems that of_find_compatible_node has a weird calling convention in
which it calls of_node_put() on the "from" node argument, instead of
leaving that up to the caller. This comes from the fact that
of_find_compatible_node with a non-NULL "from" argument it only supposed
to be used as the iterator function of for_each_compatible_node(). OF
iterator functions call of_node_get on the next OF node and of_node_put()
on the previous one.

When of_find_compatible_node calls of_node_put, it actually never
expects the refcount to drop to zero, because the call is done under the
atomic devtree_lock context, and when the refcount drops to zero it
triggers a kobject and a sysfs file deletion, which assume blocking
context.

So any driver call to of_find_compatible_node is probably buggy because
an unexpected of_node_put() takes place.

What should be done is to use the of_get_compatible_child() function.

Fixes: 5a8f09748ee7 ("net: dsa: sja1105: register the MDIO buses for 100base-T1 and 100base-TX")
Link: https://lore.kernel.org/netdev/20210814010139.kzryimmp4rizlznt@skbuf/
Suggested-by: Frank Rowand <frowand.list@gmail.com>
Suggested-by: Rob Herring <robh+dt@kernel.org>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
 drivers/net/dsa/sja1105/sja1105_mdio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/net/dsa/sja1105/sja1105_mdio.c b/drivers/net/dsa/sja1105/sja1105_mdio.c
index 19aea8fb76f6..705d3900e43a 100644
--- a/drivers/net/dsa/sja1105/sja1105_mdio.c
+++ b/drivers/net/dsa/sja1105/sja1105_mdio.c
@@ -284,8 +284,7 @@ static int sja1105_mdiobus_base_tx_register(struct sja1105_private *priv,
 	struct mii_bus *bus;
 	int rc = 0;
 
-	np = of_find_compatible_node(mdio_node, NULL,
-				     "nxp,sja1110-base-tx-mdio");
+	np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-tx-mdio");
 	if (!np)
 		return 0;
 
@@ -339,8 +338,7 @@ static int sja1105_mdiobus_base_t1_register(struct sja1105_private *priv,
 	struct mii_bus *bus;
 	int rc = 0;
 
-	np = of_find_compatible_node(mdio_node, NULL,
-				     "nxp,sja1110-base-t1-mdio");
+	np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-t1-mdio");
 	if (!np)
 		return 0;
 
-- 
2.25.1

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

Hi Vladimir,

On 8/17/21 4:52 PM, Vladimir Oltean wrote:
> It seems that of_find_compatible_node has a weird calling convention in
> which it calls of_node_put() on the "from" node argument, instead of
> leaving that up to the caller. This comes from the fact that
> of_find_compatible_node with a non-NULL "from" argument it only supposed
> to be used as the iterator function of for_each_compatible_node(). OF
> iterator functions call of_node_get on the next OF node and of_node_put()
> on the previous one.
> 
> When of_find_compatible_node calls of_node_put, it actually never
> expects the refcount to drop to zero, because the call is done under the
> atomic devtree_lock context, and when the refcount drops to zero it
> triggers a kobject and a sysfs file deletion, which assume blocking
> context.
> 
> So any driver call to of_find_compatible_node is probably buggy because
> an unexpected of_node_put() takes place.
> 
> What should be done is to use the of_get_compatible_child() function.

I have an observation that's slightly out of the scope of your patch, 
but I'll post here on the off chance that you find it relevant. 
Apologies if it's out of place.

Do these integrated NXP PHYs use a specific PHY driver, or do they just 
use the Generic PHY driver? If the former is the case, do you experience 
that the PHY driver fails to get probed during mdiobus registration if 
the kernel uses fw_devlink=on?

In my case I am writing a new subdriver for realtek-smi, a DSA driver 
which registers an internal MDIO bus analogously to sja1105, which is 
why I'm asking. I noticed a deferred probe of the PHY driver because the 
supplier (ethernet-switch) is not ready - presumably because all of this 
is happening in the probe of the switch driver. See below:

[   83.653213] device_add:3270: device: 'SMI-0': device_add
[   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
[   83.654055] device_add:3270: device: 
'platform:ethernet-switch--mdio_bus:SMI-0': device_add
[   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync 
state only consumer to ethernet-switch
[   83.654291] libphy: SMI slave MII: probed
...
[   83.659809] device_add:3270: device: 'SMI-0:00': device_add
[   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
[   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
[   83.660122] device_add:3270: device: 
'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
[   83.660274] devices_kset_move_last:2701: devices_kset: Moving 
SMI-0:00 to end of list
[   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to 
end of list
[   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a 
consumer to ethernet-switch
[   83.660350] __driver_probe_device:736: bus: 'mdio_bus': 
__driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC 
Gigabit Ethernet
[   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: 
probe deferral - supplier ethernet-switch not ready
[   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added 
to deferred list

It's not necessarily fatal because phy_attach_direct will just use the 
Generic PHY driver as a fallback, but it's obviously not the intended 
behaviour.

Perhaps this affects your driver too? Due to lack of hardware I am not 
in a position to test, but a static code analysis suggests it may be if 
you are expecting anything but Generic PHY.

Kind regards,
Alvin

> 
> Fixes: 5a8f09748ee7 ("net: dsa: sja1105: register the MDIO buses for 100base-T1 and 100base-TX")
> Link: https://lore.kernel.org/netdev/20210814010139.kzryimmp4rizlznt@skbuf/
> Suggested-by: Frank Rowand <frowand.list@gmail.com>
> Suggested-by: Rob Herring <robh+dt@kernel.org>
> Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
> ---
>   drivers/net/dsa/sja1105/sja1105_mdio.c | 6 ++----
>   1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/dsa/sja1105/sja1105_mdio.c b/drivers/net/dsa/sja1105/sja1105_mdio.c
> index 19aea8fb76f6..705d3900e43a 100644
> --- a/drivers/net/dsa/sja1105/sja1105_mdio.c
> +++ b/drivers/net/dsa/sja1105/sja1105_mdio.c
> @@ -284,8 +284,7 @@ static int sja1105_mdiobus_base_tx_register(struct sja1105_private *priv,
>   	struct mii_bus *bus;
>   	int rc = 0;
>   
> -	np = of_find_compatible_node(mdio_node, NULL,
> -				     "nxp,sja1110-base-tx-mdio");
> +	np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-tx-mdio");
>   	if (!np)
>   		return 0;
>   
> @@ -339,8 +338,7 @@ static int sja1105_mdiobus_base_t1_register(struct sja1105_private *priv,
>   	struct mii_bus *bus;
>   	int rc = 0;
>   
> -	np = of_find_compatible_node(mdio_node, NULL,
> -				     "nxp,sja1110-base-t1-mdio");
> +	np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-t1-mdio");
>   	if (!np)
>   		return 0;
>   
> 

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Andrew Lunn]

> Do these integrated NXP PHYs use a specific PHY driver, or do they just 
> use the Generic PHY driver? If the former is the case, do you experience 
> that the PHY driver fails to get probed during mdiobus registration if 
> the kernel uses fw_devlink=on?

The Marvell mv88e6xxx driver registers upto two MDIO busses, and the
PHYs on those busses make use of the marvell PHY driver. I've not
tested specifically with fw_devlink=on, but in general, the Marvell
PHY driver does get bound to these devices.

    Andrew

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

Hi Andrew,

On 8/18/21 12:05 AM, Andrew Lunn wrote:
>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
>> use the Generic PHY driver? If the former is the case, do you experience
>> that the PHY driver fails to get probed during mdiobus registration if
>> the kernel uses fw_devlink=on?
> 
> The Marvell mv88e6xxx driver registers upto two MDIO busses, and the
> PHYs on those busses make use of the marvell PHY driver. I've not
> tested specifically with fw_devlink=on, but in general, the Marvell
> PHY driver does get bound to these devices.

Thanks for your reply. I should add that with fw_devlink=permissive, the 
PHY driver gets probed just fine. It's only since v5.13 that 
fw_devlink=on has been the default[1], so it may be something new to 
check. But if you think it's working for other drivers then that means I 
will have to look into further on my end.

Kind regards,
Alvin

[1] ea718c699055 ("Revert "Revert "driver core: Set fw_devlink=on by 
default""")
 
https://lore.kernel.org/lkml/20210302211133.2244281-4-saravanak@google.com/

> 
>      Andrew
> 

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Vladimir Oltean]

Hi Alvin,

On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
> I have an observation that's slightly out of the scope of your patch,
> but I'll post here on the off chance that you find it relevant.
> Apologies if it's out of place.
>
> Do these integrated NXP PHYs use a specific PHY driver, or do they just
> use the Generic PHY driver?

They refuse to probe at all with the Generic PHY driver. I have been
caught off guard a few times now when I had a kernel built with
CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.

> If the former is the case, do you experience that the PHY driver fails
> to get probed during mdiobus registration if the kernel uses
> fw_devlink=on?

I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
time I do it. It behaves exactly as you say.

>
> In my case I am writing a new subdriver for realtek-smi, a DSA driver
> which registers an internal MDIO bus analogously to sja1105, which is
> why I'm asking. I noticed a deferred probe of the PHY driver because the
> supplier (ethernet-switch) is not ready - presumably because all of this
> is happening in the probe of the switch driver. See below:
>
> [   83.653213] device_add:3270: device: 'SMI-0': device_add
> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
> [   83.654291] libphy: SMI slave MII: probed
> ...
> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list

So it's a circular dependency? Switch cannot finish probing because it
cannot connect to PHY, which cannot probe because switch has not
finished probing, which....

So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00'
device should not be added to the deferred list, it should have everything
it needs right now (after all, it works without fw_devlink). No?

It might be the late hour over here too, but right now I just don't
know. Let me add Saravana to the discussion too, he made an impressive
analysis recently on a PHY probing issue with mdio-mux, so the PHY
library probing dependencies should still be fresh in his mind, maybe he
has an idea what's wrong.

>
> It's not necessarily fatal because phy_attach_direct will just use the
> Generic PHY driver as a fallback, but it's obviously not the intended
> behaviour.
>
> Perhaps this affects your driver too? Due to lack of hardware I am not
> in a position to test, but a static code analysis suggests it may be if
> you are expecting anything but Generic PHY.

Yeah, rest assured, you're not the only one.

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Vladimir Oltean]

On Wed, Aug 18, 2021 at 01:31:01AM +0300, Vladimir Oltean wrote:
> So it's a circular dependency? Switch cannot finish probing because it
> cannot connect to PHY, which cannot probe because switch has not
> finished probing, which....

The missing piece of the puzzle might be that MAC drivers can either
connect to the PHY at probe time, or at .ndo_open time. DSA chooses to
call phylink_of_phy_connect() inside dsa_port_phylink_register(), so at
probe time. Maybe this is a reason why it is a problem for some drivers
but not for others.

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

Hi Vladimir,

On 8/18/21 12:31 AM, Vladimir Oltean wrote:
> 
> So it's a circular dependency? Switch cannot finish probing because it
> cannot connect to PHY, which cannot probe because switch has not
> finished probing, which....
> 
> So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00'
> device should not be added to the deferred list, it should have everything
> it needs right now (after all, it works without fw_devlink). No?

If I understand correctly, it's just the probing of the PHY driver that 
is getting deferred. This happens both when everything is builtin, and 
when the switch driver (realtek-smi in my case) is built as a module. 
Additionally, when the switch is finished probing, the PHY driver 
doesn't actually get probed. I'm not sure if this is because it has been 
replaced with the Generic PHY driver in the mean time.

What's strange is that if I unload the switch driver module and reload 
it, this time the PHY driver _does_ get probed on demand. I have 
attached a debug log which illustrates the difference. Please note that 
this is using a subdriver of realtek-smi which I have not yet posted to 
the list, but the semantics are the pretty much same as rtl8366rb. See here:

1st load:

[   83.659809] device_add:3270: device: 'SMI-0:00': device_add
[   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
[   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
[   83.660122] device_add:3270: device: 
'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
[   83.660274] devices_kset_move_last:2701: devices_kset: Moving 
SMI-0:00 to end of list
[   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to 
end of list
[   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a 
consumer to ethernet-switch
[   83.660350] __driver_probe_device:736: bus: 'mdio_bus': 
__driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC 
Gigabit Ethernet
[   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: 
probe deferral - supplier ethernet-switch not ready

2nd load:

[  138.945901] device_add:3270: device: 'SMI-0:00': device_add
[  138.946003] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
[  138.946097] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
[  138.946309] __driver_probe_device:736: bus: 'mdio_bus': 
__driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC 
Gigabit Ethernet
[  138.946326] really_probe:561: bus: 'mdio_bus': really_probe: probing 
driver RTL8365MB-VC Gigabit Ethernet with device SMI-0:00

I don't know the gory details of the driver core so I am reluctant to 
provide an explanation, but I hope it makes things clear.

Kind regards,
Alvin

[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 37089 bytes --]

[   83.374196] bus_add_driver:600: bus: 'platform': add driver realtek-smi
[   83.374409] __driver_probe_device:736: bus: 'platform': __driver_probe_device: matched device ethernet-switch with driver realtek-smi
[   83.374427] really_probe:561: bus: 'platform': really_probe: probing driver realtek-smi with device ethernet-switch
[   83.374469] pinctrl_bind_pins:34: realtek-smi ethernet-switch: no pinctrl handle
[   83.374487] of_dma_configure_id:151: realtek-smi ethernet-switch: device is not dma coherent
[   83.374497] of_dma_configure_id:163: realtek-smi ethernet-switch: device is not behind an iommu
[   83.516721] realtek-smi ethernet-switch: deasserted RESET
[   83.517239] realtek-smi ethernet-switch: found an RTL8365MB-VC switch (ver=0x0040)
[   83.652325] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/interrupt-controller, index=0
[   83.652351] of_irq_parse_one:319: OF:  parent=/soc@0/bus@30000000/gpio@30230000, intsize=2
[   83.652366] of_irq_parse_one:332: OF:  intspec=31
[   83.652372] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/soc@0/bus@30000000/gpio@30230000, size=2
[   83.652386] of_irq_parse_raw:143: OF:  -> addrsize=1
[   83.652391] of_irq_parse_raw:163: OF:  -> got it !
[   83.653213] device_add:3270: device: 'SMI-0': device_add
[   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
[   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
[   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
[   83.654291] libphy: SMI slave MII: probed
[   83.659598] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@0, index=0
[   83.659687] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[   83.659699] of_irq_parse_one:332: OF:  intspec=0
[   83.659705] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[   83.659716] of_irq_parse_raw:143: OF:  -> addrsize=0
[   83.659722] of_irq_parse_raw:163: OF:  -> got it !
[   83.659809] device_add:3270: device: 'SMI-0:00': device_add
[   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
[   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
[   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
[   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
[   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
[   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
[   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
[   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
[   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
[   83.660388] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0b990 fwnode at address 0
[   83.665538] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@1, index=0
[   83.665578] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[   83.665590] of_irq_parse_one:332: OF:  intspec=1
[   83.665596] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[   83.665607] of_irq_parse_raw:143: OF:  -> addrsize=0
[   83.665613] of_irq_parse_raw:163: OF:  -> got it !
[   83.665693] device_add:3270: device: 'SMI-0:01': device_add
[   83.665765] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:01
[   83.665846] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:01
[   83.665993] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:01': device_add
[   83.666155] devices_kset_move_last:2701: devices_kset: Moving SMI-0:01 to end of list
[   83.666164] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:01 to end of list
[   83.666176] device_link_add:859: mdio_bus SMI-0:01: Linked as a consumer to ethernet-switch
[   83.666241] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:01 with driver RTL8365MB-VC Gigabit Ethernet
[   83.666258] device_links_check_suppliers:1001: mdio_bus SMI-0:01: probe deferral - supplier ethernet-switch not ready
[   83.666270] driver_deferred_probe_add:138: mdio_bus SMI-0:01: Added to deferred list
[   83.666282] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0bc50 fwnode at address 1
[   83.673533] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@2, index=0
[   83.673584] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[   83.673596] of_irq_parse_one:332: OF:  intspec=2
[   83.673602] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[   83.673613] of_irq_parse_raw:143: OF:  -> addrsize=0
[   83.673619] of_irq_parse_raw:163: OF:  -> got it !
[   83.673715] device_add:3270: device: 'SMI-0:02': device_add
[   83.673816] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:02
[   83.673890] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:02
[   83.674108] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:02': device_add
[   83.674306] devices_kset_move_last:2701: devices_kset: Moving SMI-0:02 to end of list
[   83.674315] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:02 to end of list
[   83.674327] device_link_add:859: mdio_bus SMI-0:02: Linked as a consumer to ethernet-switch
[   83.674391] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:02 with driver RTL8365MB-VC Gigabit Ethernet
[   83.674408] device_links_check_suppliers:1001: mdio_bus SMI-0:02: probe deferral - supplier ethernet-switch not ready
[   83.674420] driver_deferred_probe_add:138: mdio_bus SMI-0:02: Added to deferred list
[   83.674433] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0bf10 fwnode at address 2
[   83.681037] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@3, index=0
[   83.681076] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[   83.681088] of_irq_parse_one:332: OF:  intspec=3
[   83.681094] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[   83.681105] of_irq_parse_raw:143: OF:  -> addrsize=0
[   83.681110] of_irq_parse_raw:163: OF:  -> got it !
[   83.681195] device_add:3270: device: 'SMI-0:03': device_add
[   83.681266] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:03
[   83.681349] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:03
[   83.681490] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:03': device_add
[   83.681646] devices_kset_move_last:2701: devices_kset: Moving SMI-0:03 to end of list
[   83.681653] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:03 to end of list
[   83.681664] device_link_add:859: mdio_bus SMI-0:03: Linked as a consumer to ethernet-switch
[   83.681720] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:03 with driver RTL8365MB-VC Gigabit Ethernet
[   83.681735] device_links_check_suppliers:1001: mdio_bus SMI-0:03: probe deferral - supplier ethernet-switch not ready
[   83.681746] driver_deferred_probe_add:138: mdio_bus SMI-0:03: Added to deferred list
[   83.681757] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0c1d0 fwnode at address 3
[   83.682997] __device_link_del:876: Generic PHY SMI-0:00: Dropping the link to ethernet-switch
[   83.683012] device_unregister:3574: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_unregister
[   83.683207] driver_bound:382: driver: 'Generic PHY': driver_bound: bound to device 'SMI-0:00'
[   83.683228] driver_deferred_probe_del:148: Generic PHY SMI-0:00: Removed from deferred list
[   83.683282] devices_kset_move_last:2701: devices_kset: Moving SMI-0:01 to end of list
[   83.683292] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:01 to end of list
[   83.683302] deferred_probe_work_func:121: mdio_bus SMI-0:01: Retrying from deferred list
[   83.683332] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:01 with driver RTL8365MB-VC Gigabit Ethernet
[   83.683347] device_links_check_suppliers:1001: mdio_bus SMI-0:01: probe deferral - supplier ethernet-switch not ready
[   83.683358] driver_deferred_probe_add:138: mdio_bus SMI-0:01: Added to deferred list
[   83.683367] devices_kset_move_last:2701: devices_kset: Moving SMI-0:02 to end of list
[   83.683373] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:02 to end of list
[   83.683380] deferred_probe_work_func:121: mdio_bus SMI-0:02: Retrying from deferred list
[   83.683395] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:02 with driver RTL8365MB-VC Gigabit Ethernet
[   83.683404] device_links_check_suppliers:1001: mdio_bus SMI-0:02: probe deferral - supplier ethernet-switch not ready
[   83.683412] driver_deferred_probe_add:138: mdio_bus SMI-0:02: Added to deferred list
[   83.683419] devices_kset_move_last:2701: devices_kset: Moving SMI-0:03 to end of list
[   83.683425] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:03 to end of list
[   83.683432] deferred_probe_work_func:121: mdio_bus SMI-0:03: Retrying from deferred list
[   83.683446] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:03 with driver RTL8365MB-VC Gigabit Ethernet
[   83.683455] device_links_check_suppliers:1001: mdio_bus SMI-0:03: probe deferral - supplier ethernet-switch not ready
[   83.683463] driver_deferred_probe_add:138: mdio_bus SMI-0:03: Added to deferred list
[   83.683852] realtek-smi ethernet-switch swp0 (uninitialized): PHY [SMI-0:00] driver [Generic PHY] (irq=POLL)
[   83.683867] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp0 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[   83.683919] device_add:3270: device: 'swp0': device_add
[   83.684285] device_pm_add:136: PM: Adding info for No Bus:swp0
[   83.690497] __device_link_del:876: Generic PHY SMI-0:01: Dropping the link to ethernet-switch
[   83.690520] device_unregister:3574: device: 'platform:ethernet-switch--mdio_bus:SMI-0:01': device_unregister
[   83.690734] driver_bound:382: driver: 'Generic PHY': driver_bound: bound to device 'SMI-0:01'
[   83.690763] driver_deferred_probe_del:148: Generic PHY SMI-0:01: Removed from deferred list
[   83.691166] devices_kset_move_last:2701: devices_kset: Moving SMI-0:02 to end of list
[   83.691180] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:02 to end of list
[   83.691194] deferred_probe_work_func:121: mdio_bus SMI-0:02: Retrying from deferred list
[   83.691224] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:02 with driver RTL8365MB-VC Gigabit Ethernet
[   83.691239] device_links_check_suppliers:1001: mdio_bus SMI-0:02: probe deferral - supplier ethernet-switch not ready
[   83.691250] driver_deferred_probe_add:138: mdio_bus SMI-0:02: Added to deferred list
[   83.691259] devices_kset_move_last:2701: devices_kset: Moving SMI-0:03 to end of list
[   83.691265] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:03 to end of list
[   83.691273] deferred_probe_work_func:121: mdio_bus SMI-0:03: Retrying from deferred list
[   83.691287] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:03 with driver RTL8365MB-VC Gigabit Ethernet
[   83.691297] device_links_check_suppliers:1001: mdio_bus SMI-0:03: probe deferral - supplier ethernet-switch not ready
[   83.691305] driver_deferred_probe_add:138: mdio_bus SMI-0:03: Added to deferred list
[   83.691433] realtek-smi ethernet-switch swp1 (uninitialized): PHY [SMI-0:01] driver [Generic PHY] (irq=POLL)
[   83.691450] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp1 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[   83.691540] device_add:3270: device: 'swp1': device_add
[   83.691908] device_pm_add:136: PM: Adding info for No Bus:swp1
[   83.708741] __device_link_del:876: Generic PHY SMI-0:02: Dropping the link to ethernet-switch
[   83.708757] device_unregister:3574: device: 'platform:ethernet-switch--mdio_bus:SMI-0:02': device_unregister
[   83.708979] driver_bound:382: driver: 'Generic PHY': driver_bound: bound to device 'SMI-0:02'
[   83.709011] driver_deferred_probe_del:148: Generic PHY SMI-0:02: Removed from deferred list
[   83.709647] realtek-smi ethernet-switch swp2 (uninitialized): PHY [SMI-0:02] driver [Generic PHY] (irq=POLL)
[   83.709665] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp2 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[   83.709756] device_add:3270: device: 'swp2': device_add
[   83.710102] device_pm_add:136: PM: Adding info for No Bus:swp2
[   83.710765] devices_kset_move_last:2701: devices_kset: Moving SMI-0:03 to end of list
[   83.710784] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:03 to end of list
[   83.710799] deferred_probe_work_func:121: mdio_bus SMI-0:03: Retrying from deferred list
[   83.710832] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:03 with driver RTL8365MB-VC Gigabit Ethernet
[   83.710847] device_links_check_suppliers:1001: mdio_bus SMI-0:03: probe deferral - supplier ethernet-switch not ready
[   83.710860] driver_deferred_probe_add:138: mdio_bus SMI-0:03: Added to deferred list
[   83.713895] __device_link_del:876: Generic PHY SMI-0:03: Dropping the link to ethernet-switch
[   83.713919] device_unregister:3574: device: 'platform:ethernet-switch--mdio_bus:SMI-0:03': device_unregister
[   83.714185] driver_bound:382: driver: 'Generic PHY': driver_bound: bound to device 'SMI-0:03'
[   83.714243] driver_deferred_probe_del:148: Generic PHY SMI-0:03: Removed from deferred list
[   83.715325] realtek-smi ethernet-switch swp3 (uninitialized): PHY [SMI-0:03] driver [Generic PHY] (irq=POLL)
[   83.715349] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp3 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[   83.715412] device_add:3270: device: 'swp3': device_add
[   83.715855] device_pm_add:136: PM: Adding info for No Bus:swp3
[   83.726038] DSA: failed to set STP state 3 (-95)
[   83.726045] realtek-smi ethernet-switch: configuring for fixed/rgmii-id link mode
[   83.726058] phylink_major_config:454: realtek-smi ethernet-switch: major config rgmii-id
[   83.726068] phylink_mac_config:425: realtek-smi ethernet-switch: phylink_mac_config: mode=fixed/rgmii-id/1Gbps/Full adv=0000000,00000000,00006260 pause=03 link=0 an=1
[   83.726539] device_add:3270: device: 'platform:30be0000.ethernet--platform:ethernet-switch': device_add
[   83.726747] devices_kset_move_last:2701: devices_kset: Moving ethernet-switch to end of list
[   83.726755] device_pm_move_last:203: PM: Moving platform:ethernet-switch to end of list
[   83.726762] devices_kset_move_last:2701: devices_kset: Moving SMI-0 to end of list
[   83.726768] device_pm_move_last:203: PM: Moving No Bus:SMI-0 to end of list
[   83.726774] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
[   83.726781] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
[   83.726788] devices_kset_move_last:2701: devices_kset: Moving SMI-0:01 to end of list
[   83.726794] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:01 to end of list
[   83.726800] devices_kset_move_last:2701: devices_kset: Moving SMI-0:02 to end of list
[   83.726806] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:02 to end of list
[   83.726813] devices_kset_move_last:2701: devices_kset: Moving SMI-0:03 to end of list
[   83.726818] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:03 to end of list
[   83.726825] devices_kset_move_last:2701: devices_kset: Moving swp0 to end of list
[   83.726831] device_pm_move_last:203: PM: Moving No Bus:swp0 to end of list
[   83.726837] devices_kset_move_last:2701: devices_kset: Moving swp1 to end of list
[   83.726843] device_pm_move_last:203: PM: Moving No Bus:swp1 to end of list
[   83.726848] devices_kset_move_last:2701: devices_kset: Moving swp2 to end of list
[   83.726854] device_pm_move_last:203: PM: Moving No Bus:swp2 to end of list
[   83.726860] devices_kset_move_last:2701: devices_kset: Moving swp3 to end of list
[   83.726865] device_pm_move_last:203: PM: Moving No Bus:swp3 to end of list
[   83.726875] device_link_add:859: realtek-smi ethernet-switch: Linked as a consumer to 30be0000.ethernet
[   83.727000] DSA: tree 0 setup
[   83.727011] driver_bound:382: driver: 'realtek-smi': driver_bound: bound to device 'ethernet-switch'
[   83.727052] __device_link_del:876: realtek-smi ethernet-switch: Dropping the link to 30230000.gpio
[   83.727060] device_unregister:3574: device: 'platform:30230000.gpio--platform:ethernet-switch': device_unregister
[   83.727219] __device_link_del:876: realtek-smi ethernet-switch: Dropping the link to ethernet-switch
[   83.727227] device_unregister:3574: device: 'platform:ethernet-switch--platform:ethernet-switch': device_unregister
[   83.727436] really_probe:644: bus: 'platform': really_probe: bound device ethernet-switch to driver realtek-smi
[   83.728496] phylink_mac_config:425: realtek-smi ethernet-switch: phylink_mac_config: mode=fixed/rgmii-id/1Gbps/Full adv=0000000,00000000,00006260 pause=03 link=1 an=1
[   83.729182] realtek-smi ethernet-switch: Link is Up - 1Gbps/Full - flow control rx/tx
[   83.920969] br0: port 1(brwan0) entered disabled state
[   83.923179] br0: port 3(swp2) entered blocking state
[   83.923201] br0: port 3(swp2) entered disabled state
[   83.923594] realtek-smi ethernet-switch: enable filtering on port 2
[   83.923627] device swp2 entered promiscuous mode
[   83.924034] realtek-smi ethernet-switch: add VLAN 1 on port 2, untagged,  PVID
[   83.964396] realtek-smi ethernet-switch: add VLAN 1 on port 4, untagged, no PVID
[   83.964417] realtek-smi ethernet-switch: port is DSA or CPU port
[   84.052238] driver_bound:382: driver: 'Generic PHY': driver_bound: bound to device 'fixed-0:00'
[   84.052584] Generic PHY fixed-0:00: attached PHY driver (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[   84.053854] fec 30be0000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   84.053878] phy_state_machine:1145: Generic PHY fixed-0:00: PHY state change UP -> RUNNING
[   84.072768] device_unregister:3574: device: 'wakeup3': device_unregister
[   84.073651] device eth0 entered promiscuous mode
[   84.074771] realtek-smi ethernet-switch swp2: configuring for phy/ link mode
[   84.074786] phylink_major_config:454: realtek-smi ethernet-switch swp2: major config 
[   84.074796] phylink_mac_config:425: realtek-smi ethernet-switch swp2: phylink_mac_config: mode=phy//Unknown/Unknown adv=0000000,00000000,00000000 pause=00 link=0 an=0
[   84.081841] realtek-smi ethernet-switch: add VLAN 1 on port 2, untagged,  PVID
[   84.083377] phylink_phy_change:945: realtek-smi ethernet-switch swp2: phy link down /Unknown/Unknown
[   84.083393] phy_state_machine:1145: Generic PHY SMI-0:02: PHY state change UP -> NOLINK
[   84.086760] realtek-smi ethernet-switch: add VLAN 1 on port 4, untagged, no PVID
[   84.086772] realtek-smi ethernet-switch: port is DSA or CPU port
[   87.136746] phylink_phy_change:945: realtek-smi ethernet-switch swp2: phy link up /1Gbps/Full
[   87.136767] phy_state_machine:1145: Generic PHY SMI-0:02: PHY state change NOLINK -> RUNNING
[   87.136780] phylink_mac_config:425: realtek-smi ethernet-switch swp2: phylink_mac_config: mode=phy//1Gbps/Full adv=0000000,00000000,00000000 pause=03 link=1 an=0
[   87.136809] realtek-smi ethernet-switch swp2: Link is Up - 1Gbps/Full - flow control rx/tx
[   87.136855] br0: port 3(swp2) entered blocking state
[   87.136868] br0: port 3(swp2) entered forwarding state
[   87.171073] br0: port 1(brwan0) entered blocking state
[   87.171106] br0: port 1(brwan0) entered forwarding state
[   87.325549] realtek-smi ethernet-switch: add VLAN 1 on port 2, untagged,  PVID
[   87.329594] realtek-smi ethernet-switch: add VLAN 1 on port 4, untagged, no PVID
[   87.329612] realtek-smi ethernet-switch: port is DSA or CPU port
[   87.387104] device_add:3270: device: 'lo': device_add
[   87.387457] device_pm_add:136: PM: Adding info for No Bus:lo
[  117.873950] device_pm_remove:157: PM: Removing info for No Bus:lo
[  134.225421] bus_remove_driver:674: bus: 'platform': remove driver realtek-smi
[  134.270168] device_pm_remove:157: PM: Removing info for No Bus:swp0
[  134.370066] device_pm_remove:157: PM: Removing info for No Bus:swp1
[  134.481366] phylink_phy_change:945: realtek-smi ethernet-switch swp2: phy link down /1Gbps/Full
[  134.483955] device eth0 left promiscuous mode
[  134.484472] br0: port 3(swp2) entered disabled state
[  134.513782] device swp2 left promiscuous mode
[  134.513818] br0: port 3(swp2) entered disabled state
[  134.513963] realtek-smi ethernet-switch: del VLAN 0001 on port 2
[  134.529559] realtek-smi ethernet-switch: disable filtering on port 2
[  134.529583] DSA: failed to set STP state 3 (-95)
[  134.546619] device_pm_remove:157: PM: Removing info for No Bus:swp2
[  134.633024] br0: port 1(brwan0) entered disabled state
[  134.670303] device_pm_remove:157: PM: Removing info for No Bus:swp3
[  134.753163] realtek-smi ethernet-switch: Link is Down
[  134.753197] DSA: failed to set STP state 0 (-95)
[  134.753339] bus_remove_device:527: bus: 'mdio_bus': remove device SMI-0:00
[  134.753349] device_pm_remove:157: PM: Removing info for mdio_bus:SMI-0:00
[  134.753551] bus_remove_device:527: bus: 'mdio_bus': remove device SMI-0:01
[  134.753559] device_pm_remove:157: PM: Removing info for mdio_bus:SMI-0:01
[  134.753698] bus_remove_device:527: bus: 'mdio_bus': remove device SMI-0:02
[  134.753705] device_pm_remove:157: PM: Removing info for mdio_bus:SMI-0:02
[  134.753849] bus_remove_device:527: bus: 'mdio_bus': remove device SMI-0:03
[  134.753855] device_pm_remove:157: PM: Removing info for mdio_bus:SMI-0:03
[  134.754463] device_pm_remove:157: PM: Removing info for No Bus:SMI-0
[  134.754479] __device_link_del:876: mdio_bus SMI-0: Dropping the link to ethernet-switch
[  134.754487] device_unregister:3574: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_unregister
[  134.757201] DSA: tree 0 torn down
[  134.758604] __device_link_del:876: realtek-smi ethernet-switch: Dropping the link to 30be0000.ethernet
[  134.758622] device_unregister:3574: device: 'platform:30be0000.ethernet--platform:ethernet-switch': device_unregister
[  134.759169] driver_release:90: driver: 'realtek-smi': driver_release
[  134.793043] device_add:3270: device: 'lo': device_add
[  134.793424] device_pm_add:136: PM: Adding info for No Bus:lo
[  138.674861] bus_add_driver:600: bus: 'platform': add driver realtek-smi
[  138.675096] __driver_probe_device:736: bus: 'platform': __driver_probe_device: matched device ethernet-switch with driver realtek-smi
[  138.675112] really_probe:561: bus: 'platform': really_probe: probing driver realtek-smi with device ethernet-switch
[  138.675157] pinctrl_bind_pins:34: realtek-smi ethernet-switch: no pinctrl handle
[  138.675175] of_dma_configure_id:151: realtek-smi ethernet-switch: device is not dma coherent
[  138.675185] of_dma_configure_id:163: realtek-smi ethernet-switch: device is not behind an iommu
[  138.820701] realtek-smi ethernet-switch: deasserted RESET
[  138.821180] realtek-smi ethernet-switch: found an RTL8365MB-VC switch (ver=0x0040)
[  138.937173] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/interrupt-controller, index=0
[  138.937195] of_irq_parse_one:319: OF:  parent=/soc@0/bus@30000000/gpio@30230000, intsize=2
[  138.937209] of_irq_parse_one:332: OF:  intspec=31
[  138.937214] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/soc@0/bus@30000000/gpio@30230000, size=2
[  138.937228] of_irq_parse_raw:143: OF:  -> addrsize=1
[  138.937234] of_irq_parse_raw:163: OF:  -> got it !
[  138.938252] device_add:3270: device: 'SMI-0': device_add
[  138.938879] device_pm_add:136: PM: Adding info for No Bus:SMI-0
[  138.939064] libphy: SMI slave MII: probed
[  138.945677] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@0, index=0
[  138.945729] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[  138.945742] of_irq_parse_one:332: OF:  intspec=0
[  138.945748] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[  138.945760] of_irq_parse_raw:143: OF:  -> addrsize=0
[  138.945766] of_irq_parse_raw:163: OF:  -> got it !
[  138.945901] device_add:3270: device: 'SMI-0:00': device_add
[  138.946003] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
[  138.946097] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
[  138.946309] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
[  138.946326] really_probe:561: bus: 'mdio_bus': really_probe: probing driver RTL8365MB-VC Gigabit Ethernet with device SMI-0:00
[  138.946379] pinctrl_bind_pins:34: RTL8365MB-VC Gigabit Ethernet SMI-0:00: no pinctrl handle
[  138.947516] driver_bound:382: driver: 'RTL8365MB-VC Gigabit Ethernet': driver_bound: bound to device 'SMI-0:00'
[  138.947729] really_probe:644: bus: 'mdio_bus': really_probe: bound device SMI-0:00 to driver RTL8365MB-VC Gigabit Ethernet
[  138.947763] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0b990 fwnode at address 0
[  138.955596] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@1, index=0
[  138.955650] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[  138.955662] of_irq_parse_one:332: OF:  intspec=1
[  138.955668] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[  138.955681] of_irq_parse_raw:143: OF:  -> addrsize=0
[  138.955687] of_irq_parse_raw:163: OF:  -> got it !
[  138.955792] device_add:3270: device: 'SMI-0:01': device_add
[  138.955888] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:01
[  138.955972] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:01
[  138.956191] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:01 with driver RTL8365MB-VC Gigabit Ethernet
[  138.956210] really_probe:561: bus: 'mdio_bus': really_probe: probing driver RTL8365MB-VC Gigabit Ethernet with device SMI-0:01
[  138.956266] pinctrl_bind_pins:34: RTL8365MB-VC Gigabit Ethernet SMI-0:01: no pinctrl handle
[  138.957465] driver_bound:382: driver: 'RTL8365MB-VC Gigabit Ethernet': driver_bound: bound to device 'SMI-0:01'
[  138.957672] really_probe:644: bus: 'mdio_bus': really_probe: bound device SMI-0:01 to driver RTL8365MB-VC Gigabit Ethernet
[  138.957704] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0bc50 fwnode at address 1
[  138.965135] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@2, index=0
[  138.965176] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[  138.965188] of_irq_parse_one:332: OF:  intspec=2
[  138.965193] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[  138.965205] of_irq_parse_raw:143: OF:  -> addrsize=0
[  138.965210] of_irq_parse_raw:163: OF:  -> got it !
[  138.965286] device_add:3270: device: 'SMI-0:02': device_add
[  138.965356] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:02
[  138.965427] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:02
[  138.965670] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:02 with driver RTL8365MB-VC Gigabit Ethernet
[  138.965686] really_probe:561: bus: 'mdio_bus': really_probe: probing driver RTL8365MB-VC Gigabit Ethernet with device SMI-0:02
[  138.965730] pinctrl_bind_pins:34: RTL8365MB-VC Gigabit Ethernet SMI-0:02: no pinctrl handle
[  138.972311] driver_bound:382: driver: 'RTL8365MB-VC Gigabit Ethernet': driver_bound: bound to device 'SMI-0:02'
[  138.972481] really_probe:644: bus: 'mdio_bus': really_probe: bound device SMI-0:02 to driver RTL8365MB-VC Gigabit Ethernet
[  138.972503] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0bf10 fwnode at address 2
[  138.982999] of_irq_parse_one:293: OF: of_irq_parse_one: dev=/ethernet-switch/mdio/phy@3, index=0
[  138.983047] of_irq_parse_one:319: OF:  parent=/ethernet-switch/interrupt-controller, intsize=1
[  138.983060] of_irq_parse_one:332: OF:  intspec=3
[  138.983066] of_irq_parse_raw:124: OF: of_irq_parse_raw: ipar=/ethernet-switch/interrupt-controller, size=1
[  138.983077] of_irq_parse_raw:143: OF:  -> addrsize=0
[  138.983083] of_irq_parse_raw:163: OF:  -> got it !
[  138.983177] device_add:3270: device: 'SMI-0:03': device_add
[  138.983257] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:03
[  138.983358] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:03
[  138.983519] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:03 with driver RTL8365MB-VC Gigabit Ethernet
[  138.983534] really_probe:561: bus: 'mdio_bus': really_probe: probing driver RTL8365MB-VC Gigabit Ethernet with device SMI-0:03
[  138.983583] pinctrl_bind_pins:34: RTL8365MB-VC Gigabit Ethernet SMI-0:03: no pinctrl handle
[  138.984715] driver_bound:382: driver: 'RTL8365MB-VC Gigabit Ethernet': driver_bound: bound to device 'SMI-0:03'
[  138.985118] really_probe:644: bus: 'mdio_bus': really_probe: bound device SMI-0:03 to driver RTL8365MB-VC Gigabit Ethernet
[  138.985150] fwnode_mdiobus_phy_device_register:79: mdio_bus SMI-0: registered phy ffff00003de0c1d0 fwnode at address 3
[  138.985941] realtek-smi ethernet-switch swp0 (uninitialized): PHY [SMI-0:00] driver [RTL8365MB-VC Gigabit Ethernet] (irq=221)
[  138.985963] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp0 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[  138.986092] device_add:3270: device: 'swp0': device_add
[  138.986453] device_pm_add:136: PM: Adding info for No Bus:swp0
[  138.997467] realtek-smi ethernet-switch swp1 (uninitialized): PHY [SMI-0:01] driver [RTL8365MB-VC Gigabit Ethernet] (irq=222)
[  138.997497] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp1 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[  138.997639] device_add:3270: device: 'swp1': device_add
[  138.998008] device_pm_add:136: PM: Adding info for No Bus:swp1
[  139.006406] realtek-smi ethernet-switch swp2 (uninitialized): PHY [SMI-0:02] driver [RTL8365MB-VC Gigabit Ethernet] (irq=223)
[  139.006439] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp2 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[  139.006627] device_add:3270: device: 'swp2': device_add
[  139.008270] device_pm_add:136: PM: Adding info for No Bus:swp2
[  139.010816] realtek-smi ethernet-switch swp3 (uninitialized): PHY [SMI-0:03] driver [RTL8365MB-VC Gigabit Ethernet] (irq=224)
[  139.010914] phylink_bringup_phy:1019: realtek-smi ethernet-switch swp3 (uninitialized): phy: setting supported 0000000,00000000,000062ef advertising 0000000,00000000,000062ef
[  139.011098] device_add:3270: device: 'swp3': device_add
[  139.011743] device_pm_add:136: PM: Adding info for No Bus:swp3
[  139.013288] DSA: failed to set STP state 3 (-95)
[  139.013294] realtek-smi ethernet-switch: configuring for fixed/rgmii-id link mode
[  139.013307] phylink_major_config:454: realtek-smi ethernet-switch: major config rgmii-id
[  139.013317] phylink_mac_config:425: realtek-smi ethernet-switch: phylink_mac_config: mode=fixed/rgmii-id/1Gbps/Full adv=0000000,00000000,00006260 pause=03 link=0 an=1
[  139.013785] device_add:3270: device: 'platform:30be0000.ethernet--platform:ethernet-switch': device_add
[  139.014013] devices_kset_move_last:2701: devices_kset: Moving ethernet-switch to end of list
[  139.014021] device_pm_move_last:203: PM: Moving platform:ethernet-switch to end of list
[  139.014029] devices_kset_move_last:2701: devices_kset: Moving SMI-0 to end of list
[  139.014034] device_pm_move_last:203: PM: Moving No Bus:SMI-0 to end of list
[  139.014041] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
[  139.014047] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
[  139.014054] devices_kset_move_last:2701: devices_kset: Moving SMI-0:01 to end of list
[  139.014060] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:01 to end of list
[  139.014066] devices_kset_move_last:2701: devices_kset: Moving SMI-0:02 to end of list
[  139.014072] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:02 to end of list
[  139.014079] devices_kset_move_last:2701: devices_kset: Moving SMI-0:03 to end of list
[  139.014084] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:03 to end of list
[  139.014091] devices_kset_move_last:2701: devices_kset: Moving swp0 to end of list
[  139.014097] device_pm_move_last:203: PM: Moving No Bus:swp0 to end of list
[  139.014104] devices_kset_move_last:2701: devices_kset: Moving swp1 to end of list
[  139.014110] device_pm_move_last:203: PM: Moving No Bus:swp1 to end of list
[  139.014117] devices_kset_move_last:2701: devices_kset: Moving swp2 to end of list
[  139.014122] device_pm_move_last:203: PM: Moving No Bus:swp2 to end of list
[  139.014128] devices_kset_move_last:2701: devices_kset: Moving swp3 to end of list
[  139.014134] device_pm_move_last:203: PM: Moving No Bus:swp3 to end of list
[  139.014143] device_link_add:859: realtek-smi ethernet-switch: Linked as a consumer to 30be0000.ethernet
[  139.014548] DSA: tree 0 setup
[  139.014565] driver_bound:382: driver: 'realtek-smi': driver_bound: bound to device 'ethernet-switch'
[  139.014709] really_probe:644: bus: 'platform': really_probe: bound device ethernet-switch to driver realtek-smi
[  139.018967] phylink_mac_config:425: realtek-smi ethernet-switch: phylink_mac_config: mode=fixed/rgmii-id/1Gbps/Full adv=0000000,00000000,00006260 pause=03 link=1 an=1
[  139.019281] realtek-smi ethernet-switch: Link is Up - 1Gbps/Full - flow control rx/tx
[  139.173698] br0: port 3(swp2) entered blocking state
[  139.173740] br0: port 3(swp2) entered disabled state
[  139.174273] realtek-smi ethernet-switch: enable filtering on port 2
[  139.174326] device swp2 entered promiscuous mode
[  139.174754] realtek-smi ethernet-switch: add VLAN 1 on port 2, untagged,  PVID
[  139.221959] realtek-smi ethernet-switch: add VLAN 1 on port 4, untagged, no PVID
[  139.221989] realtek-smi ethernet-switch: port is DSA or CPU port
[  139.236864] device eth0 entered promiscuous mode
[  139.238007] realtek-smi ethernet-switch swp2: configuring for phy/ link mode
[  139.238019] phylink_major_config:454: realtek-smi ethernet-switch swp2: major config 
[  139.238029] phylink_mac_config:425: realtek-smi ethernet-switch swp2: phylink_mac_config: mode=phy//Unknown/Unknown adv=0000000,00000000,00000000 pause=00 link=0 an=0
[  139.239141] realtek-smi ethernet-switch: add VLAN 1 on port 2, untagged,  PVID
[  139.254043] realtek-smi ethernet-switch: add VLAN 1 on port 4, untagged, no PVID
[  139.254065] realtek-smi ethernet-switch: port is DSA or CPU port
[  139.257162] phylink_phy_change:945: realtek-smi ethernet-switch swp2: phy link down /Unknown/Unknown
[  139.257203] phy_state_machine:1145: RTL8365MB-VC Gigabit Ethernet SMI-0:02: PHY state change UP -> NOLINK
[  143.225157] phylink_phy_change:945: realtek-smi ethernet-switch swp2: phy link up /1Gbps/Full
[  143.225158] phylink_mac_config:425: realtek-smi ethernet-switch swp2: phylink_mac_config: mode=phy//1Gbps/Full adv=0000000,00000000,00000000 pause=03 link=1 an=0
[  143.225181] phy_state_machine:1145: RTL8365MB-VC Gigabit Ethernet SMI-0:02: PHY state change NOLINK -> RUNNING
[  143.225199] realtek-smi ethernet-switch swp2: Link is Up - 1Gbps/Full - flow control rx/tx
[  143.225252] br0: port 3(swp2) entered blocking state
[  143.225267] br0: port 3(swp2) entered forwarding state
[  143.265881] br0: port 1(brwan0) entered blocking state
[  143.265908] br0: port 1(brwan0) entered forwarding state
[  143.461233] realtek-smi ethernet-switch: add VLAN 1 on port 2, untagged,  PVID
[  143.477285] realtek-smi ethernet-switch: add VLAN 1 on port 4, untagged, no PVID
[  143.477312] realtek-smi ethernet-switch: port is DSA or CPU port

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
>
> Hi Alvin,
>
> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
> > I have an observation that's slightly out of the scope of your patch,
> > but I'll post here on the off chance that you find it relevant.
> > Apologies if it's out of place.
> >
> > Do these integrated NXP PHYs use a specific PHY driver, or do they just
> > use the Generic PHY driver?
>
> They refuse to probe at all with the Generic PHY driver. I have been
> caught off guard a few times now when I had a kernel built with
> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
>
> > If the former is the case, do you experience that the PHY driver fails
> > to get probed during mdiobus registration if the kernel uses
> > fw_devlink=on?
>
> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
> time I do it. It behaves exactly as you say.
>
> >
> > In my case I am writing a new subdriver for realtek-smi, a DSA driver
> > which registers an internal MDIO bus analogously to sja1105, which is
> > why I'm asking. I noticed a deferred probe of the PHY driver because the
> > supplier (ethernet-switch) is not ready - presumably because all of this
> > is happening in the probe of the switch driver. See below:
> >
> > [   83.653213] device_add:3270: device: 'SMI-0': device_add
> > [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
> > [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
> > [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
> > [   83.654291] libphy: SMI slave MII: probed
> > ...
> > [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
> > [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
> > [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
> > [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
> > [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
> > [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
> > [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
> > [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
> > [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
> > [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
>
> So it's a circular dependency? Switch cannot finish probing because it
> cannot connect to PHY, which cannot probe because switch has not
> finished probing, which....

Hi Vladimir/Alvin,

If there's a cyclic dependency between two devices, then fw_devlink=on
is smart enough to notice that. Once it notices a cycle, it knows that
it can't tell which one is the real dependency and which one is the
false dependency and so stops enforcing ordering between the devices
in the cycle.

But fw_devlink doesn't understand all the properties yet. Just most of
them and I'm always trying to add more. So when it only understands
the property that's causing the false dependency but not the property
that causes the real dependency, it can cause issues like this where
fw_devlink=on enforces the false dependency and the driver/code
enforces the real dependency. These are generally easy to fix -- you
just need to teach fw_devlink how to parse more properties.

This is just a preliminary analysis since I don't have all the info
yet -- so I could be wrong. With that said, I happened to be working
on adding fw_devlink support for phy-handle property and I think it
should fix your issue with fw_devlink=on. Can you give [1] a shot?

If it doesn't fix it, can one of you please point me to an upstream
dts (not dtsi) file for a platform in which you see this issue? And
ideally also the DT nodes and their drivers that are involved in this
cycle? With that info, I should be able to root cause this if the
patch above doesn't already fix it.

>
> So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00'
> device should not be added to the deferred list, it should have everything
> it needs right now (after all, it works without fw_devlink). No?
>
> It might be the late hour over here too, but right now I just don't
> know. Let me add Saravana to the discussion too, he made an impressive
> analysis recently on a PHY probing issue with mdio-mux,

Lol, thanks for the kind words.

> so the PHY
> library probing dependencies should still be fresh in his mind, maybe he
> has an idea what's wrong.

[1] - https://lore.kernel.org/lkml/20210818021717.3268255-1-saravanak@google.com/T/#u

Thanks,
Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (refs/heads/master):

On Tue, 17 Aug 2021 17:52:45 +0300 you wrote:
> It seems that of_find_compatible_node has a weird calling convention in
> which it calls of_node_put() on the "from" node argument, instead of
> leaving that up to the caller. This comes from the fact that
> of_find_compatible_node with a non-NULL "from" argument it only supposed
> to be used as the iterator function of for_each_compatible_node(). OF
> iterator functions call of_node_get on the next OF node and of_node_put()
> on the previous one.
> 
> [...]

Here is the summary with links:
  - [net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse
    https://git.kernel.org/netdev/net/c/ed5d2937a6a8

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

Hi Saravana,

On 8/18/21 4:46 AM, Saravana Kannan wrote:
> On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
>>
>> Hi Alvin,
>>
>> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
>>> I have an observation that's slightly out of the scope of your patch,
>>> but I'll post here on the off chance that you find it relevant.
>>> Apologies if it's out of place.
>>>
>>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
>>> use the Generic PHY driver?
>>
>> They refuse to probe at all with the Generic PHY driver. I have been
>> caught off guard a few times now when I had a kernel built with
>> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
>>
>>> If the former is the case, do you experience that the PHY driver fails
>>> to get probed during mdiobus registration if the kernel uses
>>> fw_devlink=on?
>>
>> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
>> time I do it. It behaves exactly as you say.
>>
>>>
>>> In my case I am writing a new subdriver for realtek-smi, a DSA driver
>>> which registers an internal MDIO bus analogously to sja1105, which is
>>> why I'm asking. I noticed a deferred probe of the PHY driver because the
>>> supplier (ethernet-switch) is not ready - presumably because all of this
>>> is happening in the probe of the switch driver. See below:
>>>
>>> [   83.653213] device_add:3270: device: 'SMI-0': device_add
>>> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
>>> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
>>> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
>>> [   83.654291] libphy: SMI slave MII: probed
>>> ...
>>> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
>>> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
>>> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
>>> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
>>> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
>>> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
>>> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
>>> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
>>> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
>>> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
>>
>> So it's a circular dependency? Switch cannot finish probing because it
>> cannot connect to PHY, which cannot probe because switch has not
>> finished probing, which....
> 
> Hi Vladimir/Alvin,
> 
> If there's a cyclic dependency between two devices, then fw_devlink=on
> is smart enough to notice that. Once it notices a cycle, it knows that
> it can't tell which one is the real dependency and which one is the
> false dependency and so stops enforcing ordering between the devices
> in the cycle.
> 
> But fw_devlink doesn't understand all the properties yet. Just most of
> them and I'm always trying to add more. So when it only understands
> the property that's causing the false dependency but not the property
> that causes the real dependency, it can cause issues like this where
> fw_devlink=on enforces the false dependency and the driver/code
> enforces the real dependency. These are generally easy to fix -- you
> just need to teach fw_devlink how to parse more properties.
> 
> This is just a preliminary analysis since I don't have all the info
> yet -- so I could be wrong. With that said, I happened to be working
> on adding fw_devlink support for phy-handle property and I think it
> should fix your issue with fw_devlink=on. Can you give [1] a shot?

I tried [1] but it did not seem to have any effect.

> 
> If it doesn't fix it, can one of you please point me to an upstream
> dts (not dtsi) file for a platform in which you see this issue? And
> ideally also the DT nodes and their drivers that are involved in this
> cycle? With that info, I should be able to root cause this if the
> patch above doesn't already fix it.

I'm working with a non-upstream dts - maybe Vladimir is using an 
upstream one? The pattern among the drivers/dts is common between our 
two cases.

But for the sake of this discussion, my dts is pretty much the same as 
what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The 
nodes of interest from that dts file are below, and the driver is in 
drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the 
Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part 
of the mdiobus registration, but that never happens. See my previous 
reply for a debug log.

/ {
	switch {
		compatible = "realtek,rtl8366rb";
		mdc-gpios = <&gpio0 21 GPIO_ACTIVE_HIGH>;
		mdio-gpios = <&gpio0 22 GPIO_ACTIVE_HIGH>;
		reset-gpios = <&gpio0 14 GPIO_ACTIVE_LOW>;
		realtek,disable-leds;

		switch_intc: interrupt-controller {
			/* GPIO 15 provides the interrupt */
			interrupt-parent = <&gpio0>;
			interrupts = <15 IRQ_TYPE_LEVEL_LOW>;
			interrupt-controller;
			#address-cells = <0>;
			#interrupt-cells = <1>;
		};

		ports {
			/* snip */
		};

		mdio {
			compatible = "realtek,smi-mdio";
			#address-cells = <1>;
			#size-cells = <0>;

			phy0: phy@0 {
				reg = <0>;
				interrupt-parent = <&switch_intc>;
				interrupts = <0>;
			};
			phy1: phy@1 {
				reg = <1>;
				interrupt-parent = <&switch_intc>;
				interrupts = <1>;
			};
			phy2: phy@2 {
				reg = <2>;
				interrupt-parent = <&switch_intc>;
				interrupts = <2>;
			};
			phy3: phy@3 {
				reg = <3>;
				interrupt-parent = <&switch_intc>;
				interrupts = <3>;
			};
			phy4: phy@4 {
				reg = <4>;
				interrupt-parent = <&switch_intc>;
				interrupts = <12>;
			};
		};
	};
};

Thanks for looking into this. Let me know if you need any other info or 
want something tested.

Kind regards,
Alvin

> 
>>
>> So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00'
>> device should not be added to the deferred list, it should have everything
>> it needs right now (after all, it works without fw_devlink). No?
>>
>> It might be the late hour over here too, but right now I just don't
>> know. Let me add Saravana to the discussion too, he made an impressive
>> analysis recently on a PHY probing issue with mdio-mux,
> 
> Lol, thanks for the kind words.
> 
>> so the PHY
>> library probing dependencies should still be fresh in his mind, maybe he
>> has an idea what's wrong.
> 
> [1] - https://lore.kernel.org/lkml/20210818021717.3268255-1-saravanak@google.com/T/#u>> 
> Thanks,
> Saravana
> 

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>
> Hi Saravana,
>
> On 8/18/21 4:46 AM, Saravana Kannan wrote:
> > On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
> >>
> >> Hi Alvin,
> >>
> >> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
> >>> I have an observation that's slightly out of the scope of your patch,
> >>> but I'll post here on the off chance that you find it relevant.
> >>> Apologies if it's out of place.
> >>>
> >>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
> >>> use the Generic PHY driver?
> >>
> >> They refuse to probe at all with the Generic PHY driver. I have been
> >> caught off guard a few times now when I had a kernel built with
> >> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
> >>
> >>> If the former is the case, do you experience that the PHY driver fails
> >>> to get probed during mdiobus registration if the kernel uses
> >>> fw_devlink=on?
> >>
> >> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
> >> time I do it. It behaves exactly as you say.
> >>
> >>>
> >>> In my case I am writing a new subdriver for realtek-smi, a DSA driver
> >>> which registers an internal MDIO bus analogously to sja1105, which is
> >>> why I'm asking. I noticed a deferred probe of the PHY driver because the
> >>> supplier (ethernet-switch) is not ready - presumably because all of this
> >>> is happening in the probe of the switch driver. See below:
> >>>
> >>> [   83.653213] device_add:3270: device: 'SMI-0': device_add
> >>> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
> >>> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
> >>> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
> >>> [   83.654291] libphy: SMI slave MII: probed
> >>> ...
> >>> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
> >>> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
> >>> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
> >>> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
> >>> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
> >>> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
> >>> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
> >>> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
> >>> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
> >>> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
> >>
> >> So it's a circular dependency? Switch cannot finish probing because it
> >> cannot connect to PHY, which cannot probe because switch has not
> >> finished probing, which....
> >
> > Hi Vladimir/Alvin,
> >
> > If there's a cyclic dependency between two devices, then fw_devlink=on
> > is smart enough to notice that. Once it notices a cycle, it knows that
> > it can't tell which one is the real dependency and which one is the
> > false dependency and so stops enforcing ordering between the devices
> > in the cycle.
> >
> > But fw_devlink doesn't understand all the properties yet. Just most of
> > them and I'm always trying to add more. So when it only understands
> > the property that's causing the false dependency but not the property
> > that causes the real dependency, it can cause issues like this where
> > fw_devlink=on enforces the false dependency and the driver/code
> > enforces the real dependency. These are generally easy to fix -- you
> > just need to teach fw_devlink how to parse more properties.
> >
> > This is just a preliminary analysis since I don't have all the info
> > yet -- so I could be wrong. With that said, I happened to be working
> > on adding fw_devlink support for phy-handle property and I think it
> > should fix your issue with fw_devlink=on. Can you give [1] a shot?
>
> I tried [1] but it did not seem to have any effect.
>
> >
> > If it doesn't fix it, can one of you please point me to an upstream
> > dts (not dtsi) file for a platform in which you see this issue? And
> > ideally also the DT nodes and their drivers that are involved in this
> > cycle? With that info, I should be able to root cause this if the
> > patch above doesn't already fix it.
>
> I'm working with a non-upstream dts - maybe Vladimir is using an
> upstream one? The pattern among the drivers/dts is common between our
> two cases.

Ideally, I can get a fully upstream example where this issue is
happening so that I can look at the actual code that's hitting this
issue and be sure my analysis is right.

>
> But for the sake of this discussion, my dts is pretty much the same as
> what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The
> nodes of interest from that dts file are below, and the driver is in
> drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the
> Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part
> of the mdiobus registration, but that never happens. See my previous
> reply for a debug log.

Your DTS might be similar to this, but the driver code also matters
for me to be sure. Anyway, I took a look at this, but my analysis
below is going to be sketchy because I'm not looking at the actual
code that's reproducing this issue.

Assuming this issue actually happens with the example you pointed to
(I don't know this yet), here's what is happening:

The main problem is that the parent device switch seems to be assuming
it's child/grandchild devices (mdiobus/PHYs) will have probed
successfully as soon as they are added. This assumption is not true
and can be broken for multiple reasons such as:

1. The driver for the child devices (PHYs in this case) could be
loaded as a module after the parent (switch) is probed. So when the
devices are added, the PHYs would not be probed.
2. The child devices could defer probe because one of their suppliers
isn't ready yet. Either because of fw_devlink=on or the framework
itself returning -EPROBE_DEFER.
3. The child devices could be getting probed asynchronously. So the
device_add() would kick off a thread to probe the child devices in a
separate thread.

(2) is what is happening in this case. fw_devlink=on sees that
"switch" implements the "switch_intc" and "switch" hasn't finished
probing yet. So it has no way of knowing that switch_intc is actually
ready. And even if switch_intc was registered as part of switch's
probe() by the time the PHYs are added, switch_intc could get
deregistered if the probe fails at a later point. So until probe()
returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
is ready. Which is good in general because you won't have to
forcefully unbind (if that is even handled correctly in the first
place) the consumers of a device if it fails probe() half way through
registering a few services.

I don't fully understand the networking frameworks, but I think
Vladimir might have a point in his earlier reply [1]. If you can make
the switch driver not assume its child PHYs are ready during the
switch's probe and instead have the switch check if the PHYs are ready
when the switch is "opened" that'd be better.

We can come up with hacks that'll delete the dependency that
fw_devlink=on is trying to enforce, but IMHO the proper fix is to have
parent drivers not assume child devices will be probed as soon as
device_add(child) returns. That's not guaranteed at all.

Btw, I do know why things work when you do the module load/unload
thing you mention in [2]. That has to do with some forced deletion of
dependencies that happens when device_bind_driver() is called when the
Generic PHY driver is used. The reason for why that's done is kind of
unrelated to the issue at hand, but the comment for
device_links_force_bind() should tell you why.

Hope that helps.

[1] - https://lore.kernel.org/netdev/20210817224008.pzdomrjaw5ewmpdg@skbuf/
[2] - https://lore.kernel.org/netdev/0c3e8814-acce-5836-3b1a-6804c21e9bf0@bang-olufsen.dk/

-Saravana

>
> / {
>         switch {
>                 compatible = "realtek,rtl8366rb";
>                 mdc-gpios = <&gpio0 21 GPIO_ACTIVE_HIGH>;
>                 mdio-gpios = <&gpio0 22 GPIO_ACTIVE_HIGH>;
>                 reset-gpios = <&gpio0 14 GPIO_ACTIVE_LOW>;
>                 realtek,disable-leds;
>
>                 switch_intc: interrupt-controller {
>                         /* GPIO 15 provides the interrupt */
>                         interrupt-parent = <&gpio0>;
>                         interrupts = <15 IRQ_TYPE_LEVEL_LOW>;
>                         interrupt-controller;
>                         #address-cells = <0>;
>                         #interrupt-cells = <1>;
>                 };
>
>                 ports {
>                         /* snip */
>                 };
>
>                 mdio {
>                         compatible = "realtek,smi-mdio";
>                         #address-cells = <1>;
>                         #size-cells = <0>;
>
>                         phy0: phy@0 {
>                                 reg = <0>;
>                                 interrupt-parent = <&switch_intc>;
>                                 interrupts = <0>;
>                         };
>                         phy1: phy@1 {
>                                 reg = <1>;
>                                 interrupt-parent = <&switch_intc>;
>                                 interrupts = <1>;
>                         };
>                         phy2: phy@2 {
>                                 reg = <2>;
>                                 interrupt-parent = <&switch_intc>;
>                                 interrupts = <2>;
>                         };
>                         phy3: phy@3 {
>                                 reg = <3>;
>                                 interrupt-parent = <&switch_intc>;
>                                 interrupts = <3>;
>                         };
>                         phy4: phy@4 {
>                                 reg = <4>;
>                                 interrupt-parent = <&switch_intc>;
>                                 interrupts = <12>;
>                         };
>                 };
>         };
> };
>
> Thanks for looking into this. Let me know if you need any other info or
> want something tested.
>
> Kind regards,
> Alvin
>
> >
> >>
> >> So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00'
> >> device should not be added to the deferred list, it should have everything
> >> it needs right now (after all, it works without fw_devlink). No?
> >>
> >> It might be the late hour over here too, but right now I just don't
> >> know. Let me add Saravana to the discussion too, he made an impressive
> >> analysis recently on a PHY probing issue with mdio-mux,
> >
> > Lol, thanks for the kind words.
> >
> >> so the PHY
> >> library probing dependencies should still be fresh in his mind, maybe he
> >> has an idea what's wrong.
> >
> > [1] - https://lore.kernel.org/lkml/20210818021717.3268255-1-saravanak@google.com/T/#u>>
> > Thanks,
> > Saravana
> >

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Vladimir Oltean]

On Wed, Aug 18, 2021 at 08:28:44PM -0700, Saravana Kannan wrote:
> On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> >
> > Hi Saravana,
> >
> > On 8/18/21 4:46 AM, Saravana Kannan wrote:
> > > On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
> > >>
> > >> Hi Alvin,
> > >>
> > >> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
> > >>> I have an observation that's slightly out of the scope of your patch,
> > >>> but I'll post here on the off chance that you find it relevant.
> > >>> Apologies if it's out of place.
> > >>>
> > >>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
> > >>> use the Generic PHY driver?
> > >>
> > >> They refuse to probe at all with the Generic PHY driver. I have been
> > >> caught off guard a few times now when I had a kernel built with
> > >> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
> > >>
> > >>> If the former is the case, do you experience that the PHY driver fails
> > >>> to get probed during mdiobus registration if the kernel uses
> > >>> fw_devlink=on?
> > >>
> > >> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
> > >> time I do it. It behaves exactly as you say.

Sorry for the delay, I wanted to reconfirm what I said (hint, I was wrong).

In my case, whatever I do, I cannot get the driver core enforce a device
link between the ethernet-switch and the PHY.

So I cannot actually see the same issue. What I was seeing was in fact
stupid testing on my part (it was working with the PHY driver as
built-in, it was working, then I made it a module, it broke, I forgot to
switch it back to module, then I thought it's broken while the PHY is
built-in).

> > >>
> > >>>
> > >>> In my case I am writing a new subdriver for realtek-smi, a DSA driver
> > >>> which registers an internal MDIO bus analogously to sja1105, which is
> > >>> why I'm asking. I noticed a deferred probe of the PHY driver because the
> > >>> supplier (ethernet-switch) is not ready - presumably because all of this
> > >>> is happening in the probe of the switch driver. See below:
> > >>>
> > >>> [   83.653213] device_add:3270: device: 'SMI-0': device_add
> > >>> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
> > >>> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
> > >>> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
> > >>> [   83.654291] libphy: SMI slave MII: probed
> > >>> ...
> > >>> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
> > >>> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
> > >>> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
> > >>> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
> > >>> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
> > >>> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
> > >>> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
> > >>> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
> > >>> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
> > >>> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
> > >>
> > >> So it's a circular dependency? Switch cannot finish probing because it
> > >> cannot connect to PHY, which cannot probe because switch has not
> > >> finished probing, which....
> > >
> > > Hi Vladimir/Alvin,
> > >
> > > If there's a cyclic dependency between two devices, then fw_devlink=on
> > > is smart enough to notice that. Once it notices a cycle, it knows that
> > > it can't tell which one is the real dependency and which one is the
> > > false dependency and so stops enforcing ordering between the devices
> > > in the cycle.
> > >
> > > But fw_devlink doesn't understand all the properties yet. Just most of
> > > them and I'm always trying to add more. So when it only understands
> > > the property that's causing the false dependency but not the property
> > > that causes the real dependency, it can cause issues like this where
> > > fw_devlink=on enforces the false dependency and the driver/code
> > > enforces the real dependency. These are generally easy to fix -- you
> > > just need to teach fw_devlink how to parse more properties.
> > >
> > > This is just a preliminary analysis since I don't have all the info
> > > yet -- so I could be wrong. With that said, I happened to be working
> > > on adding fw_devlink support for phy-handle property and I think it
> > > should fix your issue with fw_devlink=on. Can you give [1] a shot?
> >
> > I tried [1] but it did not seem to have any effect.
> >

I applied the phy-handle patch, and here are my boot logs.

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.14.0-rc5+ (tigrisor@skbuf) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #440 SMP Thu Aug 19 13:48:11 EEST 2021
[    0.000000] Machine model: NXP Layerscape LX2160ABLUEBOX3
[    0.000000] fw_devlink_setup: fw_devlink_flags = 0x120
[    0.000000] efi: UEFI not found.
[    0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000277fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x277c1c49c0-0x277c1c6fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000277fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000fbdfffff]
[    0.000000]   node   0: [mem 0x0000002080000000-0x000000277fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000277fffffff]
[    0.000000] On node 0, zone Normal: 16896 pages in unavailable ranges
[    0.000000] cma: Reserved 32 MiB at 0x00000000f9c00000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 482 pages/cpu s1933592 r8192 d32488 u1974272
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 7724544
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: fw_devlink=on root=/dev/mmcblk0p2 rw console=ttyAMA0,115200
[    0.000000] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.000000] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000f5c00000-0x00000000f9c00000] (64MB)
[    0.000000] Memory: 30595480K/31389696K available (23040K kernel code, 4562K rwdata, 10700K rodata, 9536K init, 11765K bss, 761448K reserved, 32768K cma-reserved)
[    0.000000] trace event string verifier disabled
[    0.000000] Running RCU self tests
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU lockdep checking is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=16.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=16
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006200000
[    0.000000] ITS [mem 0x06020000-0x0603ffff]
[    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @2080100000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: Using preallocated redistributor tables
[    0.000000] GICv3: using LPI property table @0x000000277fe00000
[    0.000000] GICv3: CPU0: using reserved LPI pending table @0x000000277fe10000
[    0.000000] random: get_random_bytes called from start_kernel+0x544/0x748 with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c409fb33, max_idle_ns: 440795203156 ns
[    0.000000] sched_clock: 56 bits at 25MHz, resolution 39ns, wraps every 4398046511103ns
[    0.001138] Console: colour dummy device 80x25
[    0.001160] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.001167] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.001174] ... MAX_LOCK_DEPTH:          48
[    0.001181] ... MAX_LOCKDEP_KEYS:        8192
[    0.001187] ... CLASSHASH_SIZE:          4096
[    0.001194] ... MAX_LOCKDEP_ENTRIES:     32768
[    0.001201] ... MAX_LOCKDEP_CHAINS:      65536
[    0.001207] ... CHAINHASH_SIZE:          32768
[    0.001214]  memory used by lock dependency info: 6877 kB
[    0.001220]  memory used for stack traces: 4224 kB
[    0.001227]  per task-struct memory footprint: 2688 bytes
[    0.001308] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.001322] pid_max: default: 32768 minimum: 301
[    0.002396] LSM: Security Framework initializing
[    0.002583] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.002672] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.005656] Running RCU-tasks wait API self tests
[    0.005972] rcu: Hierarchical SRCU implementation.
[    0.006409] dyndbg: Ignore empty _ddebug table in a CONFIG_DYNAMIC_DEBUG_CORE build
[    0.006490] Platform MSI: gic-its@6020000 domain created
[    0.006591] PCI/MSI: /interrupt-controller@6000000/gic-its@6020000 domain created
[    0.006680] fsl-mc MSI: gic-its@6020000 domain created
[    0.007278] EFI services will not be available.
[    0.008971] smp: Bringing up secondary CPUs ...
[    0.010766] Detected PIPT I-cache on CPU1
[    0.010791] GICv3: CPU1: found redistributor 1 region 0:0x0000000006220000
[    0.010818] GICv3: CPU1: using reserved LPI pending table @0x000000277fe20000
[    0.010951] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.013072] Detected PIPT I-cache on CPU2
[    0.013095] GICv3: CPU2: found redistributor 100 region 0:0x0000000006240000
[    0.013120] GICv3: CPU2: using reserved LPI pending table @0x000000277fe30000
[    0.013258] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[    0.015350] Detected PIPT I-cache on CPU3
[    0.015367] GICv3: CPU3: found redistributor 101 region 0:0x0000000006260000
[    0.015388] GICv3: CPU3: using reserved LPI pending table @0x000000277fe40000
[    0.015511] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[    0.017419] Detected PIPT I-cache on CPU4
[    0.017444] GICv3: CPU4: found redistributor 200 region 0:0x0000000006280000
[    0.017467] GICv3: CPU4: using reserved LPI pending table @0x000000277fe50000
[    0.017600] CPU4: Booted secondary processor 0x0000000200 [0x410fd083]
[    0.019496] Detected PIPT I-cache on CPU5
[    0.019516] GICv3: CPU5: found redistributor 201 region 0:0x00000000062a0000
[    0.019536] GICv3: CPU5: using reserved LPI pending table @0x000000277fe60000
[    0.019657] CPU5: Booted secondary processor 0x0000000201 [0x410fd083]
[    0.021614] Detected PIPT I-cache on CPU6
[    0.021643] GICv3: CPU6: found redistributor 300 region 0:0x00000000062c0000
[    0.021658] GICv3: CPU6: using reserved LPI pending table @0x000000277fe70000
[    0.021791] CPU6: Booted secondary processor 0x0000000300 [0x410fd083]
[    0.022061] Callback from call_rcu_tasks_trace() invoked.
[    0.023813] Detected PIPT I-cache on CPU7
[    0.023838] GICv3: CPU7: found redistributor 301 region 0:0x00000000062e0000
[    0.023858] GICv3: CPU7: using reserved LPI pending table @0x000000277fe80000
[    0.023978] CPU7: Booted secondary processor 0x0000000301 [0x410fd083]
[    0.025863] Detected PIPT I-cache on CPU8
[    0.025886] GICv3: CPU8: found redistributor 400 region 0:0x0000000006300000
[    0.025902] GICv3: CPU8: using reserved LPI pending table @0x000000277fe90000
[    0.026032] CPU8: Booted secondary processor 0x0000000400 [0x410fd083]
[    0.027921] Detected PIPT I-cache on CPU9
[    0.027943] GICv3: CPU9: found redistributor 401 region 0:0x0000000006320000
[    0.027956] GICv3: CPU9: using reserved LPI pending table @0x000000277fea0000
[    0.028078] CPU9: Booted secondary processor 0x0000000401 [0x410fd083]
[    0.030006] Detected PIPT I-cache on CPU10
[    0.030033] GICv3: CPU10: found redistributor 500 region 0:0x0000000006340000
[    0.030054] GICv3: CPU10: using reserved LPI pending table @0x000000277feb0000
[    0.030189] CPU10: Booted secondary processor 0x0000000500 [0x410fd083]
[    0.032076] Detected PIPT I-cache on CPU11
[    0.032099] GICv3: CPU11: found redistributor 501 region 0:0x0000000006360000
[    0.032113] GICv3: CPU11: using reserved LPI pending table @0x000000277fec0000
[    0.032235] CPU11: Booted secondary processor 0x0000000501 [0x410fd083]
[    0.034091] Detected PIPT I-cache on CPU12
[    0.034118] GICv3: CPU12: found redistributor 600 region 0:0x0000000006380000
[    0.034135] GICv3: CPU12: using reserved LPI pending table @0x000000277fed0000
[    0.034271] CPU12: Booted secondary processor 0x0000000600 [0x410fd083]
[    0.036130] Detected PIPT I-cache on CPU13
[    0.036151] GICv3: CPU13: found redistributor 601 region 0:0x00000000063a0000
[    0.036165] GICv3: CPU13: using reserved LPI pending table @0x000000277fee0000
[    0.036287] CPU13: Booted secondary processor 0x0000000601 [0x410fd083]
[    0.038161] Detected PIPT I-cache on CPU14
[    0.038189] GICv3: CPU14: found redistributor 700 region 0:0x00000000063c0000
[    0.038205] GICv3: CPU14: using reserved LPI pending table @0x000000277fef0000
[    0.038336] CPU14: Booted secondary processor 0x0000000700 [0x410fd083]
[    0.040221] Detected PIPT I-cache on CPU15
[    0.040242] GICv3: CPU15: found redistributor 701 region 0:0x00000000063e0000
[    0.040256] GICv3: CPU15: using reserved LPI pending table @0x000000277ff00000
[    0.040377] CPU15: Booted secondary processor 0x0000000701 [0x410fd083]
[    0.040608] smp: Brought up 1 node, 16 CPUs
[    0.040729] SMP: Total of 16 processors activated.
[    0.040738] CPU features: detected: 32-bit EL0 Support
[    0.040747] CPU features: detected: 32-bit EL1 Support
[    0.040756] CPU features: detected: CRC32 instructions
[    0.067039] CPU: All CPU(s) started at EL2
[    0.067176] alternatives: patching kernel code
[    0.069867] devtmpfs: initialized
[    0.099164] KASLR enabled
[    0.100410] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.100433] futex hash table entries: 4096 (order: 7, 524288 bytes, linear)
[    0.101547] pinctrl core: initialized pinctrl subsystem
[    0.104113] DMI not present or invalid.
[    0.104964] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.108803] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations
[    0.109609] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.110392] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.110509] audit: initializing netlink subsys (disabled)
[    0.110774] audit: type=2000 audit(0.108:1): state=initialized audit_enabled=0 res=1
[    0.113114] thermal_sys: Registered thermal governor 'step_wise'
[    0.115915] cpuidle: using governor menu
[    0.116336] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.116889] ASID allocator initialised with 32768 entries
[    0.119665] Serial: AMBA PL011 UART driver
[    0.131350] Machine: NXP Layerscape LX2160ABLUEBOX3
[    0.131364] SoC family: QorIQ LX2160A
[    0.131372] SoC ID: svr:0x87360020, Revision: 2.0
[    0.139246] 21c0000.serial: ttyAMA0 at MMIO 0x21c0000 (irq = 26, base_baud = 0) is a SBSA
[    1.220238] printk: console [ttyAMA0] enabled
[    1.226490] sbsa-uart 21d0000.serial: aliased and non-aliased serial devices found in device tree. Serial port enumeration may be unpredictable.
[    1.239476] 21d0000.serial: ttyAMA1 at MMIO 0x21d0000 (irq = 27, base_baud = 0) is a SBSA
[    1.251594] platform 2000000.i2c: Linked as a consumer to 2320000.gpio
[    1.283271] fsl_mc_bus 80c000000.fsl-mc: MC firmware version: 10.24.1
[    1.292523] fsl_mc_dprc dprc.1: DMA mask not set
[    1.297742] platform 2000000.i2c: Linked as a sync state only consumer to regulator-sb3v3
[    1.343358] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    1.350096] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    1.356817] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.363531] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    1.374376] cryptd: max_cpu_qlen set to 1000
[    1.450431] raid6: neonx8   gen()  5557 MB/s
[    1.522595] raid6: neonx8   xor()  3634 MB/s
[    1.594761] raid6: neonx4   gen()  5519 MB/s
[    1.666929] raid6: neonx4   xor()  3681 MB/s
[    1.739094] raid6: neonx2   gen()  4919 MB/s
[    1.811261] raid6: neonx2   xor()  3562 MB/s
[    1.883430] raid6: neonx1   gen()  3912 MB/s
[    1.955594] raid6: neonx1   xor()  2937 MB/s
[    2.027759] raid6: int64x8  gen()  3306 MB/s
[    2.099922] raid6: int64x8  xor()  1955 MB/s
[    2.172090] raid6: int64x4  gen()  3882 MB/s
[    2.244255] raid6: int64x4  xor()  2124 MB/s
[    2.316423] raid6: int64x2  gen()  3454 MB/s
[    2.388593] raid6: int64x2  xor()  1853 MB/s
[    2.460766] raid6: int64x1  gen()  2696 MB/s
[    2.532929] raid6: int64x1  xor()  1401 MB/s
[    2.537204] raid6: using algorithm neonx8 gen() 5557 MB/s
[    2.542607] raid6: .... xor() 3634 MB/s, rmw enabled
[    2.547575] raid6: using neon recovery algorithm
[    2.552869] ACPI: Interpreter disabled.
[    2.561168] iommu: Default domain type: Passthrough
[    2.566610] vgaarb: loaded
[    2.570380] SCSI subsystem initialized
[    2.575023] usbcore: registered new interface driver usbfs
[    2.580638] usbcore: registered new interface driver hub
[    2.586088] usbcore: registered new device driver usb
[    2.592112] imx-i2c 2000000.i2c: can't get pinctrl, bus recovery not supported
[    2.600269] i2c 0-0077: Linked as a sync state only consumer to regulator-sb3v3
[    2.607887] i2c i2c-0: IMX I2C adapter registered
[    2.612640] imx-i2c 2000000.i2c: Dropping the link to regulator-sb3v3
[    2.619718] imx-i2c 2050000.i2c: can't get pinctrl, bus recovery not supported
[    2.627406] i2c i2c-1: IMX I2C adapter registered
[    2.632830] pps_core: LinuxPPS API ver. 1 registered
[    2.637809] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    2.646975] PTP clock support registered
[    2.651131] EDAC MC: Ver: 3.0.0
[    2.657363] FPGA manager framework
[    2.661061] Advanced Linux Sound Architecture Driver Initialized.
[    2.668553] Bluetooth: Core ver 2.22
[    2.672194] NET: Registered PF_BLUETOOTH protocol family
[    2.677514] Bluetooth: HCI device and connection manager initialized
[    2.683893] Bluetooth: HCI socket layer initialized
[    2.688785] Bluetooth: L2CAP socket layer initialized
[    2.693888] Bluetooth: SCO socket layer initialized
[    2.699792] clocksource: Switched to clocksource arch_sys_counter
[    2.893095] VFS: Disk quotas dquot_6.6.0
[    2.897133] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.904658] pnp: PnP ACPI: disabled
[    2.925319] NET: Registered PF_INET protocol family
[    2.930703] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    2.941965] tcp_listen_portaddr_hash hash table entries: 16384 (order: 8, 1441792 bytes, linear)
[    2.952166] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    2.961285] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes, vmalloc)
[    2.974112] TCP: Hash tables configured (established 262144 bind 65536)
[    2.981465] UDP hash table entries: 16384 (order: 9, 3145728 bytes, linear)
[    2.991484] UDP-Lite hash table entries: 16384 (order: 9, 3145728 bytes, linear)
[    3.002151] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    3.009011] RPC: Registered named UNIX socket transport module.
[    3.014964] RPC: Registered udp transport module.
[    3.019678] RPC: Registered tcp transport module.
[    3.024396] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.030852] PCI: CLS 0 bytes, default 64
[    3.182026] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    3.190536] kvm [1]: IPA Size Limit: 44 bits
[    3.202176] kvm [1]: vgic-v2@c0e0000
[    3.205790] kvm [1]: GIC system register CPU interface enabled
[    3.212408] kvm [1]: vgic interrupt IRQ9
[    3.217322] kvm [1]: Hyp mode initialized successfully
[    3.233022] Initialise system trusted keyrings
[    3.237780] workingset: timestamp_bits=42 max_order=23 bucket_order=0
[    3.246308] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    3.252912] NFS: Registering the id_resolver key type
[    3.258046] Key type id_resolver registered
[    3.262255] Key type id_legacy registered
[    3.266338] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    3.273063] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    3.280763] fuse: init (API version 7.34)
[    3.285101] 9p: Installing v9fs 9p2000 file system support
[    3.309033] xor: measuring software checksum speed
[    3.314861]    8regs           :  9817 MB/sec
[    3.320228]    32regs          : 10028 MB/sec
[    3.327078]    arm64_neon      :  3973 MB/sec
[    3.331443] xor: using function: 32regs (10028 MB/sec)
[    3.336594] Key type asymmetric registered
[    3.340719] Asymmetric key parser 'x509' registered
[    3.345674] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    3.353367] io scheduler mq-deadline registered
[    3.357957] io scheduler kyber registered
[    3.382684] layerscape-pcie 3400000.pcie: host bridge /soc/pcie@3400000 ranges:
[    3.390064] layerscape-pcie 3400000.pcie:      MEM 0x8040000000..0x807fffffff -> 0x0040000000
[    3.398648] layerscape-pcie 3400000.pcie: iATU unroll: disabled
[    3.404579] layerscape-pcie 3400000.pcie: Detected iATU regions: 8 outbound, 8 inbound
[    3.512561] layerscape-pcie 3400000.pcie: Link up
[    3.517626] layerscape-pcie 3400000.pcie: PCI host bridge to bus 0000:00
[    3.524348] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.529851] pci_bus 0000:00: root bus resource [mem 0x8040000000-0x807fffffff] (bus address [0x40000000-0x7fffffff])
[    3.540459] pci 0000:00:00.0: [1957:8d80] type 01 class 0x060400
[    3.546650] pci 0000:00:00.0: supports D1 D2
[    3.550934] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.558540] pci 0000:01:00.0: [1344:6001] type 00 class 0x010802
[    3.564674] pci 0000:01:00.0: reg 0x10: [mem 0x8040000000-0x8040003fff 64bit]
[    3.572659] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    3.588748] pci 0000:00:00.0: BAR 14: assigned [mem 0x8040000000-0x80400fffff]
[    3.595997] pci 0000:01:00.0: BAR 0: assigned [mem 0x8040000000-0x8040003fff 64bit]
[    3.603714] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.608958] pci 0000:00:00.0:   bridge window [mem 0x8040000000-0x80400fffff]
[    3.616591] layerscape-pcie 3500000.pcie: host bridge /soc/pcie@3500000 ranges:
[    3.623960] layerscape-pcie 3500000.pcie:      MEM 0x8840000000..0x887fffffff -> 0x0040000000
[    3.632548] layerscape-pcie 3500000.pcie: iATU unroll: disabled
[    3.638481] layerscape-pcie 3500000.pcie: Detected iATU regions: 8 outbound, 8 inbound
[    3.746460] layerscape-pcie 3500000.pcie: Link up
[    3.751490] layerscape-pcie 3500000.pcie: PCI host bridge to bus 0001:00
[    3.758210] pci_bus 0001:00: root bus resource [bus 00-ff]
[    3.763713] pci_bus 0001:00: root bus resource [mem 0x8840000000-0x887fffffff] (bus address [0x40000000-0x7fffffff])
[    3.774286] pci 0001:00:00.0: [1957:8d80] type 01 class 0x060400
[    3.780456] pci 0001:00:00.0: supports D1 D2
[    3.784741] pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot
[    3.792301] pci 0001:01:00.0: [1957:4002] type 00 class 0x000000
[    3.798427] pci 0001:01:00.0: reg 0x10: [mem 0x8840000000-0x88400fffff 64bit pref]
[    3.806053] pci 0001:01:00.0: reg 0x18: [mem 0x8840100000-0x88401fffff]
[    3.812723] pci 0001:01:00.0: reg 0x1c: [io  0x1000-0x10ff]
[    3.818351] pci 0001:01:00.0: reg 0x20: [mem 0x8840200000-0x88402fffff]
[    3.825023] pci 0001:01:00.0: reg 0x24: [mem 0x8840300000-0x88403fffff]
[    3.831693] pci 0001:01:00.0: reg 0x30: [mem 0x8840400000-0x884040ffff pref]
[    3.839142] pci 0001:01:00.0: supports D1
[    3.843165] pci 0001:01:00.0: PME# supported from D0 D1 D3hot D3cold
[    3.849737] pci 0001:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
[    3.865871] pci 0001:00:00.0: BAR 14: assigned [mem 0x8840000000-0x88403fffff]
[    3.873111] pci 0001:00:00.0: BAR 15: assigned [mem 0x8840400000-0x88404fffff 64bit pref]
[    3.881302] pci 0001:00:00.0: BAR 13: no space for [io  size 0x1000]
[    3.887668] pci 0001:00:00.0: BAR 13: failed to assign [io  size 0x1000]
[    3.894384] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    3.899626] pci 0001:00:00.0:   bridge window [mem 0x8840000000-0x88403fffff]
[    3.906778] pci 0001:00:00.0:   bridge window [mem 0x8840400000-0x88404fffff 64bit pref]
[    3.915327] layerscape-pcie 3600000.pcie: host bridge /soc/pcie@3600000 ranges:
[    3.922695] layerscape-pcie 3600000.pcie:      MEM 0x9040000000..0x907fffffff -> 0x0040000000
[    3.931658] layerscape-pcie 3600000.pcie: iATU unroll: disabled
[    3.937587] layerscape-pcie 3600000.pcie: Detected iATU regions: 256 outbound, 24 inbound
[    4.946287] layerscape-pcie 3600000.pcie: Phy link never came up
[    4.952621] layerscape-pcie 3600000.pcie: PCI host bridge to bus 0002:00
[    4.959342] pci_bus 0002:00: root bus resource [bus 00-ff]
[    4.964846] pci_bus 0002:00: root bus resource [mem 0x9040000000-0x907fffffff] (bus address [0x40000000-0x7fffffff])
[    4.975420] pci 0002:00:00.0: [1957:8d80] type 01 class 0x060400
[    4.981603] pci 0002:00:00.0: supports D1 D2
[    4.985886] pci 0002:00:00.0: PME# supported from D0 D1 D2 D3hot
[    4.993941] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    4.999605] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges:
[    5.006973] layerscape-pcie 3800000.pcie:      MEM 0xa040000000..0xa07fffffff -> 0x0040000000
[    5.016091] layerscape-pcie 3800000.pcie: iATU unroll: disabled
[    5.022023] layerscape-pcie 3800000.pcie: Detected iATU regions: 256 outbound, 24 inbound
[    6.030799] layerscape-pcie 3800000.pcie: Phy link never came up
[    6.037113] layerscape-pcie 3800000.pcie: PCI host bridge to bus 0003:00
[    6.043833] pci_bus 0003:00: root bus resource [bus 00-ff]
[    6.049336] pci_bus 0003:00: root bus resource [mem 0xa040000000-0xa07fffffff] (bus address [0x40000000-0x7fffffff])
[    6.059912] pci 0003:00:00.0: [1957:8d80] type 01 class 0x060400
[    6.066109] pci 0003:00:00.0: supports D1 D2
[    6.070392] pci 0003:00:00.0: PME# supported from D0 D1 D2 D3hot
[    6.078457] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[    6.085845] EINJ: ACPI disabled.
[    6.121151] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    6.133080] SuperH (H)SCI(F) driver initialized
[    6.138419] msm_serial: driver initialized
[    6.147208] arm-smmu 5000000.iommu: probing hardware configuration...
[    6.153677] arm-smmu 5000000.iommu: SMMUv2 with:
[    6.158310] arm-smmu 5000000.iommu:  stage 1 translation
[    6.163635] arm-smmu 5000000.iommu:  stage 2 translation
[    6.168959] arm-smmu 5000000.iommu:  nested translation
[    6.174196] arm-smmu 5000000.iommu:  coherent table walk
[    6.179520] arm-smmu 5000000.iommu:  (IDR0.CTTW overridden by FW configuration)
[    6.186845] arm-smmu 5000000.iommu:  stream matching with 128 register groups
[    6.193999] arm-smmu 5000000.iommu:  64 context banks (0 stage-2 only)
[    6.200547] arm-smmu 5000000.iommu:  Supported page sizes: 0x61311000
[    6.207000] arm-smmu 5000000.iommu:  Stage-1: 48-bit VA -> 48-bit IPA
[    6.213453] arm-smmu 5000000.iommu:  Stage-2: 48-bit IPA -> 48-bit PA
[    6.251758] brd: module loaded
[    6.298301] loop: module loaded
[    6.303363] megasas: 07.717.02.00-rc1
[    6.315088] spi-nor spi0.0: mt35xu512aba (65536 Kbytes)
[    6.329741] spi-nor spi0.1: mt35xu512aba (65536 Kbytes)
[    6.351635] libphy: Fixed MDIO Bus: probed
[    6.361042] tun: Universal TUN/TAP device driver, 1.6
[    6.366645] CAN device driver interface
[    6.376344] thunder_xcv, ver 1.0
[    6.379680] thunder_bgx, ver 1.0
[    6.383029] nicpf, ver 1.0
[    6.387457] libphy: Freescale XGMAC MDIO Bus: probed
[    6.397953] mv88x3310 0x0000000008b96000:00: PHY failed to boot firmware, status=ffff
[    6.411222] mv88x3310 0x0000000008b96000:08: PHY failed to boot firmware, status=ffff
[    6.420253] i2c 0-0077: Linked as a sync state only consumer to 0x0000000008b96000:01
[    6.429250] i2c 0-0077: Linked as a sync state only consumer to 0x0000000008b96000:02
[    6.438742] libphy: Freescale XGMAC MDIO Bus: probed
[    6.448826] mv88x3310 0x0000000008b97000:00: PHY failed to boot firmware, status=ffff
[    6.462097] mv88x3310 0x0000000008b97000:08: PHY failed to boot firmware, status=ffff
[    6.471559] libphy: Freescale XGMAC MDIO Bus: probed
[    6.479068] libphy: Freescale XGMAC MDIO Bus: probed
[    6.486585] libphy: Freescale XGMAC MDIO Bus: probed
[    6.494898] hclge is initializing
[    6.498345] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[    6.505579] hns3: Copyright (c) 2017 Huawei Corporation.
[    6.511048] e1000: Intel(R) PRO/1000 Network Driver
[    6.515934] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    6.521805] e1000e: Intel(R) PRO/1000 Network Driver
[    6.526777] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    6.532823] igb: Intel(R) Gigabit Ethernet Network Driver
[    6.538231] igb: Copyright (c) 2007-2014 Intel Corporation.
[    6.543895] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    6.550169] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    6.556694] sky2: driver version 1.30
[    6.563181] tlan: ThunderLAN driver v1.17
[    6.567312] tlan: 0 devices installed, PCI: 0  EISA: 0
[    6.573916] VFIO - User Level meta-driver version: 0.3
[    6.580381] dwc3 3100000.usb: Linked as a consumer to 5000000.iommu
[    6.586783] dwc3 3100000.usb: Adding to iommu group 0
[    6.595924] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    6.602466] ehci-pci: EHCI PCI platform driver
[    6.606996] ehci-platform: EHCI generic platform driver
[    6.612405] ehci-orion: EHCI orion driver
[    6.616588] ehci-exynos: EHCI Exynos driver
[    6.620947] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    6.627169] ohci-pci: OHCI PCI platform driver
[    6.631702] ohci-platform: OHCI generic platform driver
[    6.637119] ohci-exynos: OHCI Exynos driver
[    6.641563] usbcore: registered new interface driver cdc_acm
[    6.647231] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[    6.655342] usbcore: registered new interface driver usb-storage
[    6.661514] usbcore: registered new interface driver usbserial_generic
[    6.668170] usbserial: USB Serial support registered for generic
[    6.674255] usbcore: registered new interface driver aircable
[    6.680053] usbserial: USB Serial support registered for aircable
[    6.686210] usbcore: registered new interface driver ark3116
[    6.691920] usbserial: USB Serial support registered for ark3116
[    6.697993] usbcore: registered new interface driver belkin_sa
[    6.703877] usbserial: USB Serial support registered for Belkin / Peracom / GoHubs USB Serial Adapter
[    6.713164] usbcore: registered new interface driver ch341
[    6.718700] usbserial: USB Serial support registered for ch341-uart
[    6.725030] usbcore: registered new interface driver cp210x
[    6.730653] usbserial: USB Serial support registered for cp210x
[    6.736636] usbcore: registered new interface driver cyberjack
[    6.742520] usbserial: USB Serial support registered for Reiner SCT Cyberjack USB card reader
[    6.751110] usbcore: registered new interface driver cypress_m8
[    6.757088] usbserial: USB Serial support registered for DeLorme Earthmate USB
[    6.764362] usbserial: USB Serial support registered for HID->COM RS232 Adapter
[    6.771720] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter
[    6.779091] usbcore: registered new interface driver digi_acceleport
[    6.785496] usbserial: USB Serial support registered for Digi 2 port USB adapter
[    6.792942] usbserial: USB Serial support registered for Digi 4 port USB adapter
[    6.800401] usbcore: registered new interface driver io_edgeport
[    6.806460] usbserial: USB Serial support registered for Edgeport 2 port adapter
[    6.813907] usbserial: USB Serial support registered for Edgeport 4 port adapter
[    6.821354] usbserial: USB Serial support registered for Edgeport 8 port adapter
[    6.828800] usbserial: USB Serial support registered for EPiC device
[    6.835215] usbcore: registered new interface driver io_ti
[    6.840753] usbserial: USB Serial support registered for Edgeport TI 1 port adapter
[    6.848461] usbserial: USB Serial support registered for Edgeport TI 2 port adapter
[    6.856184] usbcore: registered new interface driver empeg
[    6.861725] usbserial: USB Serial support registered for empeg
[    6.867632] usbcore: registered new interface driver f81534a_ctrl
[    6.873786] usbcore: registered new interface driver f81232
[    6.879409] usbserial: USB Serial support registered for f81232
[    6.885380] usbserial: USB Serial support registered for f81534a
[    6.891452] usbcore: registered new interface driver f81534
[    6.897075] usbserial: USB Serial support registered for Fintek F81532/F81534
[    6.904274] usbcore: registered new interface driver ftdi_sio
[    6.910069] usbserial: USB Serial support registered for FTDI USB Serial Device
[    6.917440] usbcore: registered new interface driver garmin_gps
[    6.923411] usbserial: USB Serial support registered for Garmin GPS usb/tty
[    6.930434] usbcore: registered new interface driver ipaq
[    6.935888] usbserial: USB Serial support registered for PocketPC PDA
[    6.942390] usbcore: registered new interface driver ipw
[    6.947751] usbserial: USB Serial support registered for IPWireless converter
[    6.954950] usbcore: registered new interface driver ir_usb
[    6.960575] usbserial: USB Serial support registered for IR Dongle
[    6.966820] usbcore: registered new interface driver iuu_phoenix
[    6.972880] usbserial: USB Serial support registered for iuu_phoenix
[    6.979295] usbcore: registered new interface driver keyspan
[    6.985005] usbserial: USB Serial support registered for Keyspan - (without firmware)
[    6.992886] usbserial: USB Serial support registered for Keyspan 1 port adapter
[    7.000246] usbserial: USB Serial support registered for Keyspan 2 port adapter
[    7.007605] usbserial: USB Serial support registered for Keyspan 4 port adapter
[    7.014976] usbcore: registered new interface driver keyspan_pda
[    7.021036] usbserial: USB Serial support registered for Keyspan PDA
[    7.027440] usbserial: USB Serial support registered for Keyspan PDA - (prerenumeration)
[    7.035593] usbcore: registered new interface driver kl5kusb105
[    7.041564] usbserial: USB Serial support registered for KL5KUSB105D / PalmConnect
[    7.049196] usbcore: registered new interface driver kobil_sct
[    7.055079] usbserial: USB Serial support registered for KOBIL USB smart card terminal
[    7.063057] usbcore: registered new interface driver mct_u232
[    7.068858] usbserial: USB Serial support registered for MCT U232
[    7.075013] usbcore: registered new interface driver metro_usb
[    7.080897] usbserial: USB Serial support registered for Metrologic USB to Serial
[    7.088444] usbcore: registered new interface driver mos7720
[    7.094154] usbserial: USB Serial support registered for Moschip 2 port adapter
[    7.101527] usbcore: registered new interface driver mos7840
[    7.107238] usbserial: USB Serial support registered for Moschip 7840/7820 USB Serial Driver
[    7.115740] usbcore: registered new interface driver mxuport
[    7.121450] usbserial: USB Serial support registered for MOXA UPort
[    7.127796] usbcore: registered new interface driver navman
[    7.133419] usbserial: USB Serial support registered for navman
[    7.139402] usbcore: registered new interface driver omninet
[    7.145122] usbserial: USB Serial support registered for ZyXEL - omni.net usb
[    7.152323] usbcore: registered new interface driver opticon
[    7.158033] usbserial: USB Serial support registered for opticon
[    7.164101] usbcore: registered new interface driver option
[    7.169727] usbserial: USB Serial support registered for GSM modem (1-port)
[    7.176751] usbcore: registered new interface driver oti6858
[    7.182461] usbserial: USB Serial support registered for oti6858
[    7.188531] usbcore: registered new interface driver pl2303
[    7.194172] usbserial: USB Serial support registered for pl2303
[    7.200156] usbcore: registered new interface driver qcaux
[    7.205694] usbserial: USB Serial support registered for qcaux
[    7.211589] usbcore: registered new interface driver qcserial
[    7.217386] usbserial: USB Serial support registered for Qualcomm USB modem
[    7.224411] usbcore: registered new interface driver quatech2
[    7.230210] usbserial: USB Serial support registered for Quatech 2nd gen USB to Serial Driver
[    7.238796] usbcore: registered new interface driver safe_serial
[    7.244854] usbserial: USB Serial support registered for safe_serial
[    7.251268] usbcore: registered new interface driver sierra
[    7.256891] usbserial: USB Serial support registered for Sierra USB modem
[    7.263742] usbcore: registered new interface driver usb_serial_simple
[    7.270323] usbserial: USB Serial support registered for carelink
[    7.276471] usbserial: USB Serial support registered for zio
[    7.282181] usbserial: USB Serial support registered for funsoft
[    7.288239] usbserial: USB Serial support registered for flashloader
[    7.294644] usbserial: USB Serial support registered for google
[    7.300626] usbserial: USB Serial support registered for libtransistor
[    7.307204] usbserial: USB Serial support registered for vivopay
[    7.313264] usbserial: USB Serial support registered for moto_modem
[    7.319583] usbserial: USB Serial support registered for motorola_tetra
[    7.326251] usbserial: USB Serial support registered for novatel_gps
[    7.332656] usbserial: USB Serial support registered for hp4x
[    7.338454] usbserial: USB Serial support registered for suunto
[    7.344424] usbserial: USB Serial support registered for siemens_mpi
[    7.350842] usbcore: registered new interface driver spcp8x5
[    7.356552] usbserial: USB Serial support registered for SPCP8x5
[    7.362620] usbcore: registered new interface driver ssu100
[    7.368248] usbserial: USB Serial support registered for Quatech SSU-100 USB to Serial Driver
[    7.376836] usbcore: registered new interface driver symbolserial
[    7.382980] usbserial: USB Serial support registered for symbol
[    7.388963] usbcore: registered new interface driver ti_usb_3410_5052
[    7.395454] usbserial: USB Serial support registered for TI USB 3410 1 port adapter
[    7.403162] usbserial: USB Serial support registered for TI USB 5052 2 port adapter
[    7.410883] usbcore: registered new interface driver upd78f0730
[    7.416854] usbserial: USB Serial support registered for upd78f0730
[    7.423183] usbcore: registered new interface driver visor
[    7.428723] usbserial: USB Serial support registered for Handspring Visor / Palm OS
[    7.436431] usbserial: USB Serial support registered for Sony Clie 5.0
[    7.443009] usbserial: USB Serial support registered for Sony Clie 3.5
[    7.449600] usbcore: registered new interface driver wishbone_serial
[    7.456007] usbserial: USB Serial support registered for wishbone_serial
[    7.462771] usbcore: registered new interface driver whiteheat
[    7.468655] usbserial: USB Serial support registered for Connect Tech - WhiteHEAT - (prerenumeration)
[    7.477926] usbserial: USB Serial support registered for Connect Tech - WhiteHEAT
[    7.485471] usbcore: registered new interface driver xsens_mt
[    7.491268] usbserial: USB Serial support registered for xsens_mt
[    7.502466] i2c /dev entries driver
[    7.510828] i2c i2c-0: Added multiplexed i2c bus 2
[    7.516281] i2c i2c-0: Added multiplexed i2c bus 3
[    7.522021] i2c i2c-0: Added multiplexed i2c bus 4
[    7.527794] i2c 5-004c: Linked as a consumer to regulator-sb3v3
[    7.534193] i2c 5-0048: Linked as a consumer to regulator-sb3v3
[    7.540291] i2c i2c-0: Added multiplexed i2c bus 5
[    7.546497] genirq: Setting trigger mode 8 for irq 55 failed (gic_set_type+0x0/0x180)
[    7.554464]
[    7.554465] =============================
[    7.554466] [ BUG: Invalid wait context ]
[    7.554467] 5.14.0-rc5+ #440 Not tainted
[    7.554469] -----------------------------
[    7.554469] swapper/0/1 is trying to lock:
[    7.554470] ffff309701582898 (&port_lock_key){....}-{3:3}, at: pl011_console_write+0x134/0x210
[    7.554475] other info that might help us debug this:
[    7.554476] context-{5:5}
[    7.554477] 6 locks held by swapper/0/1:
[    7.554478]  #0: ffff3097018861b0 (&dev->mutex){....}-{4:4}, at: __device_driver_lock+0x34/0x70
[    7.554483]  #1: ffff3097010da1b0 (&dev->mutex){....}-{4:4}, at: __device_attach+0x40/0x148
[    7.554487]  #2: ffff3097015bb688 (&desc->request_mutex){+.+.}-{4:4}, at: __setup_irq+0xb8/0x798
[    7.554491]  #3: ffff3097015bb4f8 (&irq_desc_lock_class){-...}-{2:2}, at: __setup_irq+0xdc/0x798
[    7.554496]  #4: ffffd5d447719200 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x154/0x328
[    7.554500]  #5: ffffd5d447719388 (console_owner){....}-{0:0}, at: console_unlock+0x210/0x658
[    7.554504] stack backtrace:
[    7.554505] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 5.14.0-rc5+ #440
[    7.554506] Hardware name: NXP Layerscape LX2160ABLUEBOX3 (DT)
[    7.554507] Call trace:
[    7.554508]  dump_backtrace+0x0/0x1c0
[    7.554509]  show_stack+0x18/0x28
[    7.554510]  dump_stack_lvl+0x8c/0xb8
[    7.554511]  dump_stack+0x18/0x34
[    7.554512]  __lock_acquire+0x14a8/0x1500
[    7.554513]  lock_acquire+0x314/0x3a8
[    7.554513]  _raw_spin_lock+0x6c/0x110
[    7.554514]  pl011_console_write+0x134/0x210
[    7.554515]  console_unlock+0x5f4/0x658
[    7.554516]  vprintk_emit+0x15c/0x328
[    7.554517]  vprintk_default+0x38/0x48
[    7.554518]  vprintk+0xb4/0x2a8
[    7.554519]  printk+0x68/0x8c
[    7.554520]  __irq_set_trigger+0xa0/0x190
[    7.554521]  __setup_irq+0x68c/0x798
[    7.554521]  request_threaded_irq+0xf8/0x1d0
[    7.554522]  devm_request_threaded_irq+0x84/0x108
[    7.554523]  pcf2127_probe.isra.10+0x290/0x4b8
[    7.554524]  pcf2127_i2c_probe+0x94/0xe0
[    7.554525]  i2c_device_probe+0x328/0x358
[    7.554526]  really_probe+0xb4/0x308
[    7.554527]  __driver_probe_device+0x7c/0xe8
[    7.554528]  driver_probe_device+0x8c/0x130
[    7.554529]  __device_attach_driver+0x98/0xd0
[    7.554530]  bus_for_each_drv+0x68/0xd0
[    7.554530]  __device_attach+0xec/0x148
[    7.554531]  device_initial_probe+0x14/0x20
[    7.554532]  bus_probe_device+0x9c/0xa8
[    7.554533]  device_add+0x408/0x8e8
[    7.554534]  device_register+0x20/0x30
[    7.554535]  i2c_new_client_device+0x1a0/0x288
[    7.554536]  of_i2c_register_device+0x80/0xa8
[    7.554537]  of_i2c_register_devices+0xac/0x1a0
[    7.554538]  i2c_register_adapter+0x2b8/0x6a0
[    7.554539]  i2c_add_adapter+0x80/0xd8
[    7.554540]  i2c_mux_add_adapter+0x280/0x530
[    7.554541]  pca954x_probe+0x2b0/0x470
[    7.554541]  i2c_device_probe+0x328/0x358
[    7.554542]  really_probe+0xb4/0x308
[    7.554543]  __driver_probe_device+0x7c/0xe8
[    7.554544]  driver_probe_device+0x8c/0x130
[    7.554545]  __driver_attach+0x9c/0x100
[    7.554546]  bus_for_each_dev+0x7c/0xd8
[    7.554547]  driver_attach+0x24/0x30
[    7.554548]  bus_add_driver+0x158/0x200
[    7.554549]  driver_register+0x64/0x120
[    7.554550]  i2c_register_driver+0x48/0xb0
[    7.554551]  pca954x_driver_init+0x1c/0x28
[    7.554552]  do_one_initcall+0x88/0x410
[    7.554553]  kernel_init_freeable+0x328/0x39c
[    7.554554]  kernel_init+0x24/0x128
[    7.554555]  ret_from_fork+0x10/0x18
[    7.867477] rtc-pcf2127-i2c 6-0051: failed to request alarm irq
[    7.873459] rtc-pcf2127-i2c: probe of 6-0051 failed with error -22
[    7.879679] i2c i2c-0: Added multiplexed i2c bus 6
[    7.884658] i2c i2c-0: Added multiplexed i2c bus 7
[    7.889643] i2c i2c-0: Added multiplexed i2c bus 8
[    7.894823] i2c 9-0075: Linked as a sync state only consumer to 0x0000000008b96000:01
[    7.902744] i2c 9-0075: Linked as a sync state only consumer to 0x0000000008b96000:02
[    7.911910] i2c 10-0028: Linked as a sync state only consumer to 0x0000000008b96000:01
[    7.919908] i2c 10-0028: Linked as a sync state only consumer to 0x0000000008b96000:02
[    7.928362] spi spi1.0: Linked as a sync state only consumer to 0x0000000008b96000:01
[    7.943497] sja1105 spi1.0: Probed switch chip: SJA1110A
[    7.949125] spi spi1.2: Linked as a sync state only consumer to 0x0000000008b96000:02
[    7.963340] sja1105 spi1.2: Probed switch chip: SJA1110A
[    7.968746] sc18is602 10-0028: Dropping the link to 0x0000000008b96000:01
[    7.975568] sc18is602 10-0028: Dropping the link to 0x0000000008b96000:02
[    7.982447] i2c i2c-9: Added multiplexed i2c bus 10
[    7.987515] i2c i2c-9: Added multiplexed i2c bus 11
[    7.992614] i2c i2c-9: Added multiplexed i2c bus 12
[    7.997706] i2c i2c-9: Added multiplexed i2c bus 13
[    8.002766] i2c i2c-9: Added multiplexed i2c bus 14
[    8.007829] i2c i2c-9: Added multiplexed i2c bus 15
[    8.012877] i2c i2c-9: Added multiplexed i2c bus 16
[    8.017932] i2c i2c-9: Added multiplexed i2c bus 17
[    8.022814] pca954x 9-0075: registered 8 multiplexed busses for I2C mux pca9547
[    8.030130] pca954x 9-0075: Dropping the link to 0x0000000008b96000:01
[    8.036698] pca954x 9-0075: Dropping the link to 0x0000000008b96000:02
[    8.043320] i2c i2c-0: Added multiplexed i2c bus 9
[    8.048112] pca954x 0-0077: registered 8 multiplexed busses for I2C mux pca9547
[    8.055440] pca954x 0-0077: Dropping the link to regulator-sb3v3
[    8.061480] pca954x 0-0077: Dropping the link to 0x0000000008b96000:01
[    8.068041] pca954x 0-0077: Dropping the link to 0x0000000008b96000:02
[    8.076133] i2c i2c-1: Added multiplexed i2c bus 18
[    8.081205] i2c i2c-1: Added multiplexed i2c bus 19
[    8.086263] i2c i2c-1: Added multiplexed i2c bus 20
[    8.091325] i2c i2c-1: Added multiplexed i2c bus 21
[    8.096208] pca954x 1-0077: registered 4 multiplexed busses for I2C switch pca9846
[    8.105888] sbsa-gwdt 23a0000.watchdog: Initialized with 30s timeout @ 25000050 Hz, action=0. [enabled]
[    8.118030] sdhci: Secure Digital Host Controller Interface driver
[    8.124216] sdhci: Copyright(c) Pierre Ossman
[    8.129049] Synopsys Designware Multimedia Card Interface Driver
[    8.135758] sdhci-pltfm: SDHCI platform and OF driver helper
[    8.141984] sdhci-esdhc 2150000.esdhc: Linked as a consumer to 5000000.iommu
[    8.149084] sdhci-esdhc 2150000.esdhc: Adding to iommu group 1
[    8.149105] sdhci-esdhc 2140000.esdhc: Linked as a consumer to 5000000.iommu
[    8.161986] sdhci-esdhc 2140000.esdhc: Adding to iommu group 2
[    8.168313] ledtrig-cpu: registered to indicate activity on CPUs
[    8.175015] SMCCC: SOC_ID: ARCH_FEATURES(ARCH_SOC_ID) returned error: fffffffffffffffd
[    8.183506] usbcore: registered new interface driver usbhid
[    8.187157] mmc1: SDHCI controller on 2150000.esdhc [2150000.esdhc] using ADMA 64-bit
[    8.189078] usbhid: USB HID core driver
[    8.193629] mmc0: SDHCI controller on 2140000.esdhc [2140000.esdhc] using ADMA 64-bit
[    8.213167] GACT probability NOT on
[    8.216671] Mirror/redirect action on
[    8.220366] u32 classifier
[    8.223063]     input device check on
[    8.226720]     Actions configured
[    8.230175] Initializing XFRM netlink socket
[    8.234539] NET: Registered PF_INET6 protocol family
[    8.240232] Segment Routing with IPv6
[    8.243912] In-situ OAM (IOAM) with IPv6
[    8.247865] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    8.254117] NET: Registered PF_PACKET protocol family
[    8.259181] NET: Registered PF_KEY protocol family
[    8.264005] can: controller area network core
[    8.268388] NET: Registered PF_CAN protocol family
[    8.270836] mmc0: new ultra high speed SDR104 SDHC card at address aaaa
[    8.273178] can: raw protocol
[    8.280291] mmcblk0: mmc0:aaaa SD32G 29.7 GiB
[    8.282745] can: broadcast manager protocol
[    8.291361] can: netlink gateway - max_hops=1
[    8.291988]  mmcblk0: p1 p2
[    8.295781] 8021q: 802.1Q VLAN Support v1.8
[    8.301381] random: fast init done
[    8.302700] 9pnet: Installing 9P2000 support
[    8.306072] mmc1: new HS400 MMC card at address 0001
[    8.310362] Key type dns_resolver registered
[    8.315789] mmcblk1: mmc1:0001 S0J59X 119 GiB
[    8.319819] registered taskstats version 1
[    8.324243] mmcblk1boot0: mmc1:0001 S0J59X 31.5 MiB
[    8.328078] Loading compiled-in X.509 certificates
[    8.333300] mmcblk1boot1: mmc1:0001 S0J59X 31.5 MiB
[    8.338219] Btrfs loaded, crc32c=crc32c-generic, zoned=no
[    8.342913] mmcblk1rpmb: mmc1:0001 S0J59X 4.00 MiB, chardev (511:0)
[    8.353108] fsl_mc_dprc dprc.1: Linked as a consumer to 5000000.iommu
[    8.360903] fsl_mc_dprc dprc.1: Adding to iommu group 3
[    8.461866] fsl_mc_allocator dpbp.5: Linked as a consumer to 5000000.iommu
[    8.468758] fsl_mc_allocator dpbp.5: Adding to iommu group 3
[    8.474571] fsl_mc_allocator dpbp.4: Linked as a consumer to 5000000.iommu
[    8.481453] fsl_mc_allocator dpbp.4: Adding to iommu group 3
[    8.487259] fsl_mc_allocator dpbp.3: Linked as a consumer to 5000000.iommu
[    8.494140] fsl_mc_allocator dpbp.3: Adding to iommu group 3
[    8.499948] fsl_mc_allocator dpbp.2: Linked as a consumer to 5000000.iommu
[    8.506828] fsl_mc_allocator dpbp.2: Adding to iommu group 3
[    8.512633] fsl_mc_allocator dpbp.1: Linked as a consumer to 5000000.iommu
[    8.519514] fsl_mc_allocator dpbp.1: Adding to iommu group 3
[    8.525323] fsl_mc_allocator dpbp.0: Linked as a consumer to 5000000.iommu
[    8.532204] fsl_mc_allocator dpbp.0: Adding to iommu group 3
[    8.539018] fsl_mc_allocator dpmcp.39: Linked as a consumer to 5000000.iommu
[    8.546072] fsl_mc_allocator dpmcp.39: Adding to iommu group 3
[    8.552558] fsl_mc_allocator dpmcp.38: Linked as a consumer to 5000000.iommu
[    8.559612] fsl_mc_allocator dpmcp.38: Adding to iommu group 3
[    8.566099] fsl_mc_allocator dpmcp.37: Linked as a consumer to 5000000.iommu
[    8.573153] fsl_mc_allocator dpmcp.37: Adding to iommu group 3
[    8.579639] fsl_mc_allocator dpmcp.36: Linked as a consumer to 5000000.iommu
[    8.586694] fsl_mc_allocator dpmcp.36: Adding to iommu group 3
[    8.593180] fsl_mc_allocator dpmcp.35: Linked as a consumer to 5000000.iommu
[    8.600234] fsl_mc_allocator dpmcp.35: Adding to iommu group 3
[    8.606720] fsl_mc_allocator dpmcp.34: Linked as a consumer to 5000000.iommu
[    8.613775] fsl_mc_allocator dpmcp.34: Adding to iommu group 3
[    8.620268] fsl_mc_allocator dpmcp.33: Linked as a consumer to 5000000.iommu
[    8.627322] fsl_mc_allocator dpmcp.33: Adding to iommu group 3
[    8.633811] fsl_mc_allocator dpmcp.32: Linked as a consumer to 5000000.iommu
[    8.640865] fsl_mc_allocator dpmcp.32: Adding to iommu group 3
[    8.647359] fsl_mc_allocator dpmcp.31: Linked as a consumer to 5000000.iommu
[    8.654414] fsl_mc_allocator dpmcp.31: Adding to iommu group 3
[    8.660905] fsl_mc_allocator dpmcp.30: Linked as a consumer to 5000000.iommu
[    8.667959] fsl_mc_allocator dpmcp.30: Adding to iommu group 3
[    8.674447] fsl_mc_allocator dpmcp.29: Linked as a consumer to 5000000.iommu
[    8.681503] fsl_mc_allocator dpmcp.29: Adding to iommu group 3
[    8.687999] fsl_mc_allocator dpmcp.28: Linked as a consumer to 5000000.iommu
[    8.695054] fsl_mc_allocator dpmcp.28: Adding to iommu group 3
[    8.701545] fsl_mc_allocator dpmcp.27: Linked as a consumer to 5000000.iommu
[    8.708599] fsl_mc_allocator dpmcp.27: Adding to iommu group 3
[    8.715091] fsl_mc_allocator dpmcp.26: Linked as a consumer to 5000000.iommu
[    8.722145] fsl_mc_allocator dpmcp.26: Adding to iommu group 3
[    8.728636] fsl_mc_allocator dpmcp.25: Linked as a consumer to 5000000.iommu
[    8.735690] fsl_mc_allocator dpmcp.25: Adding to iommu group 3
[    8.742185] fsl_mc_allocator dpmcp.24: Linked as a consumer to 5000000.iommu
[    8.749239] fsl_mc_allocator dpmcp.24: Adding to iommu group 3
[    8.755731] fsl_mc_allocator dpmcp.23: Linked as a consumer to 5000000.iommu
[    8.762787] fsl_mc_allocator dpmcp.23: Adding to iommu group 3
[    8.769279] fsl_mc_allocator dpmcp.22: Linked as a consumer to 5000000.iommu
[    8.776334] fsl_mc_allocator dpmcp.22: Adding to iommu group 3
[    8.782827] fsl_mc_allocator dpmcp.21: Linked as a consumer to 5000000.iommu
[    8.789882] fsl_mc_allocator dpmcp.21: Adding to iommu group 3
[    8.796383] fsl_mc_allocator dpmcp.20: Linked as a consumer to 5000000.iommu
[    8.803436] fsl_mc_allocator dpmcp.20: Adding to iommu group 3
[    8.809932] fsl_mc_allocator dpmcp.19: Linked as a consumer to 5000000.iommu
[    8.816986] fsl_mc_allocator dpmcp.19: Adding to iommu group 3
[    8.823481] fsl_mc_allocator dpmcp.18: Linked as a consumer to 5000000.iommu
[    8.830535] fsl_mc_allocator dpmcp.18: Adding to iommu group 3
[    8.837030] fsl_mc_allocator dpmcp.17: Linked as a consumer to 5000000.iommu
[    8.844085] fsl_mc_allocator dpmcp.17: Adding to iommu group 3
[    8.850581] fsl_mc_allocator dpmcp.16: Linked as a consumer to 5000000.iommu
[    8.857648] fsl_mc_allocator dpmcp.16: Adding to iommu group 3
[    8.864153] fsl_mc_allocator dpmcp.15: Linked as a consumer to 5000000.iommu
[    8.871207] fsl_mc_allocator dpmcp.15: Adding to iommu group 3
[    8.877705] fsl_mc_allocator dpmcp.14: Linked as a consumer to 5000000.iommu
[    8.884760] fsl_mc_allocator dpmcp.14: Adding to iommu group 3
[    8.891258] fsl_mc_allocator dpmcp.13: Linked as a consumer to 5000000.iommu
[    8.898313] fsl_mc_allocator dpmcp.13: Adding to iommu group 3
[    8.904810] fsl_mc_allocator dpmcp.12: Linked as a consumer to 5000000.iommu
[    8.911874] fsl_mc_allocator dpmcp.12: Adding to iommu group 3
[    8.918373] fsl_mc_allocator dpmcp.11: Linked as a consumer to 5000000.iommu
[    8.925427] fsl_mc_allocator dpmcp.11: Adding to iommu group 3
[    8.931935] fsl_mc_allocator dpmcp.10: Linked as a consumer to 5000000.iommu
[    8.938989] fsl_mc_allocator dpmcp.10: Adding to iommu group 3
[    8.945493] fsl_mc_allocator dpmcp.9: Linked as a consumer to 5000000.iommu
[    8.952460] fsl_mc_allocator dpmcp.9: Adding to iommu group 3
[    8.958875] fsl_mc_allocator dpmcp.8: Linked as a consumer to 5000000.iommu
[    8.965843] fsl_mc_allocator dpmcp.8: Adding to iommu group 3
[    8.972263] fsl_mc_allocator dpmcp.7: Linked as a consumer to 5000000.iommu
[    8.979230] fsl_mc_allocator dpmcp.7: Adding to iommu group 3
[    8.985647] fsl_mc_allocator dpmcp.6: Linked as a consumer to 5000000.iommu
[    8.992614] fsl_mc_allocator dpmcp.6: Adding to iommu group 3
[    8.999029] fsl_mc_allocator dpmcp.5: Linked as a consumer to 5000000.iommu
[    9.005997] fsl_mc_allocator dpmcp.5: Adding to iommu group 3
[    9.012420] fsl_mc_allocator dpmcp.4: Linked as a consumer to 5000000.iommu
[    9.019387] fsl_mc_allocator dpmcp.4: Adding to iommu group 3
[    9.025806] fsl_mc_allocator dpmcp.3: Linked as a consumer to 5000000.iommu
[    9.032773] fsl_mc_allocator dpmcp.3: Adding to iommu group 3
[    9.039191] fsl_mc_allocator dpmcp.2: Linked as a consumer to 5000000.iommu
[    9.046159] fsl_mc_allocator dpmcp.2: Adding to iommu group 3
[    9.052574] fsl_mc_allocator dpmcp.1: Linked as a consumer to 5000000.iommu
[    9.059541] fsl_mc_allocator dpmcp.1: Adding to iommu group 3
[    9.065458] fsl_mc_allocator dpcon.95: Linked as a consumer to 5000000.iommu
[    9.072514] fsl_mc_allocator dpcon.95: Adding to iommu group 3
[    9.078515] fsl_mc_allocator dpcon.94: Linked as a consumer to 5000000.iommu
[    9.085570] fsl_mc_allocator dpcon.94: Adding to iommu group 3
[    9.091573] fsl_mc_allocator dpcon.93: Linked as a consumer to 5000000.iommu
[    9.098628] fsl_mc_allocator dpcon.93: Adding to iommu group 3
[    9.104630] fsl_mc_allocator dpcon.92: Linked as a consumer to 5000000.iommu
[    9.111685] fsl_mc_allocator dpcon.92: Adding to iommu group 3
[    9.117690] fsl_mc_allocator dpcon.91: Linked as a consumer to 5000000.iommu
[    9.124745] fsl_mc_allocator dpcon.91: Adding to iommu group 3
[    9.130746] fsl_mc_allocator dpcon.90: Linked as a consumer to 5000000.iommu
[    9.137800] fsl_mc_allocator dpcon.90: Adding to iommu group 3
[    9.143815] fsl_mc_allocator dpcon.89: Linked as a consumer to 5000000.iommu
[    9.150868] fsl_mc_allocator dpcon.89: Adding to iommu group 3
[    9.156874] fsl_mc_allocator dpcon.88: Linked as a consumer to 5000000.iommu
[    9.163928] fsl_mc_allocator dpcon.88: Adding to iommu group 3
[    9.169944] fsl_mc_allocator dpcon.87: Linked as a consumer to 5000000.iommu
[    9.176999] fsl_mc_allocator dpcon.87: Adding to iommu group 3
[    9.183003] fsl_mc_allocator dpcon.86: Linked as a consumer to 5000000.iommu
[    9.190057] fsl_mc_allocator dpcon.86: Adding to iommu group 3
[    9.196068] fsl_mc_allocator dpcon.85: Linked as a consumer to 5000000.iommu
[    9.203122] fsl_mc_allocator dpcon.85: Adding to iommu group 3
[    9.209128] fsl_mc_allocator dpcon.84: Linked as a consumer to 5000000.iommu
[    9.216183] fsl_mc_allocator dpcon.84: Adding to iommu group 3
[    9.222192] fsl_mc_allocator dpcon.83: Linked as a consumer to 5000000.iommu
[    9.229246] fsl_mc_allocator dpcon.83: Adding to iommu group 3
[    9.235254] fsl_mc_allocator dpcon.82: Linked as a consumer to 5000000.iommu
[    9.242310] fsl_mc_allocator dpcon.82: Adding to iommu group 3
[    9.248321] fsl_mc_allocator dpcon.81: Linked as a consumer to 5000000.iommu
[    9.255376] fsl_mc_allocator dpcon.81: Adding to iommu group 3
[    9.261384] fsl_mc_allocator dpcon.80: Linked as a consumer to 5000000.iommu
[    9.268439] fsl_mc_allocator dpcon.80: Adding to iommu group 3
[    9.274450] fsl_mc_allocator dpcon.79: Linked as a consumer to 5000000.iommu
[    9.281505] fsl_mc_allocator dpcon.79: Adding to iommu group 3
[    9.287517] fsl_mc_allocator dpcon.78: Linked as a consumer to 5000000.iommu
[    9.294571] fsl_mc_allocator dpcon.78: Adding to iommu group 3
[    9.300584] fsl_mc_allocator dpcon.77: Linked as a consumer to 5000000.iommu
[    9.307638] fsl_mc_allocator dpcon.77: Adding to iommu group 3
[    9.313649] fsl_mc_allocator dpcon.76: Linked as a consumer to 5000000.iommu
[    9.320703] fsl_mc_allocator dpcon.76: Adding to iommu group 3
[    9.326717] fsl_mc_allocator dpcon.75: Linked as a consumer to 5000000.iommu
[    9.333771] fsl_mc_allocator dpcon.75: Adding to iommu group 3
[    9.339794] fsl_mc_allocator dpcon.74: Linked as a consumer to 5000000.iommu
[    9.346847] fsl_mc_allocator dpcon.74: Adding to iommu group 3
[    9.352862] fsl_mc_allocator dpcon.73: Linked as a consumer to 5000000.iommu
[    9.359917] fsl_mc_allocator dpcon.73: Adding to iommu group 3
[    9.365941] fsl_mc_allocator dpcon.72: Linked as a consumer to 5000000.iommu
[    9.372996] fsl_mc_allocator dpcon.72: Adding to iommu group 3
[    9.379017] fsl_mc_allocator dpcon.71: Linked as a consumer to 5000000.iommu
[    9.386072] fsl_mc_allocator dpcon.71: Adding to iommu group 3
[    9.392088] fsl_mc_allocator dpcon.70: Linked as a consumer to 5000000.iommu
[    9.399142] fsl_mc_allocator dpcon.70: Adding to iommu group 3
[    9.405158] fsl_mc_allocator dpcon.69: Linked as a consumer to 5000000.iommu
[    9.412213] fsl_mc_allocator dpcon.69: Adding to iommu group 3
[    9.418231] fsl_mc_allocator dpcon.68: Linked as a consumer to 5000000.iommu
[    9.425285] fsl_mc_allocator dpcon.68: Adding to iommu group 3
[    9.431312] fsl_mc_allocator dpcon.67: Linked as a consumer to 5000000.iommu
[    9.438366] fsl_mc_allocator dpcon.67: Adding to iommu group 3
[    9.444384] fsl_mc_allocator dpcon.66: Linked as a consumer to 5000000.iommu
[    9.451438] fsl_mc_allocator dpcon.66: Adding to iommu group 3
[    9.457458] fsl_mc_allocator dpcon.65: Linked as a consumer to 5000000.iommu
[    9.464512] fsl_mc_allocator dpcon.65: Adding to iommu group 3
[    9.470530] fsl_mc_allocator dpcon.64: Linked as a consumer to 5000000.iommu
[    9.477585] fsl_mc_allocator dpcon.64: Adding to iommu group 3
[    9.483605] fsl_mc_allocator dpcon.63: Linked as a consumer to 5000000.iommu
[    9.490660] fsl_mc_allocator dpcon.63: Adding to iommu group 3
[    9.496680] fsl_mc_allocator dpcon.62: Linked as a consumer to 5000000.iommu
[    9.503735] fsl_mc_allocator dpcon.62: Adding to iommu group 3
[    9.509757] fsl_mc_allocator dpcon.61: Linked as a consumer to 5000000.iommu
[    9.516811] fsl_mc_allocator dpcon.61: Adding to iommu group 3
[    9.522831] fsl_mc_allocator dpcon.60: Linked as a consumer to 5000000.iommu
[    9.529885] fsl_mc_allocator dpcon.60: Adding to iommu group 3
[    9.535917] fsl_mc_allocator dpcon.59: Linked as a consumer to 5000000.iommu
[    9.542973] fsl_mc_allocator dpcon.59: Adding to iommu group 3
[    9.548995] fsl_mc_allocator dpcon.58: Linked as a consumer to 5000000.iommu
[    9.556050] fsl_mc_allocator dpcon.58: Adding to iommu group 3
[    9.562073] fsl_mc_allocator dpcon.57: Linked as a consumer to 5000000.iommu
[    9.569128] fsl_mc_allocator dpcon.57: Adding to iommu group 3
[    9.575150] fsl_mc_allocator dpcon.56: Linked as a consumer to 5000000.iommu
[    9.582204] fsl_mc_allocator dpcon.56: Adding to iommu group 3
[    9.588232] fsl_mc_allocator dpcon.55: Linked as a consumer to 5000000.iommu
[    9.595285] fsl_mc_allocator dpcon.55: Adding to iommu group 3
[    9.601308] fsl_mc_allocator dpcon.54: Linked as a consumer to 5000000.iommu
[    9.608363] fsl_mc_allocator dpcon.54: Adding to iommu group 3
[    9.614388] fsl_mc_allocator dpcon.53: Linked as a consumer to 5000000.iommu
[    9.621443] fsl_mc_allocator dpcon.53: Adding to iommu group 3
[    9.627469] fsl_mc_allocator dpcon.52: Linked as a consumer to 5000000.iommu
[    9.634523] fsl_mc_allocator dpcon.52: Adding to iommu group 3
[    9.640551] fsl_mc_allocator dpcon.51: Linked as a consumer to 5000000.iommu
[    9.647605] fsl_mc_allocator dpcon.51: Adding to iommu group 3
[    9.653632] fsl_mc_allocator dpcon.50: Linked as a consumer to 5000000.iommu
[    9.660686] fsl_mc_allocator dpcon.50: Adding to iommu group 3
[    9.666715] fsl_mc_allocator dpcon.49: Linked as a consumer to 5000000.iommu
[    9.673771] fsl_mc_allocator dpcon.49: Adding to iommu group 3
[    9.679805] fsl_mc_allocator dpcon.48: Linked as a consumer to 5000000.iommu
[    9.686860] fsl_mc_allocator dpcon.48: Adding to iommu group 3
[    9.692897] fsl_mc_allocator dpcon.47: Linked as a consumer to 5000000.iommu
[    9.699953] fsl_mc_allocator dpcon.47: Adding to iommu group 3
[    9.705980] fsl_mc_allocator dpcon.46: Linked as a consumer to 5000000.iommu
[    9.713034] fsl_mc_allocator dpcon.46: Adding to iommu group 3
[    9.719065] fsl_mc_allocator dpcon.45: Linked as a consumer to 5000000.iommu
[    9.726119] fsl_mc_allocator dpcon.45: Adding to iommu group 3
[    9.732151] fsl_mc_allocator dpcon.44: Linked as a consumer to 5000000.iommu
[    9.739205] fsl_mc_allocator dpcon.44: Adding to iommu group 3
[    9.745234] fsl_mc_allocator dpcon.43: Linked as a consumer to 5000000.iommu
[    9.752289] fsl_mc_allocator dpcon.43: Adding to iommu group 3
[    9.758320] fsl_mc_allocator dpcon.42: Linked as a consumer to 5000000.iommu
[    9.765375] fsl_mc_allocator dpcon.42: Adding to iommu group 3
[    9.771408] fsl_mc_allocator dpcon.41: Linked as a consumer to 5000000.iommu
[    9.778462] fsl_mc_allocator dpcon.41: Adding to iommu group 3
[    9.784495] fsl_mc_allocator dpcon.40: Linked as a consumer to 5000000.iommu
[    9.791549] fsl_mc_allocator dpcon.40: Adding to iommu group 3
[    9.797584] fsl_mc_allocator dpcon.39: Linked as a consumer to 5000000.iommu
[    9.804639] fsl_mc_allocator dpcon.39: Adding to iommu group 3
[    9.810671] fsl_mc_allocator dpcon.38: Linked as a consumer to 5000000.iommu
[    9.817726] fsl_mc_allocator dpcon.38: Adding to iommu group 3
[    9.823762] fsl_mc_allocator dpcon.37: Linked as a consumer to 5000000.iommu
[    9.830817] fsl_mc_allocator dpcon.37: Adding to iommu group 3
[    9.836851] fsl_mc_allocator dpcon.36: Linked as a consumer to 5000000.iommu
[    9.843907] fsl_mc_allocator dpcon.36: Adding to iommu group 3
[    9.849943] fsl_mc_allocator dpcon.35: Linked as a consumer to 5000000.iommu
[    9.856997] fsl_mc_allocator dpcon.35: Adding to iommu group 3
[    9.863034] fsl_mc_allocator dpcon.34: Linked as a consumer to 5000000.iommu
[    9.870088] fsl_mc_allocator dpcon.34: Adding to iommu group 3
[    9.876137] fsl_mc_allocator dpcon.33: Linked as a consumer to 5000000.iommu
[    9.883191] fsl_mc_allocator dpcon.33: Adding to iommu group 3
[    9.889228] fsl_mc_allocator dpcon.32: Linked as a consumer to 5000000.iommu
[    9.896283] fsl_mc_allocator dpcon.32: Adding to iommu group 3
[    9.902321] fsl_mc_allocator dpcon.31: Linked as a consumer to 5000000.iommu
[    9.909376] fsl_mc_allocator dpcon.31: Adding to iommu group 3
[    9.915413] fsl_mc_allocator dpcon.30: Linked as a consumer to 5000000.iommu
[    9.922467] fsl_mc_allocator dpcon.30: Adding to iommu group 3
[    9.928511] fsl_mc_allocator dpcon.29: Linked as a consumer to 5000000.iommu
[    9.935565] fsl_mc_allocator dpcon.29: Adding to iommu group 3
[    9.941604] fsl_mc_allocator dpcon.28: Linked as a consumer to 5000000.iommu
[    9.948659] fsl_mc_allocator dpcon.28: Adding to iommu group 3
[    9.954705] fsl_mc_allocator dpcon.27: Linked as a consumer to 5000000.iommu
[    9.961760] fsl_mc_allocator dpcon.27: Adding to iommu group 3
[    9.967807] fsl_mc_allocator dpcon.26: Linked as a consumer to 5000000.iommu
[    9.974863] fsl_mc_allocator dpcon.26: Adding to iommu group 3
[    9.980907] fsl_mc_allocator dpcon.25: Linked as a consumer to 5000000.iommu
[    9.987962] fsl_mc_allocator dpcon.25: Adding to iommu group 3
[    9.994003] fsl_mc_allocator dpcon.24: Linked as a consumer to 5000000.iommu
[   10.001057] fsl_mc_allocator dpcon.24: Adding to iommu group 3
[   10.007101] fsl_mc_allocator dpcon.23: Linked as a consumer to 5000000.iommu
[   10.014155] fsl_mc_allocator dpcon.23: Adding to iommu group 3
[   10.020199] fsl_mc_allocator dpcon.22: Linked as a consumer to 5000000.iommu
[   10.027253] fsl_mc_allocator dpcon.22: Adding to iommu group 3
[   10.033297] fsl_mc_allocator dpcon.21: Linked as a consumer to 5000000.iommu
[   10.040352] fsl_mc_allocator dpcon.21: Adding to iommu group 3
[   10.046394] fsl_mc_allocator dpcon.20: Linked as a consumer to 5000000.iommu
[   10.053449] fsl_mc_allocator dpcon.20: Adding to iommu group 3
[   10.059496] fsl_mc_allocator dpcon.19: Linked as a consumer to 5000000.iommu
[   10.066550] fsl_mc_allocator dpcon.19: Adding to iommu group 3
[   10.072595] fsl_mc_allocator dpcon.18: Linked as a consumer to 5000000.iommu
[   10.079649] fsl_mc_allocator dpcon.18: Adding to iommu group 3
[   10.085696] fsl_mc_allocator dpcon.17: Linked as a consumer to 5000000.iommu
[   10.092751] fsl_mc_allocator dpcon.17: Adding to iommu group 3
[   10.098798] fsl_mc_allocator dpcon.16: Linked as a consumer to 5000000.iommu
[   10.105853] fsl_mc_allocator dpcon.16: Adding to iommu group 3
[   10.111909] fsl_mc_allocator dpcon.15: Linked as a consumer to 5000000.iommu
[   10.118966] fsl_mc_allocator dpcon.15: Adding to iommu group 3
[   10.125014] fsl_mc_allocator dpcon.14: Linked as a consumer to 5000000.iommu
[   10.132069] fsl_mc_allocator dpcon.14: Adding to iommu group 3
[   10.138118] fsl_mc_allocator dpcon.13: Linked as a consumer to 5000000.iommu
[   10.145174] fsl_mc_allocator dpcon.13: Adding to iommu group 3
[   10.151224] fsl_mc_allocator dpcon.12: Linked as a consumer to 5000000.iommu
[   10.158278] fsl_mc_allocator dpcon.12: Adding to iommu group 3
[   10.164327] fsl_mc_allocator dpcon.11: Linked as a consumer to 5000000.iommu
[   10.171381] fsl_mc_allocator dpcon.11: Adding to iommu group 3
[   10.177431] fsl_mc_allocator dpcon.10: Linked as a consumer to 5000000.iommu
[   10.184485] fsl_mc_allocator dpcon.10: Adding to iommu group 3
[   10.190538] fsl_mc_allocator dpcon.9: Linked as a consumer to 5000000.iommu
[   10.197506] fsl_mc_allocator dpcon.9: Adding to iommu group 3
[   10.203471] fsl_mc_allocator dpcon.8: Linked as a consumer to 5000000.iommu
[   10.210438] fsl_mc_allocator dpcon.8: Adding to iommu group 3
[   10.216412] fsl_mc_allocator dpcon.7: Linked as a consumer to 5000000.iommu
[   10.223379] fsl_mc_allocator dpcon.7: Adding to iommu group 3
[   10.229347] fsl_mc_allocator dpcon.6: Linked as a consumer to 5000000.iommu
[   10.236315] fsl_mc_allocator dpcon.6: Adding to iommu group 3
[   10.242281] fsl_mc_allocator dpcon.5: Linked as a consumer to 5000000.iommu
[   10.249249] fsl_mc_allocator dpcon.5: Adding to iommu group 3
[   10.255216] fsl_mc_allocator dpcon.4: Linked as a consumer to 5000000.iommu
[   10.262184] fsl_mc_allocator dpcon.4: Adding to iommu group 3
[   10.268152] fsl_mc_allocator dpcon.3: Linked as a consumer to 5000000.iommu
[   10.275121] fsl_mc_allocator dpcon.3: Adding to iommu group 3
[   10.281088] fsl_mc_allocator dpcon.2: Linked as a consumer to 5000000.iommu
[   10.288057] fsl_mc_allocator dpcon.2: Adding to iommu group 3
[   10.294027] fsl_mc_allocator dpcon.1: Linked as a consumer to 5000000.iommu
[   10.300995] fsl_mc_allocator dpcon.1: Adding to iommu group 3
[   10.306966] fsl_mc_allocator dpcon.0: Linked as a consumer to 5000000.iommu
[   10.313934] fsl_mc_allocator dpcon.0: Adding to iommu group 3
[   10.319917] fsl_dpaa2_eth dpni.5: Linked as a consumer to 5000000.iommu
[   10.326539] fsl_dpaa2_eth dpni.5: Adding to iommu group 3
[   10.332601] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpmcp.39
[   10.372876] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.95
[   10.412736] fsl_dpaa2_eth dpni.5: Dropping the link to dpcon.95
[   10.418700] fsl_dpaa2_eth dpni.5: Dropping the link to dpmcp.39
[   10.424909] fsl_dpaa2_eth dpni.4: Linked as a consumer to 5000000.iommu
[   10.431534] fsl_dpaa2_eth dpni.4: Adding to iommu group 3
[   10.437568] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpmcp.38
[   10.477841] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.94
[   10.517717] fsl_dpaa2_eth dpni.4: Dropping the link to dpcon.94
[   10.523700] fsl_dpaa2_eth dpni.4: Dropping the link to dpmcp.38
[   10.529940] fsl_dpaa2_eth dpni.3: Linked as a consumer to 5000000.iommu
[   10.536567] fsl_dpaa2_eth dpni.3: Adding to iommu group 3
[   10.542606] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpmcp.37
[   10.582872] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.93
[   10.622748] fsl_dpaa2_eth dpni.3: Dropping the link to dpcon.93
[   10.628723] fsl_dpaa2_eth dpni.3: Dropping the link to dpmcp.37
[   10.634950] fsl_dpaa2_eth dpni.2: Linked as a consumer to 5000000.iommu
[   10.641577] fsl_dpaa2_eth dpni.2: Adding to iommu group 3
[   10.647620] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpmcp.36
[   10.687885] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.92
[   10.727737] fsl_dpaa2_eth dpni.2: Dropping the link to dpcon.92
[   10.733702] fsl_dpaa2_eth dpni.2: Dropping the link to dpmcp.36
[   10.739915] fsl_dpaa2_eth dpni.1: Linked as a consumer to 5000000.iommu
[   10.746537] fsl_dpaa2_eth dpni.1: Adding to iommu group 3
[   10.752567] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.35
[   10.792832] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.91
[   10.832648] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.91
[   10.838614] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.35
[   10.844828] fsl_dpaa2_eth dpni.0: Linked as a consumer to 5000000.iommu
[   10.851450] fsl_dpaa2_eth dpni.0: Adding to iommu group 3
[   10.857478] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpmcp.34
[   10.897746] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.90
[   10.937569] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.90
[   10.943537] fsl_dpaa2_eth dpni.0: Dropping the link to dpmcp.34
[   10.950749] fsl_dpaa2_ptp dprtc.0: Linked as a consumer to 5000000.iommu
[   10.957462] fsl_dpaa2_ptp dprtc.0: Adding to iommu group 3
[   10.963545] fsl_dpaa2_ptp dprtc.0: Linked as a consumer to dpmcp.33
[   10.971576] ptp_qoriq: device tree node missing required elements, try automatic configuration
[   10.980337] pps pps0: new PPS source ptp0
[   10.986262] fsl_mc_dpio dpio.15: Linked as a consumer to 5000000.iommu
[   10.992811] fsl_mc_dpio dpio.15: Adding to iommu group 3
[   10.998716] fsl_mc_dpio dpio.15: Linked as a consumer to dpmcp.32
[   11.007954] fsl_mc_dpio dpio.15: probed
[   11.014046] fsl_mc_dpio dpio.14: Linked as a consumer to 5000000.iommu
[   11.020583] fsl_mc_dpio dpio.14: Adding to iommu group 3
[   11.026472] fsl_mc_dpio dpio.14: Linked as a consumer to dpmcp.31
[   11.035684] fsl_mc_dpio dpio.14: probed
[   11.041777] fsl_mc_dpio dpio.13: Linked as a consumer to 5000000.iommu
[   11.048316] fsl_mc_dpio dpio.13: Adding to iommu group 3
[   11.054206] fsl_mc_dpio dpio.13: Linked as a consumer to dpmcp.30
[   11.063417] fsl_mc_dpio dpio.13: probed
[   11.069516] fsl_mc_dpio dpio.12: Linked as a consumer to 5000000.iommu
[   11.076053] fsl_mc_dpio dpio.12: Adding to iommu group 3
[   11.081941] fsl_mc_dpio dpio.12: Linked as a consumer to dpmcp.29
[   11.091150] fsl_mc_dpio dpio.12: probed
[   11.097251] fsl_mc_dpio dpio.11: Linked as a consumer to 5000000.iommu
[   11.103786] fsl_mc_dpio dpio.11: Adding to iommu group 3
[   11.109676] fsl_mc_dpio dpio.11: Linked as a consumer to dpmcp.28
[   11.118896] fsl_mc_dpio dpio.11: probed
[   11.124999] fsl_mc_dpio dpio.10: Linked as a consumer to 5000000.iommu
[   11.131534] fsl_mc_dpio dpio.10: Adding to iommu group 3
[   11.137425] fsl_mc_dpio dpio.10: Linked as a consumer to dpmcp.27
[   11.146635] fsl_mc_dpio dpio.10: probed
[   11.152738] fsl_mc_dpio dpio.9: Linked as a consumer to 5000000.iommu
[   11.159186] fsl_mc_dpio dpio.9: Adding to iommu group 3
[   11.164990] fsl_mc_dpio dpio.9: Linked as a consumer to dpmcp.26
[   11.174114] fsl_mc_dpio dpio.9: probed
[   11.180129] fsl_mc_dpio dpio.8: Linked as a consumer to 5000000.iommu
[   11.186578] fsl_mc_dpio dpio.8: Adding to iommu group 3
[   11.192385] fsl_mc_dpio dpio.8: Linked as a consumer to dpmcp.25
[   11.201509] fsl_mc_dpio dpio.8: probed
[   11.207512] fsl_mc_dpio dpio.7: Linked as a consumer to 5000000.iommu
[   11.213960] fsl_mc_dpio dpio.7: Adding to iommu group 3
[   11.219762] fsl_mc_dpio dpio.7: Linked as a consumer to dpmcp.24
[   11.228897] fsl_mc_dpio dpio.7: probed
[   11.234904] fsl_mc_dpio dpio.6: Linked as a consumer to 5000000.iommu
[   11.241354] fsl_mc_dpio dpio.6: Adding to iommu group 3
[   11.247155] fsl_mc_dpio dpio.6: Linked as a consumer to dpmcp.23
[   11.256288] fsl_mc_dpio dpio.6: probed
[   11.262293] fsl_mc_dpio dpio.5: Linked as a consumer to 5000000.iommu
[   11.268744] fsl_mc_dpio dpio.5: Adding to iommu group 3
[   11.274546] fsl_mc_dpio dpio.5: Linked as a consumer to dpmcp.22
[   11.283670] fsl_mc_dpio dpio.5: probed
[   11.289681] fsl_mc_dpio dpio.4: Linked as a consumer to 5000000.iommu
[   11.296136] fsl_mc_dpio dpio.4: Adding to iommu group 3
[   11.301936] fsl_mc_dpio dpio.4: Linked as a consumer to dpmcp.21
[   11.311058] fsl_mc_dpio dpio.4: probed
[   11.317079] fsl_mc_dpio dpio.3: Linked as a consumer to 5000000.iommu
[   11.323528] fsl_mc_dpio dpio.3: Adding to iommu group 3
[   11.329330] fsl_mc_dpio dpio.3: Linked as a consumer to dpmcp.20
[   11.338451] fsl_mc_dpio dpio.3: probed
[   11.344472] fsl_mc_dpio dpio.2: Linked as a consumer to 5000000.iommu
[   11.350920] fsl_mc_dpio dpio.2: Adding to iommu group 3
[   11.356724] fsl_mc_dpio dpio.2: Linked as a consumer to dpmcp.19
[   11.365847] fsl_mc_dpio dpio.2: probed
[   11.371866] fsl_mc_dpio dpio.1: Linked as a consumer to 5000000.iommu
[   11.378313] fsl_mc_dpio dpio.1: Adding to iommu group 3
[   11.384118] fsl_mc_dpio dpio.1: Linked as a consumer to dpmcp.18
[   11.393241] fsl_mc_dpio dpio.1: probed
[   11.399250] fsl_mc_dpio dpio.0: Linked as a consumer to 5000000.iommu
[   11.405701] fsl_mc_dpio dpio.0: Adding to iommu group 3
[   11.411503] fsl_mc_dpio dpio.0: Linked as a consumer to dpmcp.17
[   11.420629] fsl_mc_dpio dpio.0: probed
[   11.428066] fsl_mc_dprc dprc.1: DPRC device bound to driver
[   11.433768] pcieport 0000:00:00.0: Linked as a consumer to 5000000.iommu
[   11.440498] pcieport 0000:00:00.0: Adding to iommu group 4
[   11.446159] pcieport 0000:00:00.0: PME: Signaling with IRQ 373
[   11.452239] pcieport 0001:00:00.0: Linked as a consumer to 5000000.iommu
[   11.458966] pcieport 0001:00:00.0: Adding to iommu group 5
[   11.464596] pcieport 0001:00:00.0: PME: Signaling with IRQ 374
[   11.470641] pcieport 0002:00:00.0: Linked as a consumer to 5000000.iommu
[   11.477367] pcieport 0002:00:00.0: Adding to iommu group 6
[   11.482993] pcieport 0002:00:00.0: PME: Signaling with IRQ 375
[   11.489041] pcieport 0003:00:00.0: Linked as a consumer to 5000000.iommu
[   11.495773] pcieport 0003:00:00.0: Adding to iommu group 7
[   11.501412] pcieport 0003:00:00.0: PME: Signaling with IRQ 376
[   11.513888] sja1105 spi1.0: Probed switch chip: SJA1110A
[   11.525754] sja1105 spi1.2: Probed switch chip: SJA1110A
[   11.531850] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpmcp.16
[   11.572125] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.89
[   11.580360] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.15
[   11.586997] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.88
[   11.595226] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.14
[   11.601865] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.87
[   11.610091] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.13
[   11.616727] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.86
[   11.624962] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.12
[   11.631601] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.85
[   11.639836] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.11
[   11.646471] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.84
[   11.654699] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.10
[   11.661335] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.83
[   11.669566] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.9
[   11.676122] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.82
[   11.684354] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.8
[   11.690903] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.81
[   11.699130] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.7
[   11.705679] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.80
[   11.713906] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.6
[   11.720458] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.79
[   11.728689] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.5
[   11.735240] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.78
[   11.743467] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.4
[   11.750017] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.77
[   11.758243] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.3
[   11.764795] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.76
[   11.773027] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.2
[   11.779577] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.75
[   11.787809] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.1
[   11.794360] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.74
[   11.802586] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.0
[   11.809207] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpbp.5
[   11.891236] fsl_dpaa2_eth dpni.5: Probed interface eth0
[   11.897192] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpmcp.15
[   11.938466] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.73
[   11.946693] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.15
[   11.953329] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.72
[   11.961562] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.14
[   11.968203] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.71
[   11.976448] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.13
[   11.983088] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.70
[   11.991316] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.12
[   11.997956] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.69
[   12.006184] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.11
[   12.012821] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.68
[   12.021064] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.10
[   12.027700] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.67
[   12.035933] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.9
[   12.042482] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.66
[   12.050707] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.8
[   12.057256] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.65
[   12.065488] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.7
[   12.072044] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.64
[   12.080276] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.6
[   12.086828] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.63
[   12.095056] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.5
[   12.101607] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.62
[   12.109834] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.4
[   12.116389] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.61
[   12.124621] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.3
[   12.131169] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.60
[   12.139394] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.2
[   12.145950] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.59
[   12.154177] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.1
[   12.160726] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.58
[   12.168958] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.0
[   12.175564] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpbp.4
[   12.256556] fsl_dpaa2_eth dpni.4: Probed interface eth1
[   12.262495] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpmcp.14
[   12.302777] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.57
[   12.311005] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.15
[   12.317643] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.56
[   12.325868] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.14
[   12.332506] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.55
[   12.340737] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.13
[   12.347376] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.54
[   12.355605] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.12
[   12.362245] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.53
[   12.370474] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.11
[   12.377113] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.52
[   12.385346] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.10
[   12.391989] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.51
[   12.400223] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.9
[   12.406779] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.50
[   12.415007] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.8
[   12.421556] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.49
[   12.429791] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.7
[   12.436340] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.48
[   12.444573] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.6
[   12.451121] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.47
[   12.459351] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.5
[   12.465904] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.46
[   12.474133] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.4
[   12.480684] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.45
[   12.488918] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.3
[   12.495467] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.44
[   12.503693] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.2
[   12.510243] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.43
[   12.518469] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.1
[   12.525018] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.42
[   12.533259] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.0
[   12.539873] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpbp.3
[   12.619556] fsl_dpaa2_eth dpni.3 (unnamed net_device) (uninitialized): PHY [0x0000000008b97000:00] driver [Generic Clause 45 PHY] (irq=POLL)
[   12.633450] fsl_dpaa2_eth dpni.3: Probed interface eth2
[   12.639402] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpmcp.13
[   12.679670] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.41
[   12.687902] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.15
[   12.694539] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.40
[   12.702766] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.14
[   12.709403] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.39
[   12.717633] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.13
[   12.724275] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.38
[   12.732505] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.12
[   12.739142] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.37
[   12.747370] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.11
[   12.754007] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.36
[   12.762237] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.10
[   12.768871] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.35
[   12.777105] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.9
[   12.783653] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.34
[   12.791885] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.8
[   12.798434] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.33
[   12.806661] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.7
[   12.813210] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.32
[   12.821443] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.6
[   12.827993] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.31
[   12.836224] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.5
[   12.842772] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.30
[   12.851000] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.4
[   12.857561] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.29
[   12.865793] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.3
[   12.872341] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.28
[   12.880573] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.2
[   12.887121] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.27
[   12.895348] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.1
[   12.901896] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.26
[   12.910124] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.0
[   12.916729] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpbp.2
[   12.996910] fsl_dpaa2_eth dpni.2 (unnamed net_device) (uninitialized): PHY [0x0000000008b97000:08] driver [Generic Clause 45 PHY] (irq=POLL)
[   13.010772] fsl_dpaa2_eth dpni.2: Probed interface eth3
[   13.016718] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.12
[   13.056991] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.25
[   13.065225] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.15
[   13.071869] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.24
[   13.080105] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.14
[   13.086741] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.23
[   13.094968] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.13
[   13.101605] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.22
[   13.109831] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.12
[   13.116466] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.21
[   13.124701] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.11
[   13.131340] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.20
[   13.139568] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.10
[   13.146206] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.19
[   13.154435] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.9
[   13.160988] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.18
[   13.169221] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.8
[   13.175781] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.17
[   13.184011] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.7
[   13.190563] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.16
[   13.198794] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.6
[   13.205346] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.15
[   13.213578] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.5
[   13.220136] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.14
[   13.228368] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.4
[   13.234919] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.13
[   13.243144] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.3
[   13.249692] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.12
[   13.257917] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.2
[   13.264465] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.11
[   13.272699] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.1
[   13.279252] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.10
[   13.287497] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.0
[   13.294110] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpbp.1
[   13.422401] fsl_dpaa2_eth dpni.1: Dropping the link to dpbp.1
[   13.428201] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.0
[   13.433981] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.10
[   13.439932] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.1
[   13.445710] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.11
[   13.451659] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.2
[   13.457435] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.12
[   13.463383] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.3
[   13.469156] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.13
[   13.475103] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.4
[   13.480881] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.14
[   13.486827] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.5
[   13.492601] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.15
[   13.498550] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.6
[   13.504327] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.16
[   13.510276] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.7
[   13.516051] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.17
[   13.522001] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.8
[   13.527782] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.18
[   13.533730] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.9
[   13.539507] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.19
[   13.545457] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.10
[   13.551317] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.20
[   13.557264] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.11
[   13.563127] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.21
[   13.569078] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.12
[   13.574938] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.22
[   13.580885] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.13
[   13.586748] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.23
[   13.592698] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.14
[   13.598559] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.24
[   13.604506] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.15
[   13.610377] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.25
[   13.616326] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.12
[   13.623084] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpmcp.11
[   13.663371] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.9
[   13.671516] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.15
[   13.678155] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.8
[   13.686298] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.14
[   13.692934] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.7
[   13.701077] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.13
[   13.707712] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.6
[   13.715856] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.12
[   13.722489] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.5
[   13.730628] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.11
[   13.737264] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.4
[   13.745408] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.10
[   13.752048] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.3
[   13.760191] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.9
[   13.766739] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.2
[   13.774878] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.8
[   13.781427] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.1
[   13.789572] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.7
[   13.796126] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.0
[   13.804270] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.6
[   13.810817] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.95
[   13.819043] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.5
[   13.825592] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.94
[   13.833821] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.4
[   13.840369] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.93
[   13.848599] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.3
[   13.855147] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.92
[   13.863375] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.2
[   13.869927] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.91
[   13.878161] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.1
[   13.884712] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.90
[   13.892942] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.0
[   13.899560] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpbp.0
[   13.979724] fsl_dpaa2_eth dpni.0 (unnamed net_device) (uninitialized): PHY [0x0000000008b96000:08] driver [Generic Clause 45 PHY] (irq=POLL)
[   13.993607] fsl_dpaa2_eth dpni.0: Probed interface eth4
[   14.005432] sja1105 spi1.0: Probed switch chip: SJA1110A
[   14.011175] sja1105 spi1.0: Dropping the link to 0x0000000008b96000:01
[   14.024328] sja1105 spi1.2: Probed switch chip: SJA1110A
[   14.033399] sja1105 spi1.0: H topology detected, cutting RX from DSA link 4 to CPU port 2 to prevent TX packet loops
[   14.288275] libphy: SJA1105 PCS MDIO bus: probed
[   14.445009] libphy: SJA1110 100base-T1 MDIO bus: probed
[   14.950047] sja1105 spi1.2: H topology detected, cutting RX from DSA link 1 to CPU port 2 to prevent TX packet loops
[   15.204881] libphy: SJA1105 PCS MDIO bus: probed
[   15.361009] libphy: SJA1110 100base-T1 MDIO bus: probed
[   15.874265] random: crng init done
[   15.937311] sja1105 spi1.0: configuring for fixed/rgmii-id link mode
[   15.955906] sja1105 spi1.0: Link is Up - 1Gbps/Full - flow control off
[   16.004458] sja1105 spi1.0 1ge_p1 (uninitialized): PHY [0x0000000008b96000:01] driver [Qualcomm Atheros AR8035] (irq=POLL)
[   16.026114] sja1105 spi1.0: configuring for fixed/sgmii link mode
[   16.115819] sja1105 spi1.0: Link is Up - 1Gbps/Full - flow control off
[   16.231439] sja1105 spi1.0 trx1 (uninitialized): PHY [spi1.0-base-t1:01] driver [NXP C45 TJA1103] (irq=POLL)
[   16.365831] sja1105 spi1.0 trx2 (uninitialized): PHY [spi1.0-base-t1:02] driver [NXP C45 TJA1103] (irq=POLL)
[   16.500214] sja1105 spi1.0 trx3 (uninitialized): PHY [spi1.0-base-t1:03] driver [NXP C45 TJA1103] (irq=POLL)
[   16.634609] sja1105 spi1.0 trx4 (uninitialized): PHY [spi1.0-base-t1:04] driver [NXP C45 TJA1103] (irq=POLL)
[   16.768983] sja1105 spi1.0 trx5 (uninitialized): PHY [spi1.0-base-t1:05] driver [NXP C45 TJA1103] (irq=POLL)
[   16.903366] sja1105 spi1.0 trx6 (uninitialized): PHY [spi1.0-base-t1:06] driver [NXP C45 TJA1103] (irq=POLL)
[   16.929408] sja1105 spi1.2: configuring for fixed/sgmii link mode
[   17.010299] sja1105 spi1.2: configuring for fixed/rgmii-id link mode
[   17.021336] sja1105 spi1.2: Link is Up - 1Gbps/Full - flow control off
[   17.033629] sja1105 spi1.2: Link is Up - 1Gbps/Full - flow control off
[   17.076457] sja1105 spi1.2 1ge_p2 (uninitialized): PHY [0x0000000008b96000:02] driver [Qualcomm Atheros AR8035] (irq=POLL)
[   17.212704] sja1105 spi1.2 trx7 (uninitialized): PHY [spi1.2-base-t1:01] driver [NXP C45 TJA1103] (irq=POLL)
[   17.347049] sja1105 spi1.2 trx8 (uninitialized): PHY [spi1.2-base-t1:02] driver [NXP C45 TJA1103] (irq=POLL)
[   17.481461] sja1105 spi1.2 trx9 (uninitialized): PHY [spi1.2-base-t1:03] driver [NXP C45 TJA1103] (irq=POLL)
[   17.615822] sja1105 spi1.2 trx10 (uninitialized): PHY [spi1.2-base-t1:04] driver [NXP C45 TJA1103] (irq=POLL)
[   17.750237] sja1105 spi1.2 trx11 (uninitialized): PHY [spi1.2-base-t1:05] driver [NXP C45 TJA1103] (irq=POLL)
[   17.884602] sja1105 spi1.2 trx12 (uninitialized): PHY [spi1.2-base-t1:06] driver [NXP C45 TJA1103] (irq=POLL)
[   17.895189] sja1105 spi1.0: Linked as a consumer to dpni.4
[   17.900768] sja1105 spi1.2: Linked as a consumer to dpni.5
[   17.906258] DSA: tree 0 setup
[   17.909265] sja1105 spi1.2: Dropping the link to 0x0000000008b96000:02
[   17.916595] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.10
[   17.956882] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.25
[   17.965118] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.15
[   17.971757] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.24
[   17.979993] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.14
[   17.986631] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.23
[   17.994860] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.13
[   18.001500] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.22
[   18.009733] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.12
[   18.016371] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.21
[   18.024604] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.11
[   18.031244] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.20
[   18.039474] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.10
[   18.046109] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.19
[   18.054339] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.9
[   18.060890] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.18
[   18.069125] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.8
[   18.075676] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.17
[   18.083910] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.7
[   18.090460] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.16
[   18.098689] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.6
[   18.105239] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.15
[   18.113475] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.5
[   18.120030] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.14
[   18.128266] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.4
[   18.134814] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.13
[   18.143043] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.3
[   18.149596] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.12
[   18.157825] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.2
[   18.164374] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.11
[   18.172611] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.1
[   18.179160] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.10
[   18.187389] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.0
[   18.194009] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpbp.1
[   18.321818] fsl_dpaa2_eth dpni.1: Dropping the link to dpbp.1
[   18.327621] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.0
[   18.333400] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.10
[   18.339351] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.1
[   18.345133] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.11
[   18.351081] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.2
[   18.356856] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.12
[   18.362806] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.3
[   18.368583] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.13
[   18.374530] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.4
[   18.380304] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.14
[   18.386254] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.5
[   18.392032] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.15
[   18.397979] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.6
[   18.403755] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.16
[   18.409704] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.7
[   18.415479] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.17
[   18.421427] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.8
[   18.427203] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.18
[   18.433153] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.9
[   18.438927] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.19
[   18.444876] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.10
[   18.450738] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.20
[   18.456689] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.11
[   18.462549] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.21
[   18.468497] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.12
[   18.474359] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.22
[   18.480309] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.13
[   18.486171] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.23
[   18.492118] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.14
[   18.497980] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.24
[   18.503930] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.15
[   18.509798] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.25
[   18.515749] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.10
[   18.522513] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.9
[   18.563705] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.25
[   18.571935] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.15
[   18.578573] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.24
[   18.586799] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.14
[   18.593436] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.23
[   18.601665] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.13
[   18.608302] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.22
[   18.616541] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.12
[   18.623176] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.21
[   18.631404] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.11
[   18.638039] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.20
[   18.646263] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.10
[   18.652896] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.19
[   18.661126] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.9
[   18.667674] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.18
[   18.675904] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.8
[   18.682452] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.17
[   18.690676] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.7
[   18.697222] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.16
[   18.705453] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.6
[   18.712003] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.15
[   18.720232] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.5
[   18.726779] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.14
[   18.735005] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.4
[   18.741557] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.13
[   18.749785] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.3
[   18.756332] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.12
[   18.765561] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.2
[   18.772123] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.11
[   18.780353] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.1
[   18.786901] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.10
[   18.795128] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.0
[   18.801733] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpbp.1
[   18.929911] fsl_dpaa2_eth dpni.1: Dropping the link to dpbp.1
[   18.935705] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.0
[   18.941486] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.10
[   18.947434] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.1
[   18.953213] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.11
[   18.959161] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.2
[   18.964934] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.12
[   18.970882] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.3
[   18.976655] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.13
[   18.982603] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.4
[   18.988377] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.14
[   18.994324] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.5
[   19.000098] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.15
[   19.006046] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.6
[   19.011820] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.16
[   19.017767] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.7
[   19.023540] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.17
[   19.029488] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.8
[   19.035261] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.18
[   19.041210] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.9
[   19.046984] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.19
[   19.052931] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.10
[   19.058790] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.20
[   19.064738] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.11
[   19.070598] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.21
[   19.076545] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.12
[   19.082405] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.22
[   19.088353] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.13
[   19.094212] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.23
[   19.100160] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.14
[   19.106020] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.24
[   19.111968] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.15
[   19.117829] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.25
[   19.123781] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.9
[   19.130149] ALSA device list:
[   19.133136]   No soundcards found.
[   19.341696] EXT4-fs (mmcblk0p2): recovery complete
[   19.347355] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[   19.357138] VFS: Mounted root (ext4 filesystem) on device 179:2.
[   19.363844] devtmpfs: mounted
[   19.369671] Freeing unused kernel memory: 9536K
[   19.374300] Run /sbin/init as init process
[   19.378395]   with arguments:
[   19.381356]     /sbin/init
[   19.384058]   with environment:
[   19.387188]     HOME=/
[   19.389542]     TERM=linux
INIT: version  booting
Fuse filesystem already available.
Mounting fuse control filesystem.
Starting udev
[   19.720832] udevd[480]: starting version 3.2.9
[   19.756177] udevd[481]: starting eudev-3.2.9
[   19.902508] fsl_dpaa2_eth dpni.3 dpmac10: renamed from eth2
[   19.908092] net eth2: renaming to dpmac10
[   19.984284] fsl_dpaa2_eth dpni.4 dpmac17: renamed from eth1
[   19.989997] net eth1: renaming to dpmac17
[   20.020231] fsl_dpaa2_eth dpni.2 dpmac9: renamed from eth3
[   20.025849] net eth3: renaming to dpmac9
[   20.079989] fsl_dpaa2_eth dpni.0 dpmac5: renamed from eth4
[   20.085493] net eth4: renaming to dpmac5
[   20.127996] fsl_dpaa2_eth dpni.5 dpmac18: renamed from eth0
[   20.133603] net eth0: renaming to dpmac18
[   20.166859] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   20.206953] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none.
Starting tcf-agent: OK

lx2160abluebox3 login: root
root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/
total 0
drwxr-xr-x 2 root root 0 Jan  1  1970 .
drwxr-xr-x 4 root root 0 Jan  1  1970 ..
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:00 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:00
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:01 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:01
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:02 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:02
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:08 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:08
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b97000:00 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:00
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b97000:08 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:08
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008c17000:00 -> ../../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000/0x0000000008c17000:00
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008c27000:00 -> ../../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000/0x0000000008c27000:00
lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008c2b000:00 -> ../../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000/0x0000000008c2b000:00
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:01
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:02
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:03
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:04
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:05
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:06
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:01
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:02
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:03
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:04
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:05
lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:06
root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01
spi1.0-base-t1:01/
root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01/
total 0
drwxr-xr-x  4 root root    0 Jan  1  1970 .
drwxr-xr-x 10 root root    0 Jan  1  1970 ..
lrwxrwxrwx  1 root root    0 Mar  9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103
lrwxrwxrwx  1 root root    0 Mar  9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio
@0/ethernet-phy@1
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_dev_flags
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_has_fixups
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_id
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_interface
drwxr-xr-x  2 root root    0 Mar  9 12:35 power
drwxr-xr-x  2 root root    0 Mar  9 12:35 statistics
lrwxrwxrwx  1 root root    0 Mar  9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus
-rw-r--r--  1 root root 4096 Jan  1  1970 uevent
root@lx2160abluebox3:~#
root@lx2160abluebox3:~# ls -la /sys/bus/spi/devices/spi1.0/
total 0
drwxr-xr-x  7 root root    0 Jan  1  1970 .
drwxr-xr-x  6 root root    0 Jan  1  1970 ..
lrwxrwxrwx  1 root root    0 Mar  9 12:36 driver -> ../../../../../../../../../../../bus/spi/drivers/sja1105
-rw-r--r--  1 root root 4096 Mar  9 12:36 driver_override
drwxr-xr-x  4 root root    0 Jan  1  1970 mdio_bus
-r--r--r--  1 root root 4096 Mar  9 12:36 modalias
drwxr-xr-x 10 root root    0 Jan  1  1970 net
lrwxrwxrwx  1 root root    0 Mar  9 12:36 of_node -> ../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0
drwxr-xr-x  2 root root    0 Mar  9 12:36 power
drwxr-xr-x  3 root root    0 Jan  1  1970 ptp
drwxr-xr-x  2 root root    0 Mar  9 12:36 statistics
lrwxrwxrwx  1 root root    0 Jan  1  1970 subsystem -> ../../../../../../../../../../../bus/spi
lrwxrwxrwx  1 root root    0 Mar  9 12:36 supplier:fsl-mc:dpni.4 -> ../../../../../../../../../../virtual/devlink/fsl-mc:dpni.4--spi:spi1.0
-rw-r--r--  1 root root 4096 Jan  1  1970 uevent
INIT: Id "S0" respawning too fast: disabled for 5 minutes                                                                                                                                                   c
lass/mdio_bus/
total 0
drwxr-xr-x  2 root root 0 Jan  1  1970 .
drwxr-xr-x 70 root root 0 Jan  1  1970 ..
lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008b96000 -> ../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000
lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008b97000 -> ../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000
lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008c17000 -> ../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000
lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008c27000 -> ../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000
lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008c2b000 -> ../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000
lrwxrwxrwx  1 root root 0 Jan  1  1970 fixed-0 -> ../../devices/platform/Fixed MDIO bus.0/mdio_bus/fixed-0
lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.0-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1
lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.0-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-pcs
lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.2-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1
lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.2-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-pcs
root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/
total 0
drwxr-xr-x 10 root root    0 Jan  1  1970 .
drwxr-xr-x  4 root root    0 Jan  1  1970 ..
lrwxrwxrwx  1 root root    0 Mar  9 12:37 device -> ../../../spi1.0
lrwxrwxrwx  1 root root    0 Mar  9 12:37 of_node -> ../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio@0
drwxr-xr-x  2 root root    0 Mar  9 12:37 power
drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:01
drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:02
drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:03
drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:04
drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:05
drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:06
drwxr-xr-x  2 root root    0 Mar  9 12:37 statistics
lrwxrwxrwx  1 root root    0 Jan  1  1970 subsystem -> ../../../../../../../../../../../../../class/mdio_bus
-rw-r--r--  1 root root 4096 Jan  1  1970 uevent
-r--r--r--  1 root root 4096 Mar  9 12:37 waiting_for_supplier
root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/spi1.0-base-t1\:01/
total 0
drwxr-xr-x  4 root root    0 Jan  1  1970 .
drwxr-xr-x 10 root root    0 Jan  1  1970 ..
lrwxrwxrwx  1 root root    0 Mar  9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103
lrwxrwxrwx  1 root root    0 Mar  9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio
@0/ethernet-phy@1
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_dev_flags
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_has_fixups
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_id
-r--r--r--  1 root root 4096 Mar  9 12:35 phy_interface
drwxr-xr-x  2 root root    0 Mar  9 12:35 power
drwxr-xr-x  2 root root    0 Mar  9 12:35 statistics
lrwxrwxrwx  1 root root    0 Mar  9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus
-rw-r--r--  1 root root 4096 Jan  1  1970 uevent

As you can see, there was no provider/consumer relationship enforced
between the PHY at spi1.0-base-t1\:01 and the ethernet-switch.

> > >
> > > If it doesn't fix it, can one of you please point me to an upstream
> > > dts (not dtsi) file for a platform in which you see this issue? And
> > > ideally also the DT nodes and their drivers that are involved in this
> > > cycle? With that info, I should be able to root cause this if the
> > > patch above doesn't already fix it.
> >
> > I'm working with a non-upstream dts - maybe Vladimir is using an
> > upstream one? The pattern among the drivers/dts is common between our
> > two cases.
> 
> Ideally, I can get a fully upstream example where this issue is
> happening so that I can look at the actual code that's hitting this
> issue and be sure my analysis is right.

Mine isn't upstream either, but I have no issue sharing it with you.

cat arch/arm64/boot/dts/freescale/fsl-lx2160a-bluebox3.dts

// SPDX-License-Identifier: (GPL-2.0 OR MIT)
//
// Device Tree file for LX2160A BLUEBOX3
//
// Copyright 2020 NXP

/dts-v1/;

#include "fsl-lx2160a.dtsi"

/ {
	model = "NXP Layerscape LX2160ABLUEBOX3";
	compatible = "fsl,lx2160a-rdb", "fsl,lx2160a";

	aliases {
		crypto = &crypto;
		serial0 = &uart0;
		mmc0 = &esdhc0;
		mmc1 = &esdhc1;
	};

	chosen {
		stdout-path = "serial0:115200n8";
	};

	sb_3v3: regulator-sb3v3 {
		compatible = "regulator-fixed";
		regulator-name = "MC34717-3.3VSB";
		regulator-min-microvolt = <3300000>;
		regulator-max-microvolt = <3300000>;
		regulator-boot-on;
		regulator-always-on;
	};
};

&can0 {
	status = "okay";

	can-transceiver {
		max-bitrate = <5000000>;
	};
};

&can1 {
	status = "okay";

	can-transceiver {
		max-bitrate = <5000000>;
	};
};

&crypto {
	status = "okay";
};

&emdio1 {
	status = "okay";
};

&emdio2 {
	status = "okay";
};

&esdhc0 {
	sd-uhs-sdr104;
	sd-uhs-sdr50;
	sd-uhs-sdr25;
	sd-uhs-sdr12;
	status = "okay";
};

&esdhc1 {
	mmc-hs200-1_8v;
	mmc-hs400-1_8v;
	bus-width = <8>;
	status = "okay";
};

&fspi {
	status = "okay";

	mt35xu512aba0: flash@0 {
		#address-cells = <1>;
		#size-cells = <1>;
		compatible = "jedec,spi-nor";
		m25p,fast-read;
		spi-max-frequency = <50000000>;
		reg = <0>;
		spi-rx-bus-width = <8>;
		spi-tx-bus-width = <8>;
	};

	mt35xu512aba1: flash@1 {
		#address-cells = <1>;
		#size-cells = <1>;
		compatible = "jedec,spi-nor";
		m25p,fast-read;
		spi-max-frequency = <50000000>;
		reg = <1>;
		spi-rx-bus-width = <8>;
		spi-tx-bus-width = <8>;
	};
};

&i2c0 {
	status = "okay";

	i2c-mux@77 {
		compatible = "nxp,pca9547";
		reg = <0x77>;
		#address-cells = <1>;
		#size-cells = <0>;

		i2c@1 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0x1>;
			// TODO: Add entries for EMC2305

		};

		i2c@2 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0x2>;

			power-monitor@40 {
				compatible = "ti,ina220";
				reg = <0x40>;
				shunt-resistor = <500>;
			};
		};

		i2c@3 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0x3>;

			temperature-sensor@4c {
				compatible = "nxp,sa56004";
				reg = <0x4c>;
				vcc-supply = <&sb_3v3>;
			};

			temperature-sensor@48 {
				compatible = "nxp,sa56004";
				reg = <0x48>;
				vcc-supply = <&sb_3v3>;
			};
		};

		i2c@4 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0x4>;

			rtc@51 {
				compatible = "nxp,pcf2129";
				reg = <0x51>;
				// IRQ10_B
				interrupts = <0 150 0x4>;
			};
		};

		i2c@7 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0x7>;

			i2c-mux@75 {
				compatible = "nxp,pca9547";
				reg = <0x75>;
				#address-cells = <1>;
				#size-cells = <0>;

				i2c@0 {
					#address-cells = <1>;
					#size-cells = <0>;
					reg = <0x0>;

					spi_bridge: spi@28 {
						compatible = "nxp,sc18is602b";
						reg = <0x28>;
						#address-cells = <1>;
						#size-cells = <0>;
					};
				};
			};
		};
	};
};

&i2c5 {
	status = "okay";

	i2c-mux@77 {
		compatible = "nxp,pca9846";
		reg = <0x77>;
		#address-cells = <1>;
		#size-cells = <0>;
		// TODO: Add entrier for device connected to mux
	};
};

&spi_bridge {
	/* SW1 */
	ethernet-switch@0 {
		compatible = "nxp,sja1110a";
		reg = <0>;
		spi-max-frequency = <4000000>;
		spi-cpol;
		dsa,member = <0 0>;

		ethernet-ports {
			#address-cells = <1>;
			#size-cells = <0>;

			/* Microcontroller port */
			port@0 {
				reg = <0>;
				status = "disabled";
			};

			/* SW1_P1 */
			port@1 {
				reg = <1>;
				label = "con_2x20";
				phy-mode = "sgmii";

				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};

			port@2 {
				reg = <2>;
				ethernet = <&dpmac17>;
				phy-mode = "rgmii-id";

				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};

			port@3 {
				reg = <3>;
				label = "1ge_p1";
				phy-mode = "rgmii-id";
				phy-handle = <&sw1_mii3_phy>;
			};

			sw1p4: port@4 {
				reg = <4>;
				link = <&sw2p1>;
				phy-mode = "sgmii";

				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};

			port@5 {
				reg = <5>;
				label = "trx1";
				phy-mode = "internal";
				phy-handle = <&sw1_port5_base_t1_phy>;
			};

			port@6 {
				reg = <6>;
				label = "trx2";
				phy-mode = "internal";
				phy-handle = <&sw1_port6_base_t1_phy>;
			};

			port@7 {
				reg = <7>;
				label = "trx3";
				phy-mode = "internal";
				phy-handle = <&sw1_port7_base_t1_phy>;
			};

			port@8 {
				reg = <8>;
				label = "trx4";
				phy-mode = "internal";
				phy-handle = <&sw1_port8_base_t1_phy>;
			};

			port@9 {
				reg = <9>;
				label = "trx5";
				phy-mode = "internal";
				phy-handle = <&sw1_port9_base_t1_phy>;
			};

			port@a {
				reg = <10>;
				label = "trx6";
				phy-mode = "internal";
				phy-handle = <&sw1_port10_base_t1_phy>;
			};
		};

		mdios {
			#address-cells = <1>;
			#size-cells = <0>;

			mdio@0 {
				reg = <0>;
				compatible = "nxp,sja1110-base-t1-mdio";
				#address-cells = <1>;
				#size-cells = <0>;

				sw1_port5_base_t1_phy: ethernet-phy@1 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x1>;
				};

				sw1_port6_base_t1_phy: ethernet-phy@2 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x2>;
				};

				sw1_port7_base_t1_phy: ethernet-phy@3 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x3>;
				};

				sw1_port8_base_t1_phy: ethernet-phy@4 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x4>;
				};

				sw1_port9_base_t1_phy: ethernet-phy@5 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x5>;
				};

				sw1_port10_base_t1_phy: ethernet-phy@6 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x6>;
				};
			};
		};
	};

	/* SW2 */
	ethernet-switch@2 {
		compatible = "nxp,sja1110a";
		reg = <2>;
		spi-max-frequency = <4000000>;
		spi-cpol;
		dsa,member = <0 1>;

		ethernet-ports {
			#address-cells = <1>;
			#size-cells = <0>;

			/* Microcontroller port */
			port@0 {
				reg = <0>;
				status = "disabled";
			};

			sw2p1: port@1 {
				reg = <1>;
				link = <&sw1p4>;
				phy-mode = "sgmii";

				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};

			port@2 {
				reg = <2>;
				ethernet = <&dpmac18>;
				phy-mode = "rgmii-id";

				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};

			port@3 {
				reg = <3>;
				label = "1ge_p2";
				phy-mode = "rgmii-id";
				phy-handle = <&sw2_mii3_phy>;
			};

			port@4 {
				reg = <4>;
				label = "to_sw3";
				phy-mode = "2500base-x";

				fixed-link {
					speed = <2500>;
					full-duplex;
				};
			};

			port@5 {
				reg = <5>;
				label = "trx7";
				phy-mode = "internal";
				phy-handle = <&sw2_port5_base_t1_phy>;
			};

			port@6 {
				reg = <6>;
				label = "trx8";
				phy-mode = "internal";
				phy-handle = <&sw2_port6_base_t1_phy>;
			};

			port@7 {
				reg = <7>;
				label = "trx9";
				phy-mode = "internal";
				phy-handle = <&sw2_port7_base_t1_phy>;
			};

			port@8 {
				reg = <8>;
				label = "trx10";
				phy-mode = "internal";
				phy-handle = <&sw2_port8_base_t1_phy>;
			};

			port@9 {
				reg = <9>;
				label = "trx11";
				phy-mode = "internal";
				phy-handle = <&sw2_port9_base_t1_phy>;
			};

			port@a {
				reg = <10>;
				label = "trx12";
				phy-mode = "internal";
				phy-handle = <&sw2_port10_base_t1_phy>;
			};
		};

		mdios {
			#address-cells = <1>;
			#size-cells = <0>;

			mdio@0 {
				reg = <0>;
				compatible = "nxp,sja1110-base-t1-mdio";
				#address-cells = <1>;
				#size-cells = <0>;

				sw2_port5_base_t1_phy: ethernet-phy@1 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x1>;
				};

				sw2_port6_base_t1_phy: ethernet-phy@2 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x2>;
				};

				sw2_port7_base_t1_phy: ethernet-phy@3 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x3>;
				};

				sw2_port8_base_t1_phy: ethernet-phy@4 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x4>;
				};

				sw2_port9_base_t1_phy: ethernet-phy@5 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x5>;
				};

				sw2_port10_base_t1_phy: ethernet-phy@6 {
					compatible = "ethernet-phy-ieee802.3-c45";
					reg = <0x6>;
				};
			};
		};
	};
};

&uart0 {
	status = "okay";
};

&uart1 {
	status = "okay";
};

&usb0 {
	status = "okay";
};

&emdio1 {
	aquantia_phy1: ethernet-phy@0 {
		compatible = "ethernet-phy-ieee802.3-c45";
		reg = <0x0>;
	};

	aquantia_phy2: ethernet-phy@8 {
		compatible = "ethernet-phy-ieee802.3-c45";
		reg = <0x8>;
	};

	sw1_mii3_phy: ethernet-phy@1 {
		/* AR8035 */
		compatible = "ethernet-phy-id004d.d072";
		reg = <0x5>;
	};

	sw2_mii3_phy: ethernet-phy@2 {
		/* AR8035 */
		compatible = "ethernet-phy-id004d.d072";
		reg = <0x6>;
	};
};

&emdio2 {
	aquantia_phy3: ethernet-phy@0 {
		compatible = "ethernet-phy-ieee802.3-c45";
		reg = <0x0>;
	};

	aquantia_phy4: ethernet-phy@8 {
		compatible = "ethernet-phy-ieee802.3-c45";
		reg = <0x8>;
	};
};

&dpmac5 {
	phy-handle = <&aquantia_phy2>;
	phy-connection-type = "usxgmii";
	managed = "in-band-status";
};

&dpmac6 {
	phy-handle = <&aquantia_phy1>;
	phy-connection-type = "usxgmii";
	managed = "in-band-status";
};

&dpmac9 {
	phy-handle = <&aquantia_phy4>;
	phy-connection-type = "usxgmii";
	managed = "in-band-status";
};

&dpmac10 {
	phy-handle = <&aquantia_phy3>;
	phy-connection-type = "usxgmii";
	managed = "in-band-status";
};

&dpmac17 {
	phy-connection-type = "rgmii";
	status = "okay";

	fixed-link {
		speed = <1000>;
		full-duplex;
	};
};

&dpmac18 {
	phy-connection-type = "rgmii";
	status = "okay";

	fixed-link {
		speed = <1000>;
		full-duplex;
	};
};

&pcs_mdio5 {
	status = "okay";
};

&pcs_mdio5 {
	status = "okay";
};

&pcs_mdio9 {
	status = "okay";
};

&pcs_mdio10 {
	status = "okay";
};

> 2. The child devices could defer probe because one of their suppliers
> isn't ready yet. Either because of fw_devlink=on or the framework
> itself returning -EPROBE_DEFER.
> 3. The child devices could be getting probed asynchronously. So the
> device_add() would kick off a thread to probe the child devices in a
> separate thread.
> 
> (2) is what is happening in this case. fw_devlink=on sees that
> "switch" implements the "switch_intc" and "switch" hasn't finished
> probing yet. So it has no way of knowing that switch_intc is actually
> ready. And even if switch_intc was registered as part of switch's
> probe() by the time the PHYs are added, switch_intc could get
> deregistered if the probe fails at a later point. So until probe()
> returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> is ready. Which is good in general because you won't have to
> forcefully unbind (if that is even handled correctly in the first
> place) the consumers of a device if it fails probe() half way through
> registering a few services.

I think you're right and this is the difference between my setup and
Alvin's, I don't have any sort of interrupts for the internal PHYs.

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Andrew Lunn]

> (2) is what is happening in this case. fw_devlink=on sees that
> "switch" implements the "switch_intc" and "switch" hasn't finished
> probing yet. So it has no way of knowing that switch_intc is actually
> ready. And even if switch_intc was registered as part of switch's
> probe() by the time the PHYs are added, switch_intc could get
> deregistered if the probe fails at a later point. So until probe()
> returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> is ready. Which is good in general because you won't have to
> forcefully unbind (if that is even handled correctly in the first
> place) the consumers of a device if it fails probe() half way through
> registering a few services.

There are actually a few different circular references with the way
switches work. Take for example:

&fec1 {
        phy-mode = "rmii";
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_fec1>;
        status = "okay";

        fixed-link {
                speed = <100>;
                full-duplex;
        };

        mdio1: mdio {
                #address-cells = <1>;
                #size-cells = <0>;
                clock-frequency = <12500000>;
                suppress-preamble;
                status = "okay";

                switch0: switch0@0 {
                        compatible = "marvell,mv88e6190";
                        pinctrl-0 = <&pinctrl_gpio_switch0>;
                        pinctrl-names = "default";
                        reg = <0>;
                        eeprom-length = <65536>;
                        interrupt-parent = <&gpio3>;
                        interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
                        interrupt-controller;
                        #interrupt-cells = <2>;

                        ports {
                                #address-cells = <1>;
                                #size-cells = <0>;

                                port@0 {
                                        reg = <0>;
                                        label = "cpu";
                                        ethernet = <&fec1>;

                                        fixed-link {
                                                speed = <100>;
                                                full-duplex;
                                        };
                                };

FEC is an ethernet controller. It has an MDIO bus, and on the bus is
an Ethernet switch. port 0 of the Ethernet switch is connected to the
FEC ethernet controller.

While the FEC probes, it will at some point register its MDIO bus. At
that point, the MDIO bus is probed, the switch is found, and
registered with the switch core. The switch core looks for the port
with an ethernet property and goes looking for that ethernet
interface. But that this point in time, the FEC probe has only got as
far as registering the MDIO bus. The interface itself is not
registered. So finding the interface fails, and we go into
EPROBE_DEFER for probing the switch.

It is pretty hard to solve. An Ethernet interface can be used by the
kernel itself, e.g. NFS root. At the point you call register_netdev()
in the probe function, to register the interface with the core, it
needs to be fully ready to go.  The networking stack can start using
the interface before register_netdev() even returns. So you cannot
first register the interface and then register the MDIO bus.

I once looked to see if it was possible to tell the driver core to not
even bother probing a bus as soon as it is registered, go straight to
defer probe handling. Because this is one case we know it cannot
work. But it does not seem possible.

      Andrew

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

Hi Saravana,

Thanks for your lucid analysis, it's very much appreciated. Please find 
my replies inline - but in summary, I don't think there is anything more 
I can ask of you right now.

On 8/19/21 5:28 AM, Saravana Kannan wrote:
> On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>>
>> Hi Saravana,
>>
>> On 8/18/21 4:46 AM, Saravana Kannan wrote:
>>> On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
>>>>
>>>> Hi Alvin,
>>>>
>>>> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
>>>>> I have an observation that's slightly out of the scope of your patch,
>>>>> but I'll post here on the off chance that you find it relevant.
>>>>> Apologies if it's out of place.
>>>>>
>>>>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
>>>>> use the Generic PHY driver?
>>>>
>>>> They refuse to probe at all with the Generic PHY driver. I have been
>>>> caught off guard a few times now when I had a kernel built with
>>>> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
>>>>
>>>>> If the former is the case, do you experience that the PHY driver fails
>>>>> to get probed during mdiobus registration if the kernel uses
>>>>> fw_devlink=on?
>>>>
>>>> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
>>>> time I do it. It behaves exactly as you say.
>>>>
>>>>>
>>>>> In my case I am writing a new subdriver for realtek-smi, a DSA driver
>>>>> which registers an internal MDIO bus analogously to sja1105, which is
>>>>> why I'm asking. I noticed a deferred probe of the PHY driver because the
>>>>> supplier (ethernet-switch) is not ready - presumably because all of this
>>>>> is happening in the probe of the switch driver. See below:
>>>>>
>>>>> [   83.653213] device_add:3270: device: 'SMI-0': device_add
>>>>> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
>>>>> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
>>>>> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
>>>>> [   83.654291] libphy: SMI slave MII: probed
>>>>> ...
>>>>> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
>>>>> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
>>>>> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
>>>>> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
>>>>> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
>>>>> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
>>>>> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
>>>>> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
>>>>> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
>>>>> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
>>>>
>>>> So it's a circular dependency? Switch cannot finish probing because it
>>>> cannot connect to PHY, which cannot probe because switch has not
>>>> finished probing, which....
>>>
>>> Hi Vladimir/Alvin,
>>>
>>> If there's a cyclic dependency between two devices, then fw_devlink=on
>>> is smart enough to notice that. Once it notices a cycle, it knows that
>>> it can't tell which one is the real dependency and which one is the
>>> false dependency and so stops enforcing ordering between the devices
>>> in the cycle.
>>>
>>> But fw_devlink doesn't understand all the properties yet. Just most of
>>> them and I'm always trying to add more. So when it only understands
>>> the property that's causing the false dependency but not the property
>>> that causes the real dependency, it can cause issues like this where
>>> fw_devlink=on enforces the false dependency and the driver/code
>>> enforces the real dependency. These are generally easy to fix -- you
>>> just need to teach fw_devlink how to parse more properties.
>>>
>>> This is just a preliminary analysis since I don't have all the info
>>> yet -- so I could be wrong. With that said, I happened to be working
>>> on adding fw_devlink support for phy-handle property and I think it
>>> should fix your issue with fw_devlink=on. Can you give [1] a shot?
>>
>> I tried [1] but it did not seem to have any effect.
>>
>>>
>>> If it doesn't fix it, can one of you please point me to an upstream
>>> dts (not dtsi) file for a platform in which you see this issue? And
>>> ideally also the DT nodes and their drivers that are involved in this
>>> cycle? With that info, I should be able to root cause this if the
>>> patch above doesn't already fix it.
>>
>> I'm working with a non-upstream dts - maybe Vladimir is using an
>> upstream one? The pattern among the drivers/dts is common between our
>> two cases.
> 
> Ideally, I can get a fully upstream example where this issue is
> happening so that I can look at the actual code that's hitting this
> issue and be sure my analysis is right.

Due to NDA issues at work, I am unable to publish the DTS right now. But 
based on your analysis below, I believe that the problem I am 
experiencing is exactly for the reasons you describe.

> 
>>
>> But for the sake of this discussion, my dts is pretty much the same as
>> what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The
>> nodes of interest from that dts file are below, and the driver is in
>> drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the
>> Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part
>> of the mdiobus registration, but that never happens. See my previous
>> reply for a debug log.
> 
> Your DTS might be similar to this, but the driver code also matters
> for me to be sure. Anyway, I took a look at this, but my analysis
> below is going to be sketchy because I'm not looking at the actual
> code that's reproducing this issue.

The driver code I am working with is pretty much the same in the probe 
path (the driver is modelled on rtl8366rb.c and hooks into 
realtek-smi-core.c), so it was not a waste of your time to analyze the 
upstream case. Thanks a lot.

> 
> Assuming this issue actually happens with the example you pointed to
> (I don't know this yet), here's what is happening:
> 
> The main problem is that the parent device switch seems to be assuming
> it's child/grandchild devices (mdiobus/PHYs) will have probed
> successfully as soon as they are added. This assumption is not true
> and can be broken for multiple reasons such as:
> 
> 1. The driver for the child devices (PHYs in this case) could be
> loaded as a module after the parent (switch) is probed. So when the
> devices are added, the PHYs would not be probed.
> 2. The child devices could defer probe because one of their suppliers
> isn't ready yet. Either because of fw_devlink=on or the framework
> itself returning -EPROBE_DEFER.
> 3. The child devices could be getting probed asynchronously. So the
> device_add() would kick off a thread to probe the child devices in a
> separate thread.
> 
> (2) is what is happening in this case. fw_devlink=on sees that
> "switch" implements the "switch_intc" and "switch" hasn't finished
> probing yet. So it has no way of knowing that switch_intc is actually
> ready. And even if switch_intc was registered as part of switch's
> probe() by the time the PHYs are added, switch_intc could get
> deregistered if the probe fails at a later point. So until probe()
> returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> is ready. Which is good in general because you won't have to
> forcefully unbind (if that is even handled correctly in the first
> place) the consumers of a device if it fails probe() half way through
> registering a few services.

Right, that makes perfect sense.

> 
> I don't fully understand the networking frameworks, but I think
> Vladimir might have a point in his earlier reply [1]. If you can make
> the switch driver not assume its child PHYs are ready during the
> switch's probe and instead have the switch check if the PHYs are ready
> when the switch is "opened" that'd be better.

I am pretty new to the DSA subsystem, but it appears to me that DSA is 
making this assumption: I don't think DSA switches have a concept of 
"open", as everything is handled in the dsa_register_switch() call in a 
DSA driver's probe function. And as Vladimir pointed out, this is also 
where phylink_of_phy_connect() is being called. A quick look at the 
other DSA drivers suggests that mv88e6xxx may also suffer from case (2) 
above, so it seems like a more general issue.

> 
> We can come up with hacks that'll delete the dependency that
> fw_devlink=on is trying to enforce, but IMHO the proper fix is to have
> parent drivers not assume child devices will be probed as soon as
> device_add(child) returns. That's not guaranteed at all.

You enumerated a number of reasons why, so yes, I agree that it is not 
necessarily a safe assumption to make.

> 
> Btw, I do know why things work when you do the module load/unload
> thing you mention in [2]. That has to do with some forced deletion of
> dependencies that happens when device_bind_driver() is called when the
> Generic PHY driver is used. The reason for why that's done is kind of
> unrelated to the issue at hand, but the comment for
> device_links_force_bind() should tell you why.

Thanks for clearing that up.

Kind regards,
Alvin

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

Hi Vladimir,

On 8/19/21 1:22 PM, Vladimir Oltean wrote:
> 
> Sorry for the delay, I wanted to reconfirm what I said (hint, I was wrong).

Thanks for the clarification. It also lines up with Saravana's analysis 
for my case.

> 
> In my case, whatever I do, I cannot get the driver core enforce a device
> link between the ethernet-switch and the PHY.
> 
> So I cannot actually see the same issue. What I was seeing was in fact
> stupid testing on my part (it was working with the PHY driver as
> built-in, it was working, then I made it a module, it broke, I forgot to
> switch it back to module, then I thought it's broken while the PHY is
> built-in).

Do you mean to say that you are hitting the "case (1)" that Saravana 
described? See below:

On 8/19/21 5:28 AM, Saravana Kannan wrote:
> The main problem is that the parent device switch seems to be assuming
> it's child/grandchild devices (mdiobus/PHYs) will have probed
> successfully as soon as they are added. This assumption is not true
> and can be broken for multiple reasons such as:
> 
> 1. The driver for the child devices (PHYs in this case) could be
> loaded as a module after the parent (switch) is probed. So when the
> devices are added, the PHYs would not be probed.
> 2. The child devices could defer probe because one of their suppliers
> isn't ready yet. Either because of fw_devlink=on or the framework
> itself returning -EPROBE_DEFER.
> 3. The child devices could be getting probed asynchronously. So the
> device_add() would kick off a thread to probe the child devices in a
> separate thread.

I would think that - in general - it should not matter if the PHY driver 
is built as a module.

Kind regards,
Alvin

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > (2) is what is happening in this case. fw_devlink=on sees that
> > "switch" implements the "switch_intc" and "switch" hasn't finished
> > probing yet. So it has no way of knowing that switch_intc is actually
> > ready. And even if switch_intc was registered as part of switch's
> > probe() by the time the PHYs are added, switch_intc could get
> > deregistered if the probe fails at a later point. So until probe()
> > returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> > is ready. Which is good in general because you won't have to
> > forcefully unbind (if that is even handled correctly in the first
> > place) the consumers of a device if it fails probe() half way through
> > registering a few services.

I had to read your email a couple of times before I understood it. I
think I do now, but apologies if I'm not making sense.

>
> There are actually a few different circular references with the way
> switches work. Take for example:
>
> &fec1 {
>         phy-mode = "rmii";
>         pinctrl-names = "default";
>         pinctrl-0 = <&pinctrl_fec1>;
>         status = "okay";
>
>         fixed-link {
>                 speed = <100>;
>                 full-duplex;
>         };
>
>         mdio1: mdio {
>                 #address-cells = <1>;
>                 #size-cells = <0>;
>                 clock-frequency = <12500000>;
>                 suppress-preamble;
>                 status = "okay";
>
>                 switch0: switch0@0 {
>                         compatible = "marvell,mv88e6190";
>                         pinctrl-0 = <&pinctrl_gpio_switch0>;
>                         pinctrl-names = "default";
>                         reg = <0>;
>                         eeprom-length = <65536>;
>                         interrupt-parent = <&gpio3>;
>                         interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
>                         interrupt-controller;
>                         #interrupt-cells = <2>;
>
>                         ports {
>                                 #address-cells = <1>;
>                                 #size-cells = <0>;
>
>                                 port@0 {
>                                         reg = <0>;
>                                         label = "cpu";
>                                         ethernet = <&fec1>;
>
>                                         fixed-link {
>                                                 speed = <100>;
>                                                 full-duplex;
>                                         };
>                                 };
>
> FEC is an ethernet controller. It has an MDIO bus, and on the bus is
> an Ethernet switch. port 0 of the Ethernet switch is connected to the
> FEC ethernet controller.
>
> While the FEC probes, it will at some point register its MDIO bus. At
> that point, the MDIO bus is probed, the switch is found, and
> registered with the switch core. The switch core looks for the port
> with an ethernet property and goes looking for that ethernet
> interface. But that this point in time, the FEC probe has only got as
> far as registering the MDIO bus. The interface itself is not
> registered. So finding the interface fails, and we go into
> EPROBE_DEFER for probing the switch.

Ok, I understood up to here. Couple of questions:
Is this EPROBE_DEFER causing an issue? Wouldn't the switch then
probe successfully when it's reattempted? And then things work
normally? I don't see what the problem is.

> It is pretty hard to solve. An Ethernet interface can be used by the
> kernel itself, e.g. NFS root. At the point you call register_netdev()
> in the probe function, to register the interface with the core,

Are you using "ethernet interface" and "ethernet controller"
interchangeably? Looking at some other drivers, it looks like the
ethernet controlled (FEC) is what would call register_netdev(). So
what's wrong with that happening if switch0 has not probed
successfully?

> it
> needs to be fully ready to go.  The networking stack can start using
> the interface before register_netdev() even returns. So you cannot
> first register the interface and then register the MDIO bus.
>
> I once looked to see if it was possible to tell the driver core to not
> even bother probing a bus as soon as it is registered, go straight to
> defer probe handling. Because this is one case we know it cannot
> work. But it does not seem possible.

fw_devlink doesn't understand the "ethernet" property. If I add that,
then in the example you state above, switch0's probe won't even be
called until the FEC probe returns. The change is pretty trivial
(pasted below) -- can you try it out and tell me if it does what you
need/want?

-Saravana

+++ b/drivers/of/property.c
@@ -1292,6 +1292,7 @@ DEFINE_SIMPLE_PROP(resets, "resets", "#reset-cells")
 DEFINE_SIMPLE_PROP(leds, "leds", NULL)
 DEFINE_SIMPLE_PROP(backlight, "backlight", NULL)
 DEFINE_SIMPLE_PROP(phy_handle, "phy-handle", NULL)
+DEFINE_SIMPLE_PROP(ethernet, "ethernet", NULL)
 DEFINE_SUFFIX_PROP(regulators, "-supply", NULL)
 DEFINE_SUFFIX_PROP(gpio, "-gpio", "#gpio-cells")

@@ -1381,6 +1382,7 @@ static const struct supplier_bindings
of_supplier_bindings[] = {
        { .parse_prop = parse_leds, },
        { .parse_prop = parse_backlight, },
        { .parse_prop = parse_phy_handle, },
+       { .parse_prop = parse_ethernet, },
        { .parse_prop = parse_gpio_compat, },
        { .parse_prop = parse_interrupts, },
        { .parse_prop = parse_regulators, },

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Vladimir Oltean]

On Thu, Aug 19, 2021 at 04:52:43PM -0700, Saravana Kannan wrote:
> On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote:
> >
> > > (2) is what is happening in this case. fw_devlink=on sees that
> > > "switch" implements the "switch_intc" and "switch" hasn't finished
> > > probing yet. So it has no way of knowing that switch_intc is actually
> > > ready. And even if switch_intc was registered as part of switch's
> > > probe() by the time the PHYs are added, switch_intc could get
> > > deregistered if the probe fails at a later point. So until probe()
> > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> > > is ready. Which is good in general because you won't have to
> > > forcefully unbind (if that is even handled correctly in the first
> > > place) the consumers of a device if it fails probe() half way through
> > > registering a few services.
> 
> I had to read your email a couple of times before I understood it. I
> think I do now, but apologies if I'm not making sense.
> 
> >
> > There are actually a few different circular references with the way
> > switches work. Take for example:
> >
> > &fec1 {
> >         phy-mode = "rmii";
> >         pinctrl-names = "default";
> >         pinctrl-0 = <&pinctrl_fec1>;
> >         status = "okay";
> >
> >         fixed-link {
> >                 speed = <100>;
> >                 full-duplex;
> >         };
> >
> >         mdio1: mdio {
> >                 #address-cells = <1>;
> >                 #size-cells = <0>;
> >                 clock-frequency = <12500000>;
> >                 suppress-preamble;
> >                 status = "okay";
> >
> >                 switch0: switch0@0 {
> >                         compatible = "marvell,mv88e6190";
> >                         pinctrl-0 = <&pinctrl_gpio_switch0>;
> >                         pinctrl-names = "default";
> >                         reg = <0>;
> >                         eeprom-length = <65536>;
> >                         interrupt-parent = <&gpio3>;
> >                         interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
> >                         interrupt-controller;
> >                         #interrupt-cells = <2>;
> >
> >                         ports {
> >                                 #address-cells = <1>;
> >                                 #size-cells = <0>;
> >
> >                                 port@0 {
> >                                         reg = <0>;
> >                                         label = "cpu";
> >                                         ethernet = <&fec1>;
> >
> >                                         fixed-link {
> >                                                 speed = <100>;
> >                                                 full-duplex;
> >                                         };
> >                                 };
> >
> > FEC is an ethernet controller. It has an MDIO bus, and on the bus is
> > an Ethernet switch. port 0 of the Ethernet switch is connected to the
> > FEC ethernet controller.
> >
> > While the FEC probes, it will at some point register its MDIO bus. At
> > that point, the MDIO bus is probed, the switch is found, and
> > registered with the switch core. The switch core looks for the port
> > with an ethernet property and goes looking for that ethernet
> > interface. But that this point in time, the FEC probe has only got as
> > far as registering the MDIO bus. The interface itself is not
> > registered. So finding the interface fails, and we go into
> > EPROBE_DEFER for probing the switch.
> 
> Ok, I understood up to here. Couple of questions:
> Is this EPROBE_DEFER causing an issue? Wouldn't the switch then
> probe successfully when it's reattempted? And then things work
> normally? I don't see what the problem is.

It's not an issue per se, since it's not a fully circular dependency:
the DSA master (the FEC controller) does not have any dependency on the
switch beneath it to probe (there's nothing like a phy-handle from the
FEC to the switch or to something provided by it).

A few EPROBE_DEFER iterations later the switch will finally find its DSA
master fully probed via of_find_net_device_by_node.

Andrew is wondering how to avoid those extra EPROBE_DEFER iterations.
It is weird that the entire functionality of the system depends on those
EPROBE_DEFERs, typically you'd expect that EPROBE_DEFER just serializes
asynchronous probing of drivers with interdependencies. But in this case
it serializes synchronous probing.

> > It is pretty hard to solve. An Ethernet interface can be used by the
> > kernel itself, e.g. NFS root. At the point you call register_netdev()
> > in the probe function, to register the interface with the core,
> 
> Are you using "ethernet interface" and "ethernet controller"
> interchangeably? Looking at some other drivers, it looks like the
> ethernet controlled (FEC) is what would call register_netdev(). So
> what's wrong with that happening if switch0 has not probed
> successfully?

The "interface" and "controller" terms are not really interchangeable,
an interface can also be virtual (stacked interfaces on top of physical
ones, like VLAN or DSA) while network controllers are typically physical
(unless emulated). But that is not of importance.

The context here is that you cannot solve the interdependency by
registering the DSA master (FEC) first, then its MDIO bus second (the
DSA switch probes on the DSA master's MDIO bus => if you do this,
of_find_net_device_by_node from the DSA layer would find its master the
first time). The reason you cannot do that is because you need the MDIO
bus for really basic stuff: you also have your Ethernet PHY on it, and
you need to initialize that in order to send traffic. And you need to be
able to send traffic as soon as register_netdev() completes.

So since the driver initialization sequence has a single written order
regardless of whether DSA switches are attached or not, that order is
picked to be the one where traffic works as soon as register_netdev completes.

> > it
> > needs to be fully ready to go.  The networking stack can start using
> > the interface before register_netdev() even returns. So you cannot
> > first register the interface and then register the MDIO bus.
> >
> > I once looked to see if it was possible to tell the driver core to not
> > even bother probing a bus as soon as it is registered, go straight to
> > defer probe handling. Because this is one case we know it cannot
> > work. But it does not seem possible.
> 
> fw_devlink doesn't understand the "ethernet" property. If I add that,
> then in the example you state above, switch0's probe won't even be
> called until the FEC probe returns. The change is pretty trivial
> (pasted below) -- can you try it out and tell me if it does what you
> need/want?
> 
> -Saravana
> 
> +++ b/drivers/of/property.c
> @@ -1292,6 +1292,7 @@ DEFINE_SIMPLE_PROP(resets, "resets", "#reset-cells")
>  DEFINE_SIMPLE_PROP(leds, "leds", NULL)
>  DEFINE_SIMPLE_PROP(backlight, "backlight", NULL)
>  DEFINE_SIMPLE_PROP(phy_handle, "phy-handle", NULL)
> +DEFINE_SIMPLE_PROP(ethernet, "ethernet", NULL)
>  DEFINE_SUFFIX_PROP(regulators, "-supply", NULL)
>  DEFINE_SUFFIX_PROP(gpio, "-gpio", "#gpio-cells")
> 
> @@ -1381,6 +1382,7 @@ static const struct supplier_bindings
> of_supplier_bindings[] = {
>         { .parse_prop = parse_leds, },
>         { .parse_prop = parse_backlight, },
>         { .parse_prop = parse_phy_handle, },
> +       { .parse_prop = parse_ethernet, },
>         { .parse_prop = parse_gpio_compat, },
>         { .parse_prop = parse_interrupts, },
>         { .parse_prop = parse_regulators, },

I don't have this exact setup to test, so I'll let Andrew do it, but I
have a question: DSA sets up a device link to its master in dsa_master_setup.
It does this to autoremove itself when the DSA master gets removed, but
fundamentally it does it after the entire EPROBE_DEFER shebang discussed
above has already happened. If your patch works, we can drop the manually
added device link, right?

There's also the question of what to do in case of multiple DSA masters
(multiple "ethernet" properties). Right now, if you describe two DSA
masters in the device tree, DSA will pick the first DSA master and use
just that (it doesn't have full support for more than one). But even
though the second DSA master is not used for anything, with your change,
unbinding it will also unbind the switch, will it not?

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Thu, Aug 19, 2021 at 4:23 AM Vladimir Oltean <vladimir.oltean@nxp.com> wrote:
>
> On Wed, Aug 18, 2021 at 08:28:44PM -0700, Saravana Kannan wrote:
> > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> > >
> > > Hi Saravana,
> > >
> > > On 8/18/21 4:46 AM, Saravana Kannan wrote:
> > > > On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
> > > >>
> > > >> Hi Alvin,
> > > >>
> > > >> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
> > > >>> I have an observation that's slightly out of the scope of your patch,
> > > >>> but I'll post here on the off chance that you find it relevant.
> > > >>> Apologies if it's out of place.
> > > >>>
> > > >>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
> > > >>> use the Generic PHY driver?
> > > >>
> > > >> They refuse to probe at all with the Generic PHY driver. I have been
> > > >> caught off guard a few times now when I had a kernel built with
> > > >> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
> > > >>
> > > >>> If the former is the case, do you experience that the PHY driver fails
> > > >>> to get probed during mdiobus registration if the kernel uses
> > > >>> fw_devlink=on?
> > > >>
> > > >> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
> > > >> time I do it. It behaves exactly as you say.
>
> Sorry for the delay, I wanted to reconfirm what I said (hint, I was wrong).
>
> In my case, whatever I do, I cannot get the driver core enforce a device
> link between the ethernet-switch and the PHY.
>
> So I cannot actually see the same issue. What I was seeing was in fact
> stupid testing on my part (it was working with the PHY driver as
> built-in, it was working, then I made it a module, it broke, I forgot to
> switch it back to module, then I thought it's broken while the PHY is
> built-in).

Sorry, this email is too "messy" for me and I joined this thread
halfway through. So if everything is working fine for you with
fw_devlink=on, I'd rather not dig into this. Too much stuff to do.

I'd be nice if you had put the boot logs, the DT file and the shell
command stuff into separate files and attached them. I guess I could
do that myself, but that's going to happen only when I get around to
it :)

>
> > > >>
> > > >>>
> > > >>> In my case I am writing a new subdriver for realtek-smi, a DSA driver
> > > >>> which registers an internal MDIO bus analogously to sja1105, which is
> > > >>> why I'm asking. I noticed a deferred probe of the PHY driver because the
> > > >>> supplier (ethernet-switch) is not ready - presumably because all of this
> > > >>> is happening in the probe of the switch driver. See below:
> > > >>>
> > > >>> [   83.653213] device_add:3270: device: 'SMI-0': device_add
> > > >>> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
> > > >>> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
> > > >>> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
> > > >>> [   83.654291] libphy: SMI slave MII: probed
> > > >>> ...
> > > >>> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
> > > >>> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
> > > >>> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
> > > >>> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
> > > >>> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
> > > >>> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
> > > >>> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
> > > >>> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
> > > >>> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
> > > >>> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
> > > >>
> > > >> So it's a circular dependency? Switch cannot finish probing because it
> > > >> cannot connect to PHY, which cannot probe because switch has not
> > > >> finished probing, which....
> > > >
> > > > Hi Vladimir/Alvin,
> > > >
> > > > If there's a cyclic dependency between two devices, then fw_devlink=on
> > > > is smart enough to notice that. Once it notices a cycle, it knows that
> > > > it can't tell which one is the real dependency and which one is the
> > > > false dependency and so stops enforcing ordering between the devices
> > > > in the cycle.
> > > >
> > > > But fw_devlink doesn't understand all the properties yet. Just most of
> > > > them and I'm always trying to add more. So when it only understands
> > > > the property that's causing the false dependency but not the property
> > > > that causes the real dependency, it can cause issues like this where
> > > > fw_devlink=on enforces the false dependency and the driver/code
> > > > enforces the real dependency. These are generally easy to fix -- you
> > > > just need to teach fw_devlink how to parse more properties.
> > > >
> > > > This is just a preliminary analysis since I don't have all the info
> > > > yet -- so I could be wrong. With that said, I happened to be working
> > > > on adding fw_devlink support for phy-handle property and I think it
> > > > should fix your issue with fw_devlink=on. Can you give [1] a shot?
> > >
> > > I tried [1] but it did not seem to have any effect.
> > >
>
> I applied the phy-handle patch, and here are my boot logs.
>



> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
> [    0.000000] Linux version 5.14.0-rc5+ (tigrisor@skbuf) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #440 SMP Thu Aug 19 13:48:11 EEST 2021
> [    0.000000] Machine model: NXP Layerscape LX2160ABLUEBOX3
> [    0.000000] fw_devlink_setup: fw_devlink_flags = 0x120
> [    0.000000] efi: UEFI not found.
> [    0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!

Snipped the logs

> [   19.389542]     TERM=linux
> INIT: version  booting
> Fuse filesystem already available.
> Mounting fuse control filesystem.
> Starting udev
> [   19.720832] udevd[480]: starting version 3.2.9
> [   19.756177] udevd[481]: starting eudev-3.2.9
> [   19.902508] fsl_dpaa2_eth dpni.3 dpmac10: renamed from eth2
> [   19.908092] net eth2: renaming to dpmac10
> [   19.984284] fsl_dpaa2_eth dpni.4 dpmac17: renamed from eth1
> [   19.989997] net eth1: renaming to dpmac17
> [   20.020231] fsl_dpaa2_eth dpni.2 dpmac9: renamed from eth3
> [   20.025849] net eth3: renaming to dpmac9
> [   20.079989] fsl_dpaa2_eth dpni.0 dpmac5: renamed from eth4
> [   20.085493] net eth4: renaming to dpmac5
> [   20.127996] fsl_dpaa2_eth dpni.5 dpmac18: renamed from eth0
> [   20.133603] net eth0: renaming to dpmac18
> [   20.166859] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
> [   20.206953] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none.
> Starting tcf-agent: OK
>
> lx2160abluebox3 login: root
> root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/
> total 0
> drwxr-xr-x 2 root root 0 Jan  1  1970 .
> drwxr-xr-x 4 root root 0 Jan  1  1970 ..
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:00 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:00
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:01 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:01
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:02 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:02
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b96000:08 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:08
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b97000:00 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:00
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008b97000:08 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:08
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008c17000:00 -> ../../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000/0x0000000008c17000:00
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008c27000:00 -> ../../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000/0x0000000008c27000:00
> lrwxrwxrwx 1 root root 0 Jan  1  1970 0x0000000008c2b000:00 -> ../../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000/0x0000000008c2b000:00
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:01
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:02
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:03
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:04
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:05
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.0-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:06
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:01
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:02
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:03
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:04
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:05
> lrwxrwxrwx 1 root root 0 Jan  1  1970 spi1.2-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:06
> root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01
> spi1.0-base-t1:01/
> root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01/
> total 0
> drwxr-xr-x  4 root root    0 Jan  1  1970 .
> drwxr-xr-x 10 root root    0 Jan  1  1970 ..
> lrwxrwxrwx  1 root root    0 Mar  9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103
> lrwxrwxrwx  1 root root    0 Mar  9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio
> @0/ethernet-phy@1
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_dev_flags
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_has_fixups
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_id
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_interface
> drwxr-xr-x  2 root root    0 Mar  9 12:35 power
> drwxr-xr-x  2 root root    0 Mar  9 12:35 statistics
> lrwxrwxrwx  1 root root    0 Mar  9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus
> -rw-r--r--  1 root root 4096 Jan  1  1970 uevent
> root@lx2160abluebox3:~#
> root@lx2160abluebox3:~# ls -la /sys/bus/spi/devices/spi1.0/
> total 0
> drwxr-xr-x  7 root root    0 Jan  1  1970 .
> drwxr-xr-x  6 root root    0 Jan  1  1970 ..
> lrwxrwxrwx  1 root root    0 Mar  9 12:36 driver -> ../../../../../../../../../../../bus/spi/drivers/sja1105
> -rw-r--r--  1 root root 4096 Mar  9 12:36 driver_override
> drwxr-xr-x  4 root root    0 Jan  1  1970 mdio_bus
> -r--r--r--  1 root root 4096 Mar  9 12:36 modalias
> drwxr-xr-x 10 root root    0 Jan  1  1970 net
> lrwxrwxrwx  1 root root    0 Mar  9 12:36 of_node -> ../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0
> drwxr-xr-x  2 root root    0 Mar  9 12:36 power
> drwxr-xr-x  3 root root    0 Jan  1  1970 ptp
> drwxr-xr-x  2 root root    0 Mar  9 12:36 statistics
> lrwxrwxrwx  1 root root    0 Jan  1  1970 subsystem -> ../../../../../../../../../../../bus/spi
> lrwxrwxrwx  1 root root    0 Mar  9 12:36 supplier:fsl-mc:dpni.4 -> ../../../../../../../../../../virtual/devlink/fsl-mc:dpni.4--spi:spi1.0
> -rw-r--r--  1 root root 4096 Jan  1  1970 uevent
> INIT: Id "S0" respawning too fast: disabled for 5 minutes                                                                                                                                                   c
> lass/mdio_bus/
> total 0
> drwxr-xr-x  2 root root 0 Jan  1  1970 .
> drwxr-xr-x 70 root root 0 Jan  1  1970 ..
> lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008b96000 -> ../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000
> lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008b97000 -> ../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000
> lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008c17000 -> ../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000
> lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008c27000 -> ../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000
> lrwxrwxrwx  1 root root 0 Jan  1  1970 0x0000000008c2b000 -> ../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000
> lrwxrwxrwx  1 root root 0 Jan  1  1970 fixed-0 -> ../../devices/platform/Fixed MDIO bus.0/mdio_bus/fixed-0
> lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.0-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1
> lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.0-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-pcs
> lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.2-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1
> lrwxrwxrwx  1 root root 0 Jan  1  1970 spi1.2-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-pcs
> root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/
> total 0
> drwxr-xr-x 10 root root    0 Jan  1  1970 .
> drwxr-xr-x  4 root root    0 Jan  1  1970 ..
> lrwxrwxrwx  1 root root    0 Mar  9 12:37 device -> ../../../spi1.0
> lrwxrwxrwx  1 root root    0 Mar  9 12:37 of_node -> ../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio@0
> drwxr-xr-x  2 root root    0 Mar  9 12:37 power
> drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:01
> drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:02
> drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:03
> drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:04
> drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:05
> drwxr-xr-x  4 root root    0 Jan  1  1970 spi1.0-base-t1:06
> drwxr-xr-x  2 root root    0 Mar  9 12:37 statistics
> lrwxrwxrwx  1 root root    0 Jan  1  1970 subsystem -> ../../../../../../../../../../../../../class/mdio_bus
> -rw-r--r--  1 root root 4096 Jan  1  1970 uevent
> -r--r--r--  1 root root 4096 Mar  9 12:37 waiting_for_supplier
> root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/spi1.0-base-t1\:01/
> total 0
> drwxr-xr-x  4 root root    0 Jan  1  1970 .
> drwxr-xr-x 10 root root    0 Jan  1  1970 ..
> lrwxrwxrwx  1 root root    0 Mar  9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103
> lrwxrwxrwx  1 root root    0 Mar  9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio
> @0/ethernet-phy@1
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_dev_flags
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_has_fixups
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_id
> -r--r--r--  1 root root 4096 Mar  9 12:35 phy_interface
> drwxr-xr-x  2 root root    0 Mar  9 12:35 power
> drwxr-xr-x  2 root root    0 Mar  9 12:35 statistics
> lrwxrwxrwx  1 root root    0 Mar  9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus
> -rw-r--r--  1 root root 4096 Jan  1  1970 uevent
>
> As you can see, there was no provider/consumer relationship enforced
> between the PHY at spi1.0-base-t1\:01 and the ethernet-switch.

So the only thing that's "not working" right now is that you are
saying the fw_devlink phy-handle patch doesn't seem to have an impact?

To make it easy for me, can you give me the name of this node in DT?
There are too many MDIOs and SPIs and PHY for me to keep track in my
head.

Since this isn't explicitly breaking anything, I'll get around to this
when I have time.

-Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Thu, Aug 19, 2021 at 6:42 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>
> Hi Saravana,
>
> Thanks for your lucid analysis, it's very much appreciated. Please find
> my replies inline - but in summary, I don't think there is anything more
> I can ask of you right now.

You are welcome.

>
> On 8/19/21 5:28 AM, Saravana Kannan wrote:
> > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> >>
> >> Hi Saravana,
> >>
> >> On 8/18/21 4:46 AM, Saravana Kannan wrote:
> >>> On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
> >>>>
> >>>> Hi Alvin,
> >>>>
> >>>> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
> >>>>> I have an observation that's slightly out of the scope of your patch,
> >>>>> but I'll post here on the off chance that you find it relevant.
> >>>>> Apologies if it's out of place.
> >>>>>
> >>>>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
> >>>>> use the Generic PHY driver?
> >>>>
> >>>> They refuse to probe at all with the Generic PHY driver. I have been
> >>>> caught off guard a few times now when I had a kernel built with
> >>>> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
> >>>>
> >>>>> If the former is the case, do you experience that the PHY driver fails
> >>>>> to get probed during mdiobus registration if the kernel uses
> >>>>> fw_devlink=on?
> >>>>
> >>>> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
> >>>> time I do it. It behaves exactly as you say.
> >>>>
> >>>>>
> >>>>> In my case I am writing a new subdriver for realtek-smi, a DSA driver
> >>>>> which registers an internal MDIO bus analogously to sja1105, which is
> >>>>> why I'm asking. I noticed a deferred probe of the PHY driver because the
> >>>>> supplier (ethernet-switch) is not ready - presumably because all of this
> >>>>> is happening in the probe of the switch driver. See below:
> >>>>>
> >>>>> [   83.653213] device_add:3270: device: 'SMI-0': device_add
> >>>>> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
> >>>>> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
> >>>>> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
> >>>>> [   83.654291] libphy: SMI slave MII: probed
> >>>>> ...
> >>>>> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
> >>>>> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
> >>>>> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
> >>>>> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
> >>>>> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
> >>>>> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
> >>>>> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
> >>>>> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
> >>>>> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
> >>>>> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
> >>>>
> >>>> So it's a circular dependency? Switch cannot finish probing because it
> >>>> cannot connect to PHY, which cannot probe because switch has not
> >>>> finished probing, which....
> >>>
> >>> Hi Vladimir/Alvin,
> >>>
> >>> If there's a cyclic dependency between two devices, then fw_devlink=on
> >>> is smart enough to notice that. Once it notices a cycle, it knows that
> >>> it can't tell which one is the real dependency and which one is the
> >>> false dependency and so stops enforcing ordering between the devices
> >>> in the cycle.
> >>>
> >>> But fw_devlink doesn't understand all the properties yet. Just most of
> >>> them and I'm always trying to add more. So when it only understands
> >>> the property that's causing the false dependency but not the property
> >>> that causes the real dependency, it can cause issues like this where
> >>> fw_devlink=on enforces the false dependency and the driver/code
> >>> enforces the real dependency. These are generally easy to fix -- you
> >>> just need to teach fw_devlink how to parse more properties.
> >>>
> >>> This is just a preliminary analysis since I don't have all the info
> >>> yet -- so I could be wrong. With that said, I happened to be working
> >>> on adding fw_devlink support for phy-handle property and I think it
> >>> should fix your issue with fw_devlink=on. Can you give [1] a shot?
> >>
> >> I tried [1] but it did not seem to have any effect.
> >>
> >>>
> >>> If it doesn't fix it, can one of you please point me to an upstream
> >>> dts (not dtsi) file for a platform in which you see this issue? And
> >>> ideally also the DT nodes and their drivers that are involved in this
> >>> cycle? With that info, I should be able to root cause this if the
> >>> patch above doesn't already fix it.
> >>
> >> I'm working with a non-upstream dts - maybe Vladimir is using an
> >> upstream one? The pattern among the drivers/dts is common between our
> >> two cases.
> >
> > Ideally, I can get a fully upstream example where this issue is
> > happening so that I can look at the actual code that's hitting this
> > issue and be sure my analysis is right.
>
> Due to NDA issues at work, I am unable to publish the DTS right now. But
> based on your analysis below, I believe that the problem I am
> experiencing is exactly for the reasons you describe.
>
> >
> >>
> >> But for the sake of this discussion, my dts is pretty much the same as
> >> what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The
> >> nodes of interest from that dts file are below, and the driver is in
> >> drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the
> >> Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part
> >> of the mdiobus registration, but that never happens. See my previous
> >> reply for a debug log.
> >
> > Your DTS might be similar to this, but the driver code also matters
> > for me to be sure. Anyway, I took a look at this, but my analysis
> > below is going to be sketchy because I'm not looking at the actual
> > code that's reproducing this issue.
>
> The driver code I am working with is pretty much the same in the probe
> path (the driver is modelled on rtl8366rb.c and hooks into
> realtek-smi-core.c), so it was not a waste of your time to analyze the
> upstream case. Thanks a lot.

Good to know it's useful.

>
> >
> > Assuming this issue actually happens with the example you pointed to
> > (I don't know this yet), here's what is happening:
> >
> > The main problem is that the parent device switch seems to be assuming
> > it's child/grandchild devices (mdiobus/PHYs) will have probed
> > successfully as soon as they are added. This assumption is not true
> > and can be broken for multiple reasons such as:
> >
> > 1. The driver for the child devices (PHYs in this case) could be
> > loaded as a module after the parent (switch) is probed. So when the
> > devices are added, the PHYs would not be probed.
> > 2. The child devices could defer probe because one of their suppliers
> > isn't ready yet. Either because of fw_devlink=on or the framework
> > itself returning -EPROBE_DEFER.
> > 3. The child devices could be getting probed asynchronously. So the
> > device_add() would kick off a thread to probe the child devices in a
> > separate thread.
> >
> > (2) is what is happening in this case. fw_devlink=on sees that
> > "switch" implements the "switch_intc" and "switch" hasn't finished
> > probing yet. So it has no way of knowing that switch_intc is actually
> > ready. And even if switch_intc was registered as part of switch's
> > probe() by the time the PHYs are added, switch_intc could get
> > deregistered if the probe fails at a later point. So until probe()
> > returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> > is ready. Which is good in general because you won't have to
> > forcefully unbind (if that is even handled correctly in the first
> > place) the consumers of a device if it fails probe() half way through
> > registering a few services.
>
> Right, that makes perfect sense.
>
> >
> > I don't fully understand the networking frameworks, but I think
> > Vladimir might have a point in his earlier reply [1]. If you can make
> > the switch driver not assume its child PHYs are ready during the
> > switch's probe and instead have the switch check if the PHYs are ready
> > when the switch is "opened" that'd be better.
>
> I am pretty new to the DSA subsystem,

I know absolutely nothing about DSA. Heck, I don't even know what it
stands for. It's just about as informative as "XYZ" to me :)

> but it appears to me that DSA is
> making this assumption: I don't think DSA switches have a concept of
> "open", as everything is handled in the dsa_register_switch() call in a
> DSA driver's probe function. And as Vladimir pointed out, this is also
> where phylink_of_phy_connect() is being called.

Yeah, I noticed that part.

> A quick look at the
> other DSA drivers suggests that mv88e6xxx may also suffer from case (2)
> above, so it seems like a more general issue.
>

I obviously want fw_devlink=3Don to be used by everyone and not have
people use fw_devlink=3Dpermissive. In this scenario, I think
fw_devlink=3Don is doing the right thing too. So this is where I'm
hoping the network experts can jump in and fix the general DSA issue
and I can help with the parts I understand.

-Saravana
P.S: Alvin, I accidentally replied only to you. So sending it to the
list again. If you see this email twice, that's why :)


> >
> > We can come up with hacks that'll delete the dependency that
> > fw_devlink=on is trying to enforce, but IMHO the proper fix is to have
> > parent drivers not assume child devices will be probed as soon as
> > device_add(child) returns. That's not guaranteed at all.
>
> You enumerated a number of reasons why, so yes, I agree that it is not
> necessarily a safe assumption to make.
>
> >
> > Btw, I do know why things work when you do the module load/unload
> > thing you mention in [2]. That has to do with some forced deletion of
> > dependencies that happens when device_bind_driver() is called when the
> > Generic PHY driver is used. The reason for why that's done is kind of
> > unrelated to the issue at hand, but the comment for
> > device_links_force_bind() should tell you why.
>
> Thanks for clearing that up.
>
> Kind regards,
> Alvin

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Thu, Aug 19, 2021 at 5:37 PM Vladimir Oltean <olteanv@gmail.com> wrote:
>
> On Thu, Aug 19, 2021 at 04:52:43PM -0700, Saravana Kannan wrote:
> > On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote:
> > >
> > > > (2) is what is happening in this case. fw_devlink=on sees that
> > > > "switch" implements the "switch_intc" and "switch" hasn't finished
> > > > probing yet. So it has no way of knowing that switch_intc is actually
> > > > ready. And even if switch_intc was registered as part of switch's
> > > > probe() by the time the PHYs are added, switch_intc could get
> > > > deregistered if the probe fails at a later point. So until probe()
> > > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> > > > is ready. Which is good in general because you won't have to
> > > > forcefully unbind (if that is even handled correctly in the first
> > > > place) the consumers of a device if it fails probe() half way through
> > > > registering a few services.
> >
> > I had to read your email a couple of times before I understood it. I
> > think I do now, but apologies if I'm not making sense.
> >
> > >
> > > There are actually a few different circular references with the way
> > > switches work. Take for example:
> > >
> > > &fec1 {
> > >         phy-mode = "rmii";
> > >         pinctrl-names = "default";
> > >         pinctrl-0 = <&pinctrl_fec1>;
> > >         status = "okay";
> > >
> > >         fixed-link {
> > >                 speed = <100>;
> > >                 full-duplex;
> > >         };
> > >
> > >         mdio1: mdio {
> > >                 #address-cells = <1>;
> > >                 #size-cells = <0>;
> > >                 clock-frequency = <12500000>;
> > >                 suppress-preamble;
> > >                 status = "okay";
> > >
> > >                 switch0: switch0@0 {
> > >                         compatible = "marvell,mv88e6190";
> > >                         pinctrl-0 = <&pinctrl_gpio_switch0>;
> > >                         pinctrl-names = "default";
> > >                         reg = <0>;
> > >                         eeprom-length = <65536>;
> > >                         interrupt-parent = <&gpio3>;
> > >                         interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
> > >                         interrupt-controller;
> > >                         #interrupt-cells = <2>;
> > >
> > >                         ports {
> > >                                 #address-cells = <1>;
> > >                                 #size-cells = <0>;
> > >
> > >                                 port@0 {
> > >                                         reg = <0>;
> > >                                         label = "cpu";
> > >                                         ethernet = <&fec1>;
> > >
> > >                                         fixed-link {
> > >                                                 speed = <100>;
> > >                                                 full-duplex;
> > >                                         };
> > >                                 };
> > >
> > > FEC is an ethernet controller. It has an MDIO bus, and on the bus is
> > > an Ethernet switch. port 0 of the Ethernet switch is connected to the
> > > FEC ethernet controller.
> > >
> > > While the FEC probes, it will at some point register its MDIO bus. At
> > > that point, the MDIO bus is probed, the switch is found, and
> > > registered with the switch core. The switch core looks for the port
> > > with an ethernet property and goes looking for that ethernet
> > > interface. But that this point in time, the FEC probe has only got as
> > > far as registering the MDIO bus. The interface itself is not
> > > registered. So finding the interface fails, and we go into
> > > EPROBE_DEFER for probing the switch.
> >
> > Ok, I understood up to here. Couple of questions:
> > Is this EPROBE_DEFER causing an issue? Wouldn't the switch then
> > probe successfully when it's reattempted? And then things work
> > normally? I don't see what the problem is.
>
> It's not an issue per se, since it's not a fully circular dependency:
> the DSA master (the FEC controller) does not have any dependency on the
> switch beneath it to probe (there's nothing like a phy-handle from the
> FEC to the switch or to something provided by it).
>
> A few EPROBE_DEFER iterations later the switch will finally find its DSA
> master fully probed via of_find_net_device_by_node.
>
> Andrew is wondering how to avoid those extra EPROBE_DEFER iterations.
> It is weird that the entire functionality of the system depends on those
> EPROBE_DEFERs, typically you'd expect that EPROBE_DEFER just serializes
> asynchronous probing of drivers with interdependencies. But in this case
> it serializes synchronous probing.
>
> > > It is pretty hard to solve. An Ethernet interface can be used by the
> > > kernel itself, e.g. NFS root. At the point you call register_netdev()
> > > in the probe function, to register the interface with the core,
> >
> > Are you using "ethernet interface" and "ethernet controller"
> > interchangeably? Looking at some other drivers, it looks like the
> > ethernet controlled (FEC) is what would call register_netdev(). So
> > what's wrong with that happening if switch0 has not probed
> > successfully?
>
> The "interface" and "controller" terms are not really interchangeable,
> an interface can also be virtual (stacked interfaces on top of physical
> ones, like VLAN or DSA) while network controllers are typically physical
> (unless emulated). But that is not of importance.
>
> The context here is that you cannot solve the interdependency by
> registering the DSA master (FEC) first, then its MDIO bus second (the
> DSA switch probes on the DSA master's MDIO bus => if you do this,
> of_find_net_device_by_node from the DSA layer would find its master the
> first time). The reason you cannot do that is because you need the MDIO
> bus for really basic stuff: you also have your Ethernet PHY on it, and
> you need to initialize that in order to send traffic. And you need to be
> able to send traffic as soon as register_netdev() completes.
>
> So since the driver initialization sequence has a single written order
> regardless of whether DSA switches are attached or not, that order is
> picked to be the one where traffic works as soon as register_netdev completes.
>
> > > it
> > > needs to be fully ready to go.  The networking stack can start using
> > > the interface before register_netdev() even returns. So you cannot
> > > first register the interface and then register the MDIO bus.
> > >
> > > I once looked to see if it was possible to tell the driver core to not
> > > even bother probing a bus as soon as it is registered, go straight to
> > > defer probe handling. Because this is one case we know it cannot
> > > work. But it does not seem possible.
> >
> > fw_devlink doesn't understand the "ethernet" property. If I add that,
> > then in the example you state above, switch0's probe won't even be
> > called until the FEC probe returns. The change is pretty trivial
> > (pasted below) -- can you try it out and tell me if it does what you
> > need/want?
> >
> > -Saravana
> >
> > +++ b/drivers/of/property.c
> > @@ -1292,6 +1292,7 @@ DEFINE_SIMPLE_PROP(resets, "resets", "#reset-cells")
> >  DEFINE_SIMPLE_PROP(leds, "leds", NULL)
> >  DEFINE_SIMPLE_PROP(backlight, "backlight", NULL)
> >  DEFINE_SIMPLE_PROP(phy_handle, "phy-handle", NULL)
> > +DEFINE_SIMPLE_PROP(ethernet, "ethernet", NULL)
> >  DEFINE_SUFFIX_PROP(regulators, "-supply", NULL)
> >  DEFINE_SUFFIX_PROP(gpio, "-gpio", "#gpio-cells")
> >
> > @@ -1381,6 +1382,7 @@ static const struct supplier_bindings
> > of_supplier_bindings[] = {
> >         { .parse_prop = parse_leds, },
> >         { .parse_prop = parse_backlight, },
> >         { .parse_prop = parse_phy_handle, },
> > +       { .parse_prop = parse_ethernet, },
> >         { .parse_prop = parse_gpio_compat, },
> >         { .parse_prop = parse_interrupts, },
> >         { .parse_prop = parse_regulators, },
>
> I don't have this exact setup to test, so I'll let Andrew do it, but I
> have a question: DSA sets up a device link to its master in dsa_master_setup.
> It does this to autoremove itself when the DSA master gets removed, but
> fundamentally it does it after the entire EPROBE_DEFER shebang discussed
> above has already happened. If your patch works, we can drop the manually
> added device link, right?

Yes, you can drop that code if fw_devlink=on works and forms the links
correctly (it should and if not it shouldn't be hard to fix -- just
needs a bit of time to figure out why).

> There's also the question of what to do in case of multiple DSA masters

Sigh... I'm out of my depth with all this network specific discussions
:( Especially when it comes to Linux network related frameworks.

> (multiple "ethernet" properties). Right now, if you describe two DSA
> masters in the device tree, DSA will pick the first DSA master and use
> just that (it doesn't have full support for more than one). But even
> though the second DSA master is not used for anything, with your change,
> unbinding it will also unbind the switch, will it not?

I think I understand your question. And the answer is "yes". It
actually goes one step further. Until both all the DSA masters
(devices pointed to by the "ethernet" properties) of a DSA are bound
successfully, the DSA will not even be probed in the first place. You
can set come command line args to timeout waiting for suppliers with
missing drivers, but ideally you shouldn't need to use those
(fw_devlink is fairly new and I'm continuing to work on improving
it/making it work by default).

-Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Andrew Lunn]

On Thu, Aug 19, 2021 at 04:52:43PM -0700, Saravana Kannan wrote:
> On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote:
> >
> > > (2) is what is happening in this case. fw_devlink=on sees that
> > > "switch" implements the "switch_intc" and "switch" hasn't finished
> > > probing yet. So it has no way of knowing that switch_intc is actually
> > > ready. And even if switch_intc was registered as part of switch's
> > > probe() by the time the PHYs are added, switch_intc could get
> > > deregistered if the probe fails at a later point. So until probe()
> > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> > > is ready. Which is good in general because you won't have to
> > > forcefully unbind (if that is even handled correctly in the first
> > > place) the consumers of a device if it fails probe() half way through
> > > registering a few services.
> 
> I had to read your email a couple of times before I understood it. I
> think I do now, but apologies if I'm not making sense.
> 
> >
> > There are actually a few different circular references with the way
> > switches work. Take for example:
> >
> > &fec1 {
> >         phy-mode = "rmii";
> >         pinctrl-names = "default";
> >         pinctrl-0 = <&pinctrl_fec1>;
> >         status = "okay";
> >
> >         fixed-link {
> >                 speed = <100>;
> >                 full-duplex;
> >         };
> >
> >         mdio1: mdio {
> >                 #address-cells = <1>;
> >                 #size-cells = <0>;
> >                 clock-frequency = <12500000>;
> >                 suppress-preamble;
> >                 status = "okay";
> >
> >                 switch0: switch0@0 {
> >                         compatible = "marvell,mv88e6190";
> >                         pinctrl-0 = <&pinctrl_gpio_switch0>;
> >                         pinctrl-names = "default";
> >                         reg = <0>;
> >                         eeprom-length = <65536>;
> >                         interrupt-parent = <&gpio3>;
> >                         interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
> >                         interrupt-controller;
> >                         #interrupt-cells = <2>;
> >
> >                         ports {
> >                                 #address-cells = <1>;
> >                                 #size-cells = <0>;
> >
> >                                 port@0 {
> >                                         reg = <0>;
> >                                         label = "cpu";
> >                                         ethernet = <&fec1>;
> >
> >                                         fixed-link {
> >                                                 speed = <100>;
> >                                                 full-duplex;
> >                                         };
> >                                 };
> >
> > FEC is an ethernet controller. It has an MDIO bus, and on the bus is
> > an Ethernet switch. port 0 of the Ethernet switch is connected to the
> > FEC ethernet controller.
> >
> > While the FEC probes, it will at some point register its MDIO bus. At
> > that point, the MDIO bus is probed, the switch is found, and
> > registered with the switch core. The switch core looks for the port
> > with an ethernet property and goes looking for that ethernet
> > interface. But that this point in time, the FEC probe has only got as
> > far as registering the MDIO bus. The interface itself is not
> > registered. So finding the interface fails, and we go into
> > EPROBE_DEFER for probing the switch.
> 
> Ok, I understood up to here. Couple of questions:
> Is this EPROBE_DEFER causing an issue? Wouldn't the switch then
> probe successfully when it's reattempted? And then things work
> normally? I don't see what the problem is.

Everything works on the second time around. So there is no problem,
other than we waste time trying to probe the switch, which we know is
going to fail. Depending on the setup, this can add 1/2 second to the
boot time. 

> > It is pretty hard to solve. An Ethernet interface can be used by the
> > kernel itself, e.g. NFS root. At the point you call register_netdev()
> > in the probe function, to register the interface with the core,
> 
> Are you using "ethernet interface" and "ethernet controller"
> interchangeably?

Yeh, pretty much.

> Looking at some other drivers, it looks like the
> ethernet controlled (FEC) is what would call register_netdev(). So
> what's wrong with that happening if switch0 has not probed
> successfully?
> 
> > it
> > needs to be fully ready to go.  The networking stack can start using
> > the interface before register_netdev() even returns. So you cannot
> > first register the interface and then register the MDIO bus.
> >
> > I once looked to see if it was possible to tell the driver core to not
> > even bother probing a bus as soon as it is registered, go straight to
> > defer probe handling. Because this is one case we know it cannot
> > work. But it does not seem possible.
> 
> fw_devlink doesn't understand the "ethernet" property. If I add that,
> then in the example you state above, switch0's probe won't even be
> called until the FEC probe returns. The change is pretty trivial
> (pasted below) -- can you try it out and tell me if it does what you
> need/want?

Cool. I will try this.

Thanks
	Andrew

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Thu, Aug 19, 2021 at 6:08 PM Saravana Kannan <saravanak@google.com> wrote:
>
> On Thu, Aug 19, 2021 at 6:42 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> >
> > Hi Saravana,
> >
> > Thanks for your lucid analysis, it's very much appreciated. Please find
> > my replies inline - but in summary, I don't think there is anything more
> > I can ask of you right now.
>
> You are welcome.
>
> >
> > On 8/19/21 5:28 AM, Saravana Kannan wrote:
> > > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> > >>
> > >> Hi Saravana,
> > >>
> > >> On 8/18/21 4:46 AM, Saravana Kannan wrote:
> > >>> On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote:
> > >>>>
> > >>>> Hi Alvin,
> > >>>>
> > >>>> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote:
> > >>>>> I have an observation that's slightly out of the scope of your patch,
> > >>>>> but I'll post here on the off chance that you find it relevant.
> > >>>>> Apologies if it's out of place.
> > >>>>>
> > >>>>> Do these integrated NXP PHYs use a specific PHY driver, or do they just
> > >>>>> use the Generic PHY driver?
> > >>>>
> > >>>> They refuse to probe at all with the Generic PHY driver. I have been
> > >>>> caught off guard a few times now when I had a kernel built with
> > >>>> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case.
> > >>>>
> > >>>>> If the former is the case, do you experience that the PHY driver fails
> > >>>>> to get probed during mdiobus registration if the kernel uses
> > >>>>> fw_devlink=on?
> > >>>>
> > >>>> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first
> > >>>> time I do it. It behaves exactly as you say.
> > >>>>
> > >>>>>
> > >>>>> In my case I am writing a new subdriver for realtek-smi, a DSA driver
> > >>>>> which registers an internal MDIO bus analogously to sja1105, which is
> > >>>>> why I'm asking. I noticed a deferred probe of the PHY driver because the
> > >>>>> supplier (ethernet-switch) is not ready - presumably because all of this
> > >>>>> is happening in the probe of the switch driver. See below:
> > >>>>>
> > >>>>> [   83.653213] device_add:3270: device: 'SMI-0': device_add
> > >>>>> [   83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0
> > >>>>> [   83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add
> > >>>>> [   83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch
> > >>>>> [   83.654291] libphy: SMI slave MII: probed
> > >>>>> ...
> > >>>>> [   83.659809] device_add:3270: device: 'SMI-0:00': device_add
> > >>>>> [   83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00
> > >>>>> [   83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00
> > >>>>> [   83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add
> > >>>>> [   83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list
> > >>>>> [   83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list
> > >>>>> [   83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch
> > >>>>> [   83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet
> > >>>>> [   83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready
> > >>>>> [   83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list
> > >>>>
> > >>>> So it's a circular dependency? Switch cannot finish probing because it
> > >>>> cannot connect to PHY, which cannot probe because switch has not
> > >>>> finished probing, which....
> > >>>
> > >>> Hi Vladimir/Alvin,
> > >>>
> > >>> If there's a cyclic dependency between two devices, then fw_devlink=on
> > >>> is smart enough to notice that. Once it notices a cycle, it knows that
> > >>> it can't tell which one is the real dependency and which one is the
> > >>> false dependency and so stops enforcing ordering between the devices
> > >>> in the cycle.
> > >>>
> > >>> But fw_devlink doesn't understand all the properties yet. Just most of
> > >>> them and I'm always trying to add more. So when it only understands
> > >>> the property that's causing the false dependency but not the property
> > >>> that causes the real dependency, it can cause issues like this where
> > >>> fw_devlink=on enforces the false dependency and the driver/code
> > >>> enforces the real dependency. These are generally easy to fix -- you
> > >>> just need to teach fw_devlink how to parse more properties.
> > >>>
> > >>> This is just a preliminary analysis since I don't have all the info
> > >>> yet -- so I could be wrong. With that said, I happened to be working
> > >>> on adding fw_devlink support for phy-handle property and I think it
> > >>> should fix your issue with fw_devlink=on. Can you give [1] a shot?
> > >>
> > >> I tried [1] but it did not seem to have any effect.
> > >>
> > >>>
> > >>> If it doesn't fix it, can one of you please point me to an upstream
> > >>> dts (not dtsi) file for a platform in which you see this issue? And
> > >>> ideally also the DT nodes and their drivers that are involved in this
> > >>> cycle? With that info, I should be able to root cause this if the
> > >>> patch above doesn't already fix it.
> > >>
> > >> I'm working with a non-upstream dts - maybe Vladimir is using an
> > >> upstream one? The pattern among the drivers/dts is common between our
> > >> two cases.
> > >
> > > Ideally, I can get a fully upstream example where this issue is
> > > happening so that I can look at the actual code that's hitting this
> > > issue and be sure my analysis is right.
> >
> > Due to NDA issues at work, I am unable to publish the DTS right now. But
> > based on your analysis below, I believe that the problem I am
> > experiencing is exactly for the reasons you describe.
> >
> > >
> > >>
> > >> But for the sake of this discussion, my dts is pretty much the same as
> > >> what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The
> > >> nodes of interest from that dts file are below, and the driver is in
> > >> drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the
> > >> Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part
> > >> of the mdiobus registration, but that never happens. See my previous
> > >> reply for a debug log.
> > >
> > > Your DTS might be similar to this, but the driver code also matters
> > > for me to be sure. Anyway, I took a look at this, but my analysis
> > > below is going to be sketchy because I'm not looking at the actual
> > > code that's reproducing this issue.
> >
> > The driver code I am working with is pretty much the same in the probe
> > path (the driver is modelled on rtl8366rb.c and hooks into
> > realtek-smi-core.c), so it was not a waste of your time to analyze the
> > upstream case. Thanks a lot.
>
> Good to know it's useful.
>
> >
> > >
> > > Assuming this issue actually happens with the example you pointed to
> > > (I don't know this yet), here's what is happening:
> > >
> > > The main problem is that the parent device switch seems to be assuming
> > > it's child/grandchild devices (mdiobus/PHYs) will have probed
> > > successfully as soon as they are added. This assumption is not true
> > > and can be broken for multiple reasons such as:
> > >
> > > 1. The driver for the child devices (PHYs in this case) could be
> > > loaded as a module after the parent (switch) is probed. So when the
> > > devices are added, the PHYs would not be probed.
> > > 2. The child devices could defer probe because one of their suppliers
> > > isn't ready yet. Either because of fw_devlink=on or the framework
> > > itself returning -EPROBE_DEFER.
> > > 3. The child devices could be getting probed asynchronously. So the
> > > device_add() would kick off a thread to probe the child devices in a
> > > separate thread.
> > >
> > > (2) is what is happening in this case. fw_devlink=on sees that
> > > "switch" implements the "switch_intc" and "switch" hasn't finished
> > > probing yet. So it has no way of knowing that switch_intc is actually
> > > ready. And even if switch_intc was registered as part of switch's
> > > probe() by the time the PHYs are added, switch_intc could get
> > > deregistered if the probe fails at a later point. So until probe()
> > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc)
> > > is ready. Which is good in general because you won't have to
> > > forcefully unbind (if that is even handled correctly in the first
> > > place) the consumers of a device if it fails probe() half way through
> > > registering a few services.
> >
> > Right, that makes perfect sense.
> >
> > >
> > > I don't fully understand the networking frameworks, but I think
> > > Vladimir might have a point in his earlier reply [1]. If you can make
> > > the switch driver not assume its child PHYs are ready during the
> > > switch's probe and instead have the switch check if the PHYs are ready
> > > when the switch is "opened" that'd be better.
> >
> > I am pretty new to the DSA subsystem,
>
> I know absolutely nothing about DSA. Heck, I don't even know what it
> stands for. It's just about as informative as "XYZ" to me :)
>
> > but it appears to me that DSA is
> > making this assumption: I don't think DSA switches have a concept of
> > "open", as everything is handled in the dsa_register_switch() call in a
> > DSA driver's probe function. And as Vladimir pointed out, this is also
> > where phylink_of_phy_connect() is being called.
>
> Yeah, I noticed that part.
>
> > A quick look at the
> > other DSA drivers suggests that mv88e6xxx may also suffer from case (2)
> > above, so it seems like a more general issue.
> >
>
> I obviously want fw_devlink=on to be used by everyone and not have
> people use fw_devlink=permissive. In this scenario, I think
> fw_devlink=on is doing the right thing too. So this is where I'm
> hoping the network experts can jump in and fix the general DSA issue
> and I can help with the parts I understand.

Hi Alvin,

Can you give this a shot to see if it fixes your issue? It basically
delays the registration of dsa_register_switch() until all the
consumers of this switch have probed. So it has a couple of caveats:
1. I'm hoping the PHYs are the only consumers of this switch.
2. All of them have to probe successfully before the switch will
register itself.
3. If dsa_register_switch() fails, we can't defer the probe (because
it already succeeded). But I'm not sure if it's a likely error code.

But, for now, I want to see if it fixes your issues.

-Saravana


+++ b/drivers/net/dsa/realtek-smi-core.c
@@ -454,14 +454,16 @@ static int realtek_smi_probe(struct platform_device *pdev)
        smi->ds->priv = smi;

        smi->ds->ops = var->ds_ops;
-       ret = dsa_register_switch(smi->ds);
-       if (ret) {
-               dev_err(dev, "unable to register switch ret = %d\n", ret);
-               return ret;
-       }
        return 0;
 }

+static void realtek_smi_sync_state(struct device *dev)
+{
+       struct realtek_smi *smi = dev_get_drvdata(dev);
+       if (dsa_register_switch(smi->ds))
+               dev_err(dev, "unable to register switch ret = %d\n", ret);
+}
+
 static int realtek_smi_remove(struct platform_device *pdev)
 {
        struct realtek_smi *smi = dev_get_drvdata(&pdev->dev);
@@ -492,6 +494,7 @@ static struct platform_driver realtek_smi_driver = {
        .driver = {
                .name = "realtek-smi",
                .of_match_table = of_match_ptr(realtek_smi_of_match),
+               .sync_state = realtek_smi_sync_state,
        },
        .probe  = realtek_smi_probe,
        .remove = realtek_smi_remove,

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Andrew Lunn]

> 3. If dsa_register_switch() fails, we can't defer the probe (because
> it already succeeded). But I'm not sure if it's a likely error code.

That is not a safe assumption. The tagging driver can be a module, and
not be loaded yet. So we defer probing to allow it to load.

Ethernet switches can be a big graph of parts, not a nice tree of
parts.

    Andrew

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Fri, Aug 20, 2021 at 10:54 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > 3. If dsa_register_switch() fails, we can't defer the probe (because
> > it already succeeded). But I'm not sure if it's a likely error code.
>
> That is not a safe assumption. The tagging driver can be a module, and

I don't know what "tagging driver" means. And yes, this is just a test
patch for now.

> not be loaded yet. So we defer probing to allow it to load.
>
> Ethernet switches can be a big graph of parts, not a nice tree of
> parts.

fw_devlink can handle graphs well. It has cycle detection for a reason
:) Look at slide 10 for a complicated example:
https://linuxplumbersconf.org/event/7/contributions/787/attachments/529/942/LPC_2020_-_fw_devlink.pdf

However, after sending that email I realized that fw_devlink=on would
make sure all the dependencies of this device would have probed first
before it allows this to probe. So EPROBE_DEFER can't really happen.
But for now I just want to see if it fixes the issue Alvin was seeing.

-Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

Hi Saravana,

Thanks for the follow-up. I tested your change and it does the trick: 
there is no deferral and the PHY driver gets probed first-try during the 
mdiobus registration during the call to dsa_register_switch(). I tested 
with the switch, PHY, and tagging drivers all builtin, or all modules, 
and it worked in both cases.

On 8/20/21 6:52 PM, Saravana Kannan wrote:
> Hi Alvin,
> 
> Can you give this a shot to see if it fixes your issue? It basically
> delays the registration of dsa_register_switch() until all the
> consumers of this switch have probed. So it has a couple of caveats:

Hm, weren't the only consumers the PHYs themselves? It seems like the 
main effect of your change is that - by doing the actual 
dsa_register_switch() call after the switch driver probe - the 
ethernet-switch (provider) is already probed, thereby allowing the PHY 
(consumer) to probe immediately.

> 1. I'm hoping the PHYs are the only consumers of this switch.

In my case that is true, if you count the mdio_bus as well:

/sys/devices/platform/ethernet-switch# ls -l consumer\:*
lrwxrwxrwx    1 root     root             0 Aug 22 16:00 
consumer:mdio_bus:SMI-0 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
lrwxrwxrwx    1 root     root             0 Aug 22 16:00 
consumer:mdio_bus:SMI-0:00 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
lrwxrwxrwx    1 root     root             0 Aug 22 16:00 
consumer:mdio_bus:SMI-0:01 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
lrwxrwxrwx    1 root     root             0 Aug 22 16:00 
consumer:mdio_bus:SMI-0:02 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
lrwxrwxrwx    1 root     root             0 Aug 22 16:00 
consumer:mdio_bus:SMI-0:03 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03


> 2. All of them have to probe successfully before the switch will
> register itself.

Yes.

> 3. If dsa_register_switch() fails, we can't defer the probe (because
> it already succeeded). But I'm not sure if it's a likely error code.

It's of course possible that dsa_register_switch() fails. Assuming 
fw_devlink is doing its job properly, I think the reason is most likely 
going to be something specific to the driver, such as a communication 
timeout with the switch hardware itself.

I get the impression that you don't necessarily regard this change as a 
proper fix, so I'm happy to do further tests if you choose to 
investigate further.

Kind regards,
Alvin

> 
> -Saravana
> 
> 
> +++ b/drivers/net/dsa/realtek-smi-core.c
> @@ -454,14 +454,16 @@ static int realtek_smi_probe(struct platform_device *pdev)
>          smi->ds->priv = smi;
> 
>          smi->ds->ops = var->ds_ops;
> -       ret = dsa_register_switch(smi->ds);
> -       if (ret) {
> -               dev_err(dev, "unable to register switch ret = %d\n", ret);
> -               return ret;
> -       }
>          return 0;
>   }
> 
> +static void realtek_smi_sync_state(struct device *dev)
> +{
> +       struct realtek_smi *smi = dev_get_drvdata(dev);
> +       if (dsa_register_switch(smi->ds))
> +               dev_err(dev, "unable to register switch ret = %d\n", ret);
> +}
> +
>   static int realtek_smi_remove(struct platform_device *pdev)
>   {
>          struct realtek_smi *smi = dev_get_drvdata(&pdev->dev);
> @@ -492,6 +494,7 @@ static struct platform_driver realtek_smi_driver = {
>          .driver = {
>                  .name = "realtek-smi",
>                  .of_match_table = of_match_ptr(realtek_smi_of_match),
> +               .sync_state = realtek_smi_sync_state,
>          },
>          .probe  = realtek_smi_probe,
>          .remove = realtek_smi_remove,
> 

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>
> Hi Saravana,
>
> Thanks for the follow-up. I tested your change and it does the trick:
> there is no deferral and the PHY driver gets probed first-try during the
> mdiobus registration during the call to dsa_register_switch().

I'm fairly certain the mdiobus registration happens before
dsa_register_switch(). It's in the probe call path of the DSA. The
connecting of the PHYs with the DSA is what happens when
dsa_register_switch() is called.

> I tested
> with the switch, PHY, and tagging drivers all builtin, or all modules,
> and it worked in both cases.
>
> On 8/20/21 6:52 PM, Saravana Kannan wrote:
> > Hi Alvin,
> >
> > Can you give this a shot to see if it fixes your issue? It basically
> > delays the registration of dsa_register_switch() until all the
> > consumers of this switch have probed. So it has a couple of caveats:
>
> Hm, weren't the only consumers the PHYs themselves? It seems like the
> main effect of your change is that - by doing the actual
> dsa_register_switch() call after the switch driver probe - the
> ethernet-switch (provider) is already probed, thereby allowing the PHY
> (consumer) to probe immediately.

Correct-ish -- if you modify this to account for what I said above.

>
> > 1. I'm hoping the PHYs are the only consumers of this switch.
>
> In my case that is true, if you count the mdio_bus as well:
>
> /sys/devices/platform/ethernet-switch# ls -l consumer\:*
> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> consumer:mdio_bus:SMI-0 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> consumer:mdio_bus:SMI-0:00 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> consumer:mdio_bus:SMI-0:01 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> consumer:mdio_bus:SMI-0:02 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> consumer:mdio_bus:SMI-0:03 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03

Hmm... mdio_bus being a consumer should prevent the sync_state() from
being called on "ethernet-switch". What's the value of the "status"
and "sync_state_only" files inside that mdio_bus folder?

> > 2. All of them have to probe successfully before the switch will
> > register itself.
>
> Yes.

Right, it's a yes in your case. But will it be a yes for all instances
of "realtek,rtl8366rb"?

> > 3. If dsa_register_switch() fails, we can't defer the probe (because
> > it already succeeded). But I'm not sure if it's a likely error code.
>
> It's of course possible that dsa_register_switch() fails. Assuming
> fw_devlink is doing its job properly, I think the reason is most likely
> going to be something specific to the driver, such as a communication
> timeout with the switch hardware itself.

But what if someone sets fw_devlink=permissive? Is it okay to break
this? There are ways to make this work for fw_devlink=permissive and
=on -- you check for each and decide where to call
dsa_register_switch() based on that.

> I get the impression that you don't necessarily regard this change as a
> proper fix, so I'm happy to do further tests if you choose to
> investigate further.

I thought about this in the background the past few days. I think
there are a couple of options:
1. We (community/Andrew) agree that this driver would only work with
fw_devlink=on and we can confirm that the other upstream uses of
"realtek,rtl8366rb" won't have any unprobed consumers problem and
switch to using my patch. Benefit is that it's a trivial and quick
change that gets things working again.
2. The "realtek,rtl8366rb" driver needs to be fixed to use a
"component device". A component device is a logical device that
represents a group of other devices. It's only initialized after all
these devices have probed successfully. The actual switch should be a
component device and it should call dsa_register_switch() in it's
"bind" (equivalent of probe). That way you can explicitly control what
devices need to be probed instead of depending on sync_state() that
have a bunch of caveats.

Alvin, do you want to take up (2)?

-Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Andrew Lunn]

> I thought about this in the background the past few days. I think
> there are a couple of options:
> 1. We (community/Andrew) agree that this driver would only work with
> fw_devlink=on and we can confirm that the other upstream uses of
> "realtek,rtl8366rb" won't have any unprobed consumers problem and
> switch to using my patch. Benefit is that it's a trivial and quick
> change that gets things working again.

I don't think realtek,rtl8366rb is doing anything particularly
unusual. It is not the only switch driver with an MDIO bus driver with
its internal PHYs on it.

> 2. The "realtek,rtl8366rb" driver needs to be fixed to use a
> "component device".

Again, i don't think "realtek,rtl8366rb is doing anything unusual,
compared to the other DSA drivers. If you are suggesting it needs to
make use of the component driver, you might also be suggesting that
all the switch drivers need to be component devices. I don't fully
understand the details here, but it might be, you are also suggesting
some Ethernet drivers need modifying to use the component framework?
And that is not going to fly.

This has all worked until now, things might need a few iterations with
deferral, but we get there in the end. Maybe we need to back out the
phy-handle patch? It does appear to be causing regressions.

	   Andrew

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Mon, Aug 23, 2021 at 1:44 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > I thought about this in the background the past few days. I think
> > there are a couple of options:
> > 1. We (community/Andrew) agree that this driver would only work with
> > fw_devlink=on and we can confirm that the other upstream uses of
> > "realtek,rtl8366rb" won't have any unprobed consumers problem and
> > switch to using my patch. Benefit is that it's a trivial and quick
> > change that gets things working again.
>
> I don't think realtek,rtl8366rb is doing anything particularly
> unusual. It is not the only switch driver with an MDIO bus driver with
> its internal PHYs on it.

But realtek,rtl8366rb is doing some unusual things though:
1. Expecting its child devices would get probed as soon as they are
added in its own probe().
2. The child device in turn depends on the parent node to have probed
(the interrupt dependency in this case).

There is absolutely no guarantee from the driver core that (1) would
happen. In fact, it's more likely for this to not happen.
(2) isn't fully wrong, but it's kinda weird when combined with (1)
because it causes a quasi cyclic dependency.

>
> > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a
> > "component device".
>
> Again, i don't think "realtek,rtl8366rb is doing anything unusual,
> compared to the other DSA drivers. If you are suggesting it needs to
> make use of the component driver, you might also be suggesting that
> all the switch drivers need to be component devices.

I'm saying any driver that does both (1) and (2) above needs to start
using the component devices. Otherwise, it is using a broken driver
model AND it'll be caught/enforced by fw_devlink=on. Even just doing
(1) is wrong, but fw_devlink=on doesn't catch/enforce it.

> I don't fully
> understand the details here, but it might be, you are also suggesting
> some Ethernet drivers need modifying to use the component framework?

I'm not sure if I'm saying that and more likely I'm NOT saying that
(see further below). Based on link [1] I think it's actually the
opposite of point (1) above. IIUC, in the case of FEC, you don't want
the child to probe before the parent is done (because the child
depends on the parent) and fw_devlink=on actually enforces that.

What I'm pointing out is that since dsa_register_switch() is assuming
the PHYs are ready/attaches to PHYs when it's called, if ALL the
following conditions are met, a DSA switch driver needs to use a
component driver model:
1. Switches that have child PHYs that dsa_register_switch() will
attach to when called.
2. The child PHYs depend on services provided by the parent (eg:
interrupt controller)

I don't know much about dsa_register_switch(), but if the DSA
framework could NOT attempt to attach to PHYs as soon as the switch is
registered, that might solve the problem too without needing to use
the component driver model as that'll break the cyclic dependency.

> And that is not going to fly.
>
> This has all worked until now, things might need a few iterations with
> deferral, but we get there in the end.

A few deferred probes is totally fine and fw_devlink doesn't have any
issues with that. In fact, fw_devlink would cut down on some
unnecessary deferred probes [1]. I think you are mixing two different
cases. This case is not the same as [1]. In the case of this realtek
switch driver, it's saying the deferred probe of its child PHYs is NOT
allowed. And I'm saying that's not a valid assumption and the
component device model seems like one good way to handle this
situation.

> Maybe we need to back out the
> phy-handle patch? It does appear to be causing regressions.

I don't mind if you want to do that (fixing the issue Marek reported
is easy). But that doesn't mean this realtek driver isn't wrong and
needs to be fixed. It's just that fw_devlink=on is catching this more
clearly.

[1] - https://lore.kernel.org/netdev/CAGETcx9=AyEfjX_-adgRuX=8a0MkLnj8sy2KJGhxpNCinJu4yA@mail.gmail.com/

-Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

Hi Saravana,

Sorry for the delayed response.

On 8/23/21 8:50 PM, Saravana Kannan wrote:
> On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>>
>> Hi Saravana,
>>
>> Thanks for the follow-up. I tested your change and it does the trick:
>> there is no deferral and the PHY driver gets probed first-try during the
>> mdiobus registration during the call to dsa_register_switch().
> 
> I'm fairly certain the mdiobus registration happens before
> dsa_register_switch(). It's in the probe call path of the DSA. The
> connecting of the PHYs with the DSA is what happens when
> dsa_register_switch() is called.

Hm, are you sure about this? My understanding is that mdiobus 
registration happens as follows:

dsa_register_switch()
     -> ...
         -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb
             -> realtek_smi_setup_mdio()
                 -> of_mdiobus_register()

As it stands, dsa_register_switch() is currently called from 
realtek_smi_probe(). Your patch just moves this call to 
realtek_smi_sync_state(), but per the above, the mdiobus registration 
happens inside dsa_register_switch(), meaning it doesn't happen in the 
probe call path. Or am I missing something? I'm happy to be wrong :-)

[1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/

> 
>> I tested
>> with the switch, PHY, and tagging drivers all builtin, or all modules,
>> and it worked in both cases.
>>
>> On 8/20/21 6:52 PM, Saravana Kannan wrote:
>>> Hi Alvin,
>>>
>>> Can you give this a shot to see if it fixes your issue? It basically
>>> delays the registration of dsa_register_switch() until all the
>>> consumers of this switch have probed. So it has a couple of caveats:
>>
>> Hm, weren't the only consumers the PHYs themselves? It seems like the
>> main effect of your change is that - by doing the actual
>> dsa_register_switch() call after the switch driver probe - the
>> ethernet-switch (provider) is already probed, thereby allowing the PHY
>> (consumer) to probe immediately.
> 
> Correct-ish -- if you modify this to account for what I said above.
> 
>>
>>> 1. I'm hoping the PHYs are the only consumers of this switch.
>>
>> In my case that is true, if you count the mdio_bus as well:
>>
>> /sys/devices/platform/ethernet-switch# ls -l consumer\:*
>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>> consumer:mdio_bus:SMI-0 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>> consumer:mdio_bus:SMI-0:00 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>> consumer:mdio_bus:SMI-0:01 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>> consumer:mdio_bus:SMI-0:02 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>> consumer:mdio_bus:SMI-0:03 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
> 
> Hmm... mdio_bus being a consumer should prevent the sync_state() from
> being called on "ethernet-switch". What's the value of the "status"
> and "sync_state_only" files inside that mdio_bus folder?

Without your patch:

/sys/devices/platform/ethernet-switch# ls -l consumer\:*
lrwxrwxrwx    1 root     root             0 Aug 25 13:42 
consumer:mdio_bus:SMI-0 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
/sys/devices/platform/ethernet-switch# cat consumer\:*/status
available
/sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only
1


With your patch:

0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:*
lrwxrwxrwx    1 root     root             0 Aug 25 15:03 
consumer:mdio_bus:SMI-0 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
lrwxrwxrwx    1 root     root             0 Aug 25 15:03 
consumer:mdio_bus:SMI-0:00 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
lrwxrwxrwx    1 root     root             0 Aug 25 15:03 
consumer:mdio_bus:SMI-0:01 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
lrwxrwxrwx    1 root     root             0 Aug 25 15:03 
consumer:mdio_bus:SMI-0:02 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
lrwxrwxrwx    1 root     root             0 Aug 25 15:03 
consumer:mdio_bus:SMI-0:03 -> 
../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat 
consumer\:*/status
available
active
active
active
active
0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat 
consumer\:*/sync_state_only
1
0
0
0
0

Hope that helps you understand what's going on better.

BTW, I noticed that when I build realtek-smi as a module with your 
patch, my kernel crashes if I unload the module. I haven't debugged this 
because it was just a test, nor did I get a stacktrace. LMK if you want 
more info.

> 
>>> 2. All of them have to probe successfully before the switch will
>>> register itself.
>>
>> Yes.
> 
> Right, it's a yes in your case. But will it be a yes for all instances
> of "realtek,rtl8366rb"?
> 
>>> 3. If dsa_register_switch() fails, we can't defer the probe (because
>>> it already succeeded). But I'm not sure if it's a likely error code.
>>
>> It's of course possible that dsa_register_switch() fails. Assuming
>> fw_devlink is doing its job properly, I think the reason is most likely
>> going to be something specific to the driver, such as a communication
>> timeout with the switch hardware itself.
> 
> But what if someone sets fw_devlink=permissive? Is it okay to break
> this? There are ways to make this work for fw_devlink=permissive and
> =on -- you check for each and decide where to call
> dsa_register_switch() based on that.

I am new to DSA myself so I think I am unqualified to answer whether 
it's OK to break things or not.

> 
>> I get the impression that you don't necessarily regard this change as a
>> proper fix, so I'm happy to do further tests if you choose to
>> investigate further.
> 
> I thought about this in the background the past few days. I think
> there are a couple of options:
> 1. We (community/Andrew) agree that this driver would only work with
> fw_devlink=on and we can confirm that the other upstream uses of
> "realtek,rtl8366rb" won't have any unprobed consumers problem and
> switch to using my patch. Benefit is that it's a trivial and quick
> change that gets things working again.
> 2. The "realtek,rtl8366rb" driver needs to be fixed to use a
> "component device". A component device is a logical device that
> represents a group of other devices. It's only initialized after all
> these devices have probed successfully. The actual switch should be a
> component device and it should call dsa_register_switch() in it's
> "bind" (equivalent of probe). That way you can explicitly control what
> devices need to be probed instead of depending on sync_state() that
> have a bunch of caveats.
> 
> Alvin, do you want to take up (2)?

I can give it a shot, but first:

   - It seems Andrew may also need some convincing that this is the 
right approach.
   - Are you sure that this will solve the problem? See what I wrote 
upstairs in this email.
   - I have never written - nor can I recall reading - a component 
driver, so I would appreciate if you could point me to an upstream 
example that you think is illustrative.

There's one more issue: I do not have an RTL8366 to test on - rather, I 
encountered this problem in realtek-smi while writing a new subdriver 
for it to support the RTL8365MB. So any proposed fix may be perceived as 
speculative if I cannot test on the '66 hardware. In that case this may 
have to wait until the '65MB subdriver is accepted. Many ifs and maybes, 
but don't take it to mean I'm not interested in helping. On the contrary 
- I would like to fix this bug since I am affected by it!

Kind regards,
Alvin

> 
> -Saravana
> 

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Wed, Aug 25, 2021 at 6:40 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>
> Hi Saravana,
>
> Sorry for the delayed response.
>
> On 8/23/21 8:50 PM, Saravana Kannan wrote:
> > On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> >>
> >> Hi Saravana,
> >>
> >> Thanks for the follow-up. I tested your change and it does the trick:
> >> there is no deferral and the PHY driver gets probed first-try during the
> >> mdiobus registration during the call to dsa_register_switch().
> >
> > I'm fairly certain the mdiobus registration happens before
> > dsa_register_switch(). It's in the probe call path of the DSA. The
> > connecting of the PHYs with the DSA is what happens when
> > dsa_register_switch() is called.
>
> Hm, are you sure about this? My understanding is that mdiobus
> registration happens as follows:
>
> dsa_register_switch()
>      -> ...
>          -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb
>              -> realtek_smi_setup_mdio()
>                  -> of_mdiobus_register()

My bad, you are definitely right! Thanks for correcting my understanding.

> As it stands, dsa_register_switch() is currently called from
> realtek_smi_probe(). Your patch just moves this call to
> realtek_smi_sync_state(), but per the above, the mdiobus registration
> happens inside dsa_register_switch(), meaning it doesn't happen in the
> probe call path. Or am I missing something? I'm happy to be wrong :-)

Ok, my sync_state() hack is definitely not a solution anymore. It's
just a terrible hack.

>
> [1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/
>
> >
> >> I tested
> >> with the switch, PHY, and tagging drivers all builtin, or all modules,
> >> and it worked in both cases.
> >>
> >> On 8/20/21 6:52 PM, Saravana Kannan wrote:
> >>> Hi Alvin,
> >>>
> >>> Can you give this a shot to see if it fixes your issue? It basically
> >>> delays the registration of dsa_register_switch() until all the
> >>> consumers of this switch have probed. So it has a couple of caveats:
> >>
> >> Hm, weren't the only consumers the PHYs themselves? It seems like the
> >> main effect of your change is that - by doing the actual
> >> dsa_register_switch() call after the switch driver probe - the
> >> ethernet-switch (provider) is already probed, thereby allowing the PHY
> >> (consumer) to probe immediately.
> >
> > Correct-ish -- if you modify this to account for what I said above.
> >
> >>
> >>> 1. I'm hoping the PHYs are the only consumers of this switch.
> >>
> >> In my case that is true, if you count the mdio_bus as well:
> >>
> >> /sys/devices/platform/ethernet-switch# ls -l consumer\:*
> >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> >> consumer:mdio_bus:SMI-0 ->
> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
> >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> >> consumer:mdio_bus:SMI-0:00 ->
> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
> >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> >> consumer:mdio_bus:SMI-0:01 ->
> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
> >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> >> consumer:mdio_bus:SMI-0:02 ->
> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
> >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> >> consumer:mdio_bus:SMI-0:03 ->
> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
> >
> > Hmm... mdio_bus being a consumer should prevent the sync_state() from
> > being called on "ethernet-switch". What's the value of the "status"
> > and "sync_state_only" files inside that mdio_bus folder?
>
> Without your patch:
>
> /sys/devices/platform/ethernet-switch# ls -l consumer\:*
> lrwxrwxrwx    1 root     root             0 Aug 25 13:42
> consumer:mdio_bus:SMI-0 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
> /sys/devices/platform/ethernet-switch# cat consumer\:*/status
> available
> /sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only
> 1
>
>
> With your patch:
>
> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:*
> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> consumer:mdio_bus:SMI-0 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> consumer:mdio_bus:SMI-0:00 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> consumer:mdio_bus:SMI-0:01 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> consumer:mdio_bus:SMI-0:02 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> consumer:mdio_bus:SMI-0:03 ->
> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat
> consumer\:*/status
> available
> active
> active
> active
> active
> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat
> consumer\:*/sync_state_only
> 1
> 0
> 0
> 0
> 0
>
> Hope that helps you understand what's going on better.
>
> BTW, I noticed that when I build realtek-smi as a module with your
> patch, my kernel crashes if I unload the module. I haven't debugged this
> because it was just a test, nor did I get a stacktrace. LMK if you want
> more info.

Yeah, don't bother. It should never merge.

> >
> >>> 2. All of them have to probe successfully before the switch will
> >>> register itself.
> >>
> >> Yes.
> >
> > Right, it's a yes in your case. But will it be a yes for all instances
> > of "realtek,rtl8366rb"?
> >
> >>> 3. If dsa_register_switch() fails, we can't defer the probe (because
> >>> it already succeeded). But I'm not sure if it's a likely error code.
> >>
> >> It's of course possible that dsa_register_switch() fails. Assuming
> >> fw_devlink is doing its job properly, I think the reason is most likely
> >> going to be something specific to the driver, such as a communication
> >> timeout with the switch hardware itself.
> >
> > But what if someone sets fw_devlink=permissive? Is it okay to break
> > this? There are ways to make this work for fw_devlink=permissive and
> > =on -- you check for each and decide where to call
> > dsa_register_switch() based on that.
>
> I am new to DSA myself so I think I am unqualified to answer whether
> it's OK to break things or not.
>
> >
> >> I get the impression that you don't necessarily regard this change as a
> >> proper fix, so I'm happy to do further tests if you choose to
> >> investigate further.
> >
> > I thought about this in the background the past few days. I think
> > there are a couple of options:
> > 1. We (community/Andrew) agree that this driver would only work with
> > fw_devlink=on and we can confirm that the other upstream uses of
> > "realtek,rtl8366rb" won't have any unprobed consumers problem and
> > switch to using my patch. Benefit is that it's a trivial and quick
> > change that gets things working again.
> > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a
> > "component device". A component device is a logical device that
> > represents a group of other devices. It's only initialized after all
> > these devices have probed successfully. The actual switch should be a
> > component device and it should call dsa_register_switch() in it's
> > "bind" (equivalent of probe). That way you can explicitly control what
> > devices need to be probed instead of depending on sync_state() that
> > have a bunch of caveats.
> >
> > Alvin, do you want to take up (2)?
>
> I can give it a shot, but first:
>
>    - It seems Andrew may also need some convincing that this is the
> right approach.

Agreed. Let's wait to see what Andrew thinks of my last response to him.

>    - Are you sure that this will solve the problem? See what I wrote
> upstairs in this email.

Yeah, it would solve the problem with a few changes:
1. The IRQ registration and mdio bus registration would get moved to
realtek_smi_probe() which probes "realtek,rtl8366rb".
2. The component device needs to be set up to be "made up of"
realtek,rtl8366rb and all the PHYs. So it'll wait for all of them to
finish probing before it's initialized. PHYs will initialize now
because realtek,rtl8366rb probe would finish without problems.
3. The component device init would call dsa_register_switch() which
kinda makes sense because the rtl8366rb and all the PHYs combined
together is what makes up the logical DSA switch.

If (2) and (3) could be make part of the framework itself, with (1)
like fix ups done to drivers with have these cyclic dependency issues,
then we could fix bad driver usage model (assuming device_add() will
probe the device before it returns).

>    - I have never written - nor can I recall reading - a component
> driver, so I would appreciate if you could point me to an upstream
> example that you think is illustrative.

The APIs you'll end up using are those in drivers/base/component.c.
You can also grep for component_master_ops and component_ops.
drivers/gpu/drm/msm/msm_drv.c might be one place to start.

Btw, the component API itself could afford some clean up and there was
a series [1], but it looks like Stephen has been busy and hasn't
gotten around to it?
[1] - https://lore.kernel.org/lkml/20210520002519.3538432-1-swboyd@chromium.org/

Having said all that, I might be able to give an API to tell
fw_devlink to ignore a specific child node as a supplier. So you'd
call it to ask fw_devlink to ignore the interrupt controller as a
supplier. I still maintain the current model of this driver is
definitely broken, but I'd rather just unblock this right now that
revert phy-handle support because of a few whacky corner cases like
this. I'll try to send those out this week.

> There's one more issue: I do not have an RTL8366 to test on - rather, I
> encountered this problem in realtek-smi while writing a new subdriver
> for it to support the RTL8365MB. So any proposed fix may be perceived as
> speculative if I cannot test on the '66 hardware.

If you know it fixes the issue on a very similar downstream
hardware/driver, I think it's okay to send fixes. And someone else
might be able to test it out for you.

> In that case this may
> have to wait until the '65MB subdriver is accepted. Many ifs and maybes,
> but don't take it to mean I'm not interested in helping. On the contrary
> - I would like to fix this bug since I am affected by it!

Thanks.

-Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Saravana Kannan]

On Wed, Aug 25, 2021 at 10:33 PM Saravana Kannan <saravanak@google.com> wrote:
>
> On Wed, Aug 25, 2021 at 6:40 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> >
> > Hi Saravana,
> >
> > Sorry for the delayed response.
> >
> > On 8/23/21 8:50 PM, Saravana Kannan wrote:
> > > On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
> > >>
> > >> Hi Saravana,
> > >>
> > >> Thanks for the follow-up. I tested your change and it does the trick:
> > >> there is no deferral and the PHY driver gets probed first-try during the
> > >> mdiobus registration during the call to dsa_register_switch().
> > >
> > > I'm fairly certain the mdiobus registration happens before
> > > dsa_register_switch(). It's in the probe call path of the DSA. The
> > > connecting of the PHYs with the DSA is what happens when
> > > dsa_register_switch() is called.
> >
> > Hm, are you sure about this? My understanding is that mdiobus
> > registration happens as follows:
> >
> > dsa_register_switch()
> >      -> ...
> >          -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb
> >              -> realtek_smi_setup_mdio()
> >                  -> of_mdiobus_register()
>
> My bad, you are definitely right! Thanks for correcting my understanding.
>
> > As it stands, dsa_register_switch() is currently called from
> > realtek_smi_probe(). Your patch just moves this call to
> > realtek_smi_sync_state(), but per the above, the mdiobus registration
> > happens inside dsa_register_switch(), meaning it doesn't happen in the
> > probe call path. Or am I missing something? I'm happy to be wrong :-)
>
> Ok, my sync_state() hack is definitely not a solution anymore. It's
> just a terrible hack.
>
> >
> > [1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/
> >
> > >
> > >> I tested
> > >> with the switch, PHY, and tagging drivers all builtin, or all modules,
> > >> and it worked in both cases.
> > >>
> > >> On 8/20/21 6:52 PM, Saravana Kannan wrote:
> > >>> Hi Alvin,
> > >>>
> > >>> Can you give this a shot to see if it fixes your issue? It basically
> > >>> delays the registration of dsa_register_switch() until all the
> > >>> consumers of this switch have probed. So it has a couple of caveats:
> > >>
> > >> Hm, weren't the only consumers the PHYs themselves? It seems like the
> > >> main effect of your change is that - by doing the actual
> > >> dsa_register_switch() call after the switch driver probe - the
> > >> ethernet-switch (provider) is already probed, thereby allowing the PHY
> > >> (consumer) to probe immediately.
> > >
> > > Correct-ish -- if you modify this to account for what I said above.
> > >
> > >>
> > >>> 1. I'm hoping the PHYs are the only consumers of this switch.
> > >>
> > >> In my case that is true, if you count the mdio_bus as well:
> > >>
> > >> /sys/devices/platform/ethernet-switch# ls -l consumer\:*
> > >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> > >> consumer:mdio_bus:SMI-0 ->
> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
> > >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> > >> consumer:mdio_bus:SMI-0:00 ->
> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
> > >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> > >> consumer:mdio_bus:SMI-0:01 ->
> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
> > >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> > >> consumer:mdio_bus:SMI-0:02 ->
> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
> > >> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
> > >> consumer:mdio_bus:SMI-0:03 ->
> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
> > >
> > > Hmm... mdio_bus being a consumer should prevent the sync_state() from
> > > being called on "ethernet-switch". What's the value of the "status"
> > > and "sync_state_only" files inside that mdio_bus folder?
> >
> > Without your patch:
> >
> > /sys/devices/platform/ethernet-switch# ls -l consumer\:*
> > lrwxrwxrwx    1 root     root             0 Aug 25 13:42
> > consumer:mdio_bus:SMI-0 ->
> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
> > /sys/devices/platform/ethernet-switch# cat consumer\:*/status
> > available
> > /sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only
> > 1
> >
> >
> > With your patch:
> >
> > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:*
> > lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> > consumer:mdio_bus:SMI-0 ->
> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
> > lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> > consumer:mdio_bus:SMI-0:00 ->
> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
> > lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> > consumer:mdio_bus:SMI-0:01 ->
> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
> > lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> > consumer:mdio_bus:SMI-0:02 ->
> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
> > lrwxrwxrwx    1 root     root             0 Aug 25 15:03
> > consumer:mdio_bus:SMI-0:03 ->
> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
> > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
> > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
> > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat
> > consumer\:*/status
> > available
> > active
> > active
> > active
> > active
> > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat
> > consumer\:*/sync_state_only
> > 1
> > 0
> > 0
> > 0
> > 0
> >
> > Hope that helps you understand what's going on better.
> >
> > BTW, I noticed that when I build realtek-smi as a module with your
> > patch, my kernel crashes if I unload the module. I haven't debugged this
> > because it was just a test, nor did I get a stacktrace. LMK if you want
> > more info.
>
> Yeah, don't bother. It should never merge.
>
> > >
> > >>> 2. All of them have to probe successfully before the switch will
> > >>> register itself.
> > >>
> > >> Yes.
> > >
> > > Right, it's a yes in your case. But will it be a yes for all instances
> > > of "realtek,rtl8366rb"?
> > >
> > >>> 3. If dsa_register_switch() fails, we can't defer the probe (because
> > >>> it already succeeded). But I'm not sure if it's a likely error code.
> > >>
> > >> It's of course possible that dsa_register_switch() fails. Assuming
> > >> fw_devlink is doing its job properly, I think the reason is most likely
> > >> going to be something specific to the driver, such as a communication
> > >> timeout with the switch hardware itself.
> > >
> > > But what if someone sets fw_devlink=permissive? Is it okay to break
> > > this? There are ways to make this work for fw_devlink=permissive and
> > > =on -- you check for each and decide where to call
> > > dsa_register_switch() based on that.
> >
> > I am new to DSA myself so I think I am unqualified to answer whether
> > it's OK to break things or not.
> >
> > >
> > >> I get the impression that you don't necessarily regard this change as a
> > >> proper fix, so I'm happy to do further tests if you choose to
> > >> investigate further.
> > >
> > > I thought about this in the background the past few days. I think
> > > there are a couple of options:
> > > 1. We (community/Andrew) agree that this driver would only work with
> > > fw_devlink=on and we can confirm that the other upstream uses of
> > > "realtek,rtl8366rb" won't have any unprobed consumers problem and
> > > switch to using my patch. Benefit is that it's a trivial and quick
> > > change that gets things working again.
> > > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a
> > > "component device". A component device is a logical device that
> > > represents a group of other devices. It's only initialized after all
> > > these devices have probed successfully. The actual switch should be a
> > > component device and it should call dsa_register_switch() in it's
> > > "bind" (equivalent of probe). That way you can explicitly control what
> > > devices need to be probed instead of depending on sync_state() that
> > > have a bunch of caveats.
> > >
> > > Alvin, do you want to take up (2)?
> >
> > I can give it a shot, but first:
> >
> >    - It seems Andrew may also need some convincing that this is the
> > right approach.
>
> Agreed. Let's wait to see what Andrew thinks of my last response to him.
>
> >    - Are you sure that this will solve the problem? See what I wrote
> > upstairs in this email.
>
> Yeah, it would solve the problem with a few changes:
> 1. The IRQ registration and mdio bus registration would get moved to
> realtek_smi_probe() which probes "realtek,rtl8366rb".
> 2. The component device needs to be set up to be "made up of"
> realtek,rtl8366rb and all the PHYs. So it'll wait for all of them to
> finish probing before it's initialized. PHYs will initialize now
> because realtek,rtl8366rb probe would finish without problems.
> 3. The component device init would call dsa_register_switch() which
> kinda makes sense because the rtl8366rb and all the PHYs combined
> together is what makes up the logical DSA switch.
>
> If (2) and (3) could be make part of the framework itself, with (1)
> like fix ups done to drivers with have these cyclic dependency issues,
> then we could fix bad driver usage model (assuming device_add() will
> probe the device before it returns).
>
> >    - I have never written - nor can I recall reading - a component
> > driver, so I would appreciate if you could point me to an upstream
> > example that you think is illustrative.
>
> The APIs you'll end up using are those in drivers/base/component.c.
> You can also grep for component_master_ops and component_ops.
> drivers/gpu/drm/msm/msm_drv.c might be one place to start.
>
> Btw, the component API itself could afford some clean up and there was
> a series [1], but it looks like Stephen has been busy and hasn't
> gotten around to it?
> [1] - https://lore.kernel.org/lkml/20210520002519.3538432-1-swboyd@chromium.org/
>
> Having said all that, I might be able to give an API to tell
> fw_devlink to ignore a specific child node as a supplier. So you'd
> call it to ask fw_devlink to ignore the interrupt controller as a
> supplier. I still maintain the current model of this driver is
> definitely broken, but I'd rather just unblock this right now that
> revert phy-handle support because of a few whacky corner cases like
> this. I'll try to send those out this week.

Not exactly what I described, but here's an attempt. Can you give it a shot?
https://lore.kernel.org/lkml/20210826074526.825517-1-saravanak@google.com/

-Saravana

>
> > There's one more issue: I do not have an RTL8366 to test on - rather, I
> > encountered this problem in realtek-smi while writing a new subdriver
> > for it to support the RTL8365MB. So any proposed fix may be perceived as
> > speculative if I cannot test on the '66 hardware.
>
> If you know it fixes the issue on a very similar downstream
> hardware/driver, I think it's okay to send fixes. And someone else
> might be able to test it out for you.
>
> > In that case this may
> > have to wait until the '65MB subdriver is accepted. Many ifs and maybes,
> > but don't take it to mean I'm not interested in helping. On the contrary
> > - I would like to fix this bug since I am affected by it!
>
> Thanks.
>
> -Saravana

Re: [PATCH net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse

[Alvin Šipraga]

On 8/26/21 7:33 AM, Saravana Kannan wrote:
> On Wed, Aug 25, 2021 at 6:40 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>>
>> Hi Saravana,
>>
>> Sorry for the delayed response.
>>
>> On 8/23/21 8:50 PM, Saravana Kannan wrote:
>>> On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote:
>>>>
>>>> Hi Saravana,
>>>>
>>>> Thanks for the follow-up. I tested your change and it does the trick:
>>>> there is no deferral and the PHY driver gets probed first-try during the
>>>> mdiobus registration during the call to dsa_register_switch().
>>>
>>> I'm fairly certain the mdiobus registration happens before
>>> dsa_register_switch(). It's in the probe call path of the DSA. The
>>> connecting of the PHYs with the DSA is what happens when
>>> dsa_register_switch() is called.
>>
>> Hm, are you sure about this? My understanding is that mdiobus
>> registration happens as follows:
>>
>> dsa_register_switch()
>>       -> ...
>>           -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb
>>               -> realtek_smi_setup_mdio()
>>                   -> of_mdiobus_register()
> 
> My bad, you are definitely right! Thanks for correcting my understanding.
> 
>> As it stands, dsa_register_switch() is currently called from
>> realtek_smi_probe(). Your patch just moves this call to
>> realtek_smi_sync_state(), but per the above, the mdiobus registration
>> happens inside dsa_register_switch(), meaning it doesn't happen in the
>> probe call path. Or am I missing something? I'm happy to be wrong :-)
> 
> Ok, my sync_state() hack is definitely not a solution anymore. It's
> just a terrible hack.
> 
>>
>> [1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/>>>
>>>
>>>> I tested
>>>> with the switch, PHY, and tagging drivers all builtin, or all modules,
>>>> and it worked in both cases.
>>>>
>>>> On 8/20/21 6:52 PM, Saravana Kannan wrote:
>>>>> Hi Alvin,
>>>>>
>>>>> Can you give this a shot to see if it fixes your issue? It basically
>>>>> delays the registration of dsa_register_switch() until all the
>>>>> consumers of this switch have probed. So it has a couple of caveats:
>>>>
>>>> Hm, weren't the only consumers the PHYs themselves? It seems like the
>>>> main effect of your change is that - by doing the actual
>>>> dsa_register_switch() call after the switch driver probe - the
>>>> ethernet-switch (provider) is already probed, thereby allowing the PHY
>>>> (consumer) to probe immediately.
>>>
>>> Correct-ish -- if you modify this to account for what I said above.
>>>
>>>>
>>>>> 1. I'm hoping the PHYs are the only consumers of this switch.
>>>>
>>>> In my case that is true, if you count the mdio_bus as well:
>>>>
>>>> /sys/devices/platform/ethernet-switch# ls -l consumer\:*
>>>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>>>> consumer:mdio_bus:SMI-0 ->
>>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
>>>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>>>> consumer:mdio_bus:SMI-0:00 ->
>>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
>>>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>>>> consumer:mdio_bus:SMI-0:01 ->
>>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
>>>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>>>> consumer:mdio_bus:SMI-0:02 ->
>>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
>>>> lrwxrwxrwx    1 root     root             0 Aug 22 16:00
>>>> consumer:mdio_bus:SMI-0:03 ->
>>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
>>>
>>> Hmm... mdio_bus being a consumer should prevent the sync_state() from
>>> being called on "ethernet-switch". What's the value of the "status"
>>> and "sync_state_only" files inside that mdio_bus folder?
>>
>> Without your patch:
>>
>> /sys/devices/platform/ethernet-switch# ls -l consumer\:*
>> lrwxrwxrwx    1 root     root             0 Aug 25 13:42
>> consumer:mdio_bus:SMI-0 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
>> /sys/devices/platform/ethernet-switch# cat consumer\:*/status
>> available
>> /sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only
>> 1
>>
>>
>> With your patch:
>>
>> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:*
>> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
>> consumer:mdio_bus:SMI-0 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0
>> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
>> consumer:mdio_bus:SMI-0:00 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00
>> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
>> consumer:mdio_bus:SMI-0:01 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01
>> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
>> consumer:mdio_bus:SMI-0:02 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02
>> lrwxrwxrwx    1 root     root             0 Aug 25 15:03
>> consumer:mdio_bus:SMI-0:03 ->
>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03
>> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
>> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch#
>> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat
>> consumer\:*/status
>> available
>> active
>> active
>> active
>> active
>> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat
>> consumer\:*/sync_state_only
>> 1
>> 0
>> 0
>> 0
>> 0
>>
>> Hope that helps you understand what's going on better.
>>
>> BTW, I noticed that when I build realtek-smi as a module with your
>> patch, my kernel crashes if I unload the module. I haven't debugged this
>> because it was just a test, nor did I get a stacktrace. LMK if you want
>> more info.
> 
> Yeah, don't bother. It should never merge.
> 
>>>
>>>>> 2. All of them have to probe successfully before the switch will
>>>>> register itself.
>>>>
>>>> Yes.
>>>
>>> Right, it's a yes in your case. But will it be a yes for all instances
>>> of "realtek,rtl8366rb"?
>>>
>>>>> 3. If dsa_register_switch() fails, we can't defer the probe (because
>>>>> it already succeeded). But I'm not sure if it's a likely error code.
>>>>
>>>> It's of course possible that dsa_register_switch() fails. Assuming
>>>> fw_devlink is doing its job properly, I think the reason is most likely
>>>> going to be something specific to the driver, such as a communication
>>>> timeout with the switch hardware itself.
>>>
>>> But what if someone sets fw_devlink=permissive? Is it okay to break
>>> this? There are ways to make this work for fw_devlink=permissive and
>>> =on -- you check for each and decide where to call
>>> dsa_register_switch() based on that.
>>
>> I am new to DSA myself so I think I am unqualified to answer whether
>> it's OK to break things or not.
>>
>>>
>>>> I get the impression that you don't necessarily regard this change as a
>>>> proper fix, so I'm happy to do further tests if you choose to
>>>> investigate further.
>>>
>>> I thought about this in the background the past few days. I think
>>> there are a couple of options:
>>> 1. We (community/Andrew) agree that this driver would only work with
>>> fw_devlink=on and we can confirm that the other upstream uses of
>>> "realtek,rtl8366rb" won't have any unprobed consumers problem and
>>> switch to using my patch. Benefit is that it's a trivial and quick
>>> change that gets things working again.
>>> 2. The "realtek,rtl8366rb" driver needs to be fixed to use a
>>> "component device". A component device is a logical device that
>>> represents a group of other devices. It's only initialized after all
>>> these devices have probed successfully. The actual switch should be a
>>> component device and it should call dsa_register_switch() in it's
>>> "bind" (equivalent of probe). That way you can explicitly control what
>>> devices need to be probed instead of depending on sync_state() that
>>> have a bunch of caveats.
>>>
>>> Alvin, do you want to take up (2)?
>>
>> I can give it a shot, but first:
>>
>>     - It seems Andrew may also need some convincing that this is the
>> right approach.
> 
> Agreed. Let's wait to see what Andrew thinks of my last response to him.
> 
>>     - Are you sure that this will solve the problem? See what I wrote
>> upstairs in this email.
> 
> Yeah, it would solve the problem with a few changes:
> 1. The IRQ registration and mdio bus registration would get moved to
> realtek_smi_probe() which probes "realtek,rtl8366rb".
> 2. The component device needs to be set up to be "made up of"
> realtek,rtl8366rb and all the PHYs. So it'll wait for all of them to
> finish probing before it's initialized. PHYs will initialize now
> because realtek,rtl8366rb probe would finish without problems.
> 3. The component device init would call dsa_register_switch() which
> kinda makes sense because the rtl8366rb and all the PHYs combined
> together is what makes up the logical DSA switch.
> 
> If (2) and (3) could be make part of the framework itself, with (1)
> like fix ups done to drivers with have these cyclic dependency issues,
> then we could fix bad driver usage model (assuming device_add() will
> probe the device before it returns).
> 
>>     - I have never written - nor can I recall reading - a component
>> driver, so I would appreciate if you could point me to an upstream
>> example that you think is illustrative.
> 
> The APIs you'll end up using are those in drivers/base/component.c.
> You can also grep for component_master_ops and component_ops.
> drivers/gpu/drm/msm/msm_drv.c might be one place to start.

Thanks for your pointers, I think your sketch makes sense to me. If we 
get some sort of approval from the DSA maintainers then I can give it a 
shot. Just one question though - it seems like the component master 
ought to have its own compatible string too? How is that going to work 
without breaking the existing dt-bindings for this switch?

Kind regards,
Alvin

> 
> Btw, the component API itself could afford some clean up and there was
> a series [1], but it looks like Stephen has been busy and hasn't
> gotten around to it?
> [1] - https://lore.kernel.org/lkml/20210520002519.3538432-1-swboyd@chromium.org/>> 
> Having said all that, I might be able to give an API to tell
> fw_devlink to ignore a specific child node as a supplier. So you'd
> call it to ask fw_devlink to ignore the interrupt controller as a
> supplier. I still maintain the current model of this driver is
> definitely broken, but I'd rather just unblock this right now that
> revert phy-handle support because of a few whacky corner cases like
> this. I'll try to send those out this week.
> 
>> There's one more issue: I do not have an RTL8366 to test on - rather, I
>> encountered this problem in realtek-smi while writing a new subdriver
>> for it to support the RTL8365MB. So any proposed fix may be perceived as
>> speculative if I cannot test on the '66 hardware.
> 
> If you know it fixes the issue on a very similar downstream
> hardware/driver, I think it's okay to send fixes. And someone else
> might be able to test it out for you.
> 
>> In that case this may
>> have to wait until the '65MB subdriver is accepted. Many ifs and maybes,
>> but don't take it to mean I'm not interested in helping. On the contrary
>> - I would like to fix this bug since I am affected by it!
> 
> Thanks.
> 
> -Saravana
>