[PATCHv4 net-next] bonding: add software tx timestamping support

[PATCHv4 net-next] bonding: add software tx timestamping support

[Hangbin Liu]

Currently, bonding only obtain the timestamp (ts) information of
the active slave, which is available only for modes 1, 5, and 6.
For other modes, bonding only has software rx timestamping support.

However, some users who use modes such as LACP also want tx timestamp
support. To address this issue, let's check the ts information of each
slave. If all slaves support tx timestamping, we can enable tx
timestamping support for the bond.

Suggested-by: Miroslav Lichvar <mlichvar@redhat.com>
Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>
---
v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
    RTNL. Only check _TX_SOFTWARE for the slaves.
v3: remove dev_hold/dev_put. remove the '\' for line continuation.
v2: check each slave's ts info to make sure bond support sw tx
    timestamping
---
 drivers/net/bonding/bond_main.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 00646aa315c3..9cf49b61f4b3 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 				    struct ethtool_ts_info *info)
 {
 	struct bonding *bond = netdev_priv(bond_dev);
+	struct ethtool_ts_info ts_info;
 	const struct ethtool_ops *ops;
 	struct net_device *real_dev;
+	bool sw_tx_support = false;
 	struct phy_device *phydev;
+	struct list_head *iter;
+	struct slave *slave;
 	int ret = 0;
 
+	ASSERT_RTNL();
+
 	rcu_read_lock();
 	real_dev = bond_option_active_slave_get_rcu(bond);
 	dev_hold(real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 			ret = ops->get_ts_info(real_dev, info);
 			goto out;
 		}
+	} else {
+		/* Check if all slaves support software tx timestamping */
+		rcu_read_lock();
+		bond_for_each_slave_rcu(bond, slave, iter) {
+			ret = -1;
+			ops = slave->dev->ethtool_ops;
+			phydev = slave->dev->phydev;
+
+			if (phy_has_tsinfo(phydev))
+				ret = phy_ts_info(phydev, &ts_info);
+			else if (ops->get_ts_info)
+				ret = ops->get_ts_info(slave->dev, &ts_info);
+
+			if (!ret && (ts_info.so_timestamping & SOF_TIMESTAMPING_TX_SOFTWARE)) {
+				sw_tx_support = true;
+				continue;
+			}
+
+			sw_tx_support = false;
+			break;
+		}
+		rcu_read_unlock();
 	}
 
+	ret = 0;
 	info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE |
 				SOF_TIMESTAMPING_SOFTWARE;
+	if (sw_tx_support)
+		info->so_timestamping |= SOF_TIMESTAMPING_TX_SOFTWARE;
+
 	info->phc_index = -1;
 
 out:
-- 
2.38.1

Re: [PATCHv4 net-next] bonding: add software tx timestamping support

[Jay Vosburgh]

Hangbin Liu <liuhangbin@gmail.com> wrote:

>Currently, bonding only obtain the timestamp (ts) information of
>the active slave, which is available only for modes 1, 5, and 6.
>For other modes, bonding only has software rx timestamping support.
>
>However, some users who use modes such as LACP also want tx timestamp
>support. To address this issue, let's check the ts information of each
>slave. If all slaves support tx timestamping, we can enable tx
>timestamping support for the bond.
>
>Suggested-by: Miroslav Lichvar <mlichvar@redhat.com>
>Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>

Acked-by: Jay Vosburgh <jay.vosburgh@canonical.com>

	-J

>---
>v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
>    RTNL. Only check _TX_SOFTWARE for the slaves.
>v3: remove dev_hold/dev_put. remove the '\' for line continuation.
>v2: check each slave's ts info to make sure bond support sw tx
>    timestamping
>---
> drivers/net/bonding/bond_main.c | 32 ++++++++++++++++++++++++++++++++
> 1 file changed, 32 insertions(+)
>
>diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>index 00646aa315c3..9cf49b61f4b3 100644
>--- a/drivers/net/bonding/bond_main.c
>+++ b/drivers/net/bonding/bond_main.c
>@@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
> 				    struct ethtool_ts_info *info)
> {
> 	struct bonding *bond = netdev_priv(bond_dev);
>+	struct ethtool_ts_info ts_info;
> 	const struct ethtool_ops *ops;
> 	struct net_device *real_dev;
>+	bool sw_tx_support = false;
> 	struct phy_device *phydev;
>+	struct list_head *iter;
>+	struct slave *slave;
> 	int ret = 0;
> 
>+	ASSERT_RTNL();
>+
> 	rcu_read_lock();
> 	real_dev = bond_option_active_slave_get_rcu(bond);
> 	dev_hold(real_dev);
>@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
> 			ret = ops->get_ts_info(real_dev, info);
> 			goto out;
> 		}
>+	} else {
>+		/* Check if all slaves support software tx timestamping */
>+		rcu_read_lock();
>+		bond_for_each_slave_rcu(bond, slave, iter) {
>+			ret = -1;
>+			ops = slave->dev->ethtool_ops;
>+			phydev = slave->dev->phydev;
>+
>+			if (phy_has_tsinfo(phydev))
>+				ret = phy_ts_info(phydev, &ts_info);
>+			else if (ops->get_ts_info)
>+				ret = ops->get_ts_info(slave->dev, &ts_info);
>+
>+			if (!ret && (ts_info.so_timestamping & SOF_TIMESTAMPING_TX_SOFTWARE)) {
>+				sw_tx_support = true;
>+				continue;
>+			}
>+
>+			sw_tx_support = false;
>+			break;
>+		}
>+		rcu_read_unlock();
> 	}
> 
>+	ret = 0;
> 	info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE |
> 				SOF_TIMESTAMPING_SOFTWARE;
>+	if (sw_tx_support)
>+		info->so_timestamping |= SOF_TIMESTAMPING_TX_SOFTWARE;
>+
> 	info->phc_index = -1;
> 
> out:
>-- 
>2.38.1
>

Re: [PATCHv4 net-next] bonding: add software tx timestamping support

[Jakub Kicinski]

On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
> v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
>     RTNL. Only check _TX_SOFTWARE for the slaves.

> +	ASSERT_RTNL();
> +
>  	rcu_read_lock();
>  	real_dev = bond_option_active_slave_get_rcu(bond);
>  	dev_hold(real_dev);
> @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
>  			ret = ops->get_ts_info(real_dev, info);
>  			goto out;
>  		}
> +	} else {
> +		/* Check if all slaves support software tx timestamping */
> +		rcu_read_lock();
> +		bond_for_each_slave_rcu(bond, slave, iter) {

> +			ret = -1;
> +			ops = slave->dev->ethtool_ops;
> +			phydev = slave->dev->phydev;
> +
> +			if (phy_has_tsinfo(phydev))
> +				ret = phy_ts_info(phydev, &ts_info);
> +			else if (ops->get_ts_info)
> +				ret = ops->get_ts_info(slave->dev, &ts_info);

My comment about this path being under rtnl was to point out that we
don't need the RCU protection to iterate over the slaves. This is 
a bit of a guess, I don't know bonding, but can we not use
bond_for_each_slave() ?

As a general rule we should let all driver callbacks sleep. Drivers 
may need to consult the FW or read something over a slow asynchronous
bus which requires process / non-atomic context. RCU lock puts us in 
an atomic context. And ->get_ts_info() is a driver callback.

It's not a deal breaker if we can't avoid RCU, but if we can - we should
let the drivers sleep. Sorry if I wasn't very clear previously.

Re: [PATCHv4 net-next] bonding: add software tx timestamping support

[Jay Vosburgh]

Jakub Kicinski <kuba@kernel.org> wrote:

>On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
>> v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
>>     RTNL. Only check _TX_SOFTWARE for the slaves.
>
>> +	ASSERT_RTNL();
>> +
>>  	rcu_read_lock();
>>  	real_dev = bond_option_active_slave_get_rcu(bond);
>>  	dev_hold(real_dev);
>> @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
>>  			ret = ops->get_ts_info(real_dev, info);
>>  			goto out;
>>  		}
>> +	} else {
>> +		/* Check if all slaves support software tx timestamping */
>> +		rcu_read_lock();
>> +		bond_for_each_slave_rcu(bond, slave, iter) {
>
>> +			ret = -1;
>> +			ops = slave->dev->ethtool_ops;
>> +			phydev = slave->dev->phydev;
>> +
>> +			if (phy_has_tsinfo(phydev))
>> +				ret = phy_ts_info(phydev, &ts_info);
>> +			else if (ops->get_ts_info)
>> +				ret = ops->get_ts_info(slave->dev, &ts_info);
>
>My comment about this path being under rtnl was to point out that we
>don't need the RCU protection to iterate over the slaves. This is 
>a bit of a guess, I don't know bonding, but can we not use
>bond_for_each_slave() ?

	Ah, I missed that nuance.  And, yes, you're correct,
bond_for_each_slave() works with RTNL and we don't need RCU here if RTNL
is held.

>As a general rule we should let all driver callbacks sleep. Drivers 
>may need to consult the FW or read something over a slow asynchronous
>bus which requires process / non-atomic context. RCU lock puts us in 
>an atomic context. And ->get_ts_info() is a driver callback.

	Agreed.

>It's not a deal breaker if we can't avoid RCU, but if we can - we should
>let the drivers sleep. Sorry if I wasn't very clear previously.

	Understood; I should have remembered this, as it's been an issue
arising from the "in the middle" aspect of bonding in the past.

	-J

---
	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: [PATCHv4 net-next] bonding: add software tx timestamping support

[Hangbin Liu]

On Fri, Apr 14, 2023 at 06:02:05PM -0700, Jakub Kicinski wrote:
> On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
> > v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
> >     RTNL. Only check _TX_SOFTWARE for the slaves.
> 
> > +	ASSERT_RTNL();
> > +
> >  	rcu_read_lock();
> >  	real_dev = bond_option_active_slave_get_rcu(bond);
> >  	dev_hold(real_dev);
> > @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
> >  			ret = ops->get_ts_info(real_dev, info);
> >  			goto out;
> >  		}
> > +	} else {
> > +		/* Check if all slaves support software tx timestamping */
> > +		rcu_read_lock();
> > +		bond_for_each_slave_rcu(bond, slave, iter) {
> 
> > +			ret = -1;
> > +			ops = slave->dev->ethtool_ops;
> > +			phydev = slave->dev->phydev;
> > +
> > +			if (phy_has_tsinfo(phydev))
> > +				ret = phy_ts_info(phydev, &ts_info);
> > +			else if (ops->get_ts_info)
> > +				ret = ops->get_ts_info(slave->dev, &ts_info);
> 
> My comment about this path being under rtnl was to point out that we
> don't need the RCU protection to iterate over the slaves. This is 
> a bit of a guess, I don't know bonding, but can we not use
> bond_for_each_slave() ?

Sorry, I misunderstood your comment in patchv3. I thought you agreed to use
the same use case like rlb_next_rx_slave(). I will post a new fix version.

> 
> As a general rule we should let all driver callbacks sleep. Drivers 
> may need to consult the FW or read something over a slow asynchronous
> bus which requires process / non-atomic context. RCU lock puts us in 
> an atomic context. And ->get_ts_info() is a driver callback.
> 
> It's not a deal breaker if we can't avoid RCU, but if we can - we should
> let the drivers sleep. Sorry if I wasn't very clear previously.

Thanks for the explanation.

Regards
Hangbin

Re: [PATCHv4 net-next] bonding: add software tx timestamping support

[Hangbin Liu]

On Fri, Apr 14, 2023 at 08:43:14PM -0700, Jay Vosburgh wrote:
> Jakub Kicinski <kuba@kernel.org> wrote:
> 
> >On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
> >> v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
> >>     RTNL. Only check _TX_SOFTWARE for the slaves.
> >
> >> +	ASSERT_RTNL();
> >> +
> >>  	rcu_read_lock();
> >>  	real_dev = bond_option_active_slave_get_rcu(bond);
> >>  	dev_hold(real_dev);
> >> @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
> >>  			ret = ops->get_ts_info(real_dev, info);
> >>  			goto out;
> >>  		}
> >> +	} else {
> >> +		/* Check if all slaves support software tx timestamping */
> >> +		rcu_read_lock();
> >> +		bond_for_each_slave_rcu(bond, slave, iter) {
> >
> >> +			ret = -1;
> >> +			ops = slave->dev->ethtool_ops;
> >> +			phydev = slave->dev->phydev;
> >> +
> >> +			if (phy_has_tsinfo(phydev))
> >> +				ret = phy_ts_info(phydev, &ts_info);
> >> +			else if (ops->get_ts_info)
> >> +				ret = ops->get_ts_info(slave->dev, &ts_info);
> >
> >My comment about this path being under rtnl was to point out that we
> >don't need the RCU protection to iterate over the slaves. This is 
> >a bit of a guess, I don't know bonding, but can we not use
> >bond_for_each_slave() ?
> 
> 	Ah, I missed that nuance.  And, yes, you're correct,
> bond_for_each_slave() works with RTNL and we don't need RCU here if RTNL
> is held.

Hi Jay, Jakub,

I remember why I use bond_for_each_slave_rcu() here now. In commit
9b80ccda233f ("bonding: fix missed rcu protection"), I added the
rcu_read_lock() as syzbot reported[1] the following path doesn't hold
rtnl lock.
- sock_setsockopt
  - sock_set_timestamping
    - sock_timestamping_bind_phc
      - ethtool_get_phc_vclocks
        - __ethtool_get_ts_info
	  - bond_ethtool_get_ts_info

[1] https://lore.kernel.org/netdev/20220513084819.zrg4ssnw667rhndt@skbuf/T/

Thanks
Hangbin

Re: [PATCHv4 net-next] bonding: add software tx timestamping support

[Jakub Kicinski]

On Mon, 17 Apr 2023 08:17:39 +0800 Hangbin Liu wrote:
> I remember why I use bond_for_each_slave_rcu() here now. In commit
> 9b80ccda233f ("bonding: fix missed rcu protection"), I added the
> rcu_read_lock() as syzbot reported[1] the following path doesn't hold
> rtnl lock.
> - sock_setsockopt
>   - sock_set_timestamping
>     - sock_timestamping_bind_phc
>       - ethtool_get_phc_vclocks
>         - __ethtool_get_ts_info
> 	  - bond_ethtool_get_ts_info

Well spotted, okay :(
Could you respin with this info added to the commit message and an
update to the kdoc in include/linux/ethtool.h for @get_ts_info
that it may be called with RCU, or rtnl or reference on the device?