netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Alexander Duyck <alexander.duyck@gmail.com>
To: Jarod Wilson <jarod@redhat.com>, linux-kernel@vger.kernel.org
Cc: "David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@google.com>,
	Jay Vosburgh <j.vosburgh@gmail.com>,
	Veaceslav Falico <vfalico@gmail.com>,
	Andy Gospodarek <gospo@cumulusnetworks.com>,
	Jiri Pirko <jiri@resnulli.us>,
	Nikolay Aleksandrov <razor@blackwall.org>,
	Michal Kubecek <mkubecek@suse.cz>,
	netdev@vger.kernel.org
Subject: Re: [PATCH net-next] net/core: generic support for disabling netdev features down stack
Date: Mon, 2 Nov 2015 10:04:39 -0800	[thread overview]
Message-ID: <5637A5B7.6070405@gmail.com> (raw)
In-Reply-To: <1446486818-26166-1-git-send-email-jarod@redhat.com>

On 11/02/2015 09:53 AM, Jarod Wilson wrote:
> There are some netdev features, which when disabled on an upper device,
> such as a bonding master or a bridge, must be disabled and cannot be
> re-enabled on underlying devices.
>
> This is a rework of an earlier more heavy-handed appraoch, which simply
> disables and prevents re-enabling of netdev features listed in a new
> define in include/net/netdev_features.h, NETIF_F_UPPER_DISABLES. Any upper
> device that disables a flag in that feature mask, the disabling will
> propagate down the stack, and any lower device that has any upper device
> with one of those flags disabled should not be able to enable said flag.
>
> Initially, only LRO is included for proof of concept, and because this
> code effectively does the same thing as dev_disable_lro(), though it will
> also activate from the ethtool path, which was one of the goals here.
>
> [root@dell-per730-01 ~]# ethtool -k bond0 |grep large
> large-receive-offload: on
> [root@dell-per730-01 ~]# ethtool -k p5p1 |grep large
> large-receive-offload: on
> [root@dell-per730-01 ~]# ethtool -K bond0 lro off
> [root@dell-per730-01 ~]# ethtool -k bond0 |grep large
> large-receive-offload: off
> [root@dell-per730-01 ~]# ethtool -k p5p1 |grep large
> large-receive-offload: off
>
> dmesg dump:
>
> [ 1033.277986] bond0: Disabling feature 0x0000000000008000 on lower dev p5p2.
> [ 1034.067949] bnx2x 0000:06:00.1 p5p2: using MSI-X  IRQs: sp 74  fp[0] 76 ... fp[7] 83
> [ 1034.753612] bond0: Disabling feature 0x0000000000008000 on lower dev p5p1.
> [ 1035.591019] bnx2x 0000:06:00.0 p5p1: using MSI-X  IRQs: sp 62  fp[0] 64 ... fp[7] 71
>
> This has been successfully tested with bnx2x, qlcnic and netxen network
> cards as slaves in a bond interface. Turning LRO on or off on the master
> also turns it on or off on each of the slaves, new slaves are added with
> LRO in the same state as the master, and LRO can't be toggled on the
> slaves.
>
> Also, this should largely remove the need for dev_disable_lro(), and most,
> if not all, of its call sites can be replaced by simply making sure
> NETIF_F_LRO isn't included in the relevant device's feature flags.
>
> Note that this patch is driven by bug reports from users saying it was
> confusing that bonds and slaves had different settings for the same
> features, and while it won't be 100% in sync if a lower device doesn't
> support a feature like LRO, I think this is a good step in the right
> direction.
>
> CC: "David S. Miller" <davem@davemloft.net>
> CC: Eric Dumazet <edumazet@google.com>
> CC: Jay Vosburgh <j.vosburgh@gmail.com>
> CC: Veaceslav Falico <vfalico@gmail.com>
> CC: Andy Gospodarek <gospo@cumulusnetworks.com>
> CC: Jiri Pirko <jiri@resnulli.us>
> CC: Nikolay Aleksandrov <razor@blackwall.org>
> CC: Michal Kubecek <mkubecek@suse.cz>
> CC: Alexander Duyck <alexander.duyck@gmail.com>
> CC: netdev@vger.kernel.org
> Signed-off-by: Jarod Wilson <jarod@redhat.com>
> ---
> Note: this replaces "[RFC PATCH net-next] net/core: initial support for
> stacked dev feature toggles" for consideration.
>
>   include/linux/netdev_features.h | 11 +++++++++
>   net/core/dev.c                  | 52 +++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 63 insertions(+)
>
> diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h
> index 9672781..0f5837a 100644
> --- a/include/linux/netdev_features.h
> +++ b/include/linux/netdev_features.h
> @@ -125,6 +125,11 @@ enum {
>   #define NETIF_F_HW_L2FW_DOFFLOAD	__NETIF_F(HW_L2FW_DOFFLOAD)
>   #define NETIF_F_BUSY_POLL	__NETIF_F(BUSY_POLL)
>
> +#define for_each_netdev_feature(mask_addr, feature)				\
> +	int bit;								\
> +	for_each_set_bit(bit, (unsigned long *)mask_addr, NETDEV_FEATURE_COUNT)	\
> +		feature = __NETIF_F_BIT(bit);
> +
>   /* Features valid for ethtool to change */
>   /* = all defined minus driver/device-class-related */
>   #define NETIF_F_NEVER_CHANGE	(NETIF_F_VLAN_CHALLENGED | \
> @@ -167,6 +172,12 @@ enum {
>    */
>   #define NETIF_F_ALL_FOR_ALL	(NETIF_F_NOCACHE_COPY | NETIF_F_FSO)
>
> +/*
> + * If upper/master device has these features disabled, they must be disabled
> + * on all lower/slave devices as well.
> + */
> +#define NETIF_F_UPPER_DISABLES	NETIF_F_LRO
> +
>   /* changeable features with no special hardware requirements */
>   #define NETIF_F_SOFT_FEATURES	(NETIF_F_GSO | NETIF_F_GRO)
>
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 13f49f8..3a8dbbc 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -6288,9 +6288,51 @@ static void rollback_registered(struct net_device *dev)
>   	list_del(&single);
>   }
>
> +static netdev_features_t netdev_sync_upper_features(struct net_device *lower,
> +	struct net_device *upper, netdev_features_t features)
> +{
> +	netdev_features_t upper_disables = NETIF_F_UPPER_DISABLES;
> +	netdev_features_t feature;
> +
> +	for_each_netdev_feature(&upper_disables, feature) {
> +		if (!(upper->wanted_features & feature)
> +		    && (features & feature)) {
> +			netdev_dbg(lower, "Dropping feature %pNF, upper dev %s has it off.\n",
> +				   &feature, upper->name);
> +			features &= ~feature;
> +		}
> +	}
> +
> +	return features;
> +}
> +
> +static void netdev_sync_lower_features(struct net_device *upper,
> +	struct net_device *lower, netdev_features_t features)
> +{
> +	netdev_features_t upper_disables = NETIF_F_UPPER_DISABLES;
> +	netdev_features_t feature;
> +
> +	for_each_netdev_feature(&upper_disables, feature) {
> +		if (!(features & feature) && (lower->features & feature)) {
> +			netdev_dbg(upper, "Disabling feature %pNF on lower dev %s.\n",
> +				   &feature, lower->name);
> +			upper->wanted_features &= ~feature;

Isn't this line redundant? The upper device should have already cleared 
the bit from the wanted_features?  That is unless the ndo_fix_features 
call modified it in which case we shouldn't be modifying it ourselves.

> +			lower->wanted_features &= ~feature;
> +			netdev_update_features(lower);
> +
> +			if (unlikely(lower->features & feature))
> +				netdev_WARN(upper, "failed to disable %pNF on %s!\n",
> +					    &feature, lower->name);
> +		}
> +	}
> +}
> +
>   static netdev_features_t netdev_fix_features(struct net_device *dev,
>   	netdev_features_t features)
>   {
> +	struct net_device *upper, *lower;
> +	struct list_head *iter;
> +
>   	/* Fix illegal checksum combinations */
>   	if ((features & NETIF_F_HW_CSUM) &&
>   	    (features & (NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))) {
> @@ -6345,6 +6387,16 @@ static netdev_features_t netdev_fix_features(struct net_device *dev,
>   		}
>   	}
>
> +	/* some features can't be enabled if they're off an an upper device */
> +	netdev_for_each_upper_dev_rcu(dev, upper, iter)
> +		features = netdev_sync_upper_features(dev, upper, features);
> +
> +	/* some features must be disabled on lower devices when disabled
> +	 * on an upper device (think: bonding master or bridge)
> +	 */
> +	netdev_for_each_lower_dev(dev, lower, iter)
> +		netdev_sync_lower_features(dev, lower, features);
> +

I don't know if this is the right spot for this.  You might want to look 
at placing this after the ndo_set_features call to handle things if 
there wasn't an error.  That way if a lower device for some reason has 
an issue with one of the other settings being changed you don't end up 
in a state where all the lower devices have the feature stripped while 
the upper device still reports it as being enabled.

>   #ifdef CONFIG_NET_RX_BUSY_POLL
>   	if (dev->netdev_ops->ndo_busy_poll)
>   		features |= NETIF_F_BUSY_POLL;
>

  reply	other threads:[~2015-11-02 18:04 UTC|newest]

Thread overview: 49+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-10-24  3:40 [RFC PATCH net-next] net/core: initial support for stacked dev feature toggles Jarod Wilson
2015-10-24  4:41 ` Tom Herbert
2015-10-24  5:51 ` Alexander Duyck
2015-10-26  9:42   ` Michal Kubecek
2015-10-30 16:25     ` Jarod Wilson
2015-10-30 20:02       ` Alexander Duyck
2015-11-02 17:37         ` Jarod Wilson
2015-10-30 16:35   ` Jarod Wilson
2015-10-30 20:14     ` Alexander Duyck
2015-11-02 17:53 ` [PATCH net-next] net/core: generic support for disabling netdev features down stack Jarod Wilson
2015-11-02 18:04   ` Alexander Duyck [this message]
2015-11-02 21:57     ` Jarod Wilson
2015-11-03  2:55   ` [PATCH v2 " Jarod Wilson
2015-11-03  4:41     ` David Miller
2015-11-03 10:03     ` Nikolay Aleksandrov
2015-11-03 13:52       ` Geert Uytterhoeven
2015-11-03 13:57         ` Jarod Wilson
2015-11-03 14:05           ` Nikolay Aleksandrov
2015-11-03 15:18             ` Jarod Wilson
2015-11-03 15:15     ` [PATCH net-next] net/core: fix for_each_netdev_feature Jarod Wilson
2015-11-03 15:33       ` Nikolay Aleksandrov
2015-11-03 16:34       ` David Miller
2015-11-03 20:36     ` [PATCH net-next] net/core: ensure features get disabled on new lower devs Jarod Wilson
2015-11-03 21:17       ` Alexander Duyck
2015-11-03 22:11         ` Jarod Wilson
2015-11-03 23:01           ` Alexander Duyck
2015-11-03 21:21       ` Nikolay Aleksandrov
2015-11-03 21:53       ` Michal Kubecek
2015-11-03 21:58         ` Jarod Wilson
2015-11-04  4:09       ` [PATCH v2 " Jarod Wilson
2015-11-05  2:56         ` David Miller
2015-11-13  0:26           ` Florian Fainelli
2015-11-13 10:29             ` Jiri Pirko
2015-11-13 10:51               ` Nikolay Aleksandrov
2015-11-13 13:54                 ` [PATCH net] net: fix feature changes on devices without ndo_set_features Nikolay Aleksandrov
2015-11-13 14:00                   ` Jiri Pirko
2015-11-13 14:06                   ` Andy Gospodarek
2015-11-13 14:34                   ` Jarod Wilson
2015-11-13 18:30                   ` Florian Fainelli
2015-11-15  7:25                   ` [net] " Dave Young
2015-11-16  2:01                     ` Dave Young
2015-11-16 19:56                   ` [PATCH net] " David Miller
2015-11-17 23:03                   ` Sergei Shtylyov
2015-11-17 23:10                     ` Nikolay Aleksandrov
2015-11-18 10:51                       ` Sergei Shtylyov
2015-11-13 22:31                 ` [PATCH v2 net-next] net/core: ensure features get disabled on new lower devs Laura Abbott
2015-11-17  9:02             ` Geert Uytterhoeven
2015-11-17 10:04               ` Geert Uytterhoeven
2016-04-02  2:21     ` [PATCH v2 net-next] net/core: generic support for disabling netdev features down stack Michał Mirosław

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5637A5B7.6070405@gmail.com \
    --to=alexander.duyck@gmail.com \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=gospo@cumulusnetworks.com \
    --cc=j.vosburgh@gmail.com \
    --cc=jarod@redhat.com \
    --cc=jiri@resnulli.us \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mkubecek@suse.cz \
    --cc=netdev@vger.kernel.org \
    --cc=razor@blackwall.org \
    --cc=vfalico@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).