netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jonathan Toppins <jtoppins@redhat.com>
To: Jay Vosburgh <jay.vosburgh@canonical.com>
Cc: "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Veaceslav Falico <vfalico@gmail.com>,
	Andy Gospodarek <andy@greyhouse.net>,
	"David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: bonding: link state question
Date: Sat, 7 Aug 2021 20:09:31 -0400	[thread overview]
Message-ID: <d2dfeba3-6cd6-1760-0abb-6005659ac125@redhat.com> (raw)
In-Reply-To: <22626.1628376134@famine>

On 8/7/21 6:42 PM, Jay Vosburgh wrote:
> Jonathan Toppins <jtoppins@redhat.com> wrote:
> 
>> Is there any reason why bonding should have an operstate of up when none
>> of its slaves are in an up state? In this particular scenario it seems
>> like the bonding device should at least assert NO-CARRIER, thoughts?
>>
>> $ ip -o -d link show | grep "bond5"
>> 2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc
>> fq_codel master bond5 state DOWN mode DEFAULT group default qlen 1000\
>> link/ether 8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68
>> maxmtu 9000 \    bond_slave state ACTIVE mii_status UP link_failure_count
>> 0 perm_hwaddr 8c:8c:aa:f8:62:16 queue_id 0 numtxqueues 1 numrxqueues 1
>> gso_max_size 65536 gso_max_segs 65535
>> 41: bond5: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue
>> state UP mode DEFAULT group default qlen 1000\    link/ether
>> 8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu
>> 65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0
>> peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none
> 
> 	I'm going to speculate that your problem is that miimon and
> arp_interval are both 0, and the bond then doesn't have any active
> mechanism to monitor the link state of its interfaces.  There might be a
> warning in dmesg to this effect.
> 
> 	Do you see what you'd consider to be correct behavior if miimon
> is set to 100?
> 

setting miimon = 100 does appear to fix it.

It is interesting that there is no link monitor on by default. For 
example when I enslave enp0s31f6 to a new bond with miimon == 0, 
enp0s31f6 starts admin down and will never de-assert NO-CARRIER the bond 
always results in an operstate of up. It seems like miimon = 100 should 
be the default since some modes cannot use arpmon.

Thank you for the discussion, see below for the steps taken.

$ sudo ip link set dev enp0s31f6 nomaster
$ sudo ip link add dev bond6 type bond mode balance-xor
$ sudo ip -o -d link set dev bond6 up
$ ip -o -d link show dev bond6
62: bond6: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc 
noqueue state DOWN mode DEFAULT group default qlen 1000\    link/ether 
3e:12:01:8a:ed:b1 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 
65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0 
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none 
arp_all_targets any primary_reselect always fail_over_mac none 
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select 
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 
gso_max_segs 65535
$ ip -o -d link show dev enp0s31f6
2: enp0s31f6: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN 
mode DEFAULT group default qlen 1000\    link/ether 8c:8c:aa:f8:62:16 
brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9000 numtxqueues 1 
numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
$ sudo ip -o -d link set dev enp0s31f6 master bond6
$ ip -o -d link show dev bond6
62: bond6: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000\    link/ether 
8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 
65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0 
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none 
arp_all_targets any primary_reselect always fail_over_mac none 
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select 
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 
gso_max_segs 65535
$ sudo ip link set dev enp0s31f6 nomaster
$ ip -o -d link show dev bond6
62: bond6: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc 
noqueue state DOWN mode DEFAULT group default qlen 1000\    link/ether 
ae:b8:6e:b3:ca:3f brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 
65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0 
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none 
arp_all_targets any primary_reselect always fail_over_mac none 
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select 
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 
gso_max_segs 65535


  reply	other threads:[~2021-08-08  0:09 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-07 21:26 bonding: link state question Jonathan Toppins
2021-08-07 22:42 ` Jay Vosburgh
2021-08-08  0:09   ` Jonathan Toppins [this message]
2021-08-08  4:49     ` Willy Tarreau
2021-08-09  1:31       ` Jonathan Toppins
2021-08-09 10:10         ` Willy Tarreau

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d2dfeba3-6cd6-1760-0abb-6005659ac125@redhat.com \
    --to=jtoppins@redhat.com \
    --cc=andy@greyhouse.net \
    --cc=davem@davemloft.net \
    --cc=jay.vosburgh@canonical.com \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=vfalico@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).