From: Jonathan Toppins <jtoppins@redhat.com>
To: Jay Vosburgh <jay.vosburgh@canonical.com>
Cc: "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
Veaceslav Falico <vfalico@gmail.com>,
Andy Gospodarek <andy@greyhouse.net>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: bonding: link state question
Date: Sat, 7 Aug 2021 20:09:31 -0400 [thread overview]
Message-ID: <d2dfeba3-6cd6-1760-0abb-6005659ac125@redhat.com> (raw)
In-Reply-To: <22626.1628376134@famine>
On 8/7/21 6:42 PM, Jay Vosburgh wrote:
> Jonathan Toppins <jtoppins@redhat.com> wrote:
>
>> Is there any reason why bonding should have an operstate of up when none
>> of its slaves are in an up state? In this particular scenario it seems
>> like the bonding device should at least assert NO-CARRIER, thoughts?
>>
>> $ ip -o -d link show | grep "bond5"
>> 2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc
>> fq_codel master bond5 state DOWN mode DEFAULT group default qlen 1000\
>> link/ether 8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68
>> maxmtu 9000 \ bond_slave state ACTIVE mii_status UP link_failure_count
>> 0 perm_hwaddr 8c:8c:aa:f8:62:16 queue_id 0 numtxqueues 1 numrxqueues 1
>> gso_max_size 65536 gso_max_segs 65535
>> 41: bond5: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue
>> state UP mode DEFAULT group default qlen 1000\ link/ether
>> 8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu
>> 65535 \ bond mode balance-xor miimon 0 updelay 0 downdelay 0
>> peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none
>
> I'm going to speculate that your problem is that miimon and
> arp_interval are both 0, and the bond then doesn't have any active
> mechanism to monitor the link state of its interfaces. There might be a
> warning in dmesg to this effect.
>
> Do you see what you'd consider to be correct behavior if miimon
> is set to 100?
>
setting miimon = 100 does appear to fix it.
It is interesting that there is no link monitor on by default. For
example when I enslave enp0s31f6 to a new bond with miimon == 0,
enp0s31f6 starts admin down and will never de-assert NO-CARRIER the bond
always results in an operstate of up. It seems like miimon = 100 should
be the default since some modes cannot use arpmon.
Thank you for the discussion, see below for the steps taken.
$ sudo ip link set dev enp0s31f6 nomaster
$ sudo ip link add dev bond6 type bond mode balance-xor
$ sudo ip -o -d link set dev bond6 up
$ ip -o -d link show dev bond6
62: bond6: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc
noqueue state DOWN mode DEFAULT group default qlen 1000\ link/ether
3e:12:01:8a:ed:b1 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu
65535 \ bond mode balance-xor miimon 0 updelay 0 downdelay 0
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none
arp_all_targets any primary_reselect always fail_over_mac none
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536
gso_max_segs 65535
$ ip -o -d link show dev enp0s31f6
2: enp0s31f6: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN
mode DEFAULT group default qlen 1000\ link/ether 8c:8c:aa:f8:62:16
brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9000 numtxqueues 1
numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
$ sudo ip -o -d link set dev enp0s31f6 master bond6
$ ip -o -d link show dev bond6
62: bond6: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP mode DEFAULT group default qlen 1000\ link/ether
8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu
65535 \ bond mode balance-xor miimon 0 updelay 0 downdelay 0
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none
arp_all_targets any primary_reselect always fail_over_mac none
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536
gso_max_segs 65535
$ sudo ip link set dev enp0s31f6 nomaster
$ ip -o -d link show dev bond6
62: bond6: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc
noqueue state DOWN mode DEFAULT group default qlen 1000\ link/ether
ae:b8:6e:b3:ca:3f brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu
65535 \ bond mode balance-xor miimon 0 updelay 0 downdelay 0
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none
arp_all_targets any primary_reselect always fail_over_mac none
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536
gso_max_segs 65535
next prev parent reply other threads:[~2021-08-08 0:09 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-07 21:26 bonding: link state question Jonathan Toppins
2021-08-07 22:42 ` Jay Vosburgh
2021-08-08 0:09 ` Jonathan Toppins [this message]
2021-08-08 4:49 ` Willy Tarreau
2021-08-09 1:31 ` Jonathan Toppins
2021-08-09 10:10 ` Willy Tarreau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d2dfeba3-6cd6-1760-0abb-6005659ac125@redhat.com \
--to=jtoppins@redhat.com \
--cc=andy@greyhouse.net \
--cc=davem@davemloft.net \
--cc=jay.vosburgh@canonical.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=vfalico@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.