linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Joseph Marsden <joseph@arctarus.co.uk>
To: davem@davemloft.net
Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: PROBLEM: IPv4 route exception patch breaks BIRD2
Date: Thu, 11 Jun 2020 19:10:44 +0100	[thread overview]
Message-ID: <8da4a321-a676-0464-f5cf-d467840dbfa7@arctarus.co.uk> (raw)
In-Reply-To: <1d797541-3f4a-34fd-669c-0dad6dffd4e0@arctarus.co.uk>

Hi all,

While upgrading my kernel from 5.2 to 5.4.43, I noticed a very sharp
increase in CPU usage - coming from BIRD, the routing daemon on my system.
I ran a Netlink capture and saw that BIRD was attempting to insert
routes twice into the kernel routing table. This was causing routes to
be re-inserted rapidly, and explained the increase of CPU usage.

I'm running IPv4 and IPv6 full tables - about 789,360 v4 routes and
78,185 v6 routes. About 10,000 routes fail to be re-inserted and this
number fluctuated over time as the routing table was updated. The routes
themselves are sourced from BGP peerings with upstreams on the router.

I tried downgrading BIRD itself several times before realising that it
must be a kernel issue, so I performed a bisect, and found this patch to
be the issue:

[PATCH net-next v7 04/11] ipv4: Dump route exceptions if requested
https://lore.kernel.org/netdev/8d3b68cd37fb5fddc470904cdd6793fcf480c6c1.1561131177.git.sbrivio@redhat.com/

When I added some debug logging to BIRD, I could see that before the
patch BIRD was not receiving any FIB exception routes, but after the
patch, it was receiving many.
I suspect this is unrelated to the bigger issue here - even if I patch
BIRD to reject these routes, the problem still occurs and BIRD keeps
trying to re-insert routes into the kernel routing table twice.

- Joseph

Extra information:
Kernel version: Linux version 5.4.43 (nixbld@localhost) (gcc version
9.2.0 (GCC)) #1-NixOS SMP Wed May 27 15:46:53 UTC 2020
Most recent kernel version which did not have the bug: 5.2.0
BIRD version I tested this on: 2.0.7 (also tested on 2.0.2 and other
older versions)

ver_linux output:
Linux bdr1 5.4.43 #1-NixOS SMP Wed May 27 15:46:53 UTC 2020 x86_64 GNU/Linux

Util-linux          	2.33.2
Mount               	2.33.2
Module-init-tools   	26
E2fsprogs           	1.45.5
Linux C Library     	2.30
Dynamic linker (ldd)	2.30
Procps              	3.3.16
Net-tools           	1.60
Kbd                 	2.0.4
Console-tools       	2.0.4
Sh-utils            	8.31
Udev                	243
Modules Loaded      	agpgart ata_generic ata_piix atkbd autofs4
bochs_drm bridge br_netfilter button cdrom crc16 crc32c_generic
crc_ccitt dm_mod drm drm_kms_helper drm_vram_helper dummy ehci_hcd evdev
ext4 failover fb_sys_fops floppy gre hid hid_generic i2c_core i2c_piix4
i8042 input_leds intel_agp intel_gtt ip6_gre ip6table_filter
ip6table_nat ip6table_raw ip6_tables ip6_tunnel iptable_filter
iptable_nat iptable_raw ip_tables ipv6 jbd2 joydev led_class libata
libcrc32c libps2 llc loop mac_hid macvlan mbcache mousedev net_failover
nf_conntrack nf_conntrack_netlink nf_defrag_ipv4 nf_defrag_ipv6
nf_log_common nf_log_ipv4 nf_log_ipv6 nf_nat nfnetlink overlay pata_acpi
psmouse qemu_fw_cfg rng_core rtc_cmos sch_fq_codel scsi_mod serio
serio_raw sr_mod stp syscopyarea sysfillrect sysimgblt tap ttm tun
tunnel6 uhci_hcd usb_common usbcore usbhid veth virtio virtio_balloon
virtio_blk virtio_console virtio_net virtio_pci virtio_ring virtio_rng
xfrm_algo xfrm_user x_tables xt_addrtype xt_conntrack xt_LOG
xt_MASQUERADE xt_multiport xt_nat xt_pkttype xt_tcpudp


           reply	other threads:[~2020-06-11 18:10 UTC|newest]

Thread overview: expand[flat|nested]  mbox.gz  Atom feed
 [parent not found: <1d797541-3f4a-34fd-669c-0dad6dffd4e0@arctarus.co.uk>]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8da4a321-a676-0464-f5cf-d467840dbfa7@arctarus.co.uk \
    --to=joseph@arctarus.co.uk \
    --cc=davem@davemloft.net \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).