From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga12.intel.com ([192.55.52.136]:29132 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725993AbgHGRBk (ORCPT ); Fri, 7 Aug 2020 13:01:40 -0400 Date: Fri, 7 Aug 2020 10:01:19 -0700 From: Jesse Brandeburg Subject: Re: Kernel Panic when loading XDP on i40e driver Message-ID: <20200807100119.000078ab@intel.com> In-Reply-To: References: MIME-Version: 1.0 Sender: xdp-newbies-owner@vger.kernel.org List-ID: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable To: Rafael Vargas Cc: xdp-newbies@vger.kernel.org, IWL On Fri, 7 Aug 2020 12:51:36 -0300 Rafael Vargas wrote: > Hello, >=20 > I've been using XDP for some time and now I've encountered this Kernel > Panic that is puzzling me. >=20 > Using kernel 5.8 (mainline from kernel.org) and a Lenovo Intel X722 > controller (using the i40e driver). Loading any XDP program to this > driver will immediately crash with a Kernel Panic. +Intel Wired LAN list for more coverage, this looks like a legit bug that we need to fix, thanks for reporting it! >=20 > I've seen 3 different machines getting the same problem, all with the > i40e driver. Tried getting the 2.12.6 driver version from the Intel > (kernel ships with 2.8.20-k) site and it worked on 2 machines. >=20 > The Kernel crash dump (using kdump) in the third one pointed me to > this point in the code: >=20 > i40e_xdp_setup: > ... > for (i =3D 0; i < vsi->num_queue_pairs; i++) > WRITE_ONCE(vsi->rx_rings[i]->xdp_prog, vsi->xdp_prog); >=20 > And found this note from Bj=F6rn T=F6pel in an old email: >=20 > On 2020-01-20 18:04 Bj=F6rn T=F6pel wrote: > [...] > > Long story short, the i40e crash is that the drivers tries to > > allocate 256 queues, but the HW is short on queues. The drivers > > enters a broken state, which triggers the crash. >=20 >=20 > The machine that didn't work has 72 cpus, setting maxcpus=3D64 in the > kernel cmdline made it work (but only with the 2.12.6 driver). Tried > this as ethtool showed that the device has 128 channels. Using the > kernel-5.8 i40e default driver (2.8.20-k) it crashes with any number > of cpus. >=20 > Is there anything I can do to make it work (as I've seen a lot of > people using XDP with the i40e driver in this mailing list). Or any > more data that I can provide so we can patch it? >=20 > This is the kernel log from the dump (using the i40e driver from the > 5.8.0 kernel): >=20 > =3D=3D=3D=3D=3D=3D=3D Log start =3D=3D=3D=3D=3D > [61064.083688] i40e 0000:08:00.0: DCB is not supported or FW LLDP is > disabled [61064.083690] i40e 0000:08:00.0: DCB init failed -64, > disabled [61064.084062] i40e 0000:08:00.0: failed to get tracking for > 256 queues for VSI 0 err -12 > [61064.084064] i40e 0000:08:00.0: setup of MAIN VSI failed > [61064.084091] i40e 0000:08:00.0: can't remove VEB 160 with 0 VSIs > left [61064.084096] BUG: kernel NULL pointer dereference, address: > 0000000000000000 [61064.084717] #PF: supervisor read access in kernel > mode [61064.085323] #PF: error_code(0x0000) - not-present page > [61064.085930] PGD 0 P4D 0 > [61064.086533] Oops: 0000 [#1] SMP PTI > [61064.087124] CPU: 40 PID: 5000 Comm: xdprouter_cli Kdump: loaded Not > tainted 5.8.0 #1 > [61064.087722] Hardware name: Lenovo ThinkSystem SR630 > -[7X02CTO1WW]-/-[7X02CTO1WW]-, BIOS -[IVE152L-2.51]- 01/14/2020 > [61064.088342] RIP: 0010:i40e_xdp+0x13c/0x1e0 [i40e] > [61064.088962] Code: 01 00 00 00 be 01 00 00 00 48 89 cf e8 2d dc ff > ff 31 c0 66 83 bb f6 0c 00 00 00 74 27 48 8b 93 90 0c 00 00 48 63 c8 > 83 c0 01 <48> 8b 14 ca 48 8b 8b d0 0c 00 00 48 89 4a 20 0f b7 93 f6 0c > 00 00 > [61064.090293] RSP: 0018:ffffb222e0357850 EFLAGS: 00010202 > [61064.090960] RAX: 0000000000000001 RBX: ffff92240fc3a000 RCX: > 0000000000000000 [61064.091628] RDX: 0000000000000000 RSI: > ffff92241fd98a20 RDI: ffff92241fd98a28 [61064.092289] RBP: > 000000001fca8301 R08: 0000000000000001 R09: 0000000000000630 > [61064.092945] R10: 0000000000000000 R11: 0000000000000000 R12: > 0000000000000000 [61064.093590] R13: ffffb222e0357c01 R14: > ffffb222c698e000 R15: 00000000ffffff00 [61064.094230] FS: > 00007fe85289cb40(0000) GS:ffff92241fd80000(0000) > knlGS:0000000000000000 [61064.094868] CS: 0010 DS: 0000 ES: 0000 > CR0: 0000000080050033 [61064.095498] CR2: 0000000000000000 CR3: > 00000007fa00c005 CR4: 00000000007606e0 [61064.096131] DR0: > 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [61064.096769] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 [61064.097390] PKRU: 55555554 [61064.097993] Call > Trace: [61064.098601] ? i40e_reconfig_rss_queues+0x160/0x160 [i40e] > [61064.099212] dev_xdp_install+0x12e/0x150 [61064.099825] ? > i40e_reconfig_rss_queues+0x160/0x160 [i40e] [61064.100446] > dev_change_xdp_fd+0x1a6/0x310 [61064.101073] do_setlink+0xdf5/0xe50 > [61064.101703] ? rtnl_dump_ifinfo+0x31e/0x5c0 > [61064.102335] ? __nla_validate_parse.part.6+0x57/0xa20 > [61064.102957] rtnl_setlink+0x107/0x160 > [61064.103580] rtnetlink_rcv_msg+0x291/0x360 > [61064.104201] ? __check_object_size+0x40/0x1b0 > [61064.104831] ? _cond_resched+0x16/0x40 > [61064.105440] ? kmem_cache_alloc_node+0x192/0x640 > [61064.106047] ? __alloc_skb+0x57/0x1b0 > [61064.106649] ? rtnl_calcit.isra.33+0x120/0x120 > [61064.107249] netlink_rcv_skb+0xd1/0x110 > [61064.107845] netlink_unicast+0x21d/0x300 > [61064.108439] netlink_sendmsg+0x323/0x460 > [61064.109024] sock_sendmsg+0x5b/0x60 > [61064.109602] __sys_sendto+0xd8/0x150 > [61064.110176] ? __sys_getsockname+0xb2/0xc0 > [61064.110756] ? syscall_trace_enter+0x1ad/0x2b0 > [61064.111339] ? __audit_syscall_exit+0x1e4/0x290 > [61064.111927] ? __prepare_exit_to_usermode+0x7e/0x1d0 > [61064.112522] __x64_sys_sendto+0x24/0x30 > [61064.113123] do_syscall_64+0x44/0xb0 > [61064.113722] entry_SYSCALL_64_after_hwframe+0x44/0xa9 > [61064.114320] RIP: 0033:0x7fe8522615dd > [61064.114921] Code: 79 20 00 f7 d8 64 89 01 48 83 c8 ff c3 8b 05 fa > bd 20 00 85 c0 75 3e 48 63 ff 45 31 c9 45 31 c0 4c 63 d1 b8 2c 00 00 > 00 0f 05 <48> 3d 00 f0 ff ff 77 0b c3 66 2e 0f 1f 84 00 00 00 00 00 48 > 8b 15 > [61064.116244] RSP: 002b:00007ffe67117248 EFLAGS: 00000246 ORIG_RAX: > 000000000000002c > [61064.116942] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: > 00007fe8522615dd [61064.117629] RDX: 000000000000002c RSI: > 00007ffe67117270 RDI: 0000000000000009 [61064.118325] RBP: > 00007ffe67118310 R08: 0000000000000000 R09: 0000000000000000 > [61064.119024] R10: 0000000000000000 R11: 0000000000000246 R12: > 0000560835236cc0 [61064.119731] R13: 0000560835233bf0 R14: > 00005608368aa890 R15: 0000000000000000 [61064.120439] Modules linked > in: intel_rapl_msr intel_rapl_common skx_edac nfit libnvdimm > ip6t_REJECT nf_reject_ipv6 cbc encrypted_keys ip6table_filter > ip6table_mangle ip6table_raw ip6_tables joydev x86_pkg_temp_thermal > coretemp ipt_REJECT nf_reject_ipv4 kvm_intel xt_tcpudp xt_recent > xt_conntrack iptable_filter iptable_mangle iptable_raw iptable_nat > kvm nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 > nf_defrag_ipv4 libcrc32c cdc_ether usbnet mii irqbypass > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel > crypto_simd cryptd glue_helper binfmt_misc rapl mgag200 intel_cstate > drm_vram_helper nls_ascii drm_ttm_helper snd_pcm nls_cp437 snd_timer > ttm vfat snd soundcore fat drm_kms_helper efi_pstore intel_uncore drm > mei_me lpc_ichioatdma ipmi_si iTCO_wdt efivars pcspkr sg i2c_algo_bit > iTCO_vendor_support mfd_core ipmi_devintf dca mei hid_generic > ipmi_msghandler evdev acpi_power_meter button usbhid hid loop > efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache > jbd2 [61064.120472] ses enclosure sd_mod t10_pi scsi_transport_sas > xhci_pci i40e ahci xhci_hcd libahci ptp crc32c_intel megaraid_sas > libata i2c_i801 pps_core i2c_smbus usbcore scsi_mod wmi > [61064.125520] CR2: 0000000000000000 >=20 >=20 > crash> bt > PID: 5000 TASK: ffff9223fd64c2c0 CPU: 40 COMMAND: "xdp_cli" > #0 [ffffb222e0357570] machine_kexec at ffffffffa4a5dbbb > #1 [ffffb222e03575c8] __crash_kexec at ffffffffa4b33ded > #2 [ffffb222e0357690] crash_kexec at ffffffffa4b34d38 > #3 [ffffb222e03576a8] oops_end at ffffffffa4a2d7f8 > #4 [ffffb222e03576c8] no_context at ffffffffa4a6b1f2 > #5 [ffffb222e0357738] exc_page_fault at ffffffffa51bc93f > #6 [ffffb222e03577a0] asm_exc_page_fault at ffffffffa5200ade > #7 [ffffb222e0357828] i40e_xdp at ffffffffc05e98ec [i40e] > #8 [ffffb222e0357888] dev_xdp_install at ffffffffa505925e > #9 [ffffb222e03578e8] dev_change_xdp_fd at ffffffffa5059ca6 > #10 [ffffb222e0357938] do_setlink at ffffffffa506c425 > #11 [ffffb222e0357a60] rtnl_setlink at ffffffffa506c587 > #12 [ffffb222e0357c70] rtnetlink_rcv_msg at ffffffffa50662f1 > #13 [ffffb222e0357cf0] netlink_rcv_skb at ffffffffa50b9bf1 > #14 [ffffb222e0357d40] netlink_unicast at ffffffffa50b930d > #15 [ffffb222e0357d80] netlink_sendmsg at ffffffffa50b9713 > #16 [ffffb222e0357df0] sock_sendmsg at ffffffffa502f40b > #17 [ffffb222e0357e08] __sys_sendto at ffffffffa50305a8 > #18 [ffffb222e0357f38] __x64_sys_sendto at ffffffffa5030644 > #19 [ffffb222e0357f40] do_syscall_64 at ffffffffa51b9614 > #20 [ffffb222e0357f50] entry_SYSCALL_64_after_hwframe at > ffffffffa520008c RIP: 00007fe8522615dd RSP: 00007ffe67117248 > RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 0000000000000002 RCX: > 00007fe8522615dd RDX: 000000000000002c RSI: 00007ffe67117270 RDI: > 0000000000000009 RBP: 00007ffe67118310 R8: 0000000000000000 R9: > 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: > 0000560835236cc0 R13: 0000560835233bf0 R14: 00005608368aa890 R15: > 0000000000000000 ORIG_RAX: 000000000000002c CS: 0033 SS: 002b > =3D=3D=3D=3D=3D=3D=3D Log end =3D=3D=3D=3D=3D >=20 > Best regards >=20