From: Magnus Karlsson <magnus.karlsson@gmail.com>
To: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Lorenzo Bianconi <lorenzo@kernel.org>, bpf <bpf@vger.kernel.org>,
Network Development <netdev@vger.kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
David Ahern <dsahern@kernel.org>,
Eelco Chaudron <echaudro@redhat.com>,
Jason Wang <jasowang@redhat.com>,
Alexander Duyck <alexander.duyck@gmail.com>,
Saeed Mahameed <saeed@kernel.org>,
"Fijalkowski, Maciej" <maciej.fijalkowski@intel.com>,
Tirthendu <tirthendu.sarkar@intel.com>
Subject: Re: Crash for i40e on net-next (was: [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support)
Date: Fri, 23 Apr 2021 07:28:08 +0200 [thread overview]
Message-ID: <CAJ8uoz1oEa6ZEp3QKZiPx4oUtt9-nuY4Sh6PVrEnZdu-d_PudQ@mail.gmail.com> (raw)
In-Reply-To: <20210422170508.22c58226@carbon>
On Thu, Apr 22, 2021 at 5:05 PM Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
>
> On Thu, 22 Apr 2021 16:42:23 +0200
> Jesper Dangaard Brouer <brouer@redhat.com> wrote:
>
> > On Thu, 22 Apr 2021 12:24:32 +0200
> > Magnus Karlsson <magnus.karlsson@gmail.com> wrote:
> >
> > > On Wed, Apr 21, 2021 at 5:39 PM Jesper Dangaard Brouer
> > > <brouer@redhat.com> wrote:
> > > >
> > > > On Wed, 21 Apr 2021 16:12:32 +0200
> > > > Magnus Karlsson <magnus.karlsson@gmail.com> wrote:
> > > >
> > [...]
> > > > > more than I get.
> > > >
> > > > I clearly have a bug in the i40e driver. As I wrote later, I don't see
> > > > any packets transmitted for XDP_TX. Hmm, I using Mel Gorman's tree,
> > > > which contains the i40e/ice/ixgbe bug we fixed earlier.
> >
> > Something is wrong with i40e, I changed git-tree to net-next (at
> > commit 5d869070569a) and XDP seems to have stopped working on i40e :-(
Found this out too when switching to the net tree yesterday to work on
proper packet drop tracing as you spotted/requested yesterday. The
commit below completely broke XDP support on i40e (if you do not run
with a zero-copy AF_XDP socket because that path still works). I am
working on a fix that does not just revert the patch, but fixes the
original problem without breaking XDP. Will post it and the tracing
fixes as soon as I can.
commit 12738ac4754ec92a6a45bf3677d8da780a1412b3
Author: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>
Date: Fri Mar 26 19:43:40 2021 +0100
i40e: Fix sparse errors in i40e_txrx.c
Remove error handling through pointers. Instead use plain int
to return value from i40e_run_xdp(...).
Previously:
- sparse errors were produced during compilation:
i40e_txrx.c:2338 i40e_run_xdp() error: (-2147483647) too low for ERR_PTR
i40e_txrx.c:2558 i40e_clean_rx_irq() error: 'skb' dereferencing
possible ERR_PTR()
- sk_buff* was used to return value, but it has never had valid
pointer to sk_buff. Returned value was always int handled as
a pointer.
Fixes: 0c8493d90b6b ("i40e: add XDP support for pass and drop actions")
Fixes: 2e6893123830 ("i40e: split XDP_TX tail and XDP_REDIRECT map
flushing")
Signed-off-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>
Tested-by: Dave Switzer <david.switzer@intel.com>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
> Renamed subj as this is without this patchset applied.
>
> > $ uname -a
> > Linux broadwell 5.12.0-rc7-net-next+ #600 SMP PREEMPT Thu Apr 22 15:13:15 CEST 2021 x86_64 x86_64 x86_64 GNU/Linux
> >
> > When I load any XDP prog almost no packets are let through:
> >
> > [kernel-bpf-samples]$ sudo ./xdp1 i40e2
> > libbpf: elf: skipping unrecognized data section(16) .eh_frame
> > libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
> > proto 17: 1 pkt/s
> > proto 0: 0 pkt/s
> > proto 17: 0 pkt/s
> > proto 0: 0 pkt/s
> > proto 17: 1 pkt/s
>
> Trying out xdp_redirect:
>
> [kernel-bpf-samples]$ sudo ./xdp_redirect i40e2 i40e2
> input: 7 output: 7
> libbpf: elf: skipping unrecognized data section(20) .eh_frame
> libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
> libbpf: Kernel error message: XDP program already attached
> WARN: link set xdp fd failed on 7
> ifindex 7: 7357 pkt/s
> ifindex 7: 7909 pkt/s
> ifindex 7: 7909 pkt/s
> ifindex 7: 7909 pkt/s
> ifindex 7: 7909 pkt/s
> ifindex 7: 7909 pkt/s
> ifindex 7: 6357 pkt/s
>
> And then it crash (see below) at page_frag_free+0x31 which calls
> virt_to_head_page() with a wrong addr (I guess). This is called by
> i40e_clean_tx_irq+0xc9.
Did not see a crash myself, just 4 Kpps. But the rings and DMA
mappings got completely mangled by the patch above, so could be the
same cause.
> $ ./scripts/faddr2line drivers/net/ethernet/intel/i40e/i40e.o i40e_clean_tx_irq+0xc9
> i40e_clean_tx_irq+0xc9/0x440:
> i40e_clean_tx_irq at /home/jbrouer/git/kernel/net-next/drivers/net/ethernet/intel/i40e/i40e_txrx.c:976
>
> Which is:
>
> /* unmap skb header data */
> Line:976 dma_unmap_single(tx_ring->dev,
> dma_unmap_addr(tx_buf, dma),
> dma_unmap_len(tx_buf, len),
> DMA_TO_DEVICE);
>
>
> [ 935.781751] BUG: unable to handle page fault for address: ffffebde00000008
> [ 935.788630] #PF: supervisor read access in kernel mode
> [ 935.793766] #PF: error_code(0x0000) - not-present page
> [ 935.798906] PGD 0 P4D 0
> [ 935.801445] Oops: 0000 [#1] PREEMPT SMP PTI
> [ 935.805632] CPU: 4 PID: 113 Comm: kworker/u12:9 Not tainted 5.12.0-rc7-net-next+ #600
> [ 935.813460] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
> [ 935.820937] Workqueue: events_unbound call_usermodehelper_exec_work
> [ 935.827214] RIP: 0010:page_frag_free+0x31/0x70
> [ 935.831656] Code: 00 00 80 48 01 c7 72 55 48 b8 00 00 00 80 7f 77 00 00 48 01 c7 48 b8 00 00 00 00 00 ea ff ff 48 c1 ef 0c 48 c1 e7 06 48 01 c7 <48> 8b 47 08 48 8d 50 ff a8 01 48 0f 45 fa f0 ff 4f 34 74 01 c3 48
> [ 935.850406] RSP: 0018:ffffc900001c0e50 EFLAGS: 00010286
> [ 935.855629] RAX: ffffea0000000000 RBX: ffff88813b258180 RCX: 0000000000000000
> [ 935.862764] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffebde00000000
> [ 935.869895] RBP: ffff88813b258180 R08: ffffc900001c0f38 R09: 0000000000000180
> [ 935.877028] R10: 00000000fffffe18 R11: ffffc900001c0ff8 R12: ffff88813bc403c0
> [ 935.884160] R13: 000000000000003c R14: 00000000fffffe18 R15: ffff88813b15b180
> [ 935.891295] FS: 0000000000000000(0000) GS:ffff88887fd00000(0000) knlGS:0000000000000000
> [ 935.899380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 935.905126] CR2: ffffebde00000008 CR3: 000000087e810002 CR4: 00000000003706e0
> [ 935.912259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 935.919391] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 935.926526] Call Trace:
> [ 935.928980] <IRQ>
> [ 935.930999] i40e_clean_tx_irq+0xc9/0x440 [i40e]
> [ 935.935653] i40e_napi_poll+0x101/0x410 [i40e]
> [ 935.940116] __napi_poll+0x2a/0x140
> [ 935.943607] net_rx_action+0x215/0x2d0
> [ 935.947358] ? i40e_msix_clean_rings+0x3f/0x50 [i40e]
> [ 935.952449] __do_softirq+0xe3/0x2df
> [ 935.956028] irq_exit_rcu+0xa7/0xb0
> [ 935.959519] common_interrupt+0x83/0xa0
> [ 935.963358] </IRQ>
> [ 935.965465] asm_common_interrupt+0x1e/0x40
> [ 935.969651] RIP: 0010:clear_page_erms+0x7/0x10
> [ 935.974096] Code: 48 89 47 18 48 89 47 20 48 89 47 28 48 89 47 30 48 89 47 38 48 8d 7f 40 75 d9 90 c3 0f 1f 80 00 00 00 00 b9 00 10 00 00 31 c0 <f3> aa c3 cc cc cc cc cc cc 0f 1f 44 00 00 48 85 ff 0f 84 f2 00 00
> [ 935.992845] RSP: 0018:ffffc900003bfbc8 EFLAGS: 00010246
> [ 935.998069] RAX: 0000000000000000 RBX: 0000000000000901 RCX: 0000000000000340
> [ 936.005202] RDX: 0000000000002dc2 RSI: 0000000000000000 RDI: ffff88813b4d8cc0
> [ 936.012334] RBP: ffffea0004ed3600 R08: ffff88887c625f00 R09: ffffea0004ed3600
> [ 936.019467] R10: ffff888000000000 R11: 0000160000000000 R12: ffffea0004ed3640
> [ 936.026600] R13: 0000000000000000 R14: 0000000000005c39 R15: ffffc900003bfc50
> [ 936.033738] prep_new_page+0x88/0xe0
> [ 936.037313] get_page_from_freelist+0x2c6/0x3d0
> [ 936.041847] __alloc_pages_nodemask+0x137/0x2e0
> [ 936.046380] __vmalloc_node_range+0x14f/0x270
> [ 936.050738] copy_process+0x39d/0x1ad0
> [ 936.054491] ? kernel_clone+0x8b/0x3c0
> [ 936.058244] kernel_clone+0x8b/0x3c0
> [ 936.061822] ? dequeue_entity+0xc0/0x270
> [ 936.065748] kernel_thread+0x47/0x50
> [ 936.069329] ? umh_complete+0x40/0x40
> [ 936.072992] call_usermodehelper_exec_work+0x2f/0x90
> [ 936.077960] process_one_work+0x1ad/0x380
> [ 936.081974] worker_thread+0x50/0x390
> [ 936.085638] ? process_one_work+0x380/0x380
> [ 936.089824] kthread+0x116/0x150
> [ 936.093057] ? kthread_park+0x90/0x90
> [ 936.096722] ret_from_fork+0x22/0x30
> [ 936.100307] Modules linked in: veth nf_defrag_ipv6 nf_defrag_ipv4 tun bridge stp llc rpcrdma sunrpc rdma_ucm ib_umad coretemp rdma_cm ib_ipoib kvm_intel iw_cm ib_cm kvm mlx5_ib i40iw irqbypass ib_uverbs rapl intel_cstate intel_uncore ib_core pcspkr i2c_i801 bfq i2c_smbus acpi_ipmi wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad sch_fq_codel sd_mod t10_pi ixgbe igb igc mdio mlx5_core i40e mlxfw nfp psample i2c_algo_bit ptp i2c_core pps_core hid_generic [last unloaded: bpfilter]
> [ 936.142697] CR2: ffffebde00000008
> [ 936.146015] ---[ end trace 1bffa979f2cccd16 ]---
> [ 936.156720] RIP: 0010:page_frag_free+0x31/0x70
> [ 936.161170] Code: 00 00 80 48 01 c7 72 55 48 b8 00 00 00 80 7f 77 00 00 48 01 c7 48 b8 00 00 00 00 00 ea ff ff 48 c1 ef 0c 48 c1 e7 06 48 01 c7 <48> 8b 47 08 48 8d 50 ff a8 01 48 0f 45 fa f0 ff 4f 34 74 01 c3 48
> [ 936.179919] RSP: 0018:ffffc900001c0e50 EFLAGS: 00010286
> [ 936.185140] RAX: ffffea0000000000 RBX: ffff88813b258180 RCX: 0000000000000000
> [ 936.192275] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffebde00000000
> [ 936.199407] RBP: ffff88813b258180 R08: ffffc900001c0f38 R09: 0000000000000180
> [ 936.206541] R10: 00000000fffffe18 R11: ffffc900001c0ff8 R12: ffff88813bc403c0
> [ 936.213673] R13: 000000000000003c R14: 00000000fffffe18 R15: ffff88813b15b180
> [ 936.220804] FS: 0000000000000000(0000) GS:ffff88887fd00000(0000) knlGS:0000000000000000
> [ 936.228893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 936.234638] CR2: ffffebde00000008 CR3: 000000087e810002 CR4: 00000000003706e0
> [ 936.241771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 936.248903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 936.256036] Kernel panic - not syncing: Fatal exception in interrupt
> [ 936.262401] Kernel Offset: disabled
> [ 936.271867] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>
>
> --
> Best regards,
> Jesper Dangaard Brouer
> MSc.CS, Principal Kernel Engineer at Red Hat
> LinkedIn: http://www.linkedin.com/in/brouer
>
next prev parent reply other threads:[~2021-04-23 5:28 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-08 12:50 [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 01/14] xdp: introduce mb in xdp_buff/xdp_frame Lorenzo Bianconi
2021-04-08 18:17 ` Vladimir Oltean
2021-04-09 16:03 ` Lorenzo Bianconi
2021-04-29 13:36 ` Jesper Dangaard Brouer
2021-04-29 13:54 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 02/14] xdp: add xdp_shared_info data structure Lorenzo Bianconi
2021-04-08 13:39 ` Vladimir Oltean
2021-04-08 14:26 ` Lorenzo Bianconi
2021-04-08 18:06 ` kernel test robot
2021-04-08 12:50 ` [PATCH v8 bpf-next 03/14] net: mvneta: update mb bit before passing the xdp buffer to eBPF layer Lorenzo Bianconi
2021-04-08 18:19 ` Vladimir Oltean
2021-04-09 16:24 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 04/14] xdp: add multi-buff support to xdp_return_{buff/frame} Lorenzo Bianconi
2021-04-08 18:30 ` Vladimir Oltean
2021-04-09 16:28 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 05/14] net: mvneta: add multi buffer support to XDP_TX Lorenzo Bianconi
2021-04-08 18:40 ` Vladimir Oltean
2021-04-09 16:36 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 06/14] net: mvneta: enable jumbo frames for XDP Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 07/14] net: xdp: add multi-buff support to xdp_build_skb_from_fram Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 08/14] bpf: add multi-buff support to the bpf_xdp_adjust_tail() API Lorenzo Bianconi
2021-04-08 19:15 ` Vladimir Oltean
2021-04-08 20:54 ` Vladimir Oltean
2021-04-09 18:13 ` Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 09/14] bpd: add multi-buffer support to xdp copy helpers Lorenzo Bianconi
2021-04-08 20:57 ` Vladimir Oltean
2021-04-09 18:19 ` Lorenzo Bianconi
2021-04-08 21:04 ` Vladimir Oltean
2021-04-14 8:08 ` Eelco Chaudron
2021-04-08 12:51 ` [PATCH v8 bpf-next 10/14] bpf: add new frame_length field to the XDP ctx Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 11/14] bpf: move user_size out of bpf_test_init Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 12/14] bpf: introduce multibuff support to bpf_prog_test_run_xdp() Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 13/14] bpf: test_run: add xdp_shared_info pointer in bpf_test_finish signature Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 14/14] bpf: update xdp_adjust_tail selftest to include multi-buffer Lorenzo Bianconi
2021-04-09 0:56 ` [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support John Fastabend
2021-04-09 20:16 ` Lorenzo Bianconi
2021-04-13 15:16 ` Eelco Chaudron
2021-04-16 14:27 ` Magnus Karlsson
2021-04-16 21:29 ` Lorenzo Bianconi
2021-04-16 23:00 ` Daniel Borkmann
2021-04-18 16:18 ` Jesper Dangaard Brouer
2021-04-19 6:20 ` Magnus Karlsson
2021-04-19 6:55 ` Lorenzo Bianconi
2021-04-20 13:49 ` Magnus Karlsson
2021-04-21 12:47 ` Jesper Dangaard Brouer
2021-04-21 14:12 ` Magnus Karlsson
2021-04-21 15:39 ` Jesper Dangaard Brouer
2021-04-22 10:24 ` Magnus Karlsson
2021-04-22 14:42 ` Jesper Dangaard Brouer
2021-04-22 15:05 ` Crash for i40e on net-next (was: [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support) Jesper Dangaard Brouer
2021-04-23 5:28 ` Magnus Karlsson [this message]
2021-04-23 16:43 ` Alexander Duyck
2021-04-25 9:45 ` Magnus Karlsson
2021-04-27 18:28 ` [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support Lorenzo Bianconi
2021-04-28 7:41 ` Magnus Karlsson
2021-04-29 12:49 ` Jesper Dangaard Brouer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAJ8uoz1oEa6ZEp3QKZiPx4oUtt9-nuY4Sh6PVrEnZdu-d_PudQ@mail.gmail.com \
--to=magnus.karlsson@gmail.com \
--cc=alexander.duyck@gmail.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=brouer@redhat.com \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=dsahern@kernel.org \
--cc=echaudro@redhat.com \
--cc=jasowang@redhat.com \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=lorenzo@kernel.org \
--cc=maciej.fijalkowski@intel.com \
--cc=netdev@vger.kernel.org \
--cc=saeed@kernel.org \
--cc=tirthendu.sarkar@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).