* Re: virtio-net: kernel panic in virtio_net.c
[not found] <1622688283.7488964-1-xuanzhuo@linux.alibaba.com>
@ 2021-06-03 8:57 ` Corentin Noël
2021-06-08 12:17 ` Greg KH
0 siblings, 1 reply; 21+ messages in thread
From: Corentin Noël @ 2021-06-03 8:57 UTC (permalink / raw)
To: Xuan Zhuo
Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization,
regressions, Eric Dumazet
Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit :
> On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël <
> corentin.noel@collabora.com> wrote:
> > Sure, here is the decoded trace:
> >
> > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434
> > len:3762
> > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [ 44.525254] kernel BUG at net/core/skbuff.c:110!
> > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI
> > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
> > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
> > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
> > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > 60
> > All code
> > ========
> > 0: 4f 70 50 rex.WRXB jo 0x53
> > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > 9: 50 push %rax
> > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > 10: 50 push %rax
> > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,
> > %rdi
> > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d
> > 2a:* 0f 0b ud2 <--
> > trapping
> > instruction
> > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > %rcx
> > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > 3c: 48 rex.W
> > 3d: c7 .byte 0xc7
> > 3e: c6 (bad)
> > 3f: 60 (bad)
> >
> > Code starting with the faulting instruction
> > ===========================================
> > 0: 0f 0b ud2
> > 2: 48 8b 14 24 mov (%rsp),%rdx
> > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > %rcx
> > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > 12: 48 rex.W
> > 13: c7 .byte 0xc7
> > 14: c6 (bad)
> > 15: 60 (bad)
> > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > 0000000000009ffb
> > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff979ad2aa5600
> > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > 0000000000000eb2
> > [ 44.539300] FS: 00007fdb9cb11700(0000)
> > GS:ffff979aebd00000(0000)
> > knlGS:0000000000000000
> > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > 0000000000370ee0
> > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 44.544063] Call Trace:
> > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator
> > 1)
> > net/core/skbuff.c:5252 (discriminator 1))
> > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485)
> > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849
> > drivers/net/virtio_net.c:1131)
> > [ 44.545870] ? netif_receive_skb_list_internal
> > (net/core/dev.c:5714)
> > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103)
> > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35
> > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565)
> > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
> > drivers/net/virtio_net.c:1525)
> > [ 44.548251] __napi_poll (net/core/dev.c:6985)
> > [ 44.548744] net_rx_action (net/core/dev.c:7054
> > net/core/dev.c:7139)
> > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
> > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
> > kernel/softirq.c:560)
> > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433
> > kernel/softirq.c:637
> > kernel/softirq.c:649)
> > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240
> > (discriminator 13))
> > [ 44.551991] ? asm_common_interrupt
> > (./arch/x86/include/asm/idtentry.h:638)
> > [ 44.552654] asm_common_interrupt
> > (./arch/x86/include/asm/idtentry.h:638)
> > [ 44.553276] RIP: 0033:0x7fdb981a82e4
> > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09
> > c4
> > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5
> > f9 6c
> > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5
> > f9
> > All code
> > ========
> > 0: d2 48 63 rorb %cl,0x63(%rax)
> > 3: f6 c4 41 test $0x41,%ah
> > 6: 7a 6f jp 0x77
> > 8: 0c 01 or $0x1,%al
> > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10
> > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12
> > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13
> > 1c: c4 c1 31 6a c2 vpunpckhdq
> > %xmm10,%xmm9,%xmm0
> > 21: c4 c1 19 6a d5 vpunpckhdq
> > %xmm13,%xmm12,%xmm2
> > 26: c5 f9 6c f2 vpunpcklqdq
> > %xmm2,%xmm0,%xmm6
> > 2a:* c5 79 6d c2 vpunpckhqdq
> > %xmm2,%xmm0,%xmm8
> > <-- trapping instruction
> > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0
> > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm
> > m0
> > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7
> > 3e: c5 .byte 0xc5
> > 3f: f9 stc
> >
> > Code starting with the faulting instruction
> > ===========================================
> > 0: c5 79 6d c2 vpunpckhqdq
> > %xmm2,%xmm0,%xmm8
> > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0
> > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm
> > m0
> > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7
> > 14: c5 .byte 0xc5
> > 15: f9 stc
> > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
> > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
> > 0000000000122d40
> > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
> > 000055d7049b9368
> > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
> > 00007fdb5e544040
> > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
> > 0000000000000000
> > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
> > 0000000000000000
> > [ 44.561965] Modules linked in:
> > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
> > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > 60
> > All code
> > ========
> > 0: 4f 70 50 rex.WRXB jo 0x53
> > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > 9: 50 push %rax
> > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > 10: 50 push %rax
> > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,
> > %rdi
> > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d
> > 2a:* 0f 0b ud2 <--
> > trapping
> > instruction
> > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > %rcx
> > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > 3c: 48 rex.W
> > 3d: c7 .byte 0xc7
> > 3e: c6 (bad)
> > 3f: 60 (bad)
> >
> > Code starting with the faulting instruction
> > ===========================================
> > 0: 0f 0b ud2
> > 2: 48 8b 14 24 mov (%rsp),%rdx
> > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > %rcx
> > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > 12: 48 rex.W
> > 13: c7 .byte 0xc7
> > 14: c6 (bad)
> > 15: 60 (bad)
> > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > 0000000000009ffb
> > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff979ad2aa5600
> > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > 0000000000000eb2
> > [ 44.571483] FS: 00007fdb9cb11700(0000)
> > GS:ffff979aebd00000(0000)
> > knlGS:0000000000000000
> > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > 0000000000370ee0
> > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 44.576618] Kernel panic - not syncing: Fatal exception in
> > interrupt
> > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
> > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> >
>
> Can you test this patch on the latest net branch?
>
> Thanks.
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index fa407eb8b457..78a01c71a17c 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct
> virtnet_info *vi,
> * add_recvbuf_mergeable() + get_mergeable_buf_len()
> */
> truesize = headroom ? PAGE_SIZE : truesize;
> - tailroom = truesize - len - headroom;
> + tailroom = truesize - len - headroom - (hdr_padded_len -
> hdr_len);
> buf = p - headroom;
>
> len -= hdr_len;
With this patch and the latest net branch I no longer get crashes.
Thanks a lot for this,
Corentin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-06-03 8:57 ` virtio-net: kernel panic in virtio_net.c Corentin Noël
@ 2021-06-08 12:17 ` Greg KH
[not found] ` <1623203313.4303577-1-xuanzhuo@linux.alibaba.com>
0 siblings, 1 reply; 21+ messages in thread
From: Greg KH @ 2021-06-08 12:17 UTC (permalink / raw)
To: Corentin Noël
Cc: Xuan Zhuo, stable, Michael S.Tsirkin, Jason Wang, virtualization,
regressions, Eric Dumazet
On Thu, Jun 03, 2021 at 10:57:52AM +0200, Corentin Noël wrote:
> Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit :
> > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël <
> > corentin.noel@collabora.com> wrote:
> > > Sure, here is the decoded trace:
> > >
> > > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434
> > > len:3762
> > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [ 44.525254] kernel BUG at net/core/skbuff.c:110!
> > > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI
> > > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
> > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
> > > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > > 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > > 60
> > > All code
> > > ========
> > > 0: 4f 70 50 rex.WRXB jo 0x53
> > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > > 9: 50 push %rax
> > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > > 10: 50 push %rax
> > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,
> > > %rdi
> > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d
> > > 2a:* 0f 0b ud2 <--
> > > trapping
> > > instruction
> > > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > > %rcx
> > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > > 3c: 48 rex.W
> > > 3d: c7 .byte 0xc7
> > > 3e: c6 (bad)
> > > 3f: 60 (bad)
> > >
> > > Code starting with the faulting instruction
> > > ===========================================
> > > 0: 0f 0b ud2
> > > 2: 48 8b 14 24 mov (%rsp),%rdx
> > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > > %rcx
> > > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > > 12: 48 rex.W
> > > 13: c7 .byte 0xc7
> > > 14: c6 (bad)
> > > 15: 60 (bad)
> > > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > > 0000000000009ffb
> > > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff979ad2aa5600
> > > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > > 0000000000000eb2
> > > [ 44.539300] FS: 00007fdb9cb11700(0000)
> > > GS:ffff979aebd00000(0000)
> > > knlGS:0000000000000000
> > > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > > 0000000000370ee0
> > > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 44.544063] Call Trace:
> > > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator
> > > 1)
> > > net/core/skbuff.c:5252 (discriminator 1))
> > > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485)
> > > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849
> > > drivers/net/virtio_net.c:1131)
> > > [ 44.545870] ? netif_receive_skb_list_internal
> > > (net/core/dev.c:5714)
> > > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103)
> > > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35
> > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565)
> > > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
> > > drivers/net/virtio_net.c:1525)
> > > [ 44.548251] __napi_poll (net/core/dev.c:6985)
> > > [ 44.548744] net_rx_action (net/core/dev.c:7054
> > > net/core/dev.c:7139)
> > > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
> > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
> > > kernel/softirq.c:560)
> > > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433
> > > kernel/softirq.c:637
> > > kernel/softirq.c:649)
> > > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240
> > > (discriminator 13))
> > > [ 44.551991] ? asm_common_interrupt
> > > (./arch/x86/include/asm/idtentry.h:638)
> > > [ 44.552654] asm_common_interrupt
> > > (./arch/x86/include/asm/idtentry.h:638)
> > > [ 44.553276] RIP: 0033:0x7fdb981a82e4
> > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09
> > > c4
> > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5
> > > f9 6c
> > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5
> > > f9
> > > All code
> > > ========
> > > 0: d2 48 63 rorb %cl,0x63(%rax)
> > > 3: f6 c4 41 test $0x41,%ah
> > > 6: 7a 6f jp 0x77
> > > 8: 0c 01 or $0x1,%al
> > > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10
> > > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12
> > > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13
> > > 1c: c4 c1 31 6a c2 vpunpckhdq
> > > %xmm10,%xmm9,%xmm0
> > > 21: c4 c1 19 6a d5 vpunpckhdq
> > > %xmm13,%xmm12,%xmm2
> > > 26: c5 f9 6c f2 vpunpcklqdq
> > > %xmm2,%xmm0,%xmm6
> > > 2a:* c5 79 6d c2 vpunpckhqdq
> > > %xmm2,%xmm0,%xmm8
> > > <-- trapping instruction
> > > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0
> > > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm
> > > m0
> > > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7
> > > 3e: c5 .byte 0xc5
> > > 3f: f9 stc
> > >
> > > Code starting with the faulting instruction
> > > ===========================================
> > > 0: c5 79 6d c2 vpunpckhqdq
> > > %xmm2,%xmm0,%xmm8
> > > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0
> > > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm
> > > m0
> > > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7
> > > 14: c5 .byte 0xc5
> > > 15: f9 stc
> > > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
> > > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
> > > 0000000000122d40
> > > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
> > > 000055d7049b9368
> > > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
> > > 00007fdb5e544040
> > > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
> > > 0000000000000000
> > > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
> > > 0000000000000000
> > > [ 44.561965] Modules linked in:
> > > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
> > > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > > 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > > 60
> > > All code
> > > ========
> > > 0: 4f 70 50 rex.WRXB jo 0x53
> > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > > 9: 50 push %rax
> > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > > 10: 50 push %rax
> > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,
> > > %rdi
> > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d
> > > 2a:* 0f 0b ud2 <--
> > > trapping
> > > instruction
> > > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > > %rcx
> > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > > 3c: 48 rex.W
> > > 3d: c7 .byte 0xc7
> > > 3e: c6 (bad)
> > > 3f: 60 (bad)
> > >
> > > Code starting with the faulting instruction
> > > ===========================================
> > > 0: 0f 0b ud2
> > > 2: 48 8b 14 24 mov (%rsp),%rdx
> > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,
> > > %rcx
> > > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > > 12: 48 rex.W
> > > 13: c7 .byte 0xc7
> > > 14: c6 (bad)
> > > 15: 60 (bad)
> > > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > > 0000000000009ffb
> > > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff979ad2aa5600
> > > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > > 0000000000000eb2
> > > [ 44.571483] FS: 00007fdb9cb11700(0000)
> > > GS:ffff979aebd00000(0000)
> > > knlGS:0000000000000000
> > > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > > 0000000000370ee0
> > > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 44.576618] Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > >
> >
> > Can you test this patch on the latest net branch?
> >
> > Thanks.
> >
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index fa407eb8b457..78a01c71a17c 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct
> > virtnet_info *vi,
> > * add_recvbuf_mergeable() + get_mergeable_buf_len()
> > */
> > truesize = headroom ? PAGE_SIZE : truesize;
> > - tailroom = truesize - len - headroom;
> > + tailroom = truesize - len - headroom - (hdr_padded_len -
> > hdr_len);
> > buf = p - headroom;
> >
> > len -= hdr_len;
>
> With this patch and the latest net branch I no longer get crashes.
Did this ever get properly submitted to the networking tree to get into
5.13-final?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
[not found] ` <1623203313.4303577-1-xuanzhuo@linux.alibaba.com>
@ 2021-06-09 4:50 ` Greg KH
[not found] ` <1623218897.4150124-1-xuanzhuo@linux.alibaba.com>
0 siblings, 1 reply; 21+ messages in thread
From: Greg KH @ 2021-06-09 4:50 UTC (permalink / raw)
To: Xuan Zhuo
Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
regressions, Eric Dumazet, Corentin Noël
On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > With this patch and the latest net branch I no longer get crashes.
> >
> > Did this ever get properly submitted to the networking tree to get into
> > 5.13-final?
>
> The patch has been submitted.
>
> [PATCH net] virtio-net: fix for skb_over_panic inside big mode
Submitted where? Do you have a lore.kernel.org link somewhere?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
[not found] ` <1623218897.4150124-1-xuanzhuo@linux.alibaba.com>
@ 2021-06-09 6:24 ` Greg KH
[not found] ` <1623225080.4793522-1-xuanzhuo@linux.alibaba.com>
0 siblings, 1 reply; 21+ messages in thread
From: Greg KH @ 2021-06-09 6:24 UTC (permalink / raw)
To: Xuan Zhuo
Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
regressions, Eric Dumazet, Corentin Noël
On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote:
> On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > > > With this patch and the latest net branch I no longer get crashes.
> > > >
> > > > Did this ever get properly submitted to the networking tree to get into
> > > > 5.13-final?
> > >
> > > The patch has been submitted.
> > >
> > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode
> >
> > Submitted where? Do you have a lore.kernel.org link somewhere?
>
>
> https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/
So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic
inside big mode") in Linus's tree, right?
But why is that referencing:
Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom")
when this problem was seen in stable kernels that had a different commit
backported to it?
Is there nothing needed to be done for the stable kernel trees?
confused,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
[not found] ` <1623225080.4793522-1-xuanzhuo@linux.alibaba.com>
@ 2021-06-09 8:03 ` Greg KH
0 siblings, 0 replies; 21+ messages in thread
From: Greg KH @ 2021-06-09 8:03 UTC (permalink / raw)
To: Xuan Zhuo
Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
regressions, Eric Dumazet, Corentin Noël
On Wed, Jun 09, 2021 at 03:51:20PM +0800, Xuan Zhuo wrote:
> On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote:
> > > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > > > > > With this patch and the latest net branch I no longer get crashes.
> > > > > >
> > > > > > Did this ever get properly submitted to the networking tree to get into
> > > > > > 5.13-final?
> > > > >
> > > > > The patch has been submitted.
> > > > >
> > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode
> > > >
> > > > Submitted where? Do you have a lore.kernel.org link somewhere?
> > >
> > >
> > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/
> >
> > So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic
> > inside big mode") in Linus's tree, right?
>
> YES.
>
> >
> > But why is that referencing:
> > Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom")
>
> This problem was indeed introduced in fb32856b16ad.
>
> I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the
> previous 5.12 did not have this commit fb32856b16ad.
>
> I'm not sure if it helped you.
Hm, then what resolves the reported problem that people were having with
the 5.12.y kernel release? Is that a separate issue?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
[not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com>
@ 2021-10-09 5:19 ` Greg KH
0 siblings, 0 replies; 21+ messages in thread
From: Greg KH @ 2021-10-09 5:19 UTC (permalink / raw)
To: Xuan Zhuo
Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
virtualization, regressions, Eric Dumazet, stable
On Sat, Oct 09, 2021 at 12:27:08AM +0800, Xuan Zhuo wrote:
> On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > > > occur with 5.13,
> > > > >
> > > > > I should have fixed this problem before. I don't know why, I just looked at the
> > > > > latest net code, and this commit seems to be lost.
> > > > >
> > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > > > >
> > > > > Can you test this patch again?
> > > >
> > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > > > it in it, right?
> > > >
> > >
> > > Yes, it may be lost due to conflicts during a certain merge.
> >
> > Really? I tried to apply that again to 5.14 and it did not work. So I
> > do not understand what to do here, can you try to explain it better?
>
> I took a look, and there is actually another missing patch:
>
> A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr
> B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
>
> A is replaced by another patch:
>
> commit c32325b8fdf2f979befb9fd5587918c0d5412db3
> Author: Jakub Kicinski <kuba@kernel.org>
> Date: Mon Aug 2 10:57:29 2021 -0700
>
> virtio-net: realign page_to_skb() after merges
>
> We ended up merging two versions of the same patch set:
>
> commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr")
> commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address")
>
> into net, and
>
> commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr")
> commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address")
>
> into net-next. Redo the merge from commit 126285651b7f ("Merge
> ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that
> the most recent code remains.
>
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> Acked-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
>
> So after this patch, patch B can be applied normally.
>
> So on the latest net branch, only lost
>
> 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
Again, I do not know what to do here, can you submit the needed fix to
the networking developers so this gets fixed?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-10-08 10:02 ` Michael S. Tsirkin
@ 2021-10-08 12:21 ` Corentin Noël
0 siblings, 0 replies; 21+ messages in thread
From: Corentin Noël @ 2021-10-08 12:21 UTC (permalink / raw)
To: Michael S. Tsirkin, Greg KH
Cc: Xuan Zhuo, Jason Wang, virtualization, regressions, Eric Dumazet, stable
Le vendredi 08 octobre 2021 à 06:02 -0400, Michael S. Tsirkin a écrit :
> On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote:
> > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <
> > > gregkh@linuxfoundation.org> wrote:
> > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <
> > > > > corentin.noel@collabora.com> wrote:
> > > > > > I've been experiencing crashes with 5.14-rc1 and above that
> > > > > > do not
> > > > > > occur with 5.13,
> > > > >
> > > > > I should have fixed this problem before. I don't know why, I
> > > > > just looked at the
> > > > > latest net code, and this commit seems to be lost.
> > > > >
> > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix
> > > > > for skb_over_panic inside big mode
> > > > >
> > > > > Can you test this patch again?
> > > >
> > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should
> > > > have had
> > > > it in it, right?
> > > >
> > >
> > > Yes, it may be lost due to conflicts during a certain merge.
> >
> > Really? I tried to apply that again to 5.14 and it did not
> > work. So I
> > do not understand what to do here, can you try to explain it
> > better?
> >
> > thanks,
> >
> > greg k-h
>
> Hmm, something like the following perhaps then?
> Corentin would you like to try this?
> Warning: untested.
>
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 096c2ac6b7a6..18dd9f6d107d 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct
> virtnet_info *vi,
> * add_recvbuf_mergeable() + get_mergeable_buf_len()
> */
> truesize = headroom ? PAGE_SIZE : truesize;
> - tailroom = truesize - len - headroom;
> + tailroom = truesize - headroom;
> buf = p - headroom;
>
> len -= hdr_len;
> offset += hdr_padded_len;
> p += hdr_padded_len;
> + tailroom -= hdr_padded_len + len;
>
> shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
>
>
Thank you for the patch, I started bisecting the issue but your patch
actually makes it work again.
Regards,
Corentin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-10-08 8:06 ` Greg KH
@ 2021-10-08 10:02 ` Michael S. Tsirkin
2021-10-08 12:21 ` Corentin Noël
[not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com>
1 sibling, 1 reply; 21+ messages in thread
From: Michael S. Tsirkin @ 2021-10-08 10:02 UTC (permalink / raw)
To: Greg KH
Cc: Xuan Zhuo, Corentin Noël, Jason Wang, virtualization,
regressions, Eric Dumazet, stable
On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote:
> On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > > occur with 5.13,
> > > >
> > > > I should have fixed this problem before. I don't know why, I just looked at the
> > > > latest net code, and this commit seems to be lost.
> > > >
> > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > > >
> > > > Can you test this patch again?
> > >
> > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > > it in it, right?
> > >
> >
> > Yes, it may be lost due to conflicts during a certain merge.
>
> Really? I tried to apply that again to 5.14 and it did not work. So I
> do not understand what to do here, can you try to explain it better?
>
> thanks,
>
> greg k-h
Hmm, something like the following perhaps then?
Corentin would you like to try this?
Warning: untested.
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 096c2ac6b7a6..18dd9f6d107d 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
* add_recvbuf_mergeable() + get_mergeable_buf_len()
*/
truesize = headroom ? PAGE_SIZE : truesize;
- tailroom = truesize - len - headroom;
+ tailroom = truesize - headroom;
buf = p - headroom;
len -= hdr_len;
offset += hdr_padded_len;
p += hdr_padded_len;
+ tailroom -= hdr_padded_len + len;
shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
[not found] ` <1633623446.6192446-1-xuanzhuo@linux.alibaba.com>
@ 2021-10-08 8:06 ` Greg KH
2021-10-08 10:02 ` Michael S. Tsirkin
[not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com>
0 siblings, 2 replies; 21+ messages in thread
From: Greg KH @ 2021-10-08 8:06 UTC (permalink / raw)
To: Xuan Zhuo
Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
virtualization, regressions, Eric Dumazet, stable
On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > occur with 5.13,
> > >
> > > I should have fixed this problem before. I don't know why, I just looked at the
> > > latest net code, and this commit seems to be lost.
> > >
> > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > >
> > > Can you test this patch again?
> >
> > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > it in it, right?
> >
>
> Yes, it may be lost due to conflicts during a certain merge.
Really? I tried to apply that again to 5.14 and it did not work. So I
do not understand what to do here, can you try to explain it better?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
[not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com>
@ 2021-10-07 15:25 ` Greg KH
[not found] ` <1633623446.6192446-1-xuanzhuo@linux.alibaba.com>
0 siblings, 1 reply; 21+ messages in thread
From: Greg KH @ 2021-10-07 15:25 UTC (permalink / raw)
To: Xuan Zhuo
Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
virtualization, regressions, Eric Dumazet, stable
On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > I've been experiencing crashes with 5.14-rc1 and above that do not
> > occur with 5.13,
>
> I should have fixed this problem before. I don't know why, I just looked at the
> latest net code, and this commit seems to be lost.
>
> 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
>
> Can you test this patch again?
That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
it in it, right?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-10-07 14:02 ` Corentin Noël
@ 2021-10-07 14:13 ` Greg KH
0 siblings, 0 replies; 21+ messages in thread
From: Greg KH @ 2021-10-07 14:13 UTC (permalink / raw)
To: Corentin Noël
Cc: Eric Dumazet, Michael S. Tsirkin, linux-stable, Jason Wang,
virtualization, regressions, Xuan Zhuo
On Thu, Oct 07, 2021 at 04:02:10PM +0200, Corentin Noël wrote:
> Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit :
> > On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com>
> > wrote:
> > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> > > > I've been experiencing crashes with 5.14-rc1 and above that do
> > > > not
> > > > occur with 5.13,
> >
> > What about 5.14 ?
> >
> > 5.14-rc1 has many bugs we do not want to spend time rediscovering
> > them...
> >
>
> I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could
> reproduce the crash on them all.
Great, any chance you can use 'git bisect' to find the offending commit?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-10-07 13:51 ` Eric Dumazet
@ 2021-10-07 14:02 ` Corentin Noël
2021-10-07 14:13 ` Greg KH
0 siblings, 1 reply; 21+ messages in thread
From: Corentin Noël @ 2021-10-07 14:02 UTC (permalink / raw)
To: Eric Dumazet, Michael S. Tsirkin
Cc: linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo
Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit :
> On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com>
> wrote:
> > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> > > I've been experiencing crashes with 5.14-rc1 and above that do
> > > not
> > > occur with 5.13,
>
> What about 5.14 ?
>
> 5.14-rc1 has many bugs we do not want to spend time rediscovering
> them...
>
I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could
reproduce the crash on them all.
> > > here is the crash trace:
> > >
> > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7
> > > len:3762
> > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [ 61.369192] kernel BUG at net/core/skbuff.c:111!
> > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI
> > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45
> > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [ 61.386825] FS: 0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > > 0000000000370ee0
> > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 61.393635] Call Trace:
> > > [ 61.394127] <IRQ>
> > > [ 61.394488] skb_put.cold+0x10/0x10
> > > [ 61.395095] page_to_skb+0xf7/0x410
> > > [ 61.395689] receive_buf+0x81/0x1660
> > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > > [ 61.397180] ? napi_gro_flush+0x97/0xe0
> > > [ 61.397896] ? detach_buf_split+0x67/0x120
> > > [ 61.398573] virtnet_poll+0x2cf/0x420
> > > [ 61.399197] __napi_poll+0x25/0x150
> > > [ 61.399764] net_rx_action+0x22f/0x280
> > > [ 61.400394] __do_softirq+0xba/0x257
> > > [ 61.401012] irq_exit_rcu+0x8e/0xb0
> > > [ 61.401618] common_interrupt+0x7b/0xa0
> > > [ 61.402270] </IRQ>
> > > [ 61.402620] asm_common_interrupt+0x1e/0x40
> > > [ 61.403302] RIP: 0010:default_idle+0xb/0x10
> > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae
> > > f0 0f
> > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44
> > > 00 fb
> > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c
> > > 01 48
> > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > > ffff8a5febd56f80
> > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > > ffff8a5febd5dd00
> > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > > 0000000000000000
> > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > > 0000000000000000
> > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > > 0000000000000000
> > > [ 61.414183] ? mwait_idle+0x70/0x70
> > > [ 61.414805] ? mwait_idle+0x70/0x70
> > > [ 61.415592] default_idle_call+0x2a/0xa0
> > > [ 61.416216] do_idle+0x1e8/0x250
> > > [ 61.416722] cpu_startup_entry+0x14/0x20
> > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> > > [ 61.418144] Modules linked in:
> > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45
> > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [ 61.429799] FS: 0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > > 0000000000370ee0
> > > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 61.435799] Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > >
> > > Here is my kernel config:
> > > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
> > >
> > >
> > > here is the decoded trace:
> > >
> > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7
> > > len:3762
> > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [ 61.369192] kernel BUG at net/core/skbuff.c:111!
> > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI
> > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [ 61.377222] RIP: skb_panic+0x43/0x45
> > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > All code
> > > ========
> > > 0: 4f 70 50 rex.WRXB jo 0x53
> > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > > 9: 50 push %rax
> > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > > 10: 50 push %rax
> > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
> > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394
> > > 2a:* 0f 0b ud2 <--
> > > trapping
> > > instruction
> > > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > > 3c: 48 rex.W
> > > 3d: c7 .byte 0xc7
> > > 3e: c6 (bad)
> > > 3f: 60 (bad)
> > >
> > > Code starting with the faulting instruction
> > > ===========================================
> > > 0: 0f 0b ud2
> > > 2: 48 8b 14 24 mov (%rsp),%rdx
> > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > > 12: 48 rex.W
> > > 13: c7 .byte 0xc7
> > > 14: c6 (bad)
> > > 15: 60 (bad)
> > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [ 61.386825] FS: 0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > > 0000000000370ee0
> > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 61.393635] Call Trace:
> > > [ 61.394127] <IRQ>
> > > [ 61.394488] skb_put.cold+0x10/0x10
> > > [ 61.395095] page_to_skb+0xf7/0x410
> > > [ 61.395689] receive_buf+0x81/0x1660
> > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > > [ 61.397180] ? napi_gro_flush+0x97/0xe0
> > > [ 61.397896] ? detach_buf_split+0x67/0x120
> > > [ 61.398573] virtnet_poll+0x2cf/0x420
> > > [ 61.399197] __napi_poll+0x25/0x150
> > > [ 61.399764] net_rx_action+0x22f/0x280
> > > [ 61.400394] __do_softirq+0xba/0x257
> > > [ 61.401012] irq_exit_rcu+0x8e/0xb0
> > > [ 61.401618] common_interrupt+0x7b/0xa0
> > > [ 61.402270] </IRQ>
> > > [ 61.402620] asm_common_interrupt+0x1e/0x40
> > > [ 61.403302] RIP: default_idle+0xb/0x10
> > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae
> > > f0 0f
> > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44
> > > 00 fb
> > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c
> > > 01 48
> > > All code
> > > ========
> > > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax
> > > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax)
> > > c: c3 retq
> > > d: 0f ae f0 mfence
> > > 10: 0f ae 38 clflush (%rax)
> > > 13: 0f ae f0 mfence
> > > 16: eb b9 jmp 0xffffffffffffffd1
> > > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
> > > 1f: eb 07 jmp 0x28
> > > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) #
> > > 0x443f07
> > > 28: fb sti
> > > 29: f4 hlt
> > > 2a:* c3 retq <--
> > > trapping
> > > instruction
> > > 2b: cc int3
> > > 2c: cc int3
> > > 2d: cc int3
> > > 2e: cc int3
> > > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
> > > # 0x77a42f67
> > > 36: 89 d2 mov %edx,%edx
> > > 38: 48 8b 05 d0 a1 0c
> > > 01 mov 0x10ca1d0(%rip),%rax #
> > > 0x10ca20f
> > > 3f: 48 rex.W
> > >
> > > Code starting with the faulting instruction
> > > ===========================================
> > > 0: c3 retq
> > > 1: cc int3
> > > 2: cc int3
> > > 3: cc int3
> > > 4: cc int3
> > > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
> > > # 0x77a42f3d
> > > c: 89 d2 mov %edx,%edx
> > > e: 48 8b 05 d0 a1 0c
> > > 01 mov 0x10ca1d0(%rip),%rax #
> > > 0x10ca1e5
> > > 15: 48 rex.W
> > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > > ffff8a5febd56f80
> > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > > ffff8a5febd5dd00
> > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > > 0000000000000000
> > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > > 0000000000000000
> > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > > 0000000000000000
> > > [ 61.414183] ? mwait_idle+0x70/0x70
> > > [ 61.414805] ? mwait_idle+0x70/0x70
> > > [ 61.415592] default_idle_call+0x2a/0xa0
> > > [ 61.416216] do_idle+0x1e8/0x250
> > > [ 61.416722] cpu_startup_entry+0x14/0x20
> > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> > > [ 61.418144] Modules linked in:
> > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > > [ 61.419399] RIP: skb_panic+0x43/0x45
> > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > All code
> > > ========
> > > 0: 4f 70 50 rex.WRXB jo 0x53
> > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > > 9: 50 push %rax
> > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > > 10: 50 push %rax
> > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
> > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394
> > > 2a:* 0f 0b ud2 <--
> > > trapping
> > > instruction
> > > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > > 3c: 48 rex.W
> > > 3d: c7 .byte 0xc7
> > > 3e: c6 (bad)
> > > 3f: 60 (bad)
> > >
> > > Code starting with the faulting instruction
> > > ===========================================
> > > 0: 0f 0b ud2
> > > 2: 48 8b 14 24 mov (%rsp),%rdx
> > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > > 12: 48 rex.W
> > > 13: c7 .byte 0xc7
> > > 14: c6 (bad)
> > > 15: 60 (bad)
> > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [ 61.429799] FS: 0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > >
> > > Regards,
> > > Corentin
> >
> > Don't see anything obvious.. could be a net stack change.
> > Any chance of a bisect?
> >
> > --
> > MST
> >
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-10-07 13:10 ` Michael S. Tsirkin
@ 2021-10-07 13:51 ` Eric Dumazet
2021-10-07 14:02 ` Corentin Noël
0 siblings, 1 reply; 21+ messages in thread
From: Eric Dumazet @ 2021-10-07 13:51 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Corentin Noël, linux-stable, Jason Wang, virtualization,
regressions, Xuan Zhuo
On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> > I've been experiencing crashes with 5.14-rc1 and above that do not
> > occur with 5.13,
What about 5.14 ?
5.14-rc1 has many bugs we do not want to spend time rediscovering them...
> >
> > here is the crash trace:
> >
> > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [ 61.369192] kernel BUG at net/core/skbuff.c:111!
> > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI
> > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45
> > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > 0000000000370ee0
> > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 61.393635] Call Trace:
> > [ 61.394127] <IRQ>
> > [ 61.394488] skb_put.cold+0x10/0x10
> > [ 61.395095] page_to_skb+0xf7/0x410
> > [ 61.395689] receive_buf+0x81/0x1660
> > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > [ 61.397180] ? napi_gro_flush+0x97/0xe0
> > [ 61.397896] ? detach_buf_split+0x67/0x120
> > [ 61.398573] virtnet_poll+0x2cf/0x420
> > [ 61.399197] __napi_poll+0x25/0x150
> > [ 61.399764] net_rx_action+0x22f/0x280
> > [ 61.400394] __do_softirq+0xba/0x257
> > [ 61.401012] irq_exit_rcu+0x8e/0xb0
> > [ 61.401618] common_interrupt+0x7b/0xa0
> > [ 61.402270] </IRQ>
> > [ 61.402620] asm_common_interrupt+0x1e/0x40
> > [ 61.403302] RIP: 0010:default_idle+0xb/0x10
> > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > ffff8a5febd56f80
> > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > ffff8a5febd5dd00
> > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > 0000000000000000
> > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > 0000000000000000
> > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > 0000000000000000
> > [ 61.414183] ? mwait_idle+0x70/0x70
> > [ 61.414805] ? mwait_idle+0x70/0x70
> > [ 61.415592] default_idle_call+0x2a/0xa0
> > [ 61.416216] do_idle+0x1e8/0x250
> > [ 61.416722] cpu_startup_entry+0x14/0x20
> > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> > [ 61.418144] Modules linked in:
> > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45
> > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > 0000000000370ee0
> > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
> > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> >
> > Here is my kernel config:
> > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
> >
> >
> > here is the decoded trace:
> >
> > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [ 61.369192] kernel BUG at net/core/skbuff.c:111!
> > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI
> > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > [ 61.377222] RIP: skb_panic+0x43/0x45
> > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > All code
> > ========
> > 0: 4f 70 50 rex.WRXB jo 0x53
> > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > 9: 50 push %rax
> > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > 10: 50 push %rax
> > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
> > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394
> > 2a:* 0f 0b ud2 <-- trapping
> > instruction
> > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > 3c: 48 rex.W
> > 3d: c7 .byte 0xc7
> > 3e: c6 (bad)
> > 3f: 60 (bad)
> >
> > Code starting with the faulting instruction
> > ===========================================
> > 0: 0f 0b ud2
> > 2: 48 8b 14 24 mov (%rsp),%rdx
> > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > 12: 48 rex.W
> > 13: c7 .byte 0xc7
> > 14: c6 (bad)
> > 15: 60 (bad)
> > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > 0000000000370ee0
> > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 61.393635] Call Trace:
> > [ 61.394127] <IRQ>
> > [ 61.394488] skb_put.cold+0x10/0x10
> > [ 61.395095] page_to_skb+0xf7/0x410
> > [ 61.395689] receive_buf+0x81/0x1660
> > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > [ 61.397180] ? napi_gro_flush+0x97/0xe0
> > [ 61.397896] ? detach_buf_split+0x67/0x120
> > [ 61.398573] virtnet_poll+0x2cf/0x420
> > [ 61.399197] __napi_poll+0x25/0x150
> > [ 61.399764] net_rx_action+0x22f/0x280
> > [ 61.400394] __do_softirq+0xba/0x257
> > [ 61.401012] irq_exit_rcu+0x8e/0xb0
> > [ 61.401618] common_interrupt+0x7b/0xa0
> > [ 61.402270] </IRQ>
> > [ 61.402620] asm_common_interrupt+0x1e/0x40
> > [ 61.403302] RIP: default_idle+0xb/0x10
> > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> > All code
> > ========
> > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax
> > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax)
> > c: c3 retq
> > d: 0f ae f0 mfence
> > 10: 0f ae 38 clflush (%rax)
> > 13: 0f ae f0 mfence
> > 16: eb b9 jmp 0xffffffffffffffd1
> > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
> > 1f: eb 07 jmp 0x28
> > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07
> > 28: fb sti
> > 29: f4 hlt
> > 2a:* c3 retq <-- trapping
> > instruction
> > 2b: cc int3
> > 2c: cc int3
> > 2d: cc int3
> > 2e: cc int3
> > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
> > # 0x77a42f67
> > 36: 89 d2 mov %edx,%edx
> > 38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
> > 0x10ca20f
> > 3f: 48 rex.W
> >
> > Code starting with the faulting instruction
> > ===========================================
> > 0: c3 retq
> > 1: cc int3
> > 2: cc int3
> > 3: cc int3
> > 4: cc int3
> > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
> > # 0x77a42f3d
> > c: 89 d2 mov %edx,%edx
> > e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
> > 0x10ca1e5
> > 15: 48 rex.W
> > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > ffff8a5febd56f80
> > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > ffff8a5febd5dd00
> > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > 0000000000000000
> > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > 0000000000000000
> > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > 0000000000000000
> > [ 61.414183] ? mwait_idle+0x70/0x70
> > [ 61.414805] ? mwait_idle+0x70/0x70
> > [ 61.415592] default_idle_call+0x2a/0xa0
> > [ 61.416216] do_idle+0x1e8/0x250
> > [ 61.416722] cpu_startup_entry+0x14/0x20
> > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> > [ 61.418144] Modules linked in:
> > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > [ 61.419399] RIP: skb_panic+0x43/0x45
> > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > All code
> > ========
> > 0: 4f 70 50 rex.WRXB jo 0x53
> > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> > 9: 50 push %rax
> > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> > 10: 50 push %rax
> > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
> > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394
> > 2a:* 0f 0b ud2 <-- trapping
> > instruction
> > 2c: 48 8b 14 24 mov (%rsp),%rdx
> > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> > 3c: 48 rex.W
> > 3d: c7 .byte 0xc7
> > 3e: c6 (bad)
> > 3f: 60 (bad)
> >
> > Code starting with the faulting instruction
> > ===========================================
> > 0: 0f 0b ud2
> > 2: 48 8b 14 24 mov (%rsp),%rdx
> > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> > d: e8 ab ff ff ff callq 0xffffffffffffffbd
> > 12: 48 rex.W
> > 13: c7 .byte 0xc7
> > 14: c6 (bad)
> > 15: 60 (bad)
> > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> >
> > Regards,
> > Corentin
>
> Don't see anything obvious.. could be a net stack change.
> Any chance of a bisect?
>
> --
> MST
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-10-07 12:04 Corentin Noël
@ 2021-10-07 13:10 ` Michael S. Tsirkin
2021-10-07 13:51 ` Eric Dumazet
[not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com>
1 sibling, 1 reply; 21+ messages in thread
From: Michael S. Tsirkin @ 2021-10-07 13:10 UTC (permalink / raw)
To: Corentin Noël
Cc: stable, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo
On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> I've been experiencing crashes with 5.14-rc1 and above that do not
> occur with 5.13,
>
> here is the crash trace:
>
> [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> end:0xec0 dev:<NULL>
> [ 61.369192] kernel BUG at net/core/skbuff.c:111!
> [ 61.372840] invalid opcode: 0000 [#1] SMP PTI
> [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> [ 61.377222] RIP: 0010:skb_panic+0x43/0x45
> [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [ 61.393635] Call Trace:
> [ 61.394127] <IRQ>
> [ 61.394488] skb_put.cold+0x10/0x10
> [ 61.395095] page_to_skb+0xf7/0x410
> [ 61.395689] receive_buf+0x81/0x1660
> [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> [ 61.397180] ? napi_gro_flush+0x97/0xe0
> [ 61.397896] ? detach_buf_split+0x67/0x120
> [ 61.398573] virtnet_poll+0x2cf/0x420
> [ 61.399197] __napi_poll+0x25/0x150
> [ 61.399764] net_rx_action+0x22f/0x280
> [ 61.400394] __do_softirq+0xba/0x257
> [ 61.401012] irq_exit_rcu+0x8e/0xb0
> [ 61.401618] common_interrupt+0x7b/0xa0
> [ 61.402270] </IRQ>
> [ 61.402620] asm_common_interrupt+0x1e/0x40
> [ 61.403302] RIP: 0010:default_idle+0xb/0x10
> [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> ffff8a5febd56f80
> [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> ffff8a5febd5dd00
> [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> 0000000000000000
> [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> 0000000000000000
> [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [ 61.414183] ? mwait_idle+0x70/0x70
> [ 61.414805] ? mwait_idle+0x70/0x70
> [ 61.415592] default_idle_call+0x2a/0xa0
> [ 61.416216] do_idle+0x1e8/0x250
> [ 61.416722] cpu_startup_entry+0x14/0x20
> [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> [ 61.418144] Modules linked in:
> [ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
> [ 61.419399] RIP: 0010:skb_panic+0x43/0x45
> [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt
> [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>
> Here is my kernel config:
> https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
>
>
> here is the decoded trace:
>
> [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> end:0xec0 dev:<NULL>
> [ 61.369192] kernel BUG at net/core/skbuff.c:111!
> [ 61.372840] invalid opcode: 0000 [#1] SMP PTI
> [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> [ 61.377222] RIP: skb_panic+0x43/0x45
> [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
> 0: 4f 70 50 rex.WRXB jo 0x53
> 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> 9: 50 push %rax
> a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> 10: 50 push %rax
> 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
> 25: e8 6a 43 fb ff callq 0xfffffffffffb4394
> 2a:* 0f 0b ud2 <-- trapping
> instruction
> 2c: 48 8b 14 24 mov (%rsp),%rdx
> 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> 3c: 48 rex.W
> 3d: c7 .byte 0xc7
> 3e: c6 (bad)
> 3f: 60 (bad)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 0f 0b ud2
> 2: 48 8b 14 24 mov (%rsp),%rdx
> 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> d: e8 ab ff ff ff callq 0xffffffffffffffbd
> 12: 48 rex.W
> 13: c7 .byte 0xc7
> 14: c6 (bad)
> 15: 60 (bad)
> [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [ 61.393635] Call Trace:
> [ 61.394127] <IRQ>
> [ 61.394488] skb_put.cold+0x10/0x10
> [ 61.395095] page_to_skb+0xf7/0x410
> [ 61.395689] receive_buf+0x81/0x1660
> [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> [ 61.397180] ? napi_gro_flush+0x97/0xe0
> [ 61.397896] ? detach_buf_split+0x67/0x120
> [ 61.398573] virtnet_poll+0x2cf/0x420
> [ 61.399197] __napi_poll+0x25/0x150
> [ 61.399764] net_rx_action+0x22f/0x280
> [ 61.400394] __do_softirq+0xba/0x257
> [ 61.401012] irq_exit_rcu+0x8e/0xb0
> [ 61.401618] common_interrupt+0x7b/0xa0
> [ 61.402270] </IRQ>
> [ 61.402620] asm_common_interrupt+0x1e/0x40
> [ 61.403302] RIP: default_idle+0xb/0x10
> [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> All code
> ========
> 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax
> 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax)
> c: c3 retq
> d: 0f ae f0 mfence
> 10: 0f ae 38 clflush (%rax)
> 13: 0f ae f0 mfence
> 16: eb b9 jmp 0xffffffffffffffd1
> 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
> 1f: eb 07 jmp 0x28
> 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07
> 28: fb sti
> 29: f4 hlt
> 2a:* c3 retq <-- trapping
> instruction
> 2b: cc int3
> 2c: cc int3
> 2d: cc int3
> 2e: cc int3
> 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
> # 0x77a42f67
> 36: 89 d2 mov %edx,%edx
> 38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
> 0x10ca20f
> 3f: 48 rex.W
>
> Code starting with the faulting instruction
> ===========================================
> 0: c3 retq
> 1: cc int3
> 2: cc int3
> 3: cc int3
> 4: cc int3
> 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
> # 0x77a42f3d
> c: 89 d2 mov %edx,%edx
> e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
> 0x10ca1e5
> 15: 48 rex.W
> [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> ffff8a5febd56f80
> [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> ffff8a5febd5dd00
> [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> 0000000000000000
> [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> 0000000000000000
> [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [ 61.414183] ? mwait_idle+0x70/0x70
> [ 61.414805] ? mwait_idle+0x70/0x70
> [ 61.415592] default_idle_call+0x2a/0xa0
> [ 61.416216] do_idle+0x1e8/0x250
> [ 61.416722] cpu_startup_entry+0x14/0x20
> [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> [ 61.418144] Modules linked in:
> [ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
> [ 61.419399] RIP: skb_panic+0x43/0x45
> [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
> 0: 4f 70 50 rex.WRXB jo 0x53
> 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
> 9: 50 push %rax
> a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
> 10: 50 push %rax
> 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
> 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
> 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
> 25: e8 6a 43 fb ff callq 0xfffffffffffb4394
> 2a:* 0f 0b ud2 <-- trapping
> instruction
> 2c: 48 8b 14 24 mov (%rsp),%rdx
> 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> 37: e8 ab ff ff ff callq 0xffffffffffffffe7
> 3c: 48 rex.W
> 3d: c7 .byte 0xc7
> 3e: c6 (bad)
> 3f: 60 (bad)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 0f 0b ud2
> 2: 48 8b 14 24 mov (%rsp),%rdx
> 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
> d: e8 ab ff ff ff callq 0xffffffffffffffbd
> 12: 48 rex.W
> 13: c7 .byte 0xc7
> 14: c6 (bad)
> 15: 60 (bad)
> [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
>
> Regards,
> Corentin
Don't see anything obvious.. could be a net stack change.
Any chance of a bisect?
--
MST
^ permalink raw reply [flat|nested] 21+ messages in thread
* virtio-net: kernel panic in virtio_net.c
@ 2021-10-07 12:04 Corentin Noël
2021-10-07 13:10 ` Michael S. Tsirkin
[not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com>
0 siblings, 2 replies; 21+ messages in thread
From: Corentin Noël @ 2021-10-07 12:04 UTC (permalink / raw)
To: stable
Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions,
Eric Dumazet, Xuan Zhuo
I've been experiencing crashes with 5.14-rc1 and above that do not
occur with 5.13,
here is the crash trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
[ 61.377222] RIP: 0010:skb_panic+0x43/0x45
[ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.393635] Call Trace:
[ 61.394127] <IRQ>
[ 61.394488] skb_put.cold+0x10/0x10
[ 61.395095] page_to_skb+0xf7/0x410
[ 61.395689] receive_buf+0x81/0x1660
[ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
[ 61.397180] ? napi_gro_flush+0x97/0xe0
[ 61.397896] ? detach_buf_split+0x67/0x120
[ 61.398573] virtnet_poll+0x2cf/0x420
[ 61.399197] __napi_poll+0x25/0x150
[ 61.399764] net_rx_action+0x22f/0x280
[ 61.400394] __do_softirq+0xba/0x257
[ 61.401012] irq_exit_rcu+0x8e/0xb0
[ 61.401618] common_interrupt+0x7b/0xa0
[ 61.402270] </IRQ>
[ 61.402620] asm_common_interrupt+0x1e/0x40
[ 61.403302] RIP: 0010:default_idle+0xb/0x10
[ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
[ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 61.414183] ? mwait_idle+0x70/0x70
[ 61.414805] ? mwait_idle+0x70/0x70
[ 61.415592] default_idle_call+0x2a/0xa0
[ 61.416216] do_idle+0x1e8/0x250
[ 61.416722] cpu_startup_entry+0x14/0x20
[ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
[ 61.418144] Modules linked in:
[ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
[ 61.419399] RIP: 0010:skb_panic+0x43/0x45
[ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt
[ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Here is my kernel config:
https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
here is the decoded trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
[ 61.377222] RIP: skb_panic+0x43/0x45
[ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
25: e8 6a 43 fb ff callq 0xfffffffffffb4394
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.393635] Call Trace:
[ 61.394127] <IRQ>
[ 61.394488] skb_put.cold+0x10/0x10
[ 61.395095] page_to_skb+0xf7/0x410
[ 61.395689] receive_buf+0x81/0x1660
[ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
[ 61.397180] ? napi_gro_flush+0x97/0xe0
[ 61.397896] ? detach_buf_split+0x67/0x120
[ 61.398573] virtnet_poll+0x2cf/0x420
[ 61.399197] __napi_poll+0x25/0x150
[ 61.399764] net_rx_action+0x22f/0x280
[ 61.400394] __do_softirq+0xba/0x257
[ 61.401012] irq_exit_rcu+0x8e/0xb0
[ 61.401618] common_interrupt+0x7b/0xa0
[ 61.402270] </IRQ>
[ 61.402620] asm_common_interrupt+0x1e/0x40
[ 61.403302] RIP: default_idle+0xb/0x10
[ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
All code
========
0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax
7: f0 80 60 02 df lock andb $0xdf,0x2(%rax)
c: c3 retq
d: 0f ae f0 mfence
10: 0f ae 38 clflush (%rax)
13: 0f ae f0 mfence
16: eb b9 jmp 0xffffffffffffffd1
18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
1f: eb 07 jmp 0x28
21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07
28: fb sti
29: f4 hlt
2a:* c3 retq <-- trapping
instruction
2b: cc int3
2c: cc int3
2d: cc int3
2e: cc int3
2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
# 0x77a42f67
36: 89 d2 mov %edx,%edx
38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
0x10ca20f
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: c3 retq
1: cc int3
2: cc int3
3: cc int3
4: cc int3
5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
# 0x77a42f3d
c: 89 d2 mov %edx,%edx
e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
0x10ca1e5
15: 48 rex.W
[ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 61.414183] ? mwait_idle+0x70/0x70
[ 61.414805] ? mwait_idle+0x70/0x70
[ 61.415592] default_idle_call+0x2a/0xa0
[ 61.416216] do_idle+0x1e8/0x250
[ 61.416722] cpu_startup_entry+0x14/0x20
[ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
[ 61.418144] Modules linked in:
[ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
[ 61.419399] RIP: skb_panic+0x43/0x45
[ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
25: e8 6a 43 fb ff callq 0xfffffffffffb4394
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
Regards,
Corentin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
[not found] <1622599316.2056065-1-xuanzhuo@linux.alibaba.com>
@ 2021-06-02 17:56 ` Corentin Noël
0 siblings, 0 replies; 21+ messages in thread
From: Corentin Noël @ 2021-06-02 17:56 UTC (permalink / raw)
To: Xuan Zhuo, Eric Dumazet
Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization,
regressions
Le mercredi 02 juin 2021 à 10:01 +0800, Xuan Zhuo a écrit :
> On Tue, 1 Jun 2021 19:47:44 +0200, Eric Dumazet <edumazet@google.com>
> wrote:
> > On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël
> > <corentin.noel@collabora.com> wrote:
> > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> > > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > > > > I've been experiencing crashes with 5.13 that do not occur
> > > > > with
> > > > > 5.12,
> > > > > here is the crash trace:
> > > > >
> > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > > > > len:3762
> > > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010
> > > > > tail:0xec2
> > > > > end:0xec0 dev:<NULL>
> > > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110!
> > > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI
> > > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted
> > > > > 5.13.0-
> > > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45
> > > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8
> > > > > 00 00
> > > > > 00 50
> > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7
> > > > > e8 7f
> > > > > 4c fb
> > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > > c7 c6
> > > > > e0
> > > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010
> > > > > RCX:
> > > > > 00000000ffffdfff
> > > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea
> > > > > RDI:
> > > > > 0000000000000000
> > > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08
> > > > > R09:
> > > > > 0000000000009ffb
> > > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff
> > > > > R12:
> > > > > ffff9e1e1e95b300
> > > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000
> > > > > R15:
> > > > > 0000000000000eb2
> > > > > [ 47.732541] FS: 00007f3a82b53700(0000)
> > > > > GS:ffff9e1f2bd00000(0000)
> > > > > knlGS:0000000000000000
> > > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > > 0000000080050033
> > > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004
> > > > > CR4:
> > > > > 0000000000370ee0
> > > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000
> > > > > DR2:
> > > > > 0000000000000000
> > > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0
> > > > > DR7:
> > > > > 0000000000000400
> > > > > [ 47.738318] Call Trace:
> > > > > [ 47.738812] skb_put.cold+0x10/0x10
> > > > > [ 47.739450] page_to_skb+0xe4/0x400
> > > > > [ 47.740072] receive_buf+0x86/0x1660
> > > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0
> > > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0
> > > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0
> > > > > [ 47.742357] __napi_poll+0x25/0x150
> > > > > [ 47.742844] net_rx_action+0x22f/0x280
> > > > > [ 47.743388] __do_softirq+0xba/0x264
> > > > > [ 47.743947] irq_exit_rcu+0x90/0xb0
> > > > > [ 47.744435] common_interrupt+0x40/0xa0
> > > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40
> > > > > [ 47.745582] asm_common_interrupt+0x1e/0x40
> > > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4
> > > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00
> > > > > c5 bc
> > > > > 54 c8
> > > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76
> > > > > c0 c5
> > > > > d5 fa
> > > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2
> > > > > 55 b8
> > > > > cb
> > > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff
> > > > > RCX:
> > > > > ffffffffffffffff
> > > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c
> > > > > RDI:
> > > > > 00007f3a7c0575a0
> > > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350
> > > > > R09:
> > > > > 00007f3a8c210354
> > > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef
> > > > > R12:
> > > > > 00007f3a8c210340
> > > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580
> > > > > R15:
> > > > > 00007f3a8c21033c
> > > > > [ 47.755354] Modules linked in:
> > > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45
> > > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8
> > > > > 00 00
> > > > > 00 50
> > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7
> > > > > e8 7f
> > > > > 4c fb
> > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > > c7 c6
> > > > > e0
> > > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010
> > > > > RCX:
> > > > > 00000000ffffdfff
> > > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea
> > > > > RDI:
> > > > > 0000000000000000
> > > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08
> > > > > R09:
> > > > > 0000000000009ffb
> > > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff
> > > > > R12:
> > > > > ffff9e1e1e95b300
> > > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000
> > > > > R15:
> > > > > 0000000000000eb2
> > > > > [ 47.766261] FS: 00007f3a82b53700(0000)
> > > > > GS:ffff9e1f2bd00000(0000)
> > > > > knlGS:0000000000000000
> > > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > > 0000000080050033
> > > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004
> > > > > CR4:
> > > > > 0000000000370ee0
> > > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000
> > > > > DR2:
> > > > > 0000000000000000
> > > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0
> > > > > DR7:
> > > > > 0000000000000400
> > > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in
> > > > > interrupt
> > > > > [ 47.772814] Kernel Offset: 0x35c00000 from
> > > > > 0xffffffff81000000
> > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > > >
> > > > > I've been able to bisect the issue a little bit and the issue
> > > > > disappeared after reverting the 4 following commits:
> > > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > > > > * af39c8f72301b268ad8b04bae646b6025918b82b
> > > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > > > > * f80bd740cb7c954791279590b2e810ba6c214e52
> > > > >
> > > > > Here is my kernel config:
> > > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
>
> Do you have XDP running? If so, you can try it
>
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5c37711d9f27bdc83fd5980446be7f4aa2106230
I applied this patch in top of 5.13-rc4 and it resulted in the same
crash
>
> Thanks.
>
> > > > Do you have the same problem with 5.13-rc4?
> > > >
> > > > thanks,
> > > >
> > > > greg k-h
> > >
> > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
> > >
> > > Thanks,
> > >
> >
> > Could you provide a stack trace with file names and line numbers ?
> >
> > (ie use scripts/decode_stacktrace.sh )
> >
> > Thanks.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-06-01 17:47 ` Eric Dumazet
@ 2021-06-02 17:54 ` Corentin Noël
0 siblings, 0 replies; 21+ messages in thread
From: Corentin Noël @ 2021-06-02 17:54 UTC (permalink / raw)
To: Eric Dumazet
Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization,
regressions, Xuan Zhuo
Le mardi 01 juin 2021 à 19:47 +0200, Eric Dumazet a écrit :
> On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël
> <corentin.noel@collabora.com> wrote:
> > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > > > I've been experiencing crashes with 5.13 that do not occur with
> > > > 5.12,
> > > > here is the crash trace:
> > > >
> > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > > > len:3762
> > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> > > > end:0xec0 dev:<NULL>
> > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110!
> > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI
> > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted
> > > > 5.13.0-
> > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45
> > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00
> > > > 00
> > > > 00 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8
> > > > 7f
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > c7 c6
> > > > e0
> > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > > 0000000000009ffb
> > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff9e1e1e95b300
> > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > > 0000000000000eb2
> > > > [ 47.732541] FS: 00007f3a82b53700(0000)
> > > > GS:ffff9e1f2bd00000(0000)
> > > > knlGS:0000000000000000
> > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > 0000000080050033
> > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > > 0000000000370ee0
> > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [ 47.738318] Call Trace:
> > > > [ 47.738812] skb_put.cold+0x10/0x10
> > > > [ 47.739450] page_to_skb+0xe4/0x400
> > > > [ 47.740072] receive_buf+0x86/0x1660
> > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0
> > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0
> > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0
> > > > [ 47.742357] __napi_poll+0x25/0x150
> > > > [ 47.742844] net_rx_action+0x22f/0x280
> > > > [ 47.743388] __do_softirq+0xba/0x264
> > > > [ 47.743947] irq_exit_rcu+0x90/0xb0
> > > > [ 47.744435] common_interrupt+0x40/0xa0
> > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40
> > > > [ 47.745582] asm_common_interrupt+0x1e/0x40
> > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4
> > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5
> > > > bc
> > > > 54 c8
> > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0
> > > > c5
> > > > d5 fa
> > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2
> > > > 55 b8
> > > > cb
> > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> > > > ffffffffffffffff
> > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> > > > 00007f3a7c0575a0
> > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> > > > 00007f3a8c210354
> > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> > > > 00007f3a8c210340
> > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> > > > 00007f3a8c21033c
> > > > [ 47.755354] Modules linked in:
> > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45
> > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00
> > > > 00
> > > > 00 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8
> > > > 7f
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > c7 c6
> > > > e0
> > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > > 0000000000009ffb
> > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff9e1e1e95b300
> > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > > 0000000000000eb2
> > > > [ 47.766261] FS: 00007f3a82b53700(0000)
> > > > GS:ffff9e1f2bd00000(0000)
> > > > knlGS:0000000000000000
> > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > 0000000080050033
> > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > > 0000000000370ee0
> > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in
> > > > interrupt
> > > > [ 47.772814] Kernel Offset: 0x35c00000 from
> > > > 0xffffffff81000000
> > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > >
> > > > I've been able to bisect the issue a little bit and the issue
> > > > disappeared after reverting the 4 following commits:
> > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > > > * af39c8f72301b268ad8b04bae646b6025918b82b
> > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > > > * f80bd740cb7c954791279590b2e810ba6c214e52
> > > >
> > > > Here is my kernel config:
> > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
> > >
> > > Do you have the same problem with 5.13-rc4?
> > >
> > > thanks,
> > >
> > > greg k-h
> >
> > Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
> >
> > Thanks,
> >
>
> Could you provide a stack trace with file names and line numbers ?
>
> (ie use scripts/decode_stacktrace.sh )
>
> Thanks.
Sure, here is the decoded trace:
[ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 len:3762
put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
end:0xec0 dev:<NULL>
[ 44.525254] kernel BUG at net/core/skbuff.c:110!
[ 44.525910] invalid opcode: 0000 [#1] SMP PTI
[ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
[ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
[ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110)
[ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi
25: e8 43 4c fb ff callq 0xfffffffffffb4c6d
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
[ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
0000000000009ffb
[ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff979ad2aa5600
[ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
0000000000000eb2
[ 44.539300] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000)
knlGS:0000000000000000
[ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
0000000000370ee0
[ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 44.544063] Call Trace:
[ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator 1)
net/core/skbuff.c:5252 (discriminator 1))
[ 44.544864] page_to_skb (drivers/net/virtio_net.c:485)
[ 44.545361] receive_buf (drivers/net/virtio_net.c:849
drivers/net/virtio_net.c:1131)
[ 44.545870] ? netif_receive_skb_list_internal (net/core/dev.c:5714)
[ 44.546628] ? dev_gro_receive (net/core/dev.c:6103)
[ 44.547135] ? napi_complete_done (./include/linux/list.h:35
net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565)
[ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
drivers/net/virtio_net.c:1525)
[ 44.548251] __napi_poll (net/core/dev.c:6985)
[ 44.548744] net_rx_action (net/core/dev.c:7054 net/core/dev.c:7139)
[ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
kernel/softirq.c:560)
[ 44.549762] irq_exit_rcu (kernel/softirq.c:433 kernel/softirq.c:637
kernel/softirq.c:649)
[ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240
(discriminator 13))
[ 44.551991] ? asm_common_interrupt
(./arch/x86/include/asm/idtentry.h:638)
[ 44.552654] asm_common_interrupt
(./arch/x86/include/asm/idtentry.h:638)
[ 44.553276] RIP: 0033:0x7fdb981a82e4
[ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 c4
41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 f9 6c
f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 f9
All code
========
0: d2 48 63 rorb %cl,0x63(%rax)
3: f6 c4 41 test $0x41,%ah
6: 7a 6f jp 0x77
8: 0c 01 or $0x1,%al
a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10
10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12
16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13
1c: c4 c1 31 6a c2 vpunpckhdq %xmm10,%xmm9,%xmm0
21: c4 c1 19 6a d5 vpunpckhdq %xmm13,%xmm12,%xmm2
26: c5 f9 6c f2 vpunpcklqdq %xmm2,%xmm0,%xmm6
2a:* c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8
<-- trapping instruction
2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0
33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0
39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7
3e: c5 .byte 0xc5
3f: f9 stc
Code starting with the faulting instruction
===========================================
0: c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8
4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0
9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0
f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7
14: c5 .byte 0xc5
15: f9 stc
[ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
[ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
0000000000122d40
[ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
000055d7049b9368
[ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
00007fdb5e544040
[ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
0000000000000000
[ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
0000000000000000
[ 44.561965] Modules linked in:
[ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
[ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110)
[ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi
25: e8 43 4c fb ff callq 0xfffffffffffb4c6d
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
[ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
0000000000009ffb
[ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff979ad2aa5600
[ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
0000000000000eb2
[ 44.571483] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000)
knlGS:0000000000000000
[ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
0000000000370ee0
[ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 44.576618] Kernel panic - not syncing: Fatal exception in interrupt
[ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-06-01 17:09 ` Corentin Noël
@ 2021-06-01 17:47 ` Eric Dumazet
2021-06-02 17:54 ` Corentin Noël
0 siblings, 1 reply; 21+ messages in thread
From: Eric Dumazet @ 2021-06-01 17:47 UTC (permalink / raw)
To: Corentin Noël
Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization,
regressions, Xuan Zhuo
On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël
<corentin.noel@collabora.com> wrote:
>
> Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > > I've been experiencing crashes with 5.13 that do not occur with
> > > 5.12,
> > > here is the crash trace:
> > >
> > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > > len:3762
> > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [ 47.716267] kernel BUG at net/core/skbuff.c:110!
> > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI
> > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
> > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45
> > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > > e0
> > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > 0000000000009ffb
> > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff9e1e1e95b300
> > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > 0000000000000eb2
> > > [ 47.732541] FS: 00007f3a82b53700(0000)
> > > GS:ffff9e1f2bd00000(0000)
> > > knlGS:0000000000000000
> > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > 0000000000370ee0
> > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 47.738318] Call Trace:
> > > [ 47.738812] skb_put.cold+0x10/0x10
> > > [ 47.739450] page_to_skb+0xe4/0x400
> > > [ 47.740072] receive_buf+0x86/0x1660
> > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0
> > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0
> > > [ 47.741846] virtnet_poll+0x2b8/0x3c0
> > > [ 47.742357] __napi_poll+0x25/0x150
> > > [ 47.742844] net_rx_action+0x22f/0x280
> > > [ 47.743388] __do_softirq+0xba/0x264
> > > [ 47.743947] irq_exit_rcu+0x90/0xb0
> > > [ 47.744435] common_interrupt+0x40/0xa0
> > > [ 47.744978] ? asm_common_interrupt+0x8/0x40
> > > [ 47.745582] asm_common_interrupt+0x1e/0x40
> > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4
> > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc
> > > 54 c8
> > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5
> > > d5 fa
> > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8
> > > cb
> > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> > > ffffffffffffffff
> > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> > > 00007f3a7c0575a0
> > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> > > 00007f3a8c210354
> > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> > > 00007f3a8c210340
> > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> > > 00007f3a8c21033c
> > > [ 47.755354] Modules linked in:
> > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45
> > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > > e0
> > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > 0000000000009ffb
> > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff9e1e1e95b300
> > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > 0000000000000eb2
> > > [ 47.766261] FS: 00007f3a82b53700(0000)
> > > GS:ffff9e1f2bd00000(0000)
> > > knlGS:0000000000000000
> > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > 0000000000370ee0
> > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 47.771339] Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > >
> > > I've been able to bisect the issue a little bit and the issue
> > > disappeared after reverting the 4 following commits:
> > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > > * af39c8f72301b268ad8b04bae646b6025918b82b
> > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > > * f80bd740cb7c954791279590b2e810ba6c214e52
> > >
> > > Here is my kernel config:
> > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
> >
> > Do you have the same problem with 5.13-rc4?
> >
> > thanks,
> >
> > greg k-h
>
> Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
>
> Thanks,
>
Could you provide a stack trace with file names and line numbers ?
(ie use scripts/decode_stacktrace.sh )
Thanks.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-06-01 17:07 ` Greg KH
@ 2021-06-01 17:09 ` Corentin Noël
2021-06-01 17:47 ` Eric Dumazet
0 siblings, 1 reply; 21+ messages in thread
From: Corentin Noël @ 2021-06-01 17:09 UTC (permalink / raw)
To: Greg KH
Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization,
regressions, Eric Dumazet, Xuan Zhuo
Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > I've been experiencing crashes with 5.13 that do not occur with
> > 5.12,
> > here is the crash trace:
> >
> > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > len:3762
> > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [ 47.716267] kernel BUG at net/core/skbuff.c:110!
> > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI
> > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
> > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45
> > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > e0
> > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > 0000000000009ffb
> > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff9e1e1e95b300
> > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > 0000000000000eb2
> > [ 47.732541] FS: 00007f3a82b53700(0000)
> > GS:ffff9e1f2bd00000(0000)
> > knlGS:0000000000000000
> > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > 0000000000370ee0
> > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 47.738318] Call Trace:
> > [ 47.738812] skb_put.cold+0x10/0x10
> > [ 47.739450] page_to_skb+0xe4/0x400
> > [ 47.740072] receive_buf+0x86/0x1660
> > [ 47.740693] ? inet_gro_receive+0x54/0x2c0
> > [ 47.741279] ? dev_gro_receive+0x194/0x6a0
> > [ 47.741846] virtnet_poll+0x2b8/0x3c0
> > [ 47.742357] __napi_poll+0x25/0x150
> > [ 47.742844] net_rx_action+0x22f/0x280
> > [ 47.743388] __do_softirq+0xba/0x264
> > [ 47.743947] irq_exit_rcu+0x90/0xb0
> > [ 47.744435] common_interrupt+0x40/0xa0
> > [ 47.744978] ? asm_common_interrupt+0x8/0x40
> > [ 47.745582] asm_common_interrupt+0x1e/0x40
> > [ 47.746182] RIP: 0033:0x7f3a7a276ed4
> > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc
> > 54 c8
> > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5
> > d5 fa
> > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8
> > cb
> > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> > ffffffffffffffff
> > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> > 00007f3a7c0575a0
> > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> > 00007f3a8c210354
> > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> > 00007f3a8c210340
> > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> > 00007f3a8c21033c
> > [ 47.755354] Modules linked in:
> > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45
> > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > e0
> > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > 0000000000009ffb
> > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff9e1e1e95b300
> > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > 0000000000000eb2
> > [ 47.766261] FS: 00007f3a82b53700(0000)
> > GS:ffff9e1f2bd00000(0000)
> > knlGS:0000000000000000
> > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > 0000000000370ee0
> > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 47.771339] Kernel panic - not syncing: Fatal exception in
> > interrupt
> > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
> > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> >
> > I've been able to bisect the issue a little bit and the issue
> > disappeared after reverting the 4 following commits:
> > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > * af39c8f72301b268ad8b04bae646b6025918b82b
> > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > * f80bd740cb7c954791279590b2e810ba6c214e52
> >
> > Here is my kernel config:
> > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
>
> Do you have the same problem with 5.13-rc4?
>
> thanks,
>
> greg k-h
Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
Thanks,
Corentin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c
2021-06-01 16:06 Corentin Noël
@ 2021-06-01 17:07 ` Greg KH
2021-06-01 17:09 ` Corentin Noël
0 siblings, 1 reply; 21+ messages in thread
From: Greg KH @ 2021-06-01 17:07 UTC (permalink / raw)
To: Corentin Noël
Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization,
regressions, Eric Dumazet, Xuan Zhuo
On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> I've been experiencing crashes with 5.13 that do not occur with 5.12,
> here is the crash trace:
>
> [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762
> put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> end:0xec0 dev:<NULL>
> [ 47.716267] kernel BUG at net/core/skbuff.c:110!
> [ 47.717197] invalid opcode: 0000 [#1] SMP PTI
> [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
> rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> [ 47.720656] RIP: 0010:skb_panic+0x43/0x45
> [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
> [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> 0000000000009ffb
> [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff9e1e1e95b300
> [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> 0000000000000eb2
> [ 47.732541] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
> knlGS:0000000000000000
> [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> 0000000000370ee0
> [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [ 47.738318] Call Trace:
> [ 47.738812] skb_put.cold+0x10/0x10
> [ 47.739450] page_to_skb+0xe4/0x400
> [ 47.740072] receive_buf+0x86/0x1660
> [ 47.740693] ? inet_gro_receive+0x54/0x2c0
> [ 47.741279] ? dev_gro_receive+0x194/0x6a0
> [ 47.741846] virtnet_poll+0x2b8/0x3c0
> [ 47.742357] __napi_poll+0x25/0x150
> [ 47.742844] net_rx_action+0x22f/0x280
> [ 47.743388] __do_softirq+0xba/0x264
> [ 47.743947] irq_exit_rcu+0x90/0xb0
> [ 47.744435] common_interrupt+0x40/0xa0
> [ 47.744978] ? asm_common_interrupt+0x8/0x40
> [ 47.745582] asm_common_interrupt+0x1e/0x40
> [ 47.746182] RIP: 0033:0x7f3a7a276ed4
> [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8
> c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa
> c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb
> [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> ffffffffffffffff
> [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> 00007f3a7c0575a0
> [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> 00007f3a8c210354
> [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> 00007f3a8c210340
> [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> 00007f3a8c21033c
> [ 47.755354] Modules linked in:
> [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> [ 47.756606] RIP: 0010:skb_panic+0x43/0x45
> [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
> [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> 0000000000009ffb
> [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff9e1e1e95b300
> [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> 0000000000000eb2
> [ 47.766261] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
> knlGS:0000000000000000
> [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> 0000000000370ee0
> [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [ 47.771339] Kernel panic - not syncing: Fatal exception in interrupt
> [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>
> I've been able to bisect the issue a little bit and the issue
> disappeared after reverting the 4 following commits:
> * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> * af39c8f72301b268ad8b04bae646b6025918b82b
> * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> * f80bd740cb7c954791279590b2e810ba6c214e52
>
> Here is my kernel config:
> https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
Do you have the same problem with 5.13-rc4?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 21+ messages in thread
* virtio-net: kernel panic in virtio_net.c
@ 2021-06-01 16:06 Corentin Noël
2021-06-01 17:07 ` Greg KH
0 siblings, 1 reply; 21+ messages in thread
From: Corentin Noël @ 2021-06-01 16:06 UTC (permalink / raw)
To: stable
Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions,
Eric Dumazet, Xuan Zhuo
I've been experiencing crashes with 5.13 that do not occur with 5.12,
here is the crash trace:
[ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762
put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
end:0xec0 dev:<NULL>
[ 47.716267] kernel BUG at net/core/skbuff.c:110!
[ 47.717197] invalid opcode: 0000 [#1] SMP PTI
[ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
[ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
[ 47.720656] RIP: 0010:skb_panic+0x43/0x45
[ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
[ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
[ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
0000000000009ffb
[ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff9e1e1e95b300
[ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
0000000000000eb2
[ 47.732541] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
knlGS:0000000000000000
[ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
0000000000370ee0
[ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 47.738318] Call Trace:
[ 47.738812] skb_put.cold+0x10/0x10
[ 47.739450] page_to_skb+0xe4/0x400
[ 47.740072] receive_buf+0x86/0x1660
[ 47.740693] ? inet_gro_receive+0x54/0x2c0
[ 47.741279] ? dev_gro_receive+0x194/0x6a0
[ 47.741846] virtnet_poll+0x2b8/0x3c0
[ 47.742357] __napi_poll+0x25/0x150
[ 47.742844] net_rx_action+0x22f/0x280
[ 47.743388] __do_softirq+0xba/0x264
[ 47.743947] irq_exit_rcu+0x90/0xb0
[ 47.744435] common_interrupt+0x40/0xa0
[ 47.744978] ? asm_common_interrupt+0x8/0x40
[ 47.745582] asm_common_interrupt+0x1e/0x40
[ 47.746182] RIP: 0033:0x7f3a7a276ed4
[ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8
c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa
c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb
[ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
[ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
ffffffffffffffff
[ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
00007f3a7c0575a0
[ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
00007f3a8c210354
[ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
00007f3a8c210340
[ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
00007f3a8c21033c
[ 47.755354] Modules linked in:
[ 47.755871] ---[ end trace a8b692ea99c9cd9e ]---
[ 47.756606] RIP: 0010:skb_panic+0x43/0x45
[ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
[ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
[ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
0000000000009ffb
[ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff9e1e1e95b300
[ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
0000000000000eb2
[ 47.766261] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
knlGS:0000000000000000
[ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
0000000000370ee0
[ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 47.771339] Kernel panic - not syncing: Fatal exception in interrupt
[ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
I've been able to bisect the issue a little bit and the issue
disappeared after reverting the 4 following commits:
* fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
* af39c8f72301b268ad8b04bae646b6025918b82b
* f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
* f80bd740cb7c954791279590b2e810ba6c214e52
Here is my kernel config:
https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
Regards,
Corentin
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2021-10-09 5:19 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <1622688283.7488964-1-xuanzhuo@linux.alibaba.com>
2021-06-03 8:57 ` virtio-net: kernel panic in virtio_net.c Corentin Noël
2021-06-08 12:17 ` Greg KH
[not found] ` <1623203313.4303577-1-xuanzhuo@linux.alibaba.com>
2021-06-09 4:50 ` Greg KH
[not found] ` <1623218897.4150124-1-xuanzhuo@linux.alibaba.com>
2021-06-09 6:24 ` Greg KH
[not found] ` <1623225080.4793522-1-xuanzhuo@linux.alibaba.com>
2021-06-09 8:03 ` Greg KH
2021-10-07 12:04 Corentin Noël
2021-10-07 13:10 ` Michael S. Tsirkin
2021-10-07 13:51 ` Eric Dumazet
2021-10-07 14:02 ` Corentin Noël
2021-10-07 14:13 ` Greg KH
[not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com>
2021-10-07 15:25 ` Greg KH
[not found] ` <1633623446.6192446-1-xuanzhuo@linux.alibaba.com>
2021-10-08 8:06 ` Greg KH
2021-10-08 10:02 ` Michael S. Tsirkin
2021-10-08 12:21 ` Corentin Noël
[not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com>
2021-10-09 5:19 ` Greg KH
[not found] <1622599316.2056065-1-xuanzhuo@linux.alibaba.com>
2021-06-02 17:56 ` Corentin Noël
-- strict thread matches above, loose matches on Subject: below --
2021-06-01 16:06 Corentin Noël
2021-06-01 17:07 ` Greg KH
2021-06-01 17:09 ` Corentin Noël
2021-06-01 17:47 ` Eric Dumazet
2021-06-02 17:54 ` Corentin Noël
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).