regressions.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* virtio-net: kernel panic in virtio_net.c
@ 2021-06-01 16:06 Corentin Noël
  2021-06-01 17:07 ` Greg KH
  0 siblings, 1 reply; 33+ messages in thread
From: Corentin Noël @ 2021-06-01 16:06 UTC (permalink / raw)
  To: stable
  Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions,
	Eric Dumazet, Xuan Zhuo

I've been experiencing crashes with 5.13 that do not occur with 5.12,
here is the crash trace:

[   47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762
put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
end:0xec0 dev:<NULL>
[   47.716267] kernel BUG at net/core/skbuff.c:110!
[   47.717197] invalid opcode: 0000 [#1] SMP PTI
[   47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
[   47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 
[   47.720656] RIP: 0010:skb_panic+0x43/0x45
[   47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
[   47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
[   47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
0000000000009ffb
[   47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff9e1e1e95b300
[   47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
0000000000000eb2
[   47.732541] FS:  00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
knlGS:0000000000000000
[   47.733858] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
0000000000370ee0
[   47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   47.738318] Call Trace:
[   47.738812]  skb_put.cold+0x10/0x10
[   47.739450]  page_to_skb+0xe4/0x400
[   47.740072]  receive_buf+0x86/0x1660
[   47.740693]  ? inet_gro_receive+0x54/0x2c0
[   47.741279]  ? dev_gro_receive+0x194/0x6a0
[   47.741846]  virtnet_poll+0x2b8/0x3c0
[   47.742357]  __napi_poll+0x25/0x150
[   47.742844]  net_rx_action+0x22f/0x280
[   47.743388]  __do_softirq+0xba/0x264
[   47.743947]  irq_exit_rcu+0x90/0xb0
[   47.744435]  common_interrupt+0x40/0xa0
[   47.744978]  ? asm_common_interrupt+0x8/0x40
[   47.745582]  asm_common_interrupt+0x1e/0x40
[   47.746182] RIP: 0033:0x7f3a7a276ed4
[   47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8
c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa
c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb
[   47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
[   47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
ffffffffffffffff
[   47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
00007f3a7c0575a0
[   47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
00007f3a8c210354
[   47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
00007f3a8c210340
[   47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
00007f3a8c21033c
[   47.755354] Modules linked in:
[   47.755871] ---[ end trace a8b692ea99c9cd9e ]---
[   47.756606] RIP: 0010:skb_panic+0x43/0x45
[   47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
[   47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
[   47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
0000000000009ffb
[   47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff9e1e1e95b300
[   47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
0000000000000eb2
[   47.766261] FS:  00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
knlGS:0000000000000000
[   47.767512] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
0000000000370ee0
[   47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   47.771339] Kernel panic - not syncing: Fatal exception in interrupt
[   47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)

I've been able to bisect the issue a little bit and the issue
disappeared after reverting the 4 following commits:
 * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
 * af39c8f72301b268ad8b04bae646b6025918b82b
 * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
 * f80bd740cb7c954791279590b2e810ba6c214e52

Here is my kernel config: 
https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config

Regards,
Corentin


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-01 16:06 virtio-net: kernel panic in virtio_net.c Corentin Noël
@ 2021-06-01 17:07 ` Greg KH
  2021-06-01 17:09   ` Corentin Noël
  0 siblings, 1 reply; 33+ messages in thread
From: Greg KH @ 2021-06-01 17:07 UTC (permalink / raw)
  To: Corentin Noël
  Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Xuan Zhuo

On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> I've been experiencing crashes with 5.13 that do not occur with 5.12,
> here is the crash trace:
> 
> [   47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762
> put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> end:0xec0 dev:<NULL>
> [   47.716267] kernel BUG at net/core/skbuff.c:110!
> [   47.717197] invalid opcode: 0000 [#1] SMP PTI
> [   47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
> rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> [   47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 
> [   47.720656] RIP: 0010:skb_panic+0x43/0x45
> [   47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
> [   47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> [   47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> 0000000000009ffb
> [   47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff9e1e1e95b300
> [   47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> 0000000000000eb2
> [   47.732541] FS:  00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
> knlGS:0000000000000000
> [   47.733858] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> 0000000000370ee0
> [   47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   47.738318] Call Trace:
> [   47.738812]  skb_put.cold+0x10/0x10
> [   47.739450]  page_to_skb+0xe4/0x400
> [   47.740072]  receive_buf+0x86/0x1660
> [   47.740693]  ? inet_gro_receive+0x54/0x2c0
> [   47.741279]  ? dev_gro_receive+0x194/0x6a0
> [   47.741846]  virtnet_poll+0x2b8/0x3c0
> [   47.742357]  __napi_poll+0x25/0x150
> [   47.742844]  net_rx_action+0x22f/0x280
> [   47.743388]  __do_softirq+0xba/0x264
> [   47.743947]  irq_exit_rcu+0x90/0xb0
> [   47.744435]  common_interrupt+0x40/0xa0
> [   47.744978]  ? asm_common_interrupt+0x8/0x40
> [   47.745582]  asm_common_interrupt+0x1e/0x40
> [   47.746182] RIP: 0033:0x7f3a7a276ed4
> [   47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8
> c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa
> c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb
> [   47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> [   47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> ffffffffffffffff
> [   47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> 00007f3a7c0575a0
> [   47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> 00007f3a8c210354
> [   47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> 00007f3a8c210340
> [   47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> 00007f3a8c21033c
> [   47.755354] Modules linked in:
> [   47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> [   47.756606] RIP: 0010:skb_panic+0x43/0x45
> [   47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0
> [   47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> [   47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> 0000000000009ffb
> [   47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff9e1e1e95b300
> [   47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> 0000000000000eb2
> [   47.766261] FS:  00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000)
> knlGS:0000000000000000
> [   47.767512] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> 0000000000370ee0
> [   47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   47.771339] Kernel panic - not syncing: Fatal exception in interrupt
> [   47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> 
> I've been able to bisect the issue a little bit and the issue
> disappeared after reverting the 4 following commits:
>  * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
>  * af39c8f72301b268ad8b04bae646b6025918b82b
>  * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
>  * f80bd740cb7c954791279590b2e810ba6c214e52
> 
> Here is my kernel config: 
> https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config

Do you have the same problem with 5.13-rc4?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-01 17:07 ` Greg KH
@ 2021-06-01 17:09   ` Corentin Noël
  2021-06-01 17:47     ` Eric Dumazet
  0 siblings, 1 reply; 33+ messages in thread
From: Corentin Noël @ 2021-06-01 17:09 UTC (permalink / raw)
  To: Greg KH
  Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Xuan Zhuo

Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > I've been experiencing crashes with 5.13 that do not occur with
> > 5.12,
> > here is the crash trace:
> > 
> > [   47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > len:3762
> > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [   47.716267] kernel BUG at net/core/skbuff.c:110!
> > [   47.717197] invalid opcode: 0000 [#1] SMP PTI
> > [   47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
> > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > [   47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 
> > [   47.720656] RIP: 0010:skb_panic+0x43/0x45
> > [   47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > e0
> > [   47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > [   47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > 0000000000009ffb
> > [   47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff9e1e1e95b300
> > [   47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > 0000000000000eb2
> > [   47.732541] FS:  00007f3a82b53700(0000)
> > GS:ffff9e1f2bd00000(0000)
> > knlGS:0000000000000000
> > [   47.733858] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > 0000000000370ee0
> > [   47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   47.738318] Call Trace:
> > [   47.738812]  skb_put.cold+0x10/0x10
> > [   47.739450]  page_to_skb+0xe4/0x400
> > [   47.740072]  receive_buf+0x86/0x1660
> > [   47.740693]  ? inet_gro_receive+0x54/0x2c0
> > [   47.741279]  ? dev_gro_receive+0x194/0x6a0
> > [   47.741846]  virtnet_poll+0x2b8/0x3c0
> > [   47.742357]  __napi_poll+0x25/0x150
> > [   47.742844]  net_rx_action+0x22f/0x280
> > [   47.743388]  __do_softirq+0xba/0x264
> > [   47.743947]  irq_exit_rcu+0x90/0xb0
> > [   47.744435]  common_interrupt+0x40/0xa0
> > [   47.744978]  ? asm_common_interrupt+0x8/0x40
> > [   47.745582]  asm_common_interrupt+0x1e/0x40
> > [   47.746182] RIP: 0033:0x7f3a7a276ed4
> > [   47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc
> > 54 c8
> > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5
> > d5 fa
> > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8
> > cb
> > [   47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > [   47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> > ffffffffffffffff
> > [   47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> > 00007f3a7c0575a0
> > [   47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> > 00007f3a8c210354
> > [   47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> > 00007f3a8c210340
> > [   47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> > 00007f3a8c21033c
> > [   47.755354] Modules linked in:
> > [   47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > [   47.756606] RIP: 0010:skb_panic+0x43/0x45
> > [   47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > e0
> > [   47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > [   47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > 0000000000009ffb
> > [   47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff9e1e1e95b300
> > [   47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > 0000000000000eb2
> > [   47.766261] FS:  00007f3a82b53700(0000)
> > GS:ffff9e1f2bd00000(0000)
> > knlGS:0000000000000000
> > [   47.767512] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > 0000000000370ee0
> > [   47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   47.771339] Kernel panic - not syncing: Fatal exception in
> > interrupt
> > [   47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
> > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > 
> > I've been able to bisect the issue a little bit and the issue
> > disappeared after reverting the 4 following commits:
> >  * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> >  * af39c8f72301b268ad8b04bae646b6025918b82b
> >  * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> >  * f80bd740cb7c954791279590b2e810ba6c214e52
> > 
> > Here is my kernel config: 
> > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
> 
> Do you have the same problem with 5.13-rc4?
> 
> thanks,
> 
> greg k-h

Yes I tried with rc2, rc3 and rc4 resulting to the same panic.

Thanks,

Corentin


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-01 17:09   ` Corentin Noël
@ 2021-06-01 17:47     ` Eric Dumazet
  2021-06-02  2:01       ` Xuan Zhuo
  2021-06-02 17:54       ` Corentin Noël
  0 siblings, 2 replies; 33+ messages in thread
From: Eric Dumazet @ 2021-06-01 17:47 UTC (permalink / raw)
  To: Corentin Noël
  Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization,
	regressions, Xuan Zhuo

On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël
<corentin.noel@collabora.com> wrote:
>
> Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > > I've been experiencing crashes with 5.13 that do not occur with
> > > 5.12,
> > > here is the crash trace:
> > >
> > > [   47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > > len:3762
> > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [   47.716267] kernel BUG at net/core/skbuff.c:110!
> > > [   47.717197] invalid opcode: 0000 [#1] SMP PTI
> > > [   47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
> > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > > [   47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [   47.720656] RIP: 0010:skb_panic+0x43/0x45
> > > [   47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > > e0
> > > [   47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > [   47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > 0000000000009ffb
> > > [   47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff9e1e1e95b300
> > > [   47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > 0000000000000eb2
> > > [   47.732541] FS:  00007f3a82b53700(0000)
> > > GS:ffff9e1f2bd00000(0000)
> > > knlGS:0000000000000000
> > > [   47.733858] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > 0000000000370ee0
> > > [   47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   47.738318] Call Trace:
> > > [   47.738812]  skb_put.cold+0x10/0x10
> > > [   47.739450]  page_to_skb+0xe4/0x400
> > > [   47.740072]  receive_buf+0x86/0x1660
> > > [   47.740693]  ? inet_gro_receive+0x54/0x2c0
> > > [   47.741279]  ? dev_gro_receive+0x194/0x6a0
> > > [   47.741846]  virtnet_poll+0x2b8/0x3c0
> > > [   47.742357]  __napi_poll+0x25/0x150
> > > [   47.742844]  net_rx_action+0x22f/0x280
> > > [   47.743388]  __do_softirq+0xba/0x264
> > > [   47.743947]  irq_exit_rcu+0x90/0xb0
> > > [   47.744435]  common_interrupt+0x40/0xa0
> > > [   47.744978]  ? asm_common_interrupt+0x8/0x40
> > > [   47.745582]  asm_common_interrupt+0x1e/0x40
> > > [   47.746182] RIP: 0033:0x7f3a7a276ed4
> > > [   47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc
> > > 54 c8
> > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5
> > > d5 fa
> > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8
> > > cb
> > > [   47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > > [   47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> > > ffffffffffffffff
> > > [   47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> > > 00007f3a7c0575a0
> > > [   47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> > > 00007f3a8c210354
> > > [   47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> > > 00007f3a8c210340
> > > [   47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> > > 00007f3a8c21033c
> > > [   47.755354] Modules linked in:
> > > [   47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > > [   47.756606] RIP: 0010:skb_panic+0x43/0x45
> > > [   47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > > e0
> > > [   47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > [   47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > 0000000000009ffb
> > > [   47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff9e1e1e95b300
> > > [   47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > 0000000000000eb2
> > > [   47.766261] FS:  00007f3a82b53700(0000)
> > > GS:ffff9e1f2bd00000(0000)
> > > knlGS:0000000000000000
> > > [   47.767512] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > 0000000000370ee0
> > > [   47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   47.771339] Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > > [   47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > >
> > > I've been able to bisect the issue a little bit and the issue
> > > disappeared after reverting the 4 following commits:
> > >  * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > >  * af39c8f72301b268ad8b04bae646b6025918b82b
> > >  * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > >  * f80bd740cb7c954791279590b2e810ba6c214e52
> > >
> > > Here is my kernel config:
> > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
> >
> > Do you have the same problem with 5.13-rc4?
> >
> > thanks,
> >
> > greg k-h
>
> Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
>
> Thanks,
>


Could you provide a stack trace with file names and line numbers ?

(ie use scripts/decode_stacktrace.sh )

Thanks.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-01 17:47     ` Eric Dumazet
@ 2021-06-02  2:01       ` Xuan Zhuo
  2021-06-02 17:56         ` Corentin Noël
  2021-06-02 17:54       ` Corentin Noël
  1 sibling, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-06-02  2:01 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization,
	regressions, Corentin Noël

On Tue, 1 Jun 2021 19:47:44 +0200, Eric Dumazet <edumazet@google.com> wrote:
> On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël
> <corentin.noel@collabora.com> wrote:
> >
> > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > > > I've been experiencing crashes with 5.13 that do not occur with
> > > > 5.12,
> > > > here is the crash trace:
> > > >
> > > > [   47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > > > len:3762
> > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> > > > end:0xec0 dev:<NULL>
> > > > [   47.716267] kernel BUG at net/core/skbuff.c:110!
> > > > [   47.717197] invalid opcode: 0000 [#1] SMP PTI
> > > > [   47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0-
> > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > > > [   47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > > > [   47.720656] RIP: 0010:skb_panic+0x43/0x45
> > > > [   47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > > 00 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > > > e0
> > > > [   47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > [   47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [   47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [   47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > > 0000000000009ffb
> > > > [   47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff9e1e1e95b300
> > > > [   47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > > 0000000000000eb2
> > > > [   47.732541] FS:  00007f3a82b53700(0000)
> > > > GS:ffff9e1f2bd00000(0000)
> > > > knlGS:0000000000000000
> > > > [   47.733858] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > > 0000000000370ee0
> > > > [   47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [   47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [   47.738318] Call Trace:
> > > > [   47.738812]  skb_put.cold+0x10/0x10
> > > > [   47.739450]  page_to_skb+0xe4/0x400
> > > > [   47.740072]  receive_buf+0x86/0x1660
> > > > [   47.740693]  ? inet_gro_receive+0x54/0x2c0
> > > > [   47.741279]  ? dev_gro_receive+0x194/0x6a0
> > > > [   47.741846]  virtnet_poll+0x2b8/0x3c0
> > > > [   47.742357]  __napi_poll+0x25/0x150
> > > > [   47.742844]  net_rx_action+0x22f/0x280
> > > > [   47.743388]  __do_softirq+0xba/0x264
> > > > [   47.743947]  irq_exit_rcu+0x90/0xb0
> > > > [   47.744435]  common_interrupt+0x40/0xa0
> > > > [   47.744978]  ? asm_common_interrupt+0x8/0x40
> > > > [   47.745582]  asm_common_interrupt+0x1e/0x40
> > > > [   47.746182] RIP: 0033:0x7f3a7a276ed4
> > > > [   47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc
> > > > 54 c8
> > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5
> > > > d5 fa
> > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8
> > > > cb
> > > > [   47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > > > [   47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> > > > ffffffffffffffff
> > > > [   47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> > > > 00007f3a7c0575a0
> > > > [   47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> > > > 00007f3a8c210354
> > > > [   47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> > > > 00007f3a8c210340
> > > > [   47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> > > > 00007f3a8c21033c
> > > > [   47.755354] Modules linked in:
> > > > [   47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > > > [   47.756606] RIP: 0010:skb_panic+0x43/0x45
> > > > [   47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > > 00 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6
> > > > e0
> > > > [   47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > [   47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [   47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [   47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > > 0000000000009ffb
> > > > [   47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff9e1e1e95b300
> > > > [   47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > > 0000000000000eb2
> > > > [   47.766261] FS:  00007f3a82b53700(0000)
> > > > GS:ffff9e1f2bd00000(0000)
> > > > knlGS:0000000000000000
> > > > [   47.767512] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > > 0000000000370ee0
> > > > [   47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [   47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [   47.771339] Kernel panic - not syncing: Fatal exception in
> > > > interrupt
> > > > [   47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000
> > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > >
> > > > I've been able to bisect the issue a little bit and the issue
> > > > disappeared after reverting the 4 following commits:
> > > >  * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > > >  * af39c8f72301b268ad8b04bae646b6025918b82b
> > > >  * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > > >  * f80bd740cb7c954791279590b2e810ba6c214e52
> > > >
> > > > Here is my kernel config:
> > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config

Do you have XDP running? If so, you can try it

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5c37711d9f27bdc83fd5980446be7f4aa2106230

Thanks.

> > >
> > > Do you have the same problem with 5.13-rc4?
> > >
> > > thanks,
> > >
> > > greg k-h
> >
> > Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
> >
> > Thanks,
> >
>
>
> Could you provide a stack trace with file names and line numbers ?
>
> (ie use scripts/decode_stacktrace.sh )
>
> Thanks.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-01 17:47     ` Eric Dumazet
  2021-06-02  2:01       ` Xuan Zhuo
@ 2021-06-02 17:54       ` Corentin Noël
  2021-06-03  2:44         ` Xuan Zhuo
  1 sibling, 1 reply; 33+ messages in thread
From: Corentin Noël @ 2021-06-02 17:54 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization,
	regressions, Xuan Zhuo

Le mardi 01 juin 2021 à 19:47 +0200, Eric Dumazet a écrit :
> On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël
> <corentin.noel@collabora.com> wrote:
> > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > > > I've been experiencing crashes with 5.13 that do not occur with
> > > > 5.12,
> > > > here is the crash trace:
> > > > 
> > > > [   47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > > > len:3762
> > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2
> > > > end:0xec0 dev:<NULL>
> > > > [   47.716267] kernel BUG at net/core/skbuff.c:110!
> > > > [   47.717197] invalid opcode: 0000 [#1] SMP PTI
> > > > [   47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted
> > > > 5.13.0-
> > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > > > [   47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > > > [   47.720656] RIP: 0010:skb_panic+0x43/0x45
> > > > [   47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00
> > > > 00
> > > > 00 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8
> > > > 7f
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > c7 c6
> > > > e0
> > > > [   47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > [   47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [   47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [   47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > > 0000000000009ffb
> > > > [   47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff9e1e1e95b300
> > > > [   47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > > 0000000000000eb2
> > > > [   47.732541] FS:  00007f3a82b53700(0000)
> > > > GS:ffff9e1f2bd00000(0000)
> > > > knlGS:0000000000000000
> > > > [   47.733858] CS:  0010 DS: 0000 ES: 0000 CR0:
> > > > 0000000080050033
> > > > [   47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > > 0000000000370ee0
> > > > [   47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [   47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [   47.738318] Call Trace:
> > > > [   47.738812]  skb_put.cold+0x10/0x10
> > > > [   47.739450]  page_to_skb+0xe4/0x400
> > > > [   47.740072]  receive_buf+0x86/0x1660
> > > > [   47.740693]  ? inet_gro_receive+0x54/0x2c0
> > > > [   47.741279]  ? dev_gro_receive+0x194/0x6a0
> > > > [   47.741846]  virtnet_poll+0x2b8/0x3c0
> > > > [   47.742357]  __napi_poll+0x25/0x150
> > > > [   47.742844]  net_rx_action+0x22f/0x280
> > > > [   47.743388]  __do_softirq+0xba/0x264
> > > > [   47.743947]  irq_exit_rcu+0x90/0xb0
> > > > [   47.744435]  common_interrupt+0x40/0xa0
> > > > [   47.744978]  ? asm_common_interrupt+0x8/0x40
> > > > [   47.745582]  asm_common_interrupt+0x1e/0x40
> > > > [   47.746182] RIP: 0033:0x7f3a7a276ed4
> > > > [   47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5
> > > > bc
> > > > 54 c8
> > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0
> > > > c5
> > > > d5 fa
> > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2
> > > > 55 b8
> > > > cb
> > > > [   47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > > > [   47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX:
> > > > ffffffffffffffff
> > > > [   47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI:
> > > > 00007f3a7c0575a0
> > > > [   47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09:
> > > > 00007f3a8c210354
> > > > [   47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12:
> > > > 00007f3a8c210340
> > > > [   47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15:
> > > > 00007f3a8c21033c
> > > > [   47.755354] Modules linked in:
> > > > [   47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > > > [   47.756606] RIP: 0010:skb_panic+0x43/0x45
> > > > [   47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00
> > > > 00
> > > > 00 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8
> > > > 7f
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > c7 c6
> > > > e0
> > > > [   47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > [   47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [   47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [   47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09:
> > > > 0000000000009ffb
> > > > [   47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff9e1e1e95b300
> > > > [   47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15:
> > > > 0000000000000eb2
> > > > [   47.766261] FS:  00007f3a82b53700(0000)
> > > > GS:ffff9e1f2bd00000(0000)
> > > > knlGS:0000000000000000
> > > > [   47.767512] CS:  0010 DS: 0000 ES: 0000 CR0:
> > > > 0000000080050033
> > > > [   47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4:
> > > > 0000000000370ee0
> > > > [   47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [   47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [   47.771339] Kernel panic - not syncing: Fatal exception in
> > > > interrupt
> > > > [   47.772814] Kernel Offset: 0x35c00000 from
> > > > 0xffffffff81000000
> > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > > 
> > > > I've been able to bisect the issue a little bit and the issue
> > > > disappeared after reverting the 4 following commits:
> > > >  * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > > >  * af39c8f72301b268ad8b04bae646b6025918b82b
> > > >  * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > > >  * f80bd740cb7c954791279590b2e810ba6c214e52
> > > > 
> > > > Here is my kernel config:
> > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
> > > 
> > > Do you have the same problem with 5.13-rc4?
> > > 
> > > thanks,
> > > 
> > > greg k-h
> > 
> > Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
> > 
> > Thanks,
> > 
> 
> Could you provide a stack trace with file names and line numbers ?
> 
> (ie use scripts/decode_stacktrace.sh )
> 
> Thanks.

Sure, here is the decoded trace:

[   44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 len:3762
put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
end:0xec0 dev:<NULL>
[   44.525254] kernel BUG at net/core/skbuff.c:110!
[   44.525910] invalid opcode: 0000 [#1] SMP PTI
[   44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
[   44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
[   44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) 
[ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60
All code
========
   0:	4f 70 50             	rex.WRXB jo 0x53
   3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
   9:	50                   	push   %rax
   a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
  10:	50                   	push   %rax
  11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
  17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
  1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,%rdi
  25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
  2a:*	0f 0b                	ud2    		<-- trapping
instruction
  2c:	48 8b 14 24          	mov    (%rsp),%rdx
  30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
  37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
  3c:	48                   	rex.W
  3d:	c7                   	.byte 0xc7
  3e:	c6                   	(bad)  
  3f:	60                   	(bad)  

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	48 8b 14 24          	mov    (%rsp),%rdx
   6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
   d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
  12:	48                   	rex.W
  13:	c7                   	.byte 0xc7
  14:	c6                   	(bad)  
  15:	60                   	(bad)  
[   44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
[   44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
0000000000009ffb
[   44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff979ad2aa5600
[   44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
0000000000000eb2
[   44.539300] FS:  00007fdb9cb11700(0000) GS:ffff979aebd00000(0000)
knlGS:0000000000000000
[   44.540376] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
0000000000370ee0
[   44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   44.544063] Call Trace:
[   44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator 1)
net/core/skbuff.c:5252 (discriminator 1)) 
[   44.544864] page_to_skb (drivers/net/virtio_net.c:485) 
[   44.545361] receive_buf (drivers/net/virtio_net.c:849
drivers/net/virtio_net.c:1131) 
[   44.545870] ? netif_receive_skb_list_internal (net/core/dev.c:5714) 
[   44.546628] ? dev_gro_receive (net/core/dev.c:6103) 
[   44.547135] ? napi_complete_done (./include/linux/list.h:35
net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) 
[   44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
drivers/net/virtio_net.c:1525) 
[   44.548251] __napi_poll (net/core/dev.c:6985) 
[   44.548744] net_rx_action (net/core/dev.c:7054 net/core/dev.c:7139) 
[   44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
kernel/softirq.c:560) 
[   44.549762] irq_exit_rcu (kernel/softirq.c:433 kernel/softirq.c:637
kernel/softirq.c:649) 
[   44.551384] common_interrupt (arch/x86/kernel/irq.c:240
(discriminator 13)) 
[   44.551991] ? asm_common_interrupt
(./arch/x86/include/asm/idtentry.h:638) 
[   44.552654] asm_common_interrupt
(./arch/x86/include/asm/idtentry.h:638) 
[   44.553276] RIP: 0033:0x7fdb981a82e4
[ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 c4
41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 f9 6c
f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 f9
All code
========
   0:	d2 48 63             	rorb   %cl,0x63(%rax)
   3:	f6 c4 41             	test   $0x41,%ah
   6:	7a 6f                	jp     0x77
   8:	0c 01                	or     $0x1,%al
   a:	c4 41 7a 6f 14 09    	vmovdqu (%r9,%rcx,1),%xmm10
  10:	c4 41 7a 6f 24 11    	vmovdqu (%r9,%rdx,1),%xmm12
  16:	c4 41 7a 6f 2c 31    	vmovdqu (%r9,%rsi,1),%xmm13
  1c:	c4 c1 31 6a c2       	vpunpckhdq %xmm10,%xmm9,%xmm0
  21:	c4 c1 19 6a d5       	vpunpckhdq %xmm13,%xmm12,%xmm2
  26:	c5 f9 6c f2          	vpunpcklqdq %xmm2,%xmm0,%xmm6
  2a:*	c5 79 6d c2          	vpunpckhqdq %xmm2,%xmm0,%xmm8		
<-- trapping instruction
  2e:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
  33:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xmm0
  39:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
  3e:	c5                   	.byte 0xc5
  3f:	f9                   	stc    

Code starting with the faulting instruction
===========================================
   0:	c5 79 6d c2          	vpunpckhqdq %xmm2,%xmm0,%xmm8
   4:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
   9:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xmm0
   f:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
  14:	c5                   	.byte 0xc5
  15:	f9                   	stc    
[   44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
[   44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
0000000000122d40
[   44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
000055d7049b9368
[   44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
00007fdb5e544040
[   44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
0000000000000000
[   44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
0000000000000000
[   44.561965] Modules linked in:
[   44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
[   44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) 
[ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60
All code
========
   0:	4f 70 50             	rex.WRXB jo 0x53
   3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
   9:	50                   	push   %rax
   a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
  10:	50                   	push   %rax
  11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
  17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
  1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,%rdi
  25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
  2a:*	0f 0b                	ud2    		<-- trapping
instruction
  2c:	48 8b 14 24          	mov    (%rsp),%rdx
  30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
  37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
  3c:	48                   	rex.W
  3d:	c7                   	.byte 0xc7
  3e:	c6                   	(bad)  
  3f:	60                   	(bad)  

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	48 8b 14 24          	mov    (%rsp),%rdx
   6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
   d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
  12:	48                   	rex.W
  13:	c7                   	.byte 0xc7
  14:	c6                   	(bad)  
  15:	60                   	(bad)  
[   44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
[   44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
0000000000009ffb
[   44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff979ad2aa5600
[   44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
0000000000000eb2
[   44.571483] FS:  00007fdb9cb11700(0000) GS:ffff979aebd00000(0000)
knlGS:0000000000000000
[   44.572694] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
0000000000370ee0
[   44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   44.576618] Kernel panic - not syncing: Fatal exception in interrupt
[   44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-02  2:01       ` Xuan Zhuo
@ 2021-06-02 17:56         ` Corentin Noël
  0 siblings, 0 replies; 33+ messages in thread
From: Corentin Noël @ 2021-06-02 17:56 UTC (permalink / raw)
  To: Xuan Zhuo, Eric Dumazet
  Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions

Le mercredi 02 juin 2021 à 10:01 +0800, Xuan Zhuo a écrit :
> On Tue, 1 Jun 2021 19:47:44 +0200, Eric Dumazet <edumazet@google.com>
> wrote:
> > On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël
> > <corentin.noel@collabora.com> wrote:
> > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit :
> > > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote:
> > > > > I've been experiencing crashes with 5.13 that do not occur
> > > > > with
> > > > > 5.12,
> > > > > here is the crash trace:
> > > > > 
> > > > > [   47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354
> > > > > len:3762
> > > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010
> > > > > tail:0xec2
> > > > > end:0xec0 dev:<NULL>
> > > > > [   47.716267] kernel BUG at net/core/skbuff.c:110!
> > > > > [   47.717197] invalid opcode: 0000 [#1] SMP PTI
> > > > > [   47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted
> > > > > 5.13.0-
> > > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1
> > > > > [   47.719739] Hardware name: ChromiumOS crosvm, BIOS 0
> > > > > [   47.720656] RIP: 0010:skb_panic+0x43/0x45
> > > > > [   47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8
> > > > > 00 00
> > > > > 00 50
> > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7
> > > > > e8 7f
> > > > > 4c fb
> > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > > c7 c6
> > > > > e0
> > > > > [   47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > > [   47.726735] RAX: 000000000000008b RBX: 0000000000000010
> > > > > RCX:
> > > > > 00000000ffffdfff
> > > > > [   47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea
> > > > > RDI:
> > > > > 0000000000000000
> > > > > [   47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08
> > > > > R09:
> > > > > 0000000000009ffb
> > > > > [   47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff
> > > > > R12:
> > > > > ffff9e1e1e95b300
> > > > > [   47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000
> > > > > R15:
> > > > > 0000000000000eb2
> > > > > [   47.732541] FS:  00007f3a82b53700(0000)
> > > > > GS:ffff9e1f2bd00000(0000)
> > > > > knlGS:0000000000000000
> > > > > [   47.733858] CS:  0010 DS: 0000 ES: 0000 CR0:
> > > > > 0000000080050033
> > > > > [   47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004
> > > > > CR4:
> > > > > 0000000000370ee0
> > > > > [   47.735968] DR0: 0000000000000000 DR1: 0000000000000000
> > > > > DR2:
> > > > > 0000000000000000
> > > > > [   47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0
> > > > > DR7:
> > > > > 0000000000000400
> > > > > [   47.738318] Call Trace:
> > > > > [   47.738812]  skb_put.cold+0x10/0x10
> > > > > [   47.739450]  page_to_skb+0xe4/0x400
> > > > > [   47.740072]  receive_buf+0x86/0x1660
> > > > > [   47.740693]  ? inet_gro_receive+0x54/0x2c0
> > > > > [   47.741279]  ? dev_gro_receive+0x194/0x6a0
> > > > > [   47.741846]  virtnet_poll+0x2b8/0x3c0
> > > > > [   47.742357]  __napi_poll+0x25/0x150
> > > > > [   47.742844]  net_rx_action+0x22f/0x280
> > > > > [   47.743388]  __do_softirq+0xba/0x264
> > > > > [   47.743947]  irq_exit_rcu+0x90/0xb0
> > > > > [   47.744435]  common_interrupt+0x40/0xa0
> > > > > [   47.744978]  ? asm_common_interrupt+0x8/0x40
> > > > > [   47.745582]  asm_common_interrupt+0x1e/0x40
> > > > > [   47.746182] RIP: 0033:0x7f3a7a276ed4
> > > > > [   47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00
> > > > > c5 bc
> > > > > 54 c8
> > > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76
> > > > > c0 c5
> > > > > d5 fa
> > > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2
> > > > > 55 b8
> > > > > cb
> > > > > [   47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212
> > > > > [   47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff
> > > > > RCX:
> > > > > ffffffffffffffff
> > > > > [   47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c
> > > > > RDI:
> > > > > 00007f3a7c0575a0
> > > > > [   47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350
> > > > > R09:
> > > > > 00007f3a8c210354
> > > > > [   47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef
> > > > > R12:
> > > > > 00007f3a8c210340
> > > > > [   47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580
> > > > > R15:
> > > > > 00007f3a8c21033c
> > > > > [   47.755354] Modules linked in:
> > > > > [   47.755871] ---[ end trace a8b692ea99c9cd9e ]---
> > > > > [   47.756606] RIP: 0010:skb_panic+0x43/0x45
> > > > > [   47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8
> > > > > 00 00
> > > > > 00 50
> > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7
> > > > > e8 7f
> > > > > 4c fb
> > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48
> > > > > c7 c6
> > > > > e0
> > > > > [   47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246
> > > > > [   47.760896] RAX: 000000000000008b RBX: 0000000000000010
> > > > > RCX:
> > > > > 00000000ffffdfff
> > > > > [   47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea
> > > > > RDI:
> > > > > 0000000000000000
> > > > > [   47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08
> > > > > R09:
> > > > > 0000000000009ffb
> > > > > [   47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff
> > > > > R12:
> > > > > ffff9e1e1e95b300
> > > > > [   47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000
> > > > > R15:
> > > > > 0000000000000eb2
> > > > > [   47.766261] FS:  00007f3a82b53700(0000)
> > > > > GS:ffff9e1f2bd00000(0000)
> > > > > knlGS:0000000000000000
> > > > > [   47.767512] CS:  0010 DS: 0000 ES: 0000 CR0:
> > > > > 0000000080050033
> > > > > [   47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004
> > > > > CR4:
> > > > > 0000000000370ee0
> > > > > [   47.769381] DR0: 0000000000000000 DR1: 0000000000000000
> > > > > DR2:
> > > > > 0000000000000000
> > > > > [   47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0
> > > > > DR7:
> > > > > 0000000000000400
> > > > > [   47.771339] Kernel panic - not syncing: Fatal exception in
> > > > > interrupt
> > > > > [   47.772814] Kernel Offset: 0x35c00000 from
> > > > > 0xffffffff81000000
> > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > > > 
> > > > > I've been able to bisect the issue a little bit and the issue
> > > > > disappeared after reverting the 4 following commits:
> > > > >  * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7
> > > > >  * af39c8f72301b268ad8b04bae646b6025918b82b
> > > > >  * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6
> > > > >  * f80bd740cb7c954791279590b2e810ba6c214e52
> > > > > 
> > > > > Here is my kernel config:
> > > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config
> 
> Do you have XDP running? If so, you can try it
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5c37711d9f27bdc83fd5980446be7f4aa2106230

I applied this patch in top of 5.13-rc4 and it resulted in the same
crash

> 
> Thanks.
> 
> > > > Do you have the same problem with 5.13-rc4?
> > > > 
> > > > thanks,
> > > > 
> > > > greg k-h
> > > 
> > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic.
> > > 
> > > Thanks,
> > > 
> > 
> > Could you provide a stack trace with file names and line numbers ?
> > 
> > (ie use scripts/decode_stacktrace.sh )
> > 
> > Thanks.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-02 17:54       ` Corentin Noël
@ 2021-06-03  2:44         ` Xuan Zhuo
  2021-06-03  8:57           ` Corentin Noël
  0 siblings, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-06-03  2:44 UTC (permalink / raw)
  To: Corentin Noël
  Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet

On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> Sure, here is the decoded trace:
>
> [   44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 len:3762
> put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
> end:0xec0 dev:<NULL>
> [   44.525254] kernel BUG at net/core/skbuff.c:110!
> [   44.525910] invalid opcode: 0000 [#1] SMP PTI
> [   44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
> rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
> [   44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
> [   44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
>    0:	4f 70 50             	rex.WRXB jo 0x53
>    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
>    9:	50                   	push   %rax
>    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
>   10:	50                   	push   %rax
>   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
>   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
>   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,%rdi
>   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
>   2a:*	0f 0b                	ud2    		<-- trapping
> instruction
>   2c:	48 8b 14 24          	mov    (%rsp),%rdx
>   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
>   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
>   3c:	48                   	rex.W
>   3d:	c7                   	.byte 0xc7
>   3e:	c6                   	(bad)
>   3f:	60                   	(bad)
>
> Code starting with the faulting instruction
> ===========================================
>    0:	0f 0b                	ud2
>    2:	48 8b 14 24          	mov    (%rsp),%rdx
>    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
>    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
>   12:	48                   	rex.W
>   13:	c7                   	.byte 0xc7
>   14:	c6                   	(bad)
>   15:	60                   	(bad)
> [   44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> [   44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> 0000000000009ffb
> [   44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff979ad2aa5600
> [   44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> 0000000000000eb2
> [   44.539300] FS:  00007fdb9cb11700(0000) GS:ffff979aebd00000(0000)
> knlGS:0000000000000000
> [   44.540376] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> 0000000000370ee0
> [   44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   44.544063] Call Trace:
> [   44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator 1)
> net/core/skbuff.c:5252 (discriminator 1))
> [   44.544864] page_to_skb (drivers/net/virtio_net.c:485)
> [   44.545361] receive_buf (drivers/net/virtio_net.c:849
> drivers/net/virtio_net.c:1131)
> [   44.545870] ? netif_receive_skb_list_internal (net/core/dev.c:5714)
> [   44.546628] ? dev_gro_receive (net/core/dev.c:6103)
> [   44.547135] ? napi_complete_done (./include/linux/list.h:35
> net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565)
> [   44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
> drivers/net/virtio_net.c:1525)
> [   44.548251] __napi_poll (net/core/dev.c:6985)
> [   44.548744] net_rx_action (net/core/dev.c:7054 net/core/dev.c:7139)
> [   44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
> ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
> kernel/softirq.c:560)
> [   44.549762] irq_exit_rcu (kernel/softirq.c:433 kernel/softirq.c:637
> kernel/softirq.c:649)
> [   44.551384] common_interrupt (arch/x86/kernel/irq.c:240
> (discriminator 13))
> [   44.551991] ? asm_common_interrupt
> (./arch/x86/include/asm/idtentry.h:638)
> [   44.552654] asm_common_interrupt
> (./arch/x86/include/asm/idtentry.h:638)
> [   44.553276] RIP: 0033:0x7fdb981a82e4
> [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 c4
> 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 f9 6c
> f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 f9
> All code
> ========
>    0:	d2 48 63             	rorb   %cl,0x63(%rax)
>    3:	f6 c4 41             	test   $0x41,%ah
>    6:	7a 6f                	jp     0x77
>    8:	0c 01                	or     $0x1,%al
>    a:	c4 41 7a 6f 14 09    	vmovdqu (%r9,%rcx,1),%xmm10
>   10:	c4 41 7a 6f 24 11    	vmovdqu (%r9,%rdx,1),%xmm12
>   16:	c4 41 7a 6f 2c 31    	vmovdqu (%r9,%rsi,1),%xmm13
>   1c:	c4 c1 31 6a c2       	vpunpckhdq %xmm10,%xmm9,%xmm0
>   21:	c4 c1 19 6a d5       	vpunpckhdq %xmm13,%xmm12,%xmm2
>   26:	c5 f9 6c f2          	vpunpcklqdq %xmm2,%xmm0,%xmm6
>   2a:*	c5 79 6d c2          	vpunpckhqdq %xmm2,%xmm0,%xmm8
> <-- trapping instruction
>   2e:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
>   33:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xmm0
>   39:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
>   3e:	c5                   	.byte 0xc5
>   3f:	f9                   	stc
>
> Code starting with the faulting instruction
> ===========================================
>    0:	c5 79 6d c2          	vpunpckhqdq %xmm2,%xmm0,%xmm8
>    4:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
>    9:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xmm0
>    f:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
>   14:	c5                   	.byte 0xc5
>   15:	f9                   	stc
> [   44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
> [   44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
> 0000000000122d40
> [   44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
> 000055d7049b9368
> [   44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
> 00007fdb5e544040
> [   44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
> 0000000000000000
> [   44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
> 0000000000000000
> [   44.561965] Modules linked in:
> [   44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
> [   44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
>    0:	4f 70 50             	rex.WRXB jo 0x53
>    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
>    9:	50                   	push   %rax
>    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
>   10:	50                   	push   %rax
>   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
>   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
>   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,%rdi
>   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
>   2a:*	0f 0b                	ud2    		<-- trapping
> instruction
>   2c:	48 8b 14 24          	mov    (%rsp),%rdx
>   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
>   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
>   3c:	48                   	rex.W
>   3d:	c7                   	.byte 0xc7
>   3e:	c6                   	(bad)
>   3f:	60                   	(bad)
>
> Code starting with the faulting instruction
> ===========================================
>    0:	0f 0b                	ud2
>    2:	48 8b 14 24          	mov    (%rsp),%rdx
>    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,%rcx
>    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
>   12:	48                   	rex.W
>   13:	c7                   	.byte 0xc7
>   14:	c6                   	(bad)
>   15:	60                   	(bad)
> [   44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> [   44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> 0000000000009ffb
> [   44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff979ad2aa5600
> [   44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> 0000000000000eb2
> [   44.571483] FS:  00007fdb9cb11700(0000) GS:ffff979aebd00000(0000)
> knlGS:0000000000000000
> [   44.572694] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> 0000000000370ee0
> [   44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   44.576618] Kernel panic - not syncing: Fatal exception in interrupt
> [   44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>

Can you test this patch on the latest net branch?

Thanks.

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index fa407eb8b457..78a01c71a17c 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
         * add_recvbuf_mergeable() + get_mergeable_buf_len()
         */
        truesize = headroom ? PAGE_SIZE : truesize;
-       tailroom = truesize - len - headroom;
+       tailroom = truesize - len - headroom - (hdr_padded_len - hdr_len);
        buf = p - headroom;

        len -= hdr_len;

^ permalink raw reply related	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-03  2:44         ` Xuan Zhuo
@ 2021-06-03  8:57           ` Corentin Noël
  2021-06-08 12:17             ` Greg KH
  0 siblings, 1 reply; 33+ messages in thread
From: Corentin Noël @ 2021-06-03  8:57 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet

Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit :
> On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël <
> corentin.noel@collabora.com> wrote:
> > Sure, here is the decoded trace:
> > 
> > [   44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434
> > len:3762
> > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [   44.525254] kernel BUG at net/core/skbuff.c:110!
> > [   44.525910] invalid opcode: 0000 [#1] SMP PTI
> > [   44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
> > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
> > [   44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
> > [   44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > 60
> > All code
> > ========
> >    0:	4f 70 50             	rex.WRXB jo 0x53
> >    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
> >    9:	50                   	push   %rax
> >    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
> >   10:	50                   	push   %rax
> >   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
> >   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
> >   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,
> > %rdi
> >   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
> >   2a:*	0f 0b                	ud2    		<--
> > trapping
> > instruction
> >   2c:	48 8b 14 24          	mov    (%rsp),%rdx
> >   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > %rcx
> >   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
> >   3c:	48                   	rex.W
> >   3d:	c7                   	.byte 0xc7
> >   3e:	c6                   	(bad)
> >   3f:	60                   	(bad)
> > 
> > Code starting with the faulting instruction
> > ===========================================
> >    0:	0f 0b                	ud2
> >    2:	48 8b 14 24          	mov    (%rsp),%rdx
> >    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > %rcx
> >    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
> >   12:	48                   	rex.W
> >   13:	c7                   	.byte 0xc7
> >   14:	c6                   	(bad)
> >   15:	60                   	(bad)
> > [   44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > [   44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > 0000000000009ffb
> > [   44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff979ad2aa5600
> > [   44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > 0000000000000eb2
> > [   44.539300] FS:  00007fdb9cb11700(0000)
> > GS:ffff979aebd00000(0000)
> > knlGS:0000000000000000
> > [   44.540376] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > 0000000000370ee0
> > [   44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   44.544063] Call Trace:
> > [   44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator
> > 1)
> > net/core/skbuff.c:5252 (discriminator 1))
> > [   44.544864] page_to_skb (drivers/net/virtio_net.c:485)
> > [   44.545361] receive_buf (drivers/net/virtio_net.c:849
> > drivers/net/virtio_net.c:1131)
> > [   44.545870] ? netif_receive_skb_list_internal
> > (net/core/dev.c:5714)
> > [   44.546628] ? dev_gro_receive (net/core/dev.c:6103)
> > [   44.547135] ? napi_complete_done (./include/linux/list.h:35
> > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565)
> > [   44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
> > drivers/net/virtio_net.c:1525)
> > [   44.548251] __napi_poll (net/core/dev.c:6985)
> > [   44.548744] net_rx_action (net/core/dev.c:7054
> > net/core/dev.c:7139)
> > [   44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
> > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
> > kernel/softirq.c:560)
> > [   44.549762] irq_exit_rcu (kernel/softirq.c:433
> > kernel/softirq.c:637
> > kernel/softirq.c:649)
> > [   44.551384] common_interrupt (arch/x86/kernel/irq.c:240
> > (discriminator 13))
> > [   44.551991] ? asm_common_interrupt
> > (./arch/x86/include/asm/idtentry.h:638)
> > [   44.552654] asm_common_interrupt
> > (./arch/x86/include/asm/idtentry.h:638)
> > [   44.553276] RIP: 0033:0x7fdb981a82e4
> > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09
> > c4
> > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5
> > f9 6c
> > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5
> > f9
> > All code
> > ========
> >    0:	d2 48 63             	rorb   %cl,0x63(%rax)
> >    3:	f6 c4 41             	test   $0x41,%ah
> >    6:	7a 6f                	jp     0x77
> >    8:	0c 01                	or     $0x1,%al
> >    a:	c4 41 7a 6f 14 09    	vmovdqu (%r9,%rcx,1),%xmm10
> >   10:	c4 41 7a 6f 24 11    	vmovdqu (%r9,%rdx,1),%xmm12
> >   16:	c4 41 7a 6f 2c 31    	vmovdqu (%r9,%rsi,1),%xmm13
> >   1c:	c4 c1 31 6a c2       	vpunpckhdq
> > %xmm10,%xmm9,%xmm0
> >   21:	c4 c1 19 6a d5       	vpunpckhdq
> > %xmm13,%xmm12,%xmm2
> >   26:	c5 f9 6c f2          	vpunpcklqdq
> > %xmm2,%xmm0,%xmm6
> >   2a:*	c5 79 6d c2          	vpunpckhqdq
> > %xmm2,%xmm0,%xmm8
> > <-- trapping instruction
> >   2e:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
> >   33:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xm
> > m0
> >   39:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
> >   3e:	c5                   	.byte 0xc5
> >   3f:	f9                   	stc
> > 
> > Code starting with the faulting instruction
> > ===========================================
> >    0:	c5 79 6d c2          	vpunpckhqdq
> > %xmm2,%xmm0,%xmm8
> >    4:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
> >    9:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xm
> > m0
> >    f:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
> >   14:	c5                   	.byte 0xc5
> >   15:	f9                   	stc
> > [   44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
> > [   44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
> > 0000000000122d40
> > [   44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
> > 000055d7049b9368
> > [   44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
> > 00007fdb5e544040
> > [   44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
> > 0000000000000000
> > [   44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
> > 0000000000000000
> > [   44.561965] Modules linked in:
> > [   44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
> > [   44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > 4c fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > 60
> > All code
> > ========
> >    0:	4f 70 50             	rex.WRXB jo 0x53
> >    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
> >    9:	50                   	push   %rax
> >    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
> >   10:	50                   	push   %rax
> >   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
> >   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
> >   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,
> > %rdi
> >   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
> >   2a:*	0f 0b                	ud2    		<--
> > trapping
> > instruction
> >   2c:	48 8b 14 24          	mov    (%rsp),%rdx
> >   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > %rcx
> >   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
> >   3c:	48                   	rex.W
> >   3d:	c7                   	.byte 0xc7
> >   3e:	c6                   	(bad)
> >   3f:	60                   	(bad)
> > 
> > Code starting with the faulting instruction
> > ===========================================
> >    0:	0f 0b                	ud2
> >    2:	48 8b 14 24          	mov    (%rsp),%rdx
> >    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > %rcx
> >    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
> >   12:	48                   	rex.W
> >   13:	c7                   	.byte 0xc7
> >   14:	c6                   	(bad)
> >   15:	60                   	(bad)
> > [   44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > [   44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > 0000000000009ffb
> > [   44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff979ad2aa5600
> > [   44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > 0000000000000eb2
> > [   44.571483] FS:  00007fdb9cb11700(0000)
> > GS:ffff979aebd00000(0000)
> > knlGS:0000000000000000
> > [   44.572694] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > 0000000000370ee0
> > [   44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   44.576618] Kernel panic - not syncing: Fatal exception in
> > interrupt
> > [   44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
> > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > 
> 
> Can you test this patch on the latest net branch?
> 
> Thanks.
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index fa407eb8b457..78a01c71a17c 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct
> virtnet_info *vi,
>          * add_recvbuf_mergeable() + get_mergeable_buf_len()
>          */
>         truesize = headroom ? PAGE_SIZE : truesize;
> -       tailroom = truesize - len - headroom;
> +       tailroom = truesize - len - headroom - (hdr_padded_len -
> hdr_len);
>         buf = p - headroom;
> 
>         len -= hdr_len;

With this patch and the latest net branch I no longer get crashes.

Thanks a lot for this,
Corentin


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-03  8:57           ` Corentin Noël
@ 2021-06-08 12:17             ` Greg KH
  2021-06-09  1:48               ` Xuan Zhuo
  0 siblings, 1 reply; 33+ messages in thread
From: Greg KH @ 2021-06-08 12:17 UTC (permalink / raw)
  To: Corentin Noël
  Cc: Xuan Zhuo, stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet

On Thu, Jun 03, 2021 at 10:57:52AM +0200, Corentin Noël wrote:
> Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit :
> > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël <
> > corentin.noel@collabora.com> wrote:
> > > Sure, here is the decoded trace:
> > > 
> > > [   44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434
> > > len:3762
> > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [   44.525254] kernel BUG at net/core/skbuff.c:110!
> > > [   44.525910] invalid opcode: 0000 [#1] SMP PTI
> > > [   44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
> > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
> > > [   44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [   44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > > 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > > 60
> > > All code
> > > ========
> > >    0:	4f 70 50             	rex.WRXB jo 0x53
> > >    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
> > >    9:	50                   	push   %rax
> > >    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
> > >   10:	50                   	push   %rax
> > >   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
> > >   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
> > >   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,
> > > %rdi
> > >   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
> > >   2a:*	0f 0b                	ud2    		<--
> > > trapping
> > > instruction
> > >   2c:	48 8b 14 24          	mov    (%rsp),%rdx
> > >   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > %rcx
> > >   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
> > >   3c:	48                   	rex.W
> > >   3d:	c7                   	.byte 0xc7
> > >   3e:	c6                   	(bad)
> > >   3f:	60                   	(bad)
> > > 
> > > Code starting with the faulting instruction
> > > ===========================================
> > >    0:	0f 0b                	ud2
> > >    2:	48 8b 14 24          	mov    (%rsp),%rdx
> > >    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > %rcx
> > >    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
> > >   12:	48                   	rex.W
> > >   13:	c7                   	.byte 0xc7
> > >   14:	c6                   	(bad)
> > >   15:	60                   	(bad)
> > > [   44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > > [   44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > > 0000000000009ffb
> > > [   44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff979ad2aa5600
> > > [   44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > > 0000000000000eb2
> > > [   44.539300] FS:  00007fdb9cb11700(0000)
> > > GS:ffff979aebd00000(0000)
> > > knlGS:0000000000000000
> > > [   44.540376] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > > 0000000000370ee0
> > > [   44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   44.544063] Call Trace:
> > > [   44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator
> > > 1)
> > > net/core/skbuff.c:5252 (discriminator 1))
> > > [   44.544864] page_to_skb (drivers/net/virtio_net.c:485)
> > > [   44.545361] receive_buf (drivers/net/virtio_net.c:849
> > > drivers/net/virtio_net.c:1131)
> > > [   44.545870] ? netif_receive_skb_list_internal
> > > (net/core/dev.c:5714)
> > > [   44.546628] ? dev_gro_receive (net/core/dev.c:6103)
> > > [   44.547135] ? napi_complete_done (./include/linux/list.h:35
> > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565)
> > > [   44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
> > > drivers/net/virtio_net.c:1525)
> > > [   44.548251] __napi_poll (net/core/dev.c:6985)
> > > [   44.548744] net_rx_action (net/core/dev.c:7054
> > > net/core/dev.c:7139)
> > > [   44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
> > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
> > > kernel/softirq.c:560)
> > > [   44.549762] irq_exit_rcu (kernel/softirq.c:433
> > > kernel/softirq.c:637
> > > kernel/softirq.c:649)
> > > [   44.551384] common_interrupt (arch/x86/kernel/irq.c:240
> > > (discriminator 13))
> > > [   44.551991] ? asm_common_interrupt
> > > (./arch/x86/include/asm/idtentry.h:638)
> > > [   44.552654] asm_common_interrupt
> > > (./arch/x86/include/asm/idtentry.h:638)
> > > [   44.553276] RIP: 0033:0x7fdb981a82e4
> > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09
> > > c4
> > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5
> > > f9 6c
> > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5
> > > f9
> > > All code
> > > ========
> > >    0:	d2 48 63             	rorb   %cl,0x63(%rax)
> > >    3:	f6 c4 41             	test   $0x41,%ah
> > >    6:	7a 6f                	jp     0x77
> > >    8:	0c 01                	or     $0x1,%al
> > >    a:	c4 41 7a 6f 14 09    	vmovdqu (%r9,%rcx,1),%xmm10
> > >   10:	c4 41 7a 6f 24 11    	vmovdqu (%r9,%rdx,1),%xmm12
> > >   16:	c4 41 7a 6f 2c 31    	vmovdqu (%r9,%rsi,1),%xmm13
> > >   1c:	c4 c1 31 6a c2       	vpunpckhdq
> > > %xmm10,%xmm9,%xmm0
> > >   21:	c4 c1 19 6a d5       	vpunpckhdq
> > > %xmm13,%xmm12,%xmm2
> > >   26:	c5 f9 6c f2          	vpunpcklqdq
> > > %xmm2,%xmm0,%xmm6
> > >   2a:*	c5 79 6d c2          	vpunpckhqdq
> > > %xmm2,%xmm0,%xmm8
> > > <-- trapping instruction
> > >   2e:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
> > >   33:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xm
> > > m0
> > >   39:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
> > >   3e:	c5                   	.byte 0xc5
> > >   3f:	f9                   	stc
> > > 
> > > Code starting with the faulting instruction
> > > ===========================================
> > >    0:	c5 79 6d c2          	vpunpckhqdq
> > > %xmm2,%xmm0,%xmm8
> > >    4:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
> > >    9:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xm
> > > m0
> > >    f:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
> > >   14:	c5                   	.byte 0xc5
> > >   15:	f9                   	stc
> > > [   44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
> > > [   44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
> > > 0000000000122d40
> > > [   44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
> > > 000055d7049b9368
> > > [   44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
> > > 00007fdb5e544040
> > > [   44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
> > > 0000000000000000
> > > [   44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
> > > 0000000000000000
> > > [   44.561965] Modules linked in:
> > > [   44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
> > > [   44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > > 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > > 4c fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > > 60
> > > All code
> > > ========
> > >    0:	4f 70 50             	rex.WRXB jo 0x53
> > >    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
> > >    9:	50                   	push   %rax
> > >    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
> > >   10:	50                   	push   %rax
> > >   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
> > >   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
> > >   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,
> > > %rdi
> > >   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
> > >   2a:*	0f 0b                	ud2    		<--
> > > trapping
> > > instruction
> > >   2c:	48 8b 14 24          	mov    (%rsp),%rdx
> > >   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > %rcx
> > >   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
> > >   3c:	48                   	rex.W
> > >   3d:	c7                   	.byte 0xc7
> > >   3e:	c6                   	(bad)
> > >   3f:	60                   	(bad)
> > > 
> > > Code starting with the faulting instruction
> > > ===========================================
> > >    0:	0f 0b                	ud2
> > >    2:	48 8b 14 24          	mov    (%rsp),%rdx
> > >    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > %rcx
> > >    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
> > >   12:	48                   	rex.W
> > >   13:	c7                   	.byte 0xc7
> > >   14:	c6                   	(bad)
> > >   15:	60                   	(bad)
> > > [   44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > > [   44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > > 0000000000009ffb
> > > [   44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff979ad2aa5600
> > > [   44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > > 0000000000000eb2
> > > [   44.571483] FS:  00007fdb9cb11700(0000)
> > > GS:ffff979aebd00000(0000)
> > > knlGS:0000000000000000
> > > [   44.572694] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > > 0000000000370ee0
> > > [   44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   44.576618] Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > > [   44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > 
> > 
> > Can you test this patch on the latest net branch?
> > 
> > Thanks.
> > 
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index fa407eb8b457..78a01c71a17c 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct
> > virtnet_info *vi,
> >          * add_recvbuf_mergeable() + get_mergeable_buf_len()
> >          */
> >         truesize = headroom ? PAGE_SIZE : truesize;
> > -       tailroom = truesize - len - headroom;
> > +       tailroom = truesize - len - headroom - (hdr_padded_len -
> > hdr_len);
> >         buf = p - headroom;
> > 
> >         len -= hdr_len;
> 
> With this patch and the latest net branch I no longer get crashes.

Did this ever get properly submitted to the networking tree to get into
5.13-final?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-08 12:17             ` Greg KH
@ 2021-06-09  1:48               ` Xuan Zhuo
  2021-06-09  4:50                 ` Greg KH
  0 siblings, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-06-09  1:48 UTC (permalink / raw)
  To: Greg KH
  Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Corentin Noël

On Tue, 8 Jun 2021 14:17:58 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Jun 03, 2021 at 10:57:52AM +0200, Corentin Noël wrote:
> > Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit :
> > > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël <
> > > corentin.noel@collabora.com> wrote:
> > > > Sure, here is the decoded trace:
> > > >
> > > > [   44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434
> > > > len:3762
> > > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2
> > > > end:0xec0 dev:<NULL>
> > > > [   44.525254] kernel BUG at net/core/skbuff.c:110!
> > > > [   44.525910] invalid opcode: 0000 [#1] SMP PTI
> > > > [   44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0-
> > > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1
> > > > [   44.528109] Hardware name: ChromiumOS crosvm, BIOS 0
> > > > [   44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > > > 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > > > 60
> > > > All code
> > > > ========
> > > >    0:	4f 70 50             	rex.WRXB jo 0x53
> > > >    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
> > > >    9:	50                   	push   %rax
> > > >    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
> > > >   10:	50                   	push   %rax
> > > >   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
> > > >   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
> > > >   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,
> > > > %rdi
> > > >   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
> > > >   2a:*	0f 0b                	ud2    		<--
> > > > trapping
> > > > instruction
> > > >   2c:	48 8b 14 24          	mov    (%rsp),%rdx
> > > >   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > > %rcx
> > > >   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
> > > >   3c:	48                   	rex.W
> > > >   3d:	c7                   	.byte 0xc7
> > > >   3e:	c6                   	(bad)
> > > >   3f:	60                   	(bad)
> > > >
> > > > Code starting with the faulting instruction
> > > > ===========================================
> > > >    0:	0f 0b                	ud2
> > > >    2:	48 8b 14 24          	mov    (%rsp),%rdx
> > > >    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > > %rcx
> > > >    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
> > > >   12:	48                   	rex.W
> > > >   13:	c7                   	.byte 0xc7
> > > >   14:	c6                   	(bad)
> > > >   15:	60                   	(bad)
> > > > [   44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > > > [   44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [   44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [   44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > > > 0000000000009ffb
> > > > [   44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff979ad2aa5600
> > > > [   44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > > > 0000000000000eb2
> > > > [   44.539300] FS:  00007fdb9cb11700(0000)
> > > > GS:ffff979aebd00000(0000)
> > > > knlGS:0000000000000000
> > > > [   44.540376] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > > > 0000000000370ee0
> > > > [   44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [   44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [   44.544063] Call Trace:
> > > > [   44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator
> > > > 1)
> > > > net/core/skbuff.c:5252 (discriminator 1))
> > > > [   44.544864] page_to_skb (drivers/net/virtio_net.c:485)
> > > > [   44.545361] receive_buf (drivers/net/virtio_net.c:849
> > > > drivers/net/virtio_net.c:1131)
> > > > [   44.545870] ? netif_receive_skb_list_internal
> > > > (net/core/dev.c:5714)
> > > > [   44.546628] ? dev_gro_receive (net/core/dev.c:6103)
> > > > [   44.547135] ? napi_complete_done (./include/linux/list.h:35
> > > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565)
> > > > [   44.547672] virtnet_poll (drivers/net/virtio_net.c:1427
> > > > drivers/net/virtio_net.c:1525)
> > > > [   44.548251] __napi_poll (net/core/dev.c:6985)
> > > > [   44.548744] net_rx_action (net/core/dev.c:7054
> > > > net/core/dev.c:7139)
> > > > [   44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19
> > > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142
> > > > kernel/softirq.c:560)
> > > > [   44.549762] irq_exit_rcu (kernel/softirq.c:433
> > > > kernel/softirq.c:637
> > > > kernel/softirq.c:649)
> > > > [   44.551384] common_interrupt (arch/x86/kernel/irq.c:240
> > > > (discriminator 13))
> > > > [   44.551991] ? asm_common_interrupt
> > > > (./arch/x86/include/asm/idtentry.h:638)
> > > > [   44.552654] asm_common_interrupt
> > > > (./arch/x86/include/asm/idtentry.h:638)
> > > > [   44.553276] RIP: 0033:0x7fdb981a82e4
> > > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09
> > > > c4
> > > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5
> > > > f9 6c
> > > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5
> > > > f9
> > > > All code
> > > > ========
> > > >    0:	d2 48 63             	rorb   %cl,0x63(%rax)
> > > >    3:	f6 c4 41             	test   $0x41,%ah
> > > >    6:	7a 6f                	jp     0x77
> > > >    8:	0c 01                	or     $0x1,%al
> > > >    a:	c4 41 7a 6f 14 09    	vmovdqu (%r9,%rcx,1),%xmm10
> > > >   10:	c4 41 7a 6f 24 11    	vmovdqu (%r9,%rdx,1),%xmm12
> > > >   16:	c4 41 7a 6f 2c 31    	vmovdqu (%r9,%rsi,1),%xmm13
> > > >   1c:	c4 c1 31 6a c2       	vpunpckhdq
> > > > %xmm10,%xmm9,%xmm0
> > > >   21:	c4 c1 19 6a d5       	vpunpckhdq
> > > > %xmm13,%xmm12,%xmm2
> > > >   26:	c5 f9 6c f2          	vpunpcklqdq
> > > > %xmm2,%xmm0,%xmm6
> > > >   2a:*	c5 79 6d c2          	vpunpckhqdq
> > > > %xmm2,%xmm0,%xmm8
> > > > <-- trapping instruction
> > > >   2e:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
> > > >   33:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xm
> > > > m0
> > > >   39:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
> > > >   3e:	c5                   	.byte 0xc5
> > > >   3f:	f9                   	stc
> > > >
> > > > Code starting with the faulting instruction
> > > > ===========================================
> > > >    0:	c5 79 6d c2          	vpunpckhqdq
> > > > %xmm2,%xmm0,%xmm8
> > > >    4:	c5 f9 71 d6 08       	vpsrlw $0x8,%xmm6,%xmm0
> > > >    9:	c5 f9 db 44 24 20    	vpand  0x20(%rsp),%xmm0,%xm
> > > > m0
> > > >    f:	c5 c1 71 f6 0b       	vpsllw $0xb,%xmm6,%xmm7
> > > >   14:	c5                   	.byte 0xc5
> > > >   15:	f9                   	stc
> > > > [   44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202
> > > > [   44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX:
> > > > 0000000000122d40
> > > > [   44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI:
> > > > 000055d7049b9368
> > > > [   44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09:
> > > > 00007fdb5e544040
> > > > [   44.560042] R10: 000000000000ffff R11: 000000000000ffff R12:
> > > > 0000000000000000
> > > > [   44.560991] R13: 0000000000000000 R14: 0000000000005000 R15:
> > > > 0000000000000000
> > > > [   44.561965] Modules linked in:
> > > > [   44.562426] ---[ end trace 9a32eb9d31cb21a1 ]---
> > > > [   44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110)
> > > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00
> > > > 50
> > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43
> > > > 4c fb
> > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6
> > > > 60
> > > > All code
> > > > ========
> > > >    0:	4f 70 50             	rex.WRXB jo 0x53
> > > >    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
> > > >    9:	50                   	push   %rax
> > > >    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
> > > >   10:	50                   	push   %rax
> > > >   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
> > > >   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
> > > >   1e:	48 c7 c7 f0 af cf ad 	mov    $0xffffffffadcfaff0,
> > > > %rdi
> > > >   25:	e8 43 4c fb ff       	callq  0xfffffffffffb4c6d
> > > >   2a:*	0f 0b                	ud2    		<--
> > > > trapping
> > > > instruction
> > > >   2c:	48 8b 14 24          	mov    (%rsp),%rdx
> > > >   30:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > > %rcx
> > > >   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
> > > >   3c:	48                   	rex.W
> > > >   3d:	c7                   	.byte 0xc7
> > > >   3e:	c6                   	(bad)
> > > >   3f:	60                   	(bad)
> > > >
> > > > Code starting with the faulting instruction
> > > > ===========================================
> > > >    0:	0f 0b                	ud2
> > > >    2:	48 8b 14 24          	mov    (%rsp),%rdx
> > > >    6:	48 c7 c1 20 23 b1 ad 	mov    $0xffffffffadb12320,
> > > > %rcx
> > > >    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
> > > >   12:	48                   	rex.W
> > > >   13:	c7                   	.byte 0xc7
> > > >   14:	c6                   	(bad)
> > > >   15:	60                   	(bad)
> > > > [   44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246
> > > > [   44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > > 00000000ffffdfff
> > > > [   44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > > 0000000000000000
> > > > [   44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09:
> > > > 0000000000009ffb
> > > > [   44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > > ffff979ad2aa5600
> > > > [   44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15:
> > > > 0000000000000eb2
> > > > [   44.571483] FS:  00007fdb9cb11700(0000)
> > > > GS:ffff979aebd00000(0000)
> > > > knlGS:0000000000000000
> > > > [   44.572694] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4:
> > > > 0000000000370ee0
> > > > [   44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > > 0000000000000000
> > > > [   44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > > 0000000000000400
> > > > [   44.576618] Kernel panic - not syncing: Fatal exception in
> > > > interrupt
> > > > [   44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000
> > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > >
> > >
> > > Can you test this patch on the latest net branch?
> > >
> > > Thanks.
> > >
> > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > index fa407eb8b457..78a01c71a17c 100644
> > > --- a/drivers/net/virtio_net.c
> > > +++ b/drivers/net/virtio_net.c
> > > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct
> > > virtnet_info *vi,
> > >          * add_recvbuf_mergeable() + get_mergeable_buf_len()
> > >          */
> > >         truesize = headroom ? PAGE_SIZE : truesize;
> > > -       tailroom = truesize - len - headroom;
> > > +       tailroom = truesize - len - headroom - (hdr_padded_len -
> > > hdr_len);
> > >         buf = p - headroom;
> > >
> > >         len -= hdr_len;
> >
> > With this patch and the latest net branch I no longer get crashes.
>
> Did this ever get properly submitted to the networking tree to get into
> 5.13-final?

The patch has been submitted.

	[PATCH net] virtio-net: fix for skb_over_panic inside big mode

Thanks.


>
> thanks,
>
> greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-09  1:48               ` Xuan Zhuo
@ 2021-06-09  4:50                 ` Greg KH
  2021-06-09  6:08                   ` Xuan Zhuo
  0 siblings, 1 reply; 33+ messages in thread
From: Greg KH @ 2021-06-09  4:50 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Corentin Noël

On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > With this patch and the latest net branch I no longer get crashes.
> >
> > Did this ever get properly submitted to the networking tree to get into
> > 5.13-final?
> 
> The patch has been submitted.
> 
> 	[PATCH net] virtio-net: fix for skb_over_panic inside big mode

Submitted where?  Do you have a lore.kernel.org link somewhere?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-09  4:50                 ` Greg KH
@ 2021-06-09  6:08                   ` Xuan Zhuo
  2021-06-09  6:24                     ` Greg KH
  0 siblings, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-06-09  6:08 UTC (permalink / raw)
  To: Greg KH
  Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Corentin Noël

On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > > With this patch and the latest net branch I no longer get crashes.
> > >
> > > Did this ever get properly submitted to the networking tree to get into
> > > 5.13-final?
> >
> > The patch has been submitted.
> >
> > 	[PATCH net] virtio-net: fix for skb_over_panic inside big mode
>
> Submitted where?  Do you have a lore.kernel.org link somewhere?


https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/

Thanks.

>
> thanks,
>
> greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-09  6:08                   ` Xuan Zhuo
@ 2021-06-09  6:24                     ` Greg KH
  2021-06-09  7:51                       ` Xuan Zhuo
  0 siblings, 1 reply; 33+ messages in thread
From: Greg KH @ 2021-06-09  6:24 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Corentin Noël

On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote:
> On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > > > With this patch and the latest net branch I no longer get crashes.
> > > >
> > > > Did this ever get properly submitted to the networking tree to get into
> > > > 5.13-final?
> > >
> > > The patch has been submitted.
> > >
> > > 	[PATCH net] virtio-net: fix for skb_over_panic inside big mode
> >
> > Submitted where?  Do you have a lore.kernel.org link somewhere?
> 
> 
> https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/

So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic
inside big mode") in Linus's tree, right?

But why is that referencing:
	Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom")

when this problem was seen in stable kernels that had a different commit
backported to it?

Is there nothing needed to be done for the stable kernel trees?

confused,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-09  6:24                     ` Greg KH
@ 2021-06-09  7:51                       ` Xuan Zhuo
  2021-06-09  8:03                         ` Greg KH
  0 siblings, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-06-09  7:51 UTC (permalink / raw)
  To: Greg KH
  Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Corentin Noël

On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote:
> > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > > > > With this patch and the latest net branch I no longer get crashes.
> > > > >
> > > > > Did this ever get properly submitted to the networking tree to get into
> > > > > 5.13-final?
> > > >
> > > > The patch has been submitted.
> > > >
> > > > 	[PATCH net] virtio-net: fix for skb_over_panic inside big mode
> > >
> > > Submitted where?  Do you have a lore.kernel.org link somewhere?
> >
> >
> > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/
>
> So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic
> inside big mode") in Linus's tree, right?

YES.

>
> But why is that referencing:
> 	Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom")

This problem was indeed introduced in fb32856b16ad.

I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the
previous 5.12 did not have this commit fb32856b16ad.

I'm not sure if it helped you.

Thanks.

>
> when this problem was seen in stable kernels that had a different commit
> backported to it?
>
> Is there nothing needed to be done for the stable kernel trees?
>
> confused,
>
> greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-09  7:51                       ` Xuan Zhuo
@ 2021-06-09  8:03                         ` Greg KH
  2021-06-09  8:08                           ` Xuan Zhuo
  0 siblings, 1 reply; 33+ messages in thread
From: Greg KH @ 2021-06-09  8:03 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Corentin Noël

On Wed, Jun 09, 2021 at 03:51:20PM +0800, Xuan Zhuo wrote:
> On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote:
> > > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > > > > > With this patch and the latest net branch I no longer get crashes.
> > > > > >
> > > > > > Did this ever get properly submitted to the networking tree to get into
> > > > > > 5.13-final?
> > > > >
> > > > > The patch has been submitted.
> > > > >
> > > > > 	[PATCH net] virtio-net: fix for skb_over_panic inside big mode
> > > >
> > > > Submitted where?  Do you have a lore.kernel.org link somewhere?
> > >
> > >
> > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/
> >
> > So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic
> > inside big mode") in Linus's tree, right?
> 
> YES.
> 
> >
> > But why is that referencing:
> > 	Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom")
> 
> This problem was indeed introduced in fb32856b16ad.
> 
> I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the
> previous 5.12 did not have this commit fb32856b16ad.
> 
> I'm not sure if it helped you.

Hm, then what resolves the reported problem that people were having with
the 5.12.y kernel release?  Is that a separate issue?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-06-09  8:03                         ` Greg KH
@ 2021-06-09  8:08                           ` Xuan Zhuo
  0 siblings, 0 replies; 33+ messages in thread
From: Xuan Zhuo @ 2021-06-09  8:08 UTC (permalink / raw)
  To: Greg KH
  Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization,
	regressions, Eric Dumazet, Corentin Noël

On Wed, 9 Jun 2021 10:03:53 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Wed, Jun 09, 2021 at 03:51:20PM +0800, Xuan Zhuo wrote:
> > On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote:
> > > > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote:
> > > > > > > > With this patch and the latest net branch I no longer get crashes.
> > > > > > >
> > > > > > > Did this ever get properly submitted to the networking tree to get into
> > > > > > > 5.13-final?
> > > > > >
> > > > > > The patch has been submitted.
> > > > > >
> > > > > > 	[PATCH net] virtio-net: fix for skb_over_panic inside big mode
> > > > >
> > > > > Submitted where?  Do you have a lore.kernel.org link somewhere?
> > > >
> > > >
> > > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/
> > >
> > > So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic
> > > inside big mode") in Linus's tree, right?
> >
> > YES.
> >
> > >
> > > But why is that referencing:
> > > 	Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom")
> >
> > This problem was indeed introduced in fb32856b16ad.
> >
> > I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the
> > previous 5.12 did not have this commit fb32856b16ad.
> >
> > I'm not sure if it helped you.
>
> Hm, then what resolves the reported problem that people were having with
> the 5.12.y kernel release?  Is that a separate issue?

Has anyone reported a problem with 5.12.y? I don’t seem to see it. Corentin
only reported a problem with 5.13? Did I miss something?

I confirm that 5.12.9 has no modification of fb32856b16ad.

Thanks.

>
> thanks,
>
> greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-08  7:59 ` Thorsten Leemhuis
@ 2021-10-12 12:44   ` Thorsten Leemhuis
  0 siblings, 0 replies; 33+ messages in thread
From: Thorsten Leemhuis @ 2021-10-12 12:44 UTC (permalink / raw)
  To: regressions



On 08.10.21 09:59, Thorsten Leemhuis wrote:
> On 07.10.21 14:04, Corentin Noël wrote:
>> I've been experiencing crashes with 5.14-rc1 and above that do not
>> occur with 5.13,
> 
> Feel free to ignore this message. I write it to make regzbot track above
> issue. Regzbot is the regression tracking bot I'm working on. It's still
> in the early stages and this is still one of the first few regression I
> make it track to get started and things tested in the field. That also
> why I'm sending the mail just to the regressions list (it will do its
> full magic nevertheless). For details see:
> https://linux-regtracking.leemhuis.info/post/inital-regzbot-running/
> https://linux-regtracking.leemhuis.info/post/regzbot-approach/
> 
> #regzbot ^introduced v5.13..v5.14-rc1
> #regzbot link
> https://lore.kernel.org/regressions/1633623446.6192446-1-xuanzhuo@linux.alibaba.com/
> Looks like a fix for this somehow fall through the cracks
> #regzbot ignore-activitiy
> 
> BTW, in case anyone is reading this: the regzbot's webinterface that
> gives insights into what is tracked can be found here:
> 
> https://linux-regtracking.leemhuis.info/regzbot/mainline.html
> 
> Ciao, Thorsten

#regzbot fixed-by: 732b74d647048668f0f8dc0c848f0746c69e2e2f


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-09  5:19           ` Greg KH
@ 2021-10-09  9:31             ` Xuan Zhuo
  0 siblings, 0 replies; 33+ messages in thread
From: Xuan Zhuo @ 2021-10-09  9:31 UTC (permalink / raw)
  To: Greg KH
  Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
	virtualization, regressions, Eric Dumazet, stable

On Sat, 9 Oct 2021 07:19:39 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Sat, Oct 09, 2021 at 12:27:08AM +0800, Xuan Zhuo wrote:
> > On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > > > > occur with 5.13,
> > > > > >
> > > > > > I should have fixed this problem before. I don't know why, I just looked at the
> > > > > > latest net code, and this commit seems to be lost.
> > > > > >
> > > > > >      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > > > > >
> > > > > > Can you test this patch again?
> > > > >
> > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > > > > it in it, right?
> > > > >
> > > >
> > > > Yes, it may be lost due to conflicts during a certain merge.
> > >
> > > Really?  I tried to apply that again to 5.14 and it did not work.  So I
> > > do not understand what to do here, can you try to explain it better?
> >
> > I took a look, and there is actually another missing patch:
> >
> > A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr
> > B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> >
> > A is replaced by another patch:
> >
> > 	commit c32325b8fdf2f979befb9fd5587918c0d5412db3
> > 	Author: Jakub Kicinski <kuba@kernel.org>
> > 	Date:   Mon Aug 2 10:57:29 2021 -0700
> >
> > 	    virtio-net: realign page_to_skb() after merges
> >
> > 	    We ended up merging two versions of the same patch set:
> >
> > 	    commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr")
> > 	    commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address")
> >
> > 	    into net, and
> >
> > 	    commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr")
> > 	    commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address")
> >
> > 	    into net-next. Redo the merge from commit 126285651b7f ("Merge
> > 	    ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that
> > 	    the most recent code remains.
> >
> > 	    Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > 	    Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> > 	    Acked-by: Jason Wang <jasowang@redhat.com>
> > 	    Signed-off-by: David S. Miller <davem@davemloft.net>
> >
> > So after this patch, patch B can be applied normally.
> >
> > So on the latest net branch, only lost
> >
> >           1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
>
> Again, I do not know what to do here, can you submit the needed fix to
> the networking developers so this gets fixed?

Michael has already submitted the patch.

https://lore.kernel.org/netdev/20211009091604.84141-1-mst@redhat.com/T/#u

Thanks.

>
> thanks,
>
> greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-08 16:27         ` Xuan Zhuo
@ 2021-10-09  5:19           ` Greg KH
  2021-10-09  9:31             ` Xuan Zhuo
  0 siblings, 1 reply; 33+ messages in thread
From: Greg KH @ 2021-10-09  5:19 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
	virtualization, regressions, Eric Dumazet, stable

On Sat, Oct 09, 2021 at 12:27:08AM +0800, Xuan Zhuo wrote:
> On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > > > occur with 5.13,
> > > > >
> > > > > I should have fixed this problem before. I don't know why, I just looked at the
> > > > > latest net code, and this commit seems to be lost.
> > > > >
> > > > >      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > > > >
> > > > > Can you test this patch again?
> > > >
> > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > > > it in it, right?
> > > >
> > >
> > > Yes, it may be lost due to conflicts during a certain merge.
> >
> > Really?  I tried to apply that again to 5.14 and it did not work.  So I
> > do not understand what to do here, can you try to explain it better?
> 
> I took a look, and there is actually another missing patch:
> 
> A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr
> B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> 
> A is replaced by another patch:
> 
> 	commit c32325b8fdf2f979befb9fd5587918c0d5412db3
> 	Author: Jakub Kicinski <kuba@kernel.org>
> 	Date:   Mon Aug 2 10:57:29 2021 -0700
> 
> 	    virtio-net: realign page_to_skb() after merges
> 
> 	    We ended up merging two versions of the same patch set:
> 
> 	    commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr")
> 	    commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address")
> 
> 	    into net, and
> 
> 	    commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr")
> 	    commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address")
> 
> 	    into net-next. Redo the merge from commit 126285651b7f ("Merge
> 	    ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that
> 	    the most recent code remains.
> 
> 	    Acked-by: Michael S. Tsirkin <mst@redhat.com>
> 	    Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> 	    Acked-by: Jason Wang <jasowang@redhat.com>
> 	    Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> So after this patch, patch B can be applied normally.
> 
> So on the latest net branch, only lost
> 
>           1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode

Again, I do not know what to do here, can you submit the needed fix to
the networking developers so this gets fixed?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-08  8:06       ` Greg KH
  2021-10-08 10:02         ` Michael S. Tsirkin
@ 2021-10-08 16:27         ` Xuan Zhuo
  2021-10-09  5:19           ` Greg KH
  1 sibling, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-10-08 16:27 UTC (permalink / raw)
  To: Greg KH
  Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
	virtualization, regressions, Eric Dumazet, stable

On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > > occur with 5.13,
> > > >
> > > > I should have fixed this problem before. I don't know why, I just looked at the
> > > > latest net code, and this commit seems to be lost.
> > > >
> > > >      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > > >
> > > > Can you test this patch again?
> > >
> > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > > it in it, right?
> > >
> >
> > Yes, it may be lost due to conflicts during a certain merge.
>
> Really?  I tried to apply that again to 5.14 and it did not work.  So I
> do not understand what to do here, can you try to explain it better?

I took a look, and there is actually another missing patch:

A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr
B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode

A is replaced by another patch:

	commit c32325b8fdf2f979befb9fd5587918c0d5412db3
	Author: Jakub Kicinski <kuba@kernel.org>
	Date:   Mon Aug 2 10:57:29 2021 -0700

	    virtio-net: realign page_to_skb() after merges

	    We ended up merging two versions of the same patch set:

	    commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr")
	    commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address")

	    into net, and

	    commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr")
	    commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address")

	    into net-next. Redo the merge from commit 126285651b7f ("Merge
	    ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that
	    the most recent code remains.

	    Acked-by: Michael S. Tsirkin <mst@redhat.com>
	    Signed-off-by: Jakub Kicinski <kuba@kernel.org>
	    Acked-by: Jason Wang <jasowang@redhat.com>
	    Signed-off-by: David S. Miller <davem@davemloft.net>

So after this patch, patch B can be applied normally.

So on the latest net branch, only lost

          1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode

Thanks.

>
> thanks,
>
> greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-08 10:02         ` Michael S. Tsirkin
@ 2021-10-08 12:21           ` Corentin Noël
  0 siblings, 0 replies; 33+ messages in thread
From: Corentin Noël @ 2021-10-08 12:21 UTC (permalink / raw)
  To: Michael S. Tsirkin, Greg KH
  Cc: Xuan Zhuo, Jason Wang, virtualization, regressions, Eric Dumazet, stable

Le vendredi 08 octobre 2021 à 06:02 -0400, Michael S. Tsirkin a écrit :
> On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote:
> > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <
> > > gregkh@linuxfoundation.org> wrote:
> > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <
> > > > > corentin.noel@collabora.com> wrote:
> > > > > > I've been experiencing crashes with 5.14-rc1 and above that
> > > > > > do not
> > > > > > occur with 5.13,
> > > > > 
> > > > > I should have fixed this problem before. I don't know why, I
> > > > > just looked at the
> > > > > latest net code, and this commit seems to be lost.
> > > > > 
> > > > >      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix
> > > > > for skb_over_panic inside big mode
> > > > > 
> > > > > Can you test this patch again?
> > > > 
> > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should
> > > > have had
> > > > it in it, right?
> > > > 
> > > 
> > > Yes, it may be lost due to conflicts during a certain merge.
> > 
> > Really?  I tried to apply that again to 5.14 and it did not
> > work.  So I
> > do not understand what to do here, can you try to explain it
> > better?
> > 
> > thanks,
> > 
> > greg k-h
> 
> Hmm, something like the following perhaps then?
> Corentin would you like to try this?
> Warning: untested.
> 
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 096c2ac6b7a6..18dd9f6d107d 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct
> virtnet_info *vi,
>  	 * add_recvbuf_mergeable() + get_mergeable_buf_len()
>  	 */
>  	truesize = headroom ? PAGE_SIZE : truesize;
> -	tailroom = truesize - len - headroom;
> +	tailroom = truesize - headroom;
>  	buf = p - headroom;
>  
>  	len -= hdr_len;
>  	offset += hdr_padded_len;
>  	p += hdr_padded_len;
> +	tailroom -= hdr_padded_len + len;
>  
>  	shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
>  
> 

Thank you for the patch, I started bisecting the issue but your patch
actually makes it work again.

Regards,
Corentin


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-08  8:06       ` Greg KH
@ 2021-10-08 10:02         ` Michael S. Tsirkin
  2021-10-08 12:21           ` Corentin Noël
  2021-10-08 16:27         ` Xuan Zhuo
  1 sibling, 1 reply; 33+ messages in thread
From: Michael S. Tsirkin @ 2021-10-08 10:02 UTC (permalink / raw)
  To: Greg KH
  Cc: Xuan Zhuo, Corentin Noël, Jason Wang, virtualization,
	regressions, Eric Dumazet, stable

On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote:
> On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > > occur with 5.13,
> > > >
> > > > I should have fixed this problem before. I don't know why, I just looked at the
> > > > latest net code, and this commit seems to be lost.
> > > >
> > > >      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > > >
> > > > Can you test this patch again?
> > >
> > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > > it in it, right?
> > >
> > 
> > Yes, it may be lost due to conflicts during a certain merge.
> 
> Really?  I tried to apply that again to 5.14 and it did not work.  So I
> do not understand what to do here, can you try to explain it better?
> 
> thanks,
> 
> greg k-h

Hmm, something like the following perhaps then?
Corentin would you like to try this?
Warning: untested.


diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 096c2ac6b7a6..18dd9f6d107d 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
 	 * add_recvbuf_mergeable() + get_mergeable_buf_len()
 	 */
 	truesize = headroom ? PAGE_SIZE : truesize;
-	tailroom = truesize - len - headroom;
+	tailroom = truesize - headroom;
 	buf = p - headroom;
 
 	len -= hdr_len;
 	offset += hdr_padded_len;
 	p += hdr_padded_len;
+	tailroom -= hdr_padded_len + len;
 
 	shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
 


^ permalink raw reply related	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 16:17     ` Xuan Zhuo
@ 2021-10-08  8:06       ` Greg KH
  2021-10-08 10:02         ` Michael S. Tsirkin
  2021-10-08 16:27         ` Xuan Zhuo
  0 siblings, 2 replies; 33+ messages in thread
From: Greg KH @ 2021-10-08  8:06 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
	virtualization, regressions, Eric Dumazet, stable

On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote:
> On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > > occur with 5.13,
> > >
> > > I should have fixed this problem before. I don't know why, I just looked at the
> > > latest net code, and this commit seems to be lost.
> > >
> > >      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> > >
> > > Can you test this patch again?
> >
> > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> > it in it, right?
> >
> 
> Yes, it may be lost due to conflicts during a certain merge.

Really?  I tried to apply that again to 5.14 and it did not work.  So I
do not understand what to do here, can you try to explain it better?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 12:04 Corentin Noël
  2021-10-07 13:10 ` Michael S. Tsirkin
  2021-10-07 15:06 ` Xuan Zhuo
@ 2021-10-08  7:59 ` Thorsten Leemhuis
  2021-10-12 12:44   ` Thorsten Leemhuis
  2 siblings, 1 reply; 33+ messages in thread
From: Thorsten Leemhuis @ 2021-10-08  7:59 UTC (permalink / raw)
  To: regressions

On 07.10.21 14:04, Corentin Noël wrote:
> I've been experiencing crashes with 5.14-rc1 and above that do not
> occur with 5.13,

Feel free to ignore this message. I write it to make regzbot track above
issue. Regzbot is the regression tracking bot I'm working on. It's still
in the early stages and this is still one of the first few regression I
make it track to get started and things tested in the field. That also
why I'm sending the mail just to the regressions list (it will do its
full magic nevertheless). For details see:
https://linux-regtracking.leemhuis.info/post/inital-regzbot-running/
https://linux-regtracking.leemhuis.info/post/regzbot-approach/

#regzbot ^introduced v5.13..v5.14-rc1
#regzbot link 
https://lore.kernel.org/regressions/1633623446.6192446-1-xuanzhuo@linux.alibaba.com/ 
Looks like a fix for this somehow fall through the cracks
#regzbot ignore-activitiy

BTW, in case anyone is reading this: the regzbot's webinterface that
gives insights into what is tracked can be found here:

https://linux-regtracking.leemhuis.info/regzbot/mainline.html

Ciao, Thorsten

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 15:25   ` Greg KH
@ 2021-10-07 16:17     ` Xuan Zhuo
  2021-10-08  8:06       ` Greg KH
  0 siblings, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-10-07 16:17 UTC (permalink / raw)
  To: Greg KH
  Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
	virtualization, regressions, Eric Dumazet, stable

On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > > I've been experiencing crashes with 5.14-rc1 and above that do not
> > > occur with 5.13,
> >
> > I should have fixed this problem before. I don't know why, I just looked at the
> > latest net code, and this commit seems to be lost.
> >
> >      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> >
> > Can you test this patch again?
>
> That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
> it in it, right?
>

Yes, it may be lost due to conflicts during a certain merge.

Thanks.

> thanks,
>
> greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 15:06 ` Xuan Zhuo
@ 2021-10-07 15:25   ` Greg KH
  2021-10-07 16:17     ` Xuan Zhuo
  0 siblings, 1 reply; 33+ messages in thread
From: Greg KH @ 2021-10-07 15:25 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang,
	virtualization, regressions, Eric Dumazet, stable

On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote:
> On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> > I've been experiencing crashes with 5.14-rc1 and above that do not
> > occur with 5.13,
> 
> I should have fixed this problem before. I don't know why, I just looked at the
> latest net code, and this commit seems to be lost.
> 
>      1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode
> 
> Can you test this patch again?

That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had
it in it, right?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 12:04 Corentin Noël
  2021-10-07 13:10 ` Michael S. Tsirkin
@ 2021-10-07 15:06 ` Xuan Zhuo
  2021-10-07 15:25   ` Greg KH
  2021-10-08  7:59 ` Thorsten Leemhuis
  2 siblings, 1 reply; 33+ messages in thread
From: Xuan Zhuo @ 2021-10-07 15:06 UTC (permalink / raw)
  To: Corentin Noël
  Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions,
	Eric Dumazet, stable

On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote:
> I've been experiencing crashes with 5.14-rc1 and above that do not
> occur with 5.13,

I should have fixed this problem before. I don't know why, I just looked at the
latest net code, and this commit seems to be lost.

     1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode

Can you test this patch again?

Thanks.

>
> here is the crash trace:
>
> [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> end:0xec0 dev:<NULL>
> [   61.369192] kernel BUG at net/core/skbuff.c:111!
> [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> [   61.377222] RIP: 0010:skb_panic+0x43/0x45
> [   61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   61.393635] Call Trace:
> [   61.394127]  <IRQ>
> [   61.394488]  skb_put.cold+0x10/0x10
> [   61.395095]  page_to_skb+0xf7/0x410
> [   61.395689]  receive_buf+0x81/0x1660
> [   61.396228]  ? netif_receive_skb_list_internal+0x1ad/0x2b0
> [   61.397180]  ? napi_gro_flush+0x97/0xe0
> [   61.397896]  ? detach_buf_split+0x67/0x120
> [   61.398573]  virtnet_poll+0x2cf/0x420
> [   61.399197]  __napi_poll+0x25/0x150
> [   61.399764]  net_rx_action+0x22f/0x280
> [   61.400394]  __do_softirq+0xba/0x257
> [   61.401012]  irq_exit_rcu+0x8e/0xb0
> [   61.401618]  common_interrupt+0x7b/0xa0
> [   61.402270]  </IRQ>
> [   61.402620]  asm_common_interrupt+0x1e/0x40
> [   61.403302] RIP: 0010:default_idle+0xb/0x10
> [   61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> ffff8a5febd56f80
> [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> ffff8a5febd5dd00
> [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> 0000000000000000
> [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> 0000000000000000
> [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [   61.414183]  ? mwait_idle+0x70/0x70
> [   61.414805]  ? mwait_idle+0x70/0x70
> [   61.415592]  default_idle_call+0x2a/0xa0
> [   61.416216]  do_idle+0x1e8/0x250
> [   61.416722]  cpu_startup_entry+0x14/0x20
> [   61.417347]  secondary_startup_64_no_verify+0xc2/0xcb
> [   61.418144] Modules linked in:
> [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> [   61.419399] RIP: 0010:skb_panic+0x43/0x45
> [   61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [   61.431048] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [   61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   61.435799] Kernel panic - not syncing: Fatal exception in interrupt
> [   61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>
> Here is my kernel config:
> https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
>
>
> here is the decoded trace:
>
> [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> end:0xec0 dev:<NULL>
> [   61.369192] kernel BUG at net/core/skbuff.c:111!
> [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> [   61.377222] RIP: skb_panic+0x43/0x45
> [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
>    0:	4f 70 50             	rex.WRXB jo 0x53
>    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
>    9:	50                   	push   %rax
>    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
>   10:	50                   	push   %rax
>   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
>   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
>   1e:	48 c7 c7 18 f1 cf 88 	mov    $0xffffffff88cff118,%rdi
>   25:	e8 6a 43 fb ff       	callq  0xfffffffffffb4394
>   2a:*	0f 0b                	ud2    		<-- trapping
> instruction
>   2c:	48 8b 14 24          	mov    (%rsp),%rdx
>   30:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
>   3c:	48                   	rex.W
>   3d:	c7                   	.byte 0xc7
>   3e:	c6                   	(bad)
>   3f:	60                   	(bad)
>
> Code starting with the faulting instruction
> ===========================================
>    0:	0f 0b                	ud2
>    2:	48 8b 14 24          	mov    (%rsp),%rdx
>    6:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
>   12:	48                   	rex.W
>   13:	c7                   	.byte 0xc7
>   14:	c6                   	(bad)
>   15:	60                   	(bad)
> [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   61.393635] Call Trace:
> [   61.394127]  <IRQ>
> [   61.394488] skb_put.cold+0x10/0x10
> [   61.395095] page_to_skb+0xf7/0x410
> [   61.395689] receive_buf+0x81/0x1660
> [   61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> [   61.397180] ? napi_gro_flush+0x97/0xe0
> [   61.397896] ? detach_buf_split+0x67/0x120
> [   61.398573] virtnet_poll+0x2cf/0x420
> [   61.399197] __napi_poll+0x25/0x150
> [   61.399764] net_rx_action+0x22f/0x280
> [   61.400394] __do_softirq+0xba/0x257
> [   61.401012] irq_exit_rcu+0x8e/0xb0
> [   61.401618] common_interrupt+0x7b/0xa0
> [   61.402270]  </IRQ>
> [   61.402620] asm_common_interrupt+0x1e/0x40
> [   61.403302] RIP: default_idle+0xb/0x10
> [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> All code
> ========
>    0:	8b 04 25 00 6d 01 00 	mov    0x16d00,%eax
>    7:	f0 80 60 02 df       	lock andb $0xdf,0x2(%rax)
>    c:	c3                   	retq
>    d:	0f ae f0             	mfence
>   10:	0f ae 38             	clflush (%rax)
>   13:	0f ae f0             	mfence
>   16:	eb b9                	jmp    0xffffffffffffffd1
>   18:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)
>   1f:	eb 07                	jmp    0x28
>   21:	0f 00 2d df 3e 44 00 	verw   0x443edf(%rip)        # 0x443f07
>   28:	fb                   	sti
>   29:	f4                   	hlt
>   2a:*	c3                   	retq   		<-- trapping
> instruction
>   2b:	cc                   	int3
>   2c:	cc                   	int3
>   2d:	cc                   	int3
>   2e:	cc                   	int3
>   2f:	65 8b 15 31 2f a4 77 	mov    %gs:0x77a42f31(%rip),%edx
>  # 0x77a42f67
>   36:	89 d2                	mov    %edx,%edx
>   38:	48 8b 05 d0 a1 0c 01 	mov    0x10ca1d0(%rip),%rax        #
> 0x10ca20f
>   3f:	48                   	rex.W
>
> Code starting with the faulting instruction
> ===========================================
>    0:	c3                   	retq
>    1:	cc                   	int3
>    2:	cc                   	int3
>    3:	cc                   	int3
>    4:	cc                   	int3
>    5:	65 8b 15 31 2f a4 77 	mov    %gs:0x77a42f31(%rip),%edx
>  # 0x77a42f3d
>    c:	89 d2                	mov    %edx,%edx
>    e:	48 8b 05 d0 a1 0c 01 	mov    0x10ca1d0(%rip),%rax        #
> 0x10ca1e5
>   15:	48                   	rex.W
> [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> ffff8a5febd56f80
> [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> ffff8a5febd5dd00
> [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> 0000000000000000
> [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> 0000000000000000
> [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [   61.414183] ? mwait_idle+0x70/0x70
> [   61.414805] ? mwait_idle+0x70/0x70
> [   61.415592] default_idle_call+0x2a/0xa0
> [   61.416216] do_idle+0x1e8/0x250
> [   61.416722] cpu_startup_entry+0x14/0x20
> [   61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> [   61.418144] Modules linked in:
> [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> [   61.419399] RIP: skb_panic+0x43/0x45
> [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
>    0:	4f 70 50             	rex.WRXB jo 0x53
>    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
>    9:	50                   	push   %rax
>    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
>   10:	50                   	push   %rax
>   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
>   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
>   1e:	48 c7 c7 18 f1 cf 88 	mov    $0xffffffff88cff118,%rdi
>   25:	e8 6a 43 fb ff       	callq  0xfffffffffffb4394
>   2a:*	0f 0b                	ud2    		<-- trapping
> instruction
>   2c:	48 8b 14 24          	mov    (%rsp),%rdx
>   30:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
>   3c:	48                   	rex.W
>   3d:	c7                   	.byte 0xc7
>   3e:	c6                   	(bad)
>   3f:	60                   	(bad)
>
> Code starting with the faulting instruction
> ===========================================
>    0:	0f 0b                	ud2
>    2:	48 8b 14 24          	mov    (%rsp),%rdx
>    6:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
>   12:	48                   	rex.W
>   13:	c7                   	.byte 0xc7
>   14:	c6                   	(bad)
>   15:	60                   	(bad)
> [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
>
> Regards,
> Corentin
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 14:02     ` Corentin Noël
@ 2021-10-07 14:13       ` Greg KH
  0 siblings, 0 replies; 33+ messages in thread
From: Greg KH @ 2021-10-07 14:13 UTC (permalink / raw)
  To: Corentin Noël
  Cc: Eric Dumazet, Michael S. Tsirkin, linux-stable, Jason Wang,
	virtualization, regressions, Xuan Zhuo

On Thu, Oct 07, 2021 at 04:02:10PM +0200, Corentin Noël wrote:
> Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit :
> > On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com>
> > wrote:
> > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> > > > I've been experiencing crashes with 5.14-rc1 and above that do
> > > > not
> > > > occur with 5.13,
> > 
> > What about 5.14 ?
> > 
> > 5.14-rc1 has many bugs we do not want to spend time rediscovering
> > them...
> > 
> 
> I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could
> reproduce the crash on them all.

Great, any chance you can use 'git bisect' to find the offending commit?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 13:51   ` Eric Dumazet
@ 2021-10-07 14:02     ` Corentin Noël
  2021-10-07 14:13       ` Greg KH
  0 siblings, 1 reply; 33+ messages in thread
From: Corentin Noël @ 2021-10-07 14:02 UTC (permalink / raw)
  To: Eric Dumazet, Michael S. Tsirkin
  Cc: linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo

Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit :
> On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com>
> wrote:
> > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> > > I've been experiencing crashes with 5.14-rc1 and above that do
> > > not
> > > occur with 5.13,
> 
> What about 5.14 ?
> 
> 5.14-rc1 has many bugs we do not want to spend time rediscovering
> them...
> 

I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could
reproduce the crash on them all.

> > > here is the crash trace:
> > > 
> > > [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7
> > > len:3762
> > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [   61.369192] kernel BUG at net/core/skbuff.c:111!
> > > [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> > > [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > > [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [   61.377222] RIP: 0010:skb_panic+0x43/0x45
> > > [   61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [   61.386825] FS:  0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > > [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > > 0000000000370ee0
> > > [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   61.393635] Call Trace:
> > > [   61.394127]  <IRQ>
> > > [   61.394488]  skb_put.cold+0x10/0x10
> > > [   61.395095]  page_to_skb+0xf7/0x410
> > > [   61.395689]  receive_buf+0x81/0x1660
> > > [   61.396228]  ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > > [   61.397180]  ? napi_gro_flush+0x97/0xe0
> > > [   61.397896]  ? detach_buf_split+0x67/0x120
> > > [   61.398573]  virtnet_poll+0x2cf/0x420
> > > [   61.399197]  __napi_poll+0x25/0x150
> > > [   61.399764]  net_rx_action+0x22f/0x280
> > > [   61.400394]  __do_softirq+0xba/0x257
> > > [   61.401012]  irq_exit_rcu+0x8e/0xb0
> > > [   61.401618]  common_interrupt+0x7b/0xa0
> > > [   61.402270]  </IRQ>
> > > [   61.402620]  asm_common_interrupt+0x1e/0x40
> > > [   61.403302] RIP: 0010:default_idle+0xb/0x10
> > > [   61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae
> > > f0 0f
> > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44
> > > 00 fb
> > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c
> > > 01 48
> > > [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > > [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > > ffff8a5febd56f80
> > > [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > > ffff8a5febd5dd00
> > > [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > > 0000000000000000
> > > [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > > 0000000000000000
> > > [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > > 0000000000000000
> > > [   61.414183]  ? mwait_idle+0x70/0x70
> > > [   61.414805]  ? mwait_idle+0x70/0x70
> > > [   61.415592]  default_idle_call+0x2a/0xa0
> > > [   61.416216]  do_idle+0x1e8/0x250
> > > [   61.416722]  cpu_startup_entry+0x14/0x20
> > > [   61.417347]  secondary_startup_64_no_verify+0xc2/0xcb
> > > [   61.418144] Modules linked in:
> > > [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > > [   61.419399] RIP: 0010:skb_panic+0x43/0x45
> > > [   61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [   61.429799] FS:  0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > > [   61.431048] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > > 0000000000370ee0
> > > [   61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   61.435799] Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > > [   61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > 
> > > Here is my kernel config:
> > > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
> > > 
> > > 
> > > here is the decoded trace:
> > > 
> > > [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7
> > > len:3762
> > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > > end:0xec0 dev:<NULL>
> > > [   61.369192] kernel BUG at net/core/skbuff.c:111!
> > > [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> > > [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > > [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > > [   61.377222] RIP: skb_panic+0x43/0x45
> > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > All code
> > > ========
> > >    0: 4f 70 50                rex.WRXB jo 0x53
> > >    3: 8b 87 bc 00 00 00       mov    0xbc(%rdi),%eax
> > >    9: 50                      push   %rax
> > >    a: 8b 87 b8 00 00 00       mov    0xb8(%rdi),%eax
> > >   10: 50                      push   %rax
> > >   11: ff b7 c8 00 00 00       pushq  0xc8(%rdi)
> > >   17: 4c 8b 8f c0 00 00 00    mov    0xc0(%rdi),%r9
> > >   1e: 48 c7 c7 18 f1 cf 88    mov    $0xffffffff88cff118,%rdi
> > >   25: e8 6a 43 fb ff          callq  0xfffffffffffb4394
> > >   2a:*        0f 0b                   ud2             <--
> > > trapping
> > > instruction
> > >   2c: 48 8b 14 24             mov    (%rsp),%rdx
> > >   30: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> > >   37: e8 ab ff ff ff          callq  0xffffffffffffffe7
> > >   3c: 48                      rex.W
> > >   3d: c7                      .byte 0xc7
> > >   3e: c6                      (bad)
> > >   3f: 60                      (bad)
> > > 
> > > Code starting with the faulting instruction
> > > ===========================================
> > >    0: 0f 0b                   ud2
> > >    2: 48 8b 14 24             mov    (%rsp),%rdx
> > >    6: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> > >    d: e8 ab ff ff ff          callq  0xffffffffffffffbd
> > >   12: 48                      rex.W
> > >   13: c7                      .byte 0xc7
> > >   14: c6                      (bad)
> > >   15: 60                      (bad)
> > > [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [   61.386825] FS:  0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > > [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > > 0000000000370ee0
> > > [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [   61.393635] Call Trace:
> > > [   61.394127]  <IRQ>
> > > [   61.394488] skb_put.cold+0x10/0x10
> > > [   61.395095] page_to_skb+0xf7/0x410
> > > [   61.395689] receive_buf+0x81/0x1660
> > > [   61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > > [   61.397180] ? napi_gro_flush+0x97/0xe0
> > > [   61.397896] ? detach_buf_split+0x67/0x120
> > > [   61.398573] virtnet_poll+0x2cf/0x420
> > > [   61.399197] __napi_poll+0x25/0x150
> > > [   61.399764] net_rx_action+0x22f/0x280
> > > [   61.400394] __do_softirq+0xba/0x257
> > > [   61.401012] irq_exit_rcu+0x8e/0xb0
> > > [   61.401618] common_interrupt+0x7b/0xa0
> > > [   61.402270]  </IRQ>
> > > [   61.402620] asm_common_interrupt+0x1e/0x40
> > > [   61.403302] RIP: default_idle+0xb/0x10
> > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae
> > > f0 0f
> > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44
> > > 00 fb
> > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c
> > > 01 48
> > > All code
> > > ========
> > >    0: 8b 04 25 00 6d 01 00    mov    0x16d00,%eax
> > >    7: f0 80 60 02 df          lock andb $0xdf,0x2(%rax)
> > >    c: c3                      retq
> > >    d: 0f ae f0                mfence
> > >   10: 0f ae 38                clflush (%rax)
> > >   13: 0f ae f0                mfence
> > >   16: eb b9                   jmp    0xffffffffffffffd1
> > >   18: 0f 1f 80 00 00 00 00    nopl   0x0(%rax)
> > >   1f: eb 07                   jmp    0x28
> > >   21: 0f 00 2d df 3e 44 00    verw   0x443edf(%rip)        #
> > > 0x443f07
> > >   28: fb                      sti
> > >   29: f4                      hlt
> > >   2a:*        c3                      retq            <--
> > > trapping
> > > instruction
> > >   2b: cc                      int3
> > >   2c: cc                      int3
> > >   2d: cc                      int3
> > >   2e: cc                      int3
> > >   2f: 65 8b 15 31 2f a4 77    mov    %gs:0x77a42f31(%rip),%edx
> > >  # 0x77a42f67
> > >   36: 89 d2                   mov    %edx,%edx
> > >   38: 48 8b 05 d0 a1 0c
> > > 01    mov    0x10ca1d0(%rip),%rax        #
> > > 0x10ca20f
> > >   3f: 48                      rex.W
> > > 
> > > Code starting with the faulting instruction
> > > ===========================================
> > >    0: c3                      retq
> > >    1: cc                      int3
> > >    2: cc                      int3
> > >    3: cc                      int3
> > >    4: cc                      int3
> > >    5: 65 8b 15 31 2f a4 77    mov    %gs:0x77a42f31(%rip),%edx
> > >  # 0x77a42f3d
> > >    c: 89 d2                   mov    %edx,%edx
> > >    e: 48 8b 05 d0 a1 0c
> > > 01    mov    0x10ca1d0(%rip),%rax        #
> > > 0x10ca1e5
> > >   15: 48                      rex.W
> > > [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > > [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > > ffff8a5febd56f80
> > > [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > > ffff8a5febd5dd00
> > > [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > > 0000000000000000
> > > [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > > 0000000000000000
> > > [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > > 0000000000000000
> > > [   61.414183] ? mwait_idle+0x70/0x70
> > > [   61.414805] ? mwait_idle+0x70/0x70
> > > [   61.415592] default_idle_call+0x2a/0xa0
> > > [   61.416216] do_idle+0x1e8/0x250
> > > [   61.416722] cpu_startup_entry+0x14/0x20
> > > [   61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> > > [   61.418144] Modules linked in:
> > > [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > > [   61.419399] RIP: skb_panic+0x43/0x45
> > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00
> > > 00 50
> > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a
> > > 43 fb
> > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7
> > > c6 60
> > > All code
> > > ========
> > >    0: 4f 70 50                rex.WRXB jo 0x53
> > >    3: 8b 87 bc 00 00 00       mov    0xbc(%rdi),%eax
> > >    9: 50                      push   %rax
> > >    a: 8b 87 b8 00 00 00       mov    0xb8(%rdi),%eax
> > >   10: 50                      push   %rax
> > >   11: ff b7 c8 00 00 00       pushq  0xc8(%rdi)
> > >   17: 4c 8b 8f c0 00 00 00    mov    0xc0(%rdi),%r9
> > >   1e: 48 c7 c7 18 f1 cf 88    mov    $0xffffffff88cff118,%rdi
> > >   25: e8 6a 43 fb ff          callq  0xfffffffffffb4394
> > >   2a:*        0f 0b                   ud2             <--
> > > trapping
> > > instruction
> > >   2c: 48 8b 14 24             mov    (%rsp),%rdx
> > >   30: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> > >   37: e8 ab ff ff ff          callq  0xffffffffffffffe7
> > >   3c: 48                      rex.W
> > >   3d: c7                      .byte 0xc7
> > >   3e: c6                      (bad)
> > >   3f: 60                      (bad)
> > > 
> > > Code starting with the faulting instruction
> > > ===========================================
> > >    0: 0f 0b                   ud2
> > >    2: 48 8b 14 24             mov    (%rsp),%rdx
> > >    6: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> > >    d: e8 ab ff ff ff          callq  0xffffffffffffffbd
> > >   12: 48                      rex.W
> > >   13: c7                      .byte 0xc7
> > >   14: c6                      (bad)
> > >   15: 60                      (bad)
> > > [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > > [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > > 00000000ffffdfff
> > > [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > > 0000000000000000
> > > [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > > 0000000000009ffb
> > > [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > > ffff8a5ec7461200
> > > [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > > 0000000000000eb2
> > > [   61.429799] FS:  0000000000000000(0000)
> > > GS:ffff8a5febd40000(0000)
> > > knlGS:0000000000000000
> > > 
> > > Regards,
> > > Corentin
> > 
> > Don't see anything obvious.. could be a net stack change.
> > Any chance of a bisect?
> > 
> > --
> > MST
> > 


^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 13:10 ` Michael S. Tsirkin
@ 2021-10-07 13:51   ` Eric Dumazet
  2021-10-07 14:02     ` Corentin Noël
  0 siblings, 1 reply; 33+ messages in thread
From: Eric Dumazet @ 2021-10-07 13:51 UTC (permalink / raw)
  To: Michael S. Tsirkin
  Cc: Corentin Noël, linux-stable, Jason Wang, virtualization,
	regressions, Xuan Zhuo

On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> > I've been experiencing crashes with 5.14-rc1 and above that do not
> > occur with 5.13,

What about 5.14 ?

5.14-rc1 has many bugs we do not want to spend time rediscovering them...

> >
> > here is the crash trace:
> >
> > [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [   61.369192] kernel BUG at net/core/skbuff.c:111!
> > [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> > [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > [   61.377222] RIP: 0010:skb_panic+0x43/0x45
> > [   61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> > [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > 0000000000370ee0
> > [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   61.393635] Call Trace:
> > [   61.394127]  <IRQ>
> > [   61.394488]  skb_put.cold+0x10/0x10
> > [   61.395095]  page_to_skb+0xf7/0x410
> > [   61.395689]  receive_buf+0x81/0x1660
> > [   61.396228]  ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > [   61.397180]  ? napi_gro_flush+0x97/0xe0
> > [   61.397896]  ? detach_buf_split+0x67/0x120
> > [   61.398573]  virtnet_poll+0x2cf/0x420
> > [   61.399197]  __napi_poll+0x25/0x150
> > [   61.399764]  net_rx_action+0x22f/0x280
> > [   61.400394]  __do_softirq+0xba/0x257
> > [   61.401012]  irq_exit_rcu+0x8e/0xb0
> > [   61.401618]  common_interrupt+0x7b/0xa0
> > [   61.402270]  </IRQ>
> > [   61.402620]  asm_common_interrupt+0x1e/0x40
> > [   61.403302] RIP: 0010:default_idle+0xb/0x10
> > [   61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> > [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > ffff8a5febd56f80
> > [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > ffff8a5febd5dd00
> > [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > 0000000000000000
> > [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > 0000000000000000
> > [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > 0000000000000000
> > [   61.414183]  ? mwait_idle+0x70/0x70
> > [   61.414805]  ? mwait_idle+0x70/0x70
> > [   61.415592]  default_idle_call+0x2a/0xa0
> > [   61.416216]  do_idle+0x1e8/0x250
> > [   61.416722]  cpu_startup_entry+0x14/0x20
> > [   61.417347]  secondary_startup_64_no_verify+0xc2/0xcb
> > [   61.418144] Modules linked in:
> > [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > [   61.419399] RIP: 0010:skb_panic+0x43/0x45
> > [   61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> > [   61.431048] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > 0000000000370ee0
> > [   61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   61.435799] Kernel panic - not syncing: Fatal exception in interrupt
> > [   61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
> > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> >
> > Here is my kernel config:
> > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
> >
> >
> > here is the decoded trace:
> >
> > [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> > end:0xec0 dev:<NULL>
> > [   61.369192] kernel BUG at net/core/skbuff.c:111!
> > [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> > [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> > [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> > [   61.377222] RIP: skb_panic+0x43/0x45
> > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > All code
> > ========
> >    0: 4f 70 50                rex.WRXB jo 0x53
> >    3: 8b 87 bc 00 00 00       mov    0xbc(%rdi),%eax
> >    9: 50                      push   %rax
> >    a: 8b 87 b8 00 00 00       mov    0xb8(%rdi),%eax
> >   10: 50                      push   %rax
> >   11: ff b7 c8 00 00 00       pushq  0xc8(%rdi)
> >   17: 4c 8b 8f c0 00 00 00    mov    0xc0(%rdi),%r9
> >   1e: 48 c7 c7 18 f1 cf 88    mov    $0xffffffff88cff118,%rdi
> >   25: e8 6a 43 fb ff          callq  0xfffffffffffb4394
> >   2a:*        0f 0b                   ud2             <-- trapping
> > instruction
> >   2c: 48 8b 14 24             mov    (%rsp),%rdx
> >   30: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> >   37: e8 ab ff ff ff          callq  0xffffffffffffffe7
> >   3c: 48                      rex.W
> >   3d: c7                      .byte 0xc7
> >   3e: c6                      (bad)
> >   3f: 60                      (bad)
> >
> > Code starting with the faulting instruction
> > ===========================================
> >    0: 0f 0b                   ud2
> >    2: 48 8b 14 24             mov    (%rsp),%rdx
> >    6: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> >    d: e8 ab ff ff ff          callq  0xffffffffffffffbd
> >   12: 48                      rex.W
> >   13: c7                      .byte 0xc7
> >   14: c6                      (bad)
> >   15: 60                      (bad)
> > [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> > [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> > 0000000000370ee0
> > [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [   61.393635] Call Trace:
> > [   61.394127]  <IRQ>
> > [   61.394488] skb_put.cold+0x10/0x10
> > [   61.395095] page_to_skb+0xf7/0x410
> > [   61.395689] receive_buf+0x81/0x1660
> > [   61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
> > [   61.397180] ? napi_gro_flush+0x97/0xe0
> > [   61.397896] ? detach_buf_split+0x67/0x120
> > [   61.398573] virtnet_poll+0x2cf/0x420
> > [   61.399197] __napi_poll+0x25/0x150
> > [   61.399764] net_rx_action+0x22f/0x280
> > [   61.400394] __do_softirq+0xba/0x257
> > [   61.401012] irq_exit_rcu+0x8e/0xb0
> > [   61.401618] common_interrupt+0x7b/0xa0
> > [   61.402270]  </IRQ>
> > [   61.402620] asm_common_interrupt+0x1e/0x40
> > [   61.403302] RIP: default_idle+0xb/0x10
> > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> > All code
> > ========
> >    0: 8b 04 25 00 6d 01 00    mov    0x16d00,%eax
> >    7: f0 80 60 02 df          lock andb $0xdf,0x2(%rax)
> >    c: c3                      retq
> >    d: 0f ae f0                mfence
> >   10: 0f ae 38                clflush (%rax)
> >   13: 0f ae f0                mfence
> >   16: eb b9                   jmp    0xffffffffffffffd1
> >   18: 0f 1f 80 00 00 00 00    nopl   0x0(%rax)
> >   1f: eb 07                   jmp    0x28
> >   21: 0f 00 2d df 3e 44 00    verw   0x443edf(%rip)        # 0x443f07
> >   28: fb                      sti
> >   29: f4                      hlt
> >   2a:*        c3                      retq            <-- trapping
> > instruction
> >   2b: cc                      int3
> >   2c: cc                      int3
> >   2d: cc                      int3
> >   2e: cc                      int3
> >   2f: 65 8b 15 31 2f a4 77    mov    %gs:0x77a42f31(%rip),%edx
> >  # 0x77a42f67
> >   36: 89 d2                   mov    %edx,%edx
> >   38: 48 8b 05 d0 a1 0c 01    mov    0x10ca1d0(%rip),%rax        #
> > 0x10ca20f
> >   3f: 48                      rex.W
> >
> > Code starting with the faulting instruction
> > ===========================================
> >    0: c3                      retq
> >    1: cc                      int3
> >    2: cc                      int3
> >    3: cc                      int3
> >    4: cc                      int3
> >    5: 65 8b 15 31 2f a4 77    mov    %gs:0x77a42f31(%rip),%edx
> >  # 0x77a42f3d
> >    c: 89 d2                   mov    %edx,%edx
> >    e: 48 8b 05 d0 a1 0c 01    mov    0x10ca1d0(%rip),%rax        #
> > 0x10ca1e5
> >   15: 48                      rex.W
> > [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> > [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> > ffff8a5febd56f80
> > [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> > ffff8a5febd5dd00
> > [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> > 0000000000000000
> > [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> > 0000000000000000
> > [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> > 0000000000000000
> > [   61.414183] ? mwait_idle+0x70/0x70
> > [   61.414805] ? mwait_idle+0x70/0x70
> > [   61.415592] default_idle_call+0x2a/0xa0
> > [   61.416216] do_idle+0x1e8/0x250
> > [   61.416722] cpu_startup_entry+0x14/0x20
> > [   61.417347] secondary_startup_64_no_verify+0xc2/0xcb
> > [   61.418144] Modules linked in:
> > [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> > [   61.419399] RIP: skb_panic+0x43/0x45
> > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> > All code
> > ========
> >    0: 4f 70 50                rex.WRXB jo 0x53
> >    3: 8b 87 bc 00 00 00       mov    0xbc(%rdi),%eax
> >    9: 50                      push   %rax
> >    a: 8b 87 b8 00 00 00       mov    0xb8(%rdi),%eax
> >   10: 50                      push   %rax
> >   11: ff b7 c8 00 00 00       pushq  0xc8(%rdi)
> >   17: 4c 8b 8f c0 00 00 00    mov    0xc0(%rdi),%r9
> >   1e: 48 c7 c7 18 f1 cf 88    mov    $0xffffffff88cff118,%rdi
> >   25: e8 6a 43 fb ff          callq  0xfffffffffffb4394
> >   2a:*        0f 0b                   ud2             <-- trapping
> > instruction
> >   2c: 48 8b 14 24             mov    (%rsp),%rdx
> >   30: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> >   37: e8 ab ff ff ff          callq  0xffffffffffffffe7
> >   3c: 48                      rex.W
> >   3d: c7                      .byte 0xc7
> >   3e: c6                      (bad)
> >   3f: 60                      (bad)
> >
> > Code starting with the faulting instruction
> > ===========================================
> >    0: 0f 0b                   ud2
> >    2: 48 8b 14 24             mov    (%rsp),%rdx
> >    6: 48 c7 c1 20 35 b1 88    mov    $0xffffffff88b13520,%rcx
> >    d: e8 ab ff ff ff          callq  0xffffffffffffffbd
> >   12: 48                      rex.W
> >   13: c7                      .byte 0xc7
> >   14: c6                      (bad)
> >   15: 60                      (bad)
> > [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> > [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> > 00000000ffffdfff
> > [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> > 0000000000000000
> > [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> > 0000000000009ffb
> > [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> > ffff8a5ec7461200
> > [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> > 0000000000000eb2
> > [   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> > knlGS:0000000000000000
> >
> > Regards,
> > Corentin
>
> Don't see anything obvious.. could be a net stack change.
> Any chance of a bisect?
>
> --
> MST
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: virtio-net: kernel panic in virtio_net.c
  2021-10-07 12:04 Corentin Noël
@ 2021-10-07 13:10 ` Michael S. Tsirkin
  2021-10-07 13:51   ` Eric Dumazet
  2021-10-07 15:06 ` Xuan Zhuo
  2021-10-08  7:59 ` Thorsten Leemhuis
  2 siblings, 1 reply; 33+ messages in thread
From: Michael S. Tsirkin @ 2021-10-07 13:10 UTC (permalink / raw)
  To: Corentin Noël
  Cc: stable, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo

On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote:
> I've been experiencing crashes with 5.14-rc1 and above that do not
> occur with 5.13,
> 
> here is the crash trace:
> 
> [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> end:0xec0 dev:<NULL>
> [   61.369192] kernel BUG at net/core/skbuff.c:111!
> [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 
> [   61.377222] RIP: 0010:skb_panic+0x43/0x45
> [   61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   61.393635] Call Trace:
> [   61.394127]  <IRQ>
> [   61.394488]  skb_put.cold+0x10/0x10
> [   61.395095]  page_to_skb+0xf7/0x410
> [   61.395689]  receive_buf+0x81/0x1660
> [   61.396228]  ? netif_receive_skb_list_internal+0x1ad/0x2b0
> [   61.397180]  ? napi_gro_flush+0x97/0xe0
> [   61.397896]  ? detach_buf_split+0x67/0x120
> [   61.398573]  virtnet_poll+0x2cf/0x420
> [   61.399197]  __napi_poll+0x25/0x150
> [   61.399764]  net_rx_action+0x22f/0x280
> [   61.400394]  __do_softirq+0xba/0x257
> [   61.401012]  irq_exit_rcu+0x8e/0xb0
> [   61.401618]  common_interrupt+0x7b/0xa0
> [   61.402270]  </IRQ>
> [   61.402620]  asm_common_interrupt+0x1e/0x40
> [   61.403302] RIP: 0010:default_idle+0xb/0x10
> [   61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> ffff8a5febd56f80
> [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> ffff8a5febd5dd00
> [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> 0000000000000000
> [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> 0000000000000000
> [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [   61.414183]  ? mwait_idle+0x70/0x70
> [   61.414805]  ? mwait_idle+0x70/0x70
> [   61.415592]  default_idle_call+0x2a/0xa0
> [   61.416216]  do_idle+0x1e8/0x250
> [   61.416722]  cpu_startup_entry+0x14/0x20
> [   61.417347]  secondary_startup_64_no_verify+0xc2/0xcb
> [   61.418144] Modules linked in:
> [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> [   61.419399] RIP: 0010:skb_panic+0x43/0x45
> [   61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [   61.431048] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [   61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   61.435799] Kernel panic - not syncing: Fatal exception in interrupt
> [   61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> 
> Here is my kernel config: 
> https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
> 
> 
> here is the decoded trace:
> 
> [   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
> put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
> end:0xec0 dev:<NULL>
> [   61.369192] kernel BUG at net/core/skbuff.c:111!
> [   61.372840] invalid opcode: 0000 [#1] SMP PTI
> [   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
> rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
> [   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
> [   61.377222] RIP: skb_panic+0x43/0x45 
> [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
>    0:	4f 70 50             	rex.WRXB jo 0x53
>    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
>    9:	50                   	push   %rax
>    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
>   10:	50                   	push   %rax
>   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
>   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
>   1e:	48 c7 c7 18 f1 cf 88 	mov    $0xffffffff88cff118,%rdi
>   25:	e8 6a 43 fb ff       	callq  0xfffffffffffb4394
>   2a:*	0f 0b                	ud2    		<-- trapping
> instruction
>   2c:	48 8b 14 24          	mov    (%rsp),%rdx
>   30:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
>   3c:	48                   	rex.W
>   3d:	c7                   	.byte 0xc7
>   3e:	c6                   	(bad)  
>   3f:	60                   	(bad)  
> 
> Code starting with the faulting instruction
> ===========================================
>    0:	0f 0b                	ud2    
>    2:	48 8b 14 24          	mov    (%rsp),%rdx
>    6:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
>   12:	48                   	rex.W
>   13:	c7                   	.byte 0xc7
>   14:	c6                   	(bad)  
>   15:	60                   	(bad)  
> [   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> [   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
> 0000000000370ee0
> [   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   61.393635] Call Trace:
> [   61.394127]  <IRQ>
> [   61.394488] skb_put.cold+0x10/0x10 
> [   61.395095] page_to_skb+0xf7/0x410 
> [   61.395689] receive_buf+0x81/0x1660 
> [   61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 
> [   61.397180] ? napi_gro_flush+0x97/0xe0 
> [   61.397896] ? detach_buf_split+0x67/0x120 
> [   61.398573] virtnet_poll+0x2cf/0x420 
> [   61.399197] __napi_poll+0x25/0x150 
> [   61.399764] net_rx_action+0x22f/0x280 
> [   61.400394] __do_softirq+0xba/0x257 
> [   61.401012] irq_exit_rcu+0x8e/0xb0 
> [   61.401618] common_interrupt+0x7b/0xa0 
> [   61.402270]  </IRQ>
> [   61.402620] asm_common_interrupt+0x1e/0x40 
> [   61.403302] RIP: default_idle+0xb/0x10 
> [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
> ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
> f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
> All code
> ========
>    0:	8b 04 25 00 6d 01 00 	mov    0x16d00,%eax
>    7:	f0 80 60 02 df       	lock andb $0xdf,0x2(%rax)
>    c:	c3                   	retq   
>    d:	0f ae f0             	mfence 
>   10:	0f ae 38             	clflush (%rax)
>   13:	0f ae f0             	mfence 
>   16:	eb b9                	jmp    0xffffffffffffffd1
>   18:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)
>   1f:	eb 07                	jmp    0x28
>   21:	0f 00 2d df 3e 44 00 	verw   0x443edf(%rip)        # 0x443f07
>   28:	fb                   	sti    
>   29:	f4                   	hlt    
>   2a:*	c3                   	retq   		<-- trapping
> instruction
>   2b:	cc                   	int3   
>   2c:	cc                   	int3   
>   2d:	cc                   	int3   
>   2e:	cc                   	int3   
>   2f:	65 8b 15 31 2f a4 77 	mov    %gs:0x77a42f31(%rip),%edx       
>  # 0x77a42f67
>   36:	89 d2                	mov    %edx,%edx
>   38:	48 8b 05 d0 a1 0c 01 	mov    0x10ca1d0(%rip),%rax        #
> 0x10ca20f
>   3f:	48                   	rex.W
> 
> Code starting with the faulting instruction
> ===========================================
>    0:	c3                   	retq   
>    1:	cc                   	int3   
>    2:	cc                   	int3   
>    3:	cc                   	int3   
>    4:	cc                   	int3   
>    5:	65 8b 15 31 2f a4 77 	mov    %gs:0x77a42f31(%rip),%edx       
>  # 0x77a42f3d
>    c:	89 d2                	mov    %edx,%edx
>    e:	48 8b 05 d0 a1 0c 01 	mov    0x10ca1d0(%rip),%rax        #
> 0x10ca1e5
>   15:	48                   	rex.W
> [   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
> [   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
> ffff8a5febd56f80
> [   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
> ffff8a5febd5dd00
> [   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
> 0000000000000000
> [   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
> 0000000000000000
> [   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [   61.414183] ? mwait_idle+0x70/0x70 
> [   61.414805] ? mwait_idle+0x70/0x70 
> [   61.415592] default_idle_call+0x2a/0xa0 
> [   61.416216] do_idle+0x1e8/0x250 
> [   61.416722] cpu_startup_entry+0x14/0x20 
> [   61.417347] secondary_startup_64_no_verify+0xc2/0xcb 
> [   61.418144] Modules linked in:
> [   61.418622] ---[ end trace 3741c3e580a52bbd ]---
> [   61.419399] RIP: skb_panic+0x43/0x45 
> [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
> ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
> ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
> All code
> ========
>    0:	4f 70 50             	rex.WRXB jo 0x53
>    3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
>    9:	50                   	push   %rax
>    a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
>   10:	50                   	push   %rax
>   11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
>   17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
>   1e:	48 c7 c7 18 f1 cf 88 	mov    $0xffffffff88cff118,%rdi
>   25:	e8 6a 43 fb ff       	callq  0xfffffffffffb4394
>   2a:*	0f 0b                	ud2    		<-- trapping
> instruction
>   2c:	48 8b 14 24          	mov    (%rsp),%rdx
>   30:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>   37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
>   3c:	48                   	rex.W
>   3d:	c7                   	.byte 0xc7
>   3e:	c6                   	(bad)  
>   3f:	60                   	(bad)  
> 
> Code starting with the faulting instruction
> ===========================================
>    0:	0f 0b                	ud2    
>    2:	48 8b 14 24          	mov    (%rsp),%rdx
>    6:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
>    d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
>   12:	48                   	rex.W
>   13:	c7                   	.byte 0xc7
>   14:	c6                   	(bad)  
>   15:	60                   	(bad)  
> [   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
> [   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
> 00000000ffffdfff
> [   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
> 0000000000000000
> [   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
> 0000000000009ffb
> [   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
> ffff8a5ec7461200
> [   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
> 0000000000000eb2
> [   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
> knlGS:0000000000000000
> 
> Regards,
> Corentin

Don't see anything obvious.. could be a net stack change.
Any chance of a bisect?

-- 
MST


^ permalink raw reply	[flat|nested] 33+ messages in thread

* virtio-net: kernel panic in virtio_net.c
@ 2021-10-07 12:04 Corentin Noël
  2021-10-07 13:10 ` Michael S. Tsirkin
                   ` (2 more replies)
  0 siblings, 3 replies; 33+ messages in thread
From: Corentin Noël @ 2021-10-07 12:04 UTC (permalink / raw)
  To: stable
  Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions,
	Eric Dumazet, Xuan Zhuo

I've been experiencing crashes with 5.14-rc1 and above that do not
occur with 5.13,

here is the crash trace:

[   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[   61.369192] kernel BUG at net/core/skbuff.c:111!
[   61.372840] invalid opcode: 0000 [#1] SMP PTI
[   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 
[   61.377222] RIP: 0010:skb_panic+0x43/0x45
[   61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   61.393635] Call Trace:
[   61.394127]  <IRQ>
[   61.394488]  skb_put.cold+0x10/0x10
[   61.395095]  page_to_skb+0xf7/0x410
[   61.395689]  receive_buf+0x81/0x1660
[   61.396228]  ? netif_receive_skb_list_internal+0x1ad/0x2b0
[   61.397180]  ? napi_gro_flush+0x97/0xe0
[   61.397896]  ? detach_buf_split+0x67/0x120
[   61.398573]  virtnet_poll+0x2cf/0x420
[   61.399197]  __napi_poll+0x25/0x150
[   61.399764]  net_rx_action+0x22f/0x280
[   61.400394]  __do_softirq+0xba/0x257
[   61.401012]  irq_exit_rcu+0x8e/0xb0
[   61.401618]  common_interrupt+0x7b/0xa0
[   61.402270]  </IRQ>
[   61.402620]  asm_common_interrupt+0x1e/0x40
[   61.403302] RIP: 0010:default_idle+0xb/0x10
[   61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
[   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[   61.414183]  ? mwait_idle+0x70/0x70
[   61.414805]  ? mwait_idle+0x70/0x70
[   61.415592]  default_idle_call+0x2a/0xa0
[   61.416216]  do_idle+0x1e8/0x250
[   61.416722]  cpu_startup_entry+0x14/0x20
[   61.417347]  secondary_startup_64_no_verify+0xc2/0xcb
[   61.418144] Modules linked in:
[   61.418622] ---[ end trace 3741c3e580a52bbd ]---
[   61.419399] RIP: 0010:skb_panic+0x43/0x45
[   61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[   61.431048] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[   61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   61.435799] Kernel panic - not syncing: Fatal exception in interrupt
[   61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Here is my kernel config: 
https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config


here is the decoded trace:

[   61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[   61.369192] kernel BUG at net/core/skbuff.c:111!
[   61.372840] invalid opcode: 0000 [#1] SMP PTI
[   61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[   61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
[   61.377222] RIP: skb_panic+0x43/0x45 
[ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
   0:	4f 70 50             	rex.WRXB jo 0x53
   3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
   9:	50                   	push   %rax
   a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
  10:	50                   	push   %rax
  11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
  17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
  1e:	48 c7 c7 18 f1 cf 88 	mov    $0xffffffff88cff118,%rdi
  25:	e8 6a 43 fb ff       	callq  0xfffffffffffb4394
  2a:*	0f 0b                	ud2    		<-- trapping
instruction
  2c:	48 8b 14 24          	mov    (%rsp),%rdx
  30:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
  37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
  3c:	48                   	rex.W
  3d:	c7                   	.byte 0xc7
  3e:	c6                   	(bad)  
  3f:	60                   	(bad)  

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	48 8b 14 24          	mov    (%rsp),%rdx
   6:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
   d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
  12:	48                   	rex.W
  13:	c7                   	.byte 0xc7
  14:	c6                   	(bad)  
  15:	60                   	(bad)  
[   61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[   61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[   61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[   61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[   61.386825] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[   61.388055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[   61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   61.393635] Call Trace:
[   61.394127]  <IRQ>
[   61.394488] skb_put.cold+0x10/0x10 
[   61.395095] page_to_skb+0xf7/0x410 
[   61.395689] receive_buf+0x81/0x1660 
[   61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 
[   61.397180] ? napi_gro_flush+0x97/0xe0 
[   61.397896] ? detach_buf_split+0x67/0x120 
[   61.398573] virtnet_poll+0x2cf/0x420 
[   61.399197] __napi_poll+0x25/0x150 
[   61.399764] net_rx_action+0x22f/0x280 
[   61.400394] __do_softirq+0xba/0x257 
[   61.401012] irq_exit_rcu+0x8e/0xb0 
[   61.401618] common_interrupt+0x7b/0xa0 
[   61.402270]  </IRQ>
[   61.402620] asm_common_interrupt+0x1e/0x40 
[   61.403302] RIP: default_idle+0xb/0x10 
[ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
All code
========
   0:	8b 04 25 00 6d 01 00 	mov    0x16d00,%eax
   7:	f0 80 60 02 df       	lock andb $0xdf,0x2(%rax)
   c:	c3                   	retq   
   d:	0f ae f0             	mfence 
  10:	0f ae 38             	clflush (%rax)
  13:	0f ae f0             	mfence 
  16:	eb b9                	jmp    0xffffffffffffffd1
  18:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)
  1f:	eb 07                	jmp    0x28
  21:	0f 00 2d df 3e 44 00 	verw   0x443edf(%rip)        # 0x443f07
  28:	fb                   	sti    
  29:	f4                   	hlt    
  2a:*	c3                   	retq   		<-- trapping
instruction
  2b:	cc                   	int3   
  2c:	cc                   	int3   
  2d:	cc                   	int3   
  2e:	cc                   	int3   
  2f:	65 8b 15 31 2f a4 77 	mov    %gs:0x77a42f31(%rip),%edx       
 # 0x77a42f67
  36:	89 d2                	mov    %edx,%edx
  38:	48 8b 05 d0 a1 0c 01 	mov    0x10ca1d0(%rip),%rax        #
0x10ca20f
  3f:	48                   	rex.W

Code starting with the faulting instruction
===========================================
   0:	c3                   	retq   
   1:	cc                   	int3   
   2:	cc                   	int3   
   3:	cc                   	int3   
   4:	cc                   	int3   
   5:	65 8b 15 31 2f a4 77 	mov    %gs:0x77a42f31(%rip),%edx       
 # 0x77a42f3d
   c:	89 d2                	mov    %edx,%edx
   e:	48 8b 05 d0 a1 0c 01 	mov    0x10ca1d0(%rip),%rax        #
0x10ca1e5
  15:	48                   	rex.W
[   61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[   61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[   61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[   61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[   61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[   61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[   61.414183] ? mwait_idle+0x70/0x70 
[   61.414805] ? mwait_idle+0x70/0x70 
[   61.415592] default_idle_call+0x2a/0xa0 
[   61.416216] do_idle+0x1e8/0x250 
[   61.416722] cpu_startup_entry+0x14/0x20 
[   61.417347] secondary_startup_64_no_verify+0xc2/0xcb 
[   61.418144] Modules linked in:
[   61.418622] ---[ end trace 3741c3e580a52bbd ]---
[   61.419399] RIP: skb_panic+0x43/0x45 
[ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
   0:	4f 70 50             	rex.WRXB jo 0x53
   3:	8b 87 bc 00 00 00    	mov    0xbc(%rdi),%eax
   9:	50                   	push   %rax
   a:	8b 87 b8 00 00 00    	mov    0xb8(%rdi),%eax
  10:	50                   	push   %rax
  11:	ff b7 c8 00 00 00    	pushq  0xc8(%rdi)
  17:	4c 8b 8f c0 00 00 00 	mov    0xc0(%rdi),%r9
  1e:	48 c7 c7 18 f1 cf 88 	mov    $0xffffffff88cff118,%rdi
  25:	e8 6a 43 fb ff       	callq  0xfffffffffffb4394
  2a:*	0f 0b                	ud2    		<-- trapping
instruction
  2c:	48 8b 14 24          	mov    (%rsp),%rdx
  30:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
  37:	e8 ab ff ff ff       	callq  0xffffffffffffffe7
  3c:	48                   	rex.W
  3d:	c7                   	.byte 0xc7
  3e:	c6                   	(bad)  
  3f:	60                   	(bad)  

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	48 8b 14 24          	mov    (%rsp),%rdx
   6:	48 c7 c1 20 35 b1 88 	mov    $0xffffffff88b13520,%rcx
   d:	e8 ab ff ff ff       	callq  0xffffffffffffffbd
  12:	48                   	rex.W
  13:	c7                   	.byte 0xc7
  14:	c6                   	(bad)  
  15:	60                   	(bad)  
[   61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[   61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[   61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[   61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[   61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[   61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[   61.429799] FS:  0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000

Regards,
Corentin


^ permalink raw reply	[flat|nested] 33+ messages in thread

end of thread, other threads:[~2021-10-12 12:44 UTC | newest]

Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-01 16:06 virtio-net: kernel panic in virtio_net.c Corentin Noël
2021-06-01 17:07 ` Greg KH
2021-06-01 17:09   ` Corentin Noël
2021-06-01 17:47     ` Eric Dumazet
2021-06-02  2:01       ` Xuan Zhuo
2021-06-02 17:56         ` Corentin Noël
2021-06-02 17:54       ` Corentin Noël
2021-06-03  2:44         ` Xuan Zhuo
2021-06-03  8:57           ` Corentin Noël
2021-06-08 12:17             ` Greg KH
2021-06-09  1:48               ` Xuan Zhuo
2021-06-09  4:50                 ` Greg KH
2021-06-09  6:08                   ` Xuan Zhuo
2021-06-09  6:24                     ` Greg KH
2021-06-09  7:51                       ` Xuan Zhuo
2021-06-09  8:03                         ` Greg KH
2021-06-09  8:08                           ` Xuan Zhuo
2021-10-07 12:04 Corentin Noël
2021-10-07 13:10 ` Michael S. Tsirkin
2021-10-07 13:51   ` Eric Dumazet
2021-10-07 14:02     ` Corentin Noël
2021-10-07 14:13       ` Greg KH
2021-10-07 15:06 ` Xuan Zhuo
2021-10-07 15:25   ` Greg KH
2021-10-07 16:17     ` Xuan Zhuo
2021-10-08  8:06       ` Greg KH
2021-10-08 10:02         ` Michael S. Tsirkin
2021-10-08 12:21           ` Corentin Noël
2021-10-08 16:27         ` Xuan Zhuo
2021-10-09  5:19           ` Greg KH
2021-10-09  9:31             ` Xuan Zhuo
2021-10-08  7:59 ` Thorsten Leemhuis
2021-10-12 12:44   ` Thorsten Leemhuis

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).