* virtio-net: kernel panic in virtio_net.c @ 2021-06-01 16:06 Corentin Noël 2021-06-01 17:07 ` Greg KH 0 siblings, 1 reply; 33+ messages in thread From: Corentin Noël @ 2021-06-01 16:06 UTC (permalink / raw) To: stable Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo I've been experiencing crashes with 5.13 that do not occur with 5.12, here is the crash trace: [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762 put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 end:0xec0 dev:<NULL> [ 47.716267] kernel BUG at net/core/skbuff.c:110! [ 47.717197] invalid opcode: 0000 [#1] SMP PTI [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: 0000000000009ffb [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff9e1e1e95b300 [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: 0000000000000eb2 [ 47.732541] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) knlGS:0000000000000000 [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: 0000000000370ee0 [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 47.738318] Call Trace: [ 47.738812] skb_put.cold+0x10/0x10 [ 47.739450] page_to_skb+0xe4/0x400 [ 47.740072] receive_buf+0x86/0x1660 [ 47.740693] ? inet_gro_receive+0x54/0x2c0 [ 47.741279] ? dev_gro_receive+0x194/0x6a0 [ 47.741846] virtnet_poll+0x2b8/0x3c0 [ 47.742357] __napi_poll+0x25/0x150 [ 47.742844] net_rx_action+0x22f/0x280 [ 47.743388] __do_softirq+0xba/0x264 [ 47.743947] irq_exit_rcu+0x90/0xb0 [ 47.744435] common_interrupt+0x40/0xa0 [ 47.744978] ? asm_common_interrupt+0x8/0x40 [ 47.745582] asm_common_interrupt+0x1e/0x40 [ 47.746182] RIP: 0033:0x7f3a7a276ed4 [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8 c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: ffffffffffffffff [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: 00007f3a7c0575a0 [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: 00007f3a8c210354 [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: 00007f3a8c210340 [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: 00007f3a8c21033c [ 47.755354] Modules linked in: [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: 0000000000009ffb [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff9e1e1e95b300 [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: 0000000000000eb2 [ 47.766261] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) knlGS:0000000000000000 [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: 0000000000370ee0 [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 47.771339] Kernel panic - not syncing: Fatal exception in interrupt [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) I've been able to bisect the issue a little bit and the issue disappeared after reverting the 4 following commits: * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 * af39c8f72301b268ad8b04bae646b6025918b82b * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 * f80bd740cb7c954791279590b2e810ba6c214e52 Here is my kernel config: https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config Regards, Corentin ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 16:06 virtio-net: kernel panic in virtio_net.c Corentin Noël @ 2021-06-01 17:07 ` Greg KH 2021-06-01 17:09 ` Corentin Noël 0 siblings, 1 reply; 33+ messages in thread From: Greg KH @ 2021-06-01 17:07 UTC (permalink / raw) To: Corentin Noël Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > I've been experiencing crashes with 5.13 that do not occur with 5.12, > here is the crash trace: > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762 > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > 0000000000009ffb > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff9e1e1e95b300 > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > 0000000000000eb2 > [ 47.732541] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) > knlGS:0000000000000000 > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > 0000000000370ee0 > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 47.738318] Call Trace: > [ 47.738812] skb_put.cold+0x10/0x10 > [ 47.739450] page_to_skb+0xe4/0x400 > [ 47.740072] receive_buf+0x86/0x1660 > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > [ 47.742357] __napi_poll+0x25/0x150 > [ 47.742844] net_rx_action+0x22f/0x280 > [ 47.743388] __do_softirq+0xba/0x264 > [ 47.743947] irq_exit_rcu+0x90/0xb0 > [ 47.744435] common_interrupt+0x40/0xa0 > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > [ 47.745582] asm_common_interrupt+0x1e/0x40 > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8 > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > ffffffffffffffff > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > 00007f3a7c0575a0 > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > 00007f3a8c210354 > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > 00007f3a8c210340 > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > 00007f3a8c21033c > [ 47.755354] Modules linked in: > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > 0000000000009ffb > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff9e1e1e95b300 > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > 0000000000000eb2 > [ 47.766261] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) > knlGS:0000000000000000 > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > 0000000000370ee0 > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 47.771339] Kernel panic - not syncing: Fatal exception in interrupt > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > I've been able to bisect the issue a little bit and the issue > disappeared after reverting the 4 following commits: > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > * af39c8f72301b268ad8b04bae646b6025918b82b > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > * f80bd740cb7c954791279590b2e810ba6c214e52 > > Here is my kernel config: > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config Do you have the same problem with 5.13-rc4? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 17:07 ` Greg KH @ 2021-06-01 17:09 ` Corentin Noël 2021-06-01 17:47 ` Eric Dumazet 0 siblings, 1 reply; 33+ messages in thread From: Corentin Noël @ 2021-06-01 17:09 UTC (permalink / raw) To: Greg KH Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > I've been experiencing crashes with 5.13 that do not occur with > > 5.12, > > here is the crash trace: > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > len:3762 > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > e0 > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > 0000000000009ffb > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff9e1e1e95b300 > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > 0000000000000eb2 > > [ 47.732541] FS: 00007f3a82b53700(0000) > > GS:ffff9e1f2bd00000(0000) > > knlGS:0000000000000000 > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > 0000000000370ee0 > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 47.738318] Call Trace: > > [ 47.738812] skb_put.cold+0x10/0x10 > > [ 47.739450] page_to_skb+0xe4/0x400 > > [ 47.740072] receive_buf+0x86/0x1660 > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > [ 47.742357] __napi_poll+0x25/0x150 > > [ 47.742844] net_rx_action+0x22f/0x280 > > [ 47.743388] __do_softirq+0xba/0x264 > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > [ 47.744435] common_interrupt+0x40/0xa0 > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc > > 54 c8 > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 > > d5 fa > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 > > cb > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > > ffffffffffffffff > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > > 00007f3a7c0575a0 > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > > 00007f3a8c210354 > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > > 00007f3a8c210340 > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > > 00007f3a8c21033c > > [ 47.755354] Modules linked in: > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > e0 > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > 0000000000009ffb > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff9e1e1e95b300 > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > 0000000000000eb2 > > [ 47.766261] FS: 00007f3a82b53700(0000) > > GS:ffff9e1f2bd00000(0000) > > knlGS:0000000000000000 > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > 0000000000370ee0 > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > interrupt > > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > I've been able to bisect the issue a little bit and the issue > > disappeared after reverting the 4 following commits: > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > * af39c8f72301b268ad8b04bae646b6025918b82b > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > Here is my kernel config: > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > Do you have the same problem with 5.13-rc4? > > thanks, > > greg k-h Yes I tried with rc2, rc3 and rc4 resulting to the same panic. Thanks, Corentin ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 17:09 ` Corentin Noël @ 2021-06-01 17:47 ` Eric Dumazet 2021-06-02 2:01 ` Xuan Zhuo 2021-06-02 17:54 ` Corentin Noël 0 siblings, 2 replies; 33+ messages in thread From: Eric Dumazet @ 2021-06-01 17:47 UTC (permalink / raw) To: Corentin Noël Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Xuan Zhuo On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël <corentin.noel@collabora.com> wrote: > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > > I've been experiencing crashes with 5.13 that do not occur with > > > 5.12, > > > here is the crash trace: > > > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > > len:3762 > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > > e0 > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > 0000000000009ffb > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff9e1e1e95b300 > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > 0000000000000eb2 > > > [ 47.732541] FS: 00007f3a82b53700(0000) > > > GS:ffff9e1f2bd00000(0000) > > > knlGS:0000000000000000 > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > 0000000000370ee0 > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 47.738318] Call Trace: > > > [ 47.738812] skb_put.cold+0x10/0x10 > > > [ 47.739450] page_to_skb+0xe4/0x400 > > > [ 47.740072] receive_buf+0x86/0x1660 > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > > [ 47.742357] __napi_poll+0x25/0x150 > > > [ 47.742844] net_rx_action+0x22f/0x280 > > > [ 47.743388] __do_softirq+0xba/0x264 > > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > > [ 47.744435] common_interrupt+0x40/0xa0 > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc > > > 54 c8 > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 > > > d5 fa > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 > > > cb > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > > > ffffffffffffffff > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > > > 00007f3a7c0575a0 > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > > > 00007f3a8c210354 > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > > > 00007f3a8c210340 > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > > > 00007f3a8c21033c > > > [ 47.755354] Modules linked in: > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > > e0 > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > 0000000000009ffb > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff9e1e1e95b300 > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > 0000000000000eb2 > > > [ 47.766261] FS: 00007f3a82b53700(0000) > > > GS:ffff9e1f2bd00000(0000) > > > knlGS:0000000000000000 > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > 0000000000370ee0 > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > > interrupt > > > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > I've been able to bisect the issue a little bit and the issue > > > disappeared after reverting the 4 following commits: > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > > * af39c8f72301b268ad8b04bae646b6025918b82b > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > > > Here is my kernel config: > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > > > Do you have the same problem with 5.13-rc4? > > > > thanks, > > > > greg k-h > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic. > > Thanks, > Could you provide a stack trace with file names and line numbers ? (ie use scripts/decode_stacktrace.sh ) Thanks. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 17:47 ` Eric Dumazet @ 2021-06-02 2:01 ` Xuan Zhuo 2021-06-02 17:56 ` Corentin Noël 2021-06-02 17:54 ` Corentin Noël 1 sibling, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-06-02 2:01 UTC (permalink / raw) To: Eric Dumazet Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Corentin Noël On Tue, 1 Jun 2021 19:47:44 +0200, Eric Dumazet <edumazet@google.com> wrote: > On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël > <corentin.noel@collabora.com> wrote: > > > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > > > I've been experiencing crashes with 5.13 that do not occur with > > > > 5.12, > > > > here is the crash trace: > > > > > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > > > len:3762 > > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > > > > end:0xec0 dev:<NULL> > > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- > > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > > 00 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > > > e0 > > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > > 0000000000009ffb > > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff9e1e1e95b300 > > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > > 0000000000000eb2 > > > > [ 47.732541] FS: 00007f3a82b53700(0000) > > > > GS:ffff9e1f2bd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > > 0000000000370ee0 > > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 47.738318] Call Trace: > > > > [ 47.738812] skb_put.cold+0x10/0x10 > > > > [ 47.739450] page_to_skb+0xe4/0x400 > > > > [ 47.740072] receive_buf+0x86/0x1660 > > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > > > [ 47.742357] __napi_poll+0x25/0x150 > > > > [ 47.742844] net_rx_action+0x22f/0x280 > > > > [ 47.743388] __do_softirq+0xba/0x264 > > > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > > > [ 47.744435] common_interrupt+0x40/0xa0 > > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc > > > > 54 c8 > > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 > > > > d5 fa > > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 > > > > cb > > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > > > > ffffffffffffffff > > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > > > > 00007f3a7c0575a0 > > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > > > > 00007f3a8c210354 > > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > > > > 00007f3a8c210340 > > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > > > > 00007f3a8c21033c > > > > [ 47.755354] Modules linked in: > > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > > 00 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > > > e0 > > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > > 0000000000009ffb > > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff9e1e1e95b300 > > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > > 0000000000000eb2 > > > > [ 47.766261] FS: 00007f3a82b53700(0000) > > > > GS:ffff9e1f2bd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > > 0000000000370ee0 > > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > > > interrupt > > > > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > > I've been able to bisect the issue a little bit and the issue > > > > disappeared after reverting the 4 following commits: > > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > > > * af39c8f72301b268ad8b04bae646b6025918b82b > > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > > > > > Here is my kernel config: > > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config Do you have XDP running? If so, you can try it https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5c37711d9f27bdc83fd5980446be7f4aa2106230 Thanks. > > > > > > Do you have the same problem with 5.13-rc4? > > > > > > thanks, > > > > > > greg k-h > > > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic. > > > > Thanks, > > > > > Could you provide a stack trace with file names and line numbers ? > > (ie use scripts/decode_stacktrace.sh ) > > Thanks. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-02 2:01 ` Xuan Zhuo @ 2021-06-02 17:56 ` Corentin Noël 0 siblings, 0 replies; 33+ messages in thread From: Corentin Noël @ 2021-06-02 17:56 UTC (permalink / raw) To: Xuan Zhuo, Eric Dumazet Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions Le mercredi 02 juin 2021 à 10:01 +0800, Xuan Zhuo a écrit : > On Tue, 1 Jun 2021 19:47:44 +0200, Eric Dumazet <edumazet@google.com> > wrote: > > On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël > > <corentin.noel@collabora.com> wrote: > > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > > > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > > > > I've been experiencing crashes with 5.13 that do not occur > > > > > with > > > > > 5.12, > > > > > here is the crash trace: > > > > > > > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > > > > len:3762 > > > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 > > > > > tail:0xec2 > > > > > end:0xec0 dev:<NULL> > > > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted > > > > > 5.13.0- > > > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 > > > > > 00 00 > > > > > 00 50 > > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 > > > > > e8 7f > > > > > 4c fb > > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > > c7 c6 > > > > > e0 > > > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 > > > > > RCX: > > > > > 00000000ffffdfff > > > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea > > > > > RDI: > > > > > 0000000000000000 > > > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 > > > > > R09: > > > > > 0000000000009ffb > > > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff > > > > > R12: > > > > > ffff9e1e1e95b300 > > > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 > > > > > R15: > > > > > 0000000000000eb2 > > > > > [ 47.732541] FS: 00007f3a82b53700(0000) > > > > > GS:ffff9e1f2bd00000(0000) > > > > > knlGS:0000000000000000 > > > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > > 0000000080050033 > > > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 > > > > > CR4: > > > > > 0000000000370ee0 > > > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 > > > > > DR2: > > > > > 0000000000000000 > > > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 > > > > > DR7: > > > > > 0000000000000400 > > > > > [ 47.738318] Call Trace: > > > > > [ 47.738812] skb_put.cold+0x10/0x10 > > > > > [ 47.739450] page_to_skb+0xe4/0x400 > > > > > [ 47.740072] receive_buf+0x86/0x1660 > > > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > > > > [ 47.742357] __napi_poll+0x25/0x150 > > > > > [ 47.742844] net_rx_action+0x22f/0x280 > > > > > [ 47.743388] __do_softirq+0xba/0x264 > > > > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > > > > [ 47.744435] common_interrupt+0x40/0xa0 > > > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > > > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 > > > > > c5 bc > > > > > 54 c8 > > > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 > > > > > c0 c5 > > > > > d5 fa > > > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 > > > > > 55 b8 > > > > > cb > > > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff > > > > > RCX: > > > > > ffffffffffffffff > > > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c > > > > > RDI: > > > > > 00007f3a7c0575a0 > > > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 > > > > > R09: > > > > > 00007f3a8c210354 > > > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef > > > > > R12: > > > > > 00007f3a8c210340 > > > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 > > > > > R15: > > > > > 00007f3a8c21033c > > > > > [ 47.755354] Modules linked in: > > > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 > > > > > 00 00 > > > > > 00 50 > > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 > > > > > e8 7f > > > > > 4c fb > > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > > c7 c6 > > > > > e0 > > > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 > > > > > RCX: > > > > > 00000000ffffdfff > > > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea > > > > > RDI: > > > > > 0000000000000000 > > > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 > > > > > R09: > > > > > 0000000000009ffb > > > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff > > > > > R12: > > > > > ffff9e1e1e95b300 > > > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 > > > > > R15: > > > > > 0000000000000eb2 > > > > > [ 47.766261] FS: 00007f3a82b53700(0000) > > > > > GS:ffff9e1f2bd00000(0000) > > > > > knlGS:0000000000000000 > > > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > > 0000000080050033 > > > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 > > > > > CR4: > > > > > 0000000000370ee0 > > > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 > > > > > DR2: > > > > > 0000000000000000 > > > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 > > > > > DR7: > > > > > 0000000000000400 > > > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > > > > interrupt > > > > > [ 47.772814] Kernel Offset: 0x35c00000 from > > > > > 0xffffffff81000000 > > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > > > > I've been able to bisect the issue a little bit and the issue > > > > > disappeared after reverting the 4 following commits: > > > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > > > > * af39c8f72301b268ad8b04bae646b6025918b82b > > > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > > > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > > > > > > > Here is my kernel config: > > > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > Do you have XDP running? If so, you can try it > > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5c37711d9f27bdc83fd5980446be7f4aa2106230 I applied this patch in top of 5.13-rc4 and it resulted in the same crash > > Thanks. > > > > > Do you have the same problem with 5.13-rc4? > > > > > > > > thanks, > > > > > > > > greg k-h > > > > > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic. > > > > > > Thanks, > > > > > > > Could you provide a stack trace with file names and line numbers ? > > > > (ie use scripts/decode_stacktrace.sh ) > > > > Thanks. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 17:47 ` Eric Dumazet 2021-06-02 2:01 ` Xuan Zhuo @ 2021-06-02 17:54 ` Corentin Noël 2021-06-03 2:44 ` Xuan Zhuo 1 sibling, 1 reply; 33+ messages in thread From: Corentin Noël @ 2021-06-02 17:54 UTC (permalink / raw) To: Eric Dumazet Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Xuan Zhuo Le mardi 01 juin 2021 à 19:47 +0200, Eric Dumazet a écrit : > On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël > <corentin.noel@collabora.com> wrote: > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > > > I've been experiencing crashes with 5.13 that do not occur with > > > > 5.12, > > > > here is the crash trace: > > > > > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > > > len:3762 > > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > > > > end:0xec0 dev:<NULL> > > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted > > > > 5.13.0- > > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 > > > > 00 > > > > 00 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 > > > > 7f > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > c7 c6 > > > > e0 > > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > > 0000000000009ffb > > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff9e1e1e95b300 > > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > > 0000000000000eb2 > > > > [ 47.732541] FS: 00007f3a82b53700(0000) > > > > GS:ffff9e1f2bd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > 0000000080050033 > > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > > 0000000000370ee0 > > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 47.738318] Call Trace: > > > > [ 47.738812] skb_put.cold+0x10/0x10 > > > > [ 47.739450] page_to_skb+0xe4/0x400 > > > > [ 47.740072] receive_buf+0x86/0x1660 > > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > > > [ 47.742357] __napi_poll+0x25/0x150 > > > > [ 47.742844] net_rx_action+0x22f/0x280 > > > > [ 47.743388] __do_softirq+0xba/0x264 > > > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > > > [ 47.744435] common_interrupt+0x40/0xa0 > > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 > > > > bc > > > > 54 c8 > > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 > > > > c5 > > > > d5 fa > > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 > > > > 55 b8 > > > > cb > > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > > > > ffffffffffffffff > > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > > > > 00007f3a7c0575a0 > > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > > > > 00007f3a8c210354 > > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > > > > 00007f3a8c210340 > > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > > > > 00007f3a8c21033c > > > > [ 47.755354] Modules linked in: > > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 > > > > 00 > > > > 00 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 > > > > 7f > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > c7 c6 > > > > e0 > > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > > 0000000000009ffb > > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff9e1e1e95b300 > > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > > 0000000000000eb2 > > > > [ 47.766261] FS: 00007f3a82b53700(0000) > > > > GS:ffff9e1f2bd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > 0000000080050033 > > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > > 0000000000370ee0 > > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > > > interrupt > > > > [ 47.772814] Kernel Offset: 0x35c00000 from > > > > 0xffffffff81000000 > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > > I've been able to bisect the issue a little bit and the issue > > > > disappeared after reverting the 4 following commits: > > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > > > * af39c8f72301b268ad8b04bae646b6025918b82b > > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > > > > > Here is my kernel config: > > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > > > > > Do you have the same problem with 5.13-rc4? > > > > > > thanks, > > > > > > greg k-h > > > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic. > > > > Thanks, > > > > Could you provide a stack trace with file names and line numbers ? > > (ie use scripts/decode_stacktrace.sh ) > > Thanks. Sure, here is the decoded trace: [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 len:3762 put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 end:0xec0 dev:<NULL> [ 44.525254] kernel BUG at net/core/skbuff.c:110! [ 44.525910] invalid opcode: 0000 [#1] SMP PTI [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60 All code ======== 0: 4f 70 50 rex.WRXB jo 0x53 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax 9: 50 push %rax a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax 10: 50 push %rax 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 8b 14 24 mov (%rsp),%rdx 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx 37: e8 ab ff ff ff callq 0xffffffffffffffe7 3c: 48 rex.W 3d: c7 .byte 0xc7 3e: c6 (bad) 3f: 60 (bad) Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 48 8b 14 24 mov (%rsp),%rdx 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx d: e8 ab ff ff ff callq 0xffffffffffffffbd 12: 48 rex.W 13: c7 .byte 0xc7 14: c6 (bad) 15: 60 (bad) [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: 0000000000009ffb [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff979ad2aa5600 [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: 0000000000000eb2 [ 44.539300] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000) knlGS:0000000000000000 [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: 0000000000370ee0 [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 44.544063] Call Trace: [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator 1) net/core/skbuff.c:5252 (discriminator 1)) [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 drivers/net/virtio_net.c:1131) [ 44.545870] ? netif_receive_skb_list_internal (net/core/dev.c:5714) [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 drivers/net/virtio_net.c:1525) [ 44.548251] __napi_poll (net/core/dev.c:6985) [ 44.548744] net_rx_action (net/core/dev.c:7054 net/core/dev.c:7139) [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:560) [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 kernel/softirq.c:637 kernel/softirq.c:649) [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 13)) [ 44.551991] ? asm_common_interrupt (./arch/x86/include/asm/idtentry.h:638) [ 44.552654] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:638) [ 44.553276] RIP: 0033:0x7fdb981a82e4 [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 c4 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 f9 6c f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 f9 All code ======== 0: d2 48 63 rorb %cl,0x63(%rax) 3: f6 c4 41 test $0x41,%ah 6: 7a 6f jp 0x77 8: 0c 01 or $0x1,%al a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 1c: c4 c1 31 6a c2 vpunpckhdq %xmm10,%xmm9,%xmm0 21: c4 c1 19 6a d5 vpunpckhdq %xmm13,%xmm12,%xmm2 26: c5 f9 6c f2 vpunpcklqdq %xmm2,%xmm0,%xmm6 2a:* c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8 <-- trapping instruction 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 3e: c5 .byte 0xc5 3f: f9 stc Code starting with the faulting instruction =========================================== 0: c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0 f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 14: c5 .byte 0xc5 15: f9 stc [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: 0000000000122d40 [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: 000055d7049b9368 [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: 00007fdb5e544040 [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: 0000000000000000 [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: 0000000000000000 [ 44.561965] Modules linked in: [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60 All code ======== 0: 4f 70 50 rex.WRXB jo 0x53 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax 9: 50 push %rax a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax 10: 50 push %rax 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 8b 14 24 mov (%rsp),%rdx 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx 37: e8 ab ff ff ff callq 0xffffffffffffffe7 3c: 48 rex.W 3d: c7 .byte 0xc7 3e: c6 (bad) 3f: 60 (bad) Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 48 8b 14 24 mov (%rsp),%rdx 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx d: e8 ab ff ff ff callq 0xffffffffffffffbd 12: 48 rex.W 13: c7 .byte 0xc7 14: c6 (bad) 15: 60 (bad) [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: 0000000000009ffb [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff979ad2aa5600 [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: 0000000000000eb2 [ 44.571483] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000) knlGS:0000000000000000 [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: 0000000000370ee0 [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 44.576618] Kernel panic - not syncing: Fatal exception in interrupt [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-02 17:54 ` Corentin Noël @ 2021-06-03 2:44 ` Xuan Zhuo 2021-06-03 8:57 ` Corentin Noël 0 siblings, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-06-03 2:44 UTC (permalink / raw) To: Corentin Noël Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > Sure, here is the decoded trace: > > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 len:3762 > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 44.525254] kernel BUG at net/core/skbuff.c:110! > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > 0000000000009ffb > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff979ad2aa5600 > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > 0000000000000eb2 > [ 44.539300] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000) > knlGS:0000000000000000 > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > 0000000000370ee0 > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 44.544063] Call Trace: > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator 1) > net/core/skbuff.c:5252 (discriminator 1)) > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 > drivers/net/virtio_net.c:1131) > [ 44.545870] ? netif_receive_skb_list_internal (net/core/dev.c:5714) > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 > drivers/net/virtio_net.c:1525) > [ 44.548251] __napi_poll (net/core/dev.c:6985) > [ 44.548744] net_rx_action (net/core/dev.c:7054 net/core/dev.c:7139) > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 > kernel/softirq.c:560) > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 kernel/softirq.c:637 > kernel/softirq.c:649) > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 > (discriminator 13)) > [ 44.551991] ? asm_common_interrupt > (./arch/x86/include/asm/idtentry.h:638) > [ 44.552654] asm_common_interrupt > (./arch/x86/include/asm/idtentry.h:638) > [ 44.553276] RIP: 0033:0x7fdb981a82e4 > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 c4 > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 f9 6c > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 f9 > All code > ======== > 0: d2 48 63 rorb %cl,0x63(%rax) > 3: f6 c4 41 test $0x41,%ah > 6: 7a 6f jp 0x77 > 8: 0c 01 or $0x1,%al > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 > 1c: c4 c1 31 6a c2 vpunpckhdq %xmm10,%xmm9,%xmm0 > 21: c4 c1 19 6a d5 vpunpckhdq %xmm13,%xmm12,%xmm2 > 26: c5 f9 6c f2 vpunpcklqdq %xmm2,%xmm0,%xmm6 > 2a:* c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8 > <-- trapping instruction > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0 > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > 3e: c5 .byte 0xc5 > 3f: f9 stc > > Code starting with the faulting instruction > =========================================== > 0: c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8 > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0 > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > 14: c5 .byte 0xc5 > 15: f9 stc > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: > 0000000000122d40 > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: > 000055d7049b9368 > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: > 00007fdb5e544040 > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: > 0000000000000000 > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: > 0000000000000000 > [ 44.561965] Modules linked in: > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > 0000000000009ffb > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff979ad2aa5600 > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > 0000000000000eb2 > [ 44.571483] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000) > knlGS:0000000000000000 > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > 0000000000370ee0 > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 44.576618] Kernel panic - not syncing: Fatal exception in interrupt > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > Can you test this patch on the latest net branch? Thanks. diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index fa407eb8b457..78a01c71a17c 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi, * add_recvbuf_mergeable() + get_mergeable_buf_len() */ truesize = headroom ? PAGE_SIZE : truesize; - tailroom = truesize - len - headroom; + tailroom = truesize - len - headroom - (hdr_padded_len - hdr_len); buf = p - headroom; len -= hdr_len; ^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-03 2:44 ` Xuan Zhuo @ 2021-06-03 8:57 ` Corentin Noël 2021-06-08 12:17 ` Greg KH 0 siblings, 1 reply; 33+ messages in thread From: Corentin Noël @ 2021-06-03 8:57 UTC (permalink / raw) To: Xuan Zhuo Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit : > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël < > corentin.noel@collabora.com> wrote: > > Sure, here is the decoded trace: > > > > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 > > len:3762 > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 44.525254] kernel BUG at net/core/skbuff.c:110! > > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI > > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 > > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > %rdi > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > 2a:* 0f 0b ud2 <-- > > trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > 0000000000009ffb > > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff979ad2aa5600 > > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > 0000000000000eb2 > > [ 44.539300] FS: 00007fdb9cb11700(0000) > > GS:ffff979aebd00000(0000) > > knlGS:0000000000000000 > > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > 0000000000370ee0 > > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 44.544063] Call Trace: > > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator > > 1) > > net/core/skbuff.c:5252 (discriminator 1)) > > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) > > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 > > drivers/net/virtio_net.c:1131) > > [ 44.545870] ? netif_receive_skb_list_internal > > (net/core/dev.c:5714) > > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) > > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) > > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 > > drivers/net/virtio_net.c:1525) > > [ 44.548251] __napi_poll (net/core/dev.c:6985) > > [ 44.548744] net_rx_action (net/core/dev.c:7054 > > net/core/dev.c:7139) > > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 > > kernel/softirq.c:560) > > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 > > kernel/softirq.c:637 > > kernel/softirq.c:649) > > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 > > (discriminator 13)) > > [ 44.551991] ? asm_common_interrupt > > (./arch/x86/include/asm/idtentry.h:638) > > [ 44.552654] asm_common_interrupt > > (./arch/x86/include/asm/idtentry.h:638) > > [ 44.553276] RIP: 0033:0x7fdb981a82e4 > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 > > c4 > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 > > f9 6c > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 > > f9 > > All code > > ======== > > 0: d2 48 63 rorb %cl,0x63(%rax) > > 3: f6 c4 41 test $0x41,%ah > > 6: 7a 6f jp 0x77 > > 8: 0c 01 or $0x1,%al > > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 > > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 > > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 > > 1c: c4 c1 31 6a c2 vpunpckhdq > > %xmm10,%xmm9,%xmm0 > > 21: c4 c1 19 6a d5 vpunpckhdq > > %xmm13,%xmm12,%xmm2 > > 26: c5 f9 6c f2 vpunpcklqdq > > %xmm2,%xmm0,%xmm6 > > 2a:* c5 79 6d c2 vpunpckhqdq > > %xmm2,%xmm0,%xmm8 > > <-- trapping instruction > > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > m0 > > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > 3e: c5 .byte 0xc5 > > 3f: f9 stc > > > > Code starting with the faulting instruction > > =========================================== > > 0: c5 79 6d c2 vpunpckhqdq > > %xmm2,%xmm0,%xmm8 > > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > m0 > > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > 14: c5 .byte 0xc5 > > 15: f9 stc > > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 > > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: > > 0000000000122d40 > > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: > > 000055d7049b9368 > > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: > > 00007fdb5e544040 > > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: > > 0000000000000000 > > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: > > 0000000000000000 > > [ 44.561965] Modules linked in: > > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- > > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > %rdi > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > 2a:* 0f 0b ud2 <-- > > trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > 0000000000009ffb > > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff979ad2aa5600 > > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > 0000000000000eb2 > > [ 44.571483] FS: 00007fdb9cb11700(0000) > > GS:ffff979aebd00000(0000) > > knlGS:0000000000000000 > > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > 0000000000370ee0 > > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 44.576618] Kernel panic - not syncing: Fatal exception in > > interrupt > > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > Can you test this patch on the latest net branch? > > Thanks. > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index fa407eb8b457..78a01c71a17c 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct > virtnet_info *vi, > * add_recvbuf_mergeable() + get_mergeable_buf_len() > */ > truesize = headroom ? PAGE_SIZE : truesize; > - tailroom = truesize - len - headroom; > + tailroom = truesize - len - headroom - (hdr_padded_len - > hdr_len); > buf = p - headroom; > > len -= hdr_len; With this patch and the latest net branch I no longer get crashes. Thanks a lot for this, Corentin ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-03 8:57 ` Corentin Noël @ 2021-06-08 12:17 ` Greg KH 2021-06-09 1:48 ` Xuan Zhuo 0 siblings, 1 reply; 33+ messages in thread From: Greg KH @ 2021-06-08 12:17 UTC (permalink / raw) To: Corentin Noël Cc: Xuan Zhuo, stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet On Thu, Jun 03, 2021 at 10:57:52AM +0200, Corentin Noël wrote: > Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit : > > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël < > > corentin.noel@collabora.com> wrote: > > > Sure, here is the decoded trace: > > > > > > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 > > > len:3762 > > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 44.525254] kernel BUG at net/core/skbuff.c:110! > > > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI > > > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- > > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 > > > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > > 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > > 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > > %rdi > > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > > 0000000000009ffb > > > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff979ad2aa5600 > > > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > > 0000000000000eb2 > > > [ 44.539300] FS: 00007fdb9cb11700(0000) > > > GS:ffff979aebd00000(0000) > > > knlGS:0000000000000000 > > > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > > 0000000000370ee0 > > > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 44.544063] Call Trace: > > > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator > > > 1) > > > net/core/skbuff.c:5252 (discriminator 1)) > > > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) > > > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 > > > drivers/net/virtio_net.c:1131) > > > [ 44.545870] ? netif_receive_skb_list_internal > > > (net/core/dev.c:5714) > > > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) > > > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 > > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) > > > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 > > > drivers/net/virtio_net.c:1525) > > > [ 44.548251] __napi_poll (net/core/dev.c:6985) > > > [ 44.548744] net_rx_action (net/core/dev.c:7054 > > > net/core/dev.c:7139) > > > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 > > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 > > > kernel/softirq.c:560) > > > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 > > > kernel/softirq.c:637 > > > kernel/softirq.c:649) > > > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 > > > (discriminator 13)) > > > [ 44.551991] ? asm_common_interrupt > > > (./arch/x86/include/asm/idtentry.h:638) > > > [ 44.552654] asm_common_interrupt > > > (./arch/x86/include/asm/idtentry.h:638) > > > [ 44.553276] RIP: 0033:0x7fdb981a82e4 > > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 > > > c4 > > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 > > > f9 6c > > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 > > > f9 > > > All code > > > ======== > > > 0: d2 48 63 rorb %cl,0x63(%rax) > > > 3: f6 c4 41 test $0x41,%ah > > > 6: 7a 6f jp 0x77 > > > 8: 0c 01 or $0x1,%al > > > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 > > > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 > > > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 > > > 1c: c4 c1 31 6a c2 vpunpckhdq > > > %xmm10,%xmm9,%xmm0 > > > 21: c4 c1 19 6a d5 vpunpckhdq > > > %xmm13,%xmm12,%xmm2 > > > 26: c5 f9 6c f2 vpunpcklqdq > > > %xmm2,%xmm0,%xmm6 > > > 2a:* c5 79 6d c2 vpunpckhqdq > > > %xmm2,%xmm0,%xmm8 > > > <-- trapping instruction > > > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > > m0 > > > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > > 3e: c5 .byte 0xc5 > > > 3f: f9 stc > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: c5 79 6d c2 vpunpckhqdq > > > %xmm2,%xmm0,%xmm8 > > > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > > m0 > > > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > > 14: c5 .byte 0xc5 > > > 15: f9 stc > > > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 > > > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: > > > 0000000000122d40 > > > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: > > > 000055d7049b9368 > > > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: > > > 00007fdb5e544040 > > > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: > > > 0000000000000000 > > > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: > > > 0000000000000000 > > > [ 44.561965] Modules linked in: > > > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- > > > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > > 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > > 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > > %rdi > > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > > 0000000000009ffb > > > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff979ad2aa5600 > > > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > > 0000000000000eb2 > > > [ 44.571483] FS: 00007fdb9cb11700(0000) > > > GS:ffff979aebd00000(0000) > > > knlGS:0000000000000000 > > > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > > 0000000000370ee0 > > > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 44.576618] Kernel panic - not syncing: Fatal exception in > > > interrupt > > > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > Can you test this patch on the latest net branch? > > > > Thanks. > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > index fa407eb8b457..78a01c71a17c 100644 > > --- a/drivers/net/virtio_net.c > > +++ b/drivers/net/virtio_net.c > > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct > > virtnet_info *vi, > > * add_recvbuf_mergeable() + get_mergeable_buf_len() > > */ > > truesize = headroom ? PAGE_SIZE : truesize; > > - tailroom = truesize - len - headroom; > > + tailroom = truesize - len - headroom - (hdr_padded_len - > > hdr_len); > > buf = p - headroom; > > > > len -= hdr_len; > > With this patch and the latest net branch I no longer get crashes. Did this ever get properly submitted to the networking tree to get into 5.13-final? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-08 12:17 ` Greg KH @ 2021-06-09 1:48 ` Xuan Zhuo 2021-06-09 4:50 ` Greg KH 0 siblings, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-06-09 1:48 UTC (permalink / raw) To: Greg KH Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Tue, 8 Jun 2021 14:17:58 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > On Thu, Jun 03, 2021 at 10:57:52AM +0200, Corentin Noël wrote: > > Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit : > > > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël < > > > corentin.noel@collabora.com> wrote: > > > > Sure, here is the decoded trace: > > > > > > > > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 > > > > len:3762 > > > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 > > > > end:0xec0 dev:<NULL> > > > > [ 44.525254] kernel BUG at net/core/skbuff.c:110! > > > > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI > > > > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- > > > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 > > > > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 > > > > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > > > 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > > > 60 > > > > All code > > > > ======== > > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > > 9: 50 push %rax > > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > > 10: 50 push %rax > > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > > > %rdi > > > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > > > 2a:* 0f 0b ud2 <-- > > > > trapping > > > > instruction > > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > > %rcx > > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > > 3c: 48 rex.W > > > > 3d: c7 .byte 0xc7 > > > > 3e: c6 (bad) > > > > 3f: 60 (bad) > > > > > > > > Code starting with the faulting instruction > > > > =========================================== > > > > 0: 0f 0b ud2 > > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > > %rcx > > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > > 12: 48 rex.W > > > > 13: c7 .byte 0xc7 > > > > 14: c6 (bad) > > > > 15: 60 (bad) > > > > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > > > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > > > 0000000000009ffb > > > > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff979ad2aa5600 > > > > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > > > 0000000000000eb2 > > > > [ 44.539300] FS: 00007fdb9cb11700(0000) > > > > GS:ffff979aebd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > > > 0000000000370ee0 > > > > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 44.544063] Call Trace: > > > > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator > > > > 1) > > > > net/core/skbuff.c:5252 (discriminator 1)) > > > > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) > > > > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 > > > > drivers/net/virtio_net.c:1131) > > > > [ 44.545870] ? netif_receive_skb_list_internal > > > > (net/core/dev.c:5714) > > > > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) > > > > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 > > > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) > > > > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 > > > > drivers/net/virtio_net.c:1525) > > > > [ 44.548251] __napi_poll (net/core/dev.c:6985) > > > > [ 44.548744] net_rx_action (net/core/dev.c:7054 > > > > net/core/dev.c:7139) > > > > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 > > > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 > > > > kernel/softirq.c:560) > > > > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 > > > > kernel/softirq.c:637 > > > > kernel/softirq.c:649) > > > > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 > > > > (discriminator 13)) > > > > [ 44.551991] ? asm_common_interrupt > > > > (./arch/x86/include/asm/idtentry.h:638) > > > > [ 44.552654] asm_common_interrupt > > > > (./arch/x86/include/asm/idtentry.h:638) > > > > [ 44.553276] RIP: 0033:0x7fdb981a82e4 > > > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 > > > > c4 > > > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 > > > > f9 6c > > > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 > > > > f9 > > > > All code > > > > ======== > > > > 0: d2 48 63 rorb %cl,0x63(%rax) > > > > 3: f6 c4 41 test $0x41,%ah > > > > 6: 7a 6f jp 0x77 > > > > 8: 0c 01 or $0x1,%al > > > > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 > > > > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 > > > > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 > > > > 1c: c4 c1 31 6a c2 vpunpckhdq > > > > %xmm10,%xmm9,%xmm0 > > > > 21: c4 c1 19 6a d5 vpunpckhdq > > > > %xmm13,%xmm12,%xmm2 > > > > 26: c5 f9 6c f2 vpunpcklqdq > > > > %xmm2,%xmm0,%xmm6 > > > > 2a:* c5 79 6d c2 vpunpckhqdq > > > > %xmm2,%xmm0,%xmm8 > > > > <-- trapping instruction > > > > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > > > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > > > m0 > > > > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > > > 3e: c5 .byte 0xc5 > > > > 3f: f9 stc > > > > > > > > Code starting with the faulting instruction > > > > =========================================== > > > > 0: c5 79 6d c2 vpunpckhqdq > > > > %xmm2,%xmm0,%xmm8 > > > > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > > > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > > > m0 > > > > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > > > 14: c5 .byte 0xc5 > > > > 15: f9 stc > > > > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 > > > > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: > > > > 0000000000122d40 > > > > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: > > > > 000055d7049b9368 > > > > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: > > > > 00007fdb5e544040 > > > > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: > > > > 0000000000000000 > > > > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: > > > > 0000000000000000 > > > > [ 44.561965] Modules linked in: > > > > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- > > > > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > > > 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > > > 60 > > > > All code > > > > ======== > > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > > 9: 50 push %rax > > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > > 10: 50 push %rax > > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > > > %rdi > > > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > > > 2a:* 0f 0b ud2 <-- > > > > trapping > > > > instruction > > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > > %rcx > > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > > 3c: 48 rex.W > > > > 3d: c7 .byte 0xc7 > > > > 3e: c6 (bad) > > > > 3f: 60 (bad) > > > > > > > > Code starting with the faulting instruction > > > > =========================================== > > > > 0: 0f 0b ud2 > > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > > %rcx > > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > > 12: 48 rex.W > > > > 13: c7 .byte 0xc7 > > > > 14: c6 (bad) > > > > 15: 60 (bad) > > > > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > > > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > > > 0000000000009ffb > > > > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff979ad2aa5600 > > > > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > > > 0000000000000eb2 > > > > [ 44.571483] FS: 00007fdb9cb11700(0000) > > > > GS:ffff979aebd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > > > 0000000000370ee0 > > > > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 44.576618] Kernel panic - not syncing: Fatal exception in > > > > interrupt > > > > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > > > > Can you test this patch on the latest net branch? > > > > > > Thanks. > > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > > index fa407eb8b457..78a01c71a17c 100644 > > > --- a/drivers/net/virtio_net.c > > > +++ b/drivers/net/virtio_net.c > > > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct > > > virtnet_info *vi, > > > * add_recvbuf_mergeable() + get_mergeable_buf_len() > > > */ > > > truesize = headroom ? PAGE_SIZE : truesize; > > > - tailroom = truesize - len - headroom; > > > + tailroom = truesize - len - headroom - (hdr_padded_len - > > > hdr_len); > > > buf = p - headroom; > > > > > > len -= hdr_len; > > > > With this patch and the latest net branch I no longer get crashes. > > Did this ever get properly submitted to the networking tree to get into > 5.13-final? The patch has been submitted. [PATCH net] virtio-net: fix for skb_over_panic inside big mode Thanks. > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-09 1:48 ` Xuan Zhuo @ 2021-06-09 4:50 ` Greg KH 2021-06-09 6:08 ` Xuan Zhuo 0 siblings, 1 reply; 33+ messages in thread From: Greg KH @ 2021-06-09 4:50 UTC (permalink / raw) To: Xuan Zhuo Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > With this patch and the latest net branch I no longer get crashes. > > > > Did this ever get properly submitted to the networking tree to get into > > 5.13-final? > > The patch has been submitted. > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode Submitted where? Do you have a lore.kernel.org link somewhere? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-09 4:50 ` Greg KH @ 2021-06-09 6:08 ` Xuan Zhuo 2021-06-09 6:24 ` Greg KH 0 siblings, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-06-09 6:08 UTC (permalink / raw) To: Greg KH Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > > With this patch and the latest net branch I no longer get crashes. > > > > > > Did this ever get properly submitted to the networking tree to get into > > > 5.13-final? > > > > The patch has been submitted. > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode > > Submitted where? Do you have a lore.kernel.org link somewhere? https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/ Thanks. > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-09 6:08 ` Xuan Zhuo @ 2021-06-09 6:24 ` Greg KH 2021-06-09 7:51 ` Xuan Zhuo 0 siblings, 1 reply; 33+ messages in thread From: Greg KH @ 2021-06-09 6:24 UTC (permalink / raw) To: Xuan Zhuo Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote: > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > > > With this patch and the latest net branch I no longer get crashes. > > > > > > > > Did this ever get properly submitted to the networking tree to get into > > > > 5.13-final? > > > > > > The patch has been submitted. > > > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode > > > > Submitted where? Do you have a lore.kernel.org link somewhere? > > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/ So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic inside big mode") in Linus's tree, right? But why is that referencing: Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom") when this problem was seen in stable kernels that had a different commit backported to it? Is there nothing needed to be done for the stable kernel trees? confused, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-09 6:24 ` Greg KH @ 2021-06-09 7:51 ` Xuan Zhuo 2021-06-09 8:03 ` Greg KH 0 siblings, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-06-09 7:51 UTC (permalink / raw) To: Greg KH Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote: > > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > > > > With this patch and the latest net branch I no longer get crashes. > > > > > > > > > > Did this ever get properly submitted to the networking tree to get into > > > > > 5.13-final? > > > > > > > > The patch has been submitted. > > > > > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode > > > > > > Submitted where? Do you have a lore.kernel.org link somewhere? > > > > > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/ > > So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic > inside big mode") in Linus's tree, right? YES. > > But why is that referencing: > Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom") This problem was indeed introduced in fb32856b16ad. I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the previous 5.12 did not have this commit fb32856b16ad. I'm not sure if it helped you. Thanks. > > when this problem was seen in stable kernels that had a different commit > backported to it? > > Is there nothing needed to be done for the stable kernel trees? > > confused, > > greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-09 7:51 ` Xuan Zhuo @ 2021-06-09 8:03 ` Greg KH 2021-06-09 8:08 ` Xuan Zhuo 0 siblings, 1 reply; 33+ messages in thread From: Greg KH @ 2021-06-09 8:03 UTC (permalink / raw) To: Xuan Zhuo Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, Jun 09, 2021 at 03:51:20PM +0800, Xuan Zhuo wrote: > On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote: > > > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > > > > > With this patch and the latest net branch I no longer get crashes. > > > > > > > > > > > > Did this ever get properly submitted to the networking tree to get into > > > > > > 5.13-final? > > > > > > > > > > The patch has been submitted. > > > > > > > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode > > > > > > > > Submitted where? Do you have a lore.kernel.org link somewhere? > > > > > > > > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/ > > > > So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic > > inside big mode") in Linus's tree, right? > > YES. > > > > > But why is that referencing: > > Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom") > > This problem was indeed introduced in fb32856b16ad. > > I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the > previous 5.12 did not have this commit fb32856b16ad. > > I'm not sure if it helped you. Hm, then what resolves the reported problem that people were having with the 5.12.y kernel release? Is that a separate issue? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-09 8:03 ` Greg KH @ 2021-06-09 8:08 ` Xuan Zhuo 0 siblings, 0 replies; 33+ messages in thread From: Xuan Zhuo @ 2021-06-09 8:08 UTC (permalink / raw) To: Greg KH Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, 9 Jun 2021 10:03:53 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > On Wed, Jun 09, 2021 at 03:51:20PM +0800, Xuan Zhuo wrote: > > On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote: > > > > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > > > > > > With this patch and the latest net branch I no longer get crashes. > > > > > > > > > > > > > > Did this ever get properly submitted to the networking tree to get into > > > > > > > 5.13-final? > > > > > > > > > > > > The patch has been submitted. > > > > > > > > > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode > > > > > > > > > > Submitted where? Do you have a lore.kernel.org link somewhere? > > > > > > > > > > > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/ > > > > > > So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic > > > inside big mode") in Linus's tree, right? > > > > YES. > > > > > > > > But why is that referencing: > > > Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom") > > > > This problem was indeed introduced in fb32856b16ad. > > > > I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the > > previous 5.12 did not have this commit fb32856b16ad. > > > > I'm not sure if it helped you. > > Hm, then what resolves the reported problem that people were having with > the 5.12.y kernel release? Is that a separate issue? Has anyone reported a problem with 5.12.y? I don’t seem to see it. Corentin only reported a problem with 5.13? Did I miss something? I confirm that 5.12.9 has no modification of fb32856b16ad. Thanks. > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* virtio-net: kernel panic in virtio_net.c
@ 2021-10-07 12:04 Corentin Noël
2021-10-07 13:10 ` Michael S. Tsirkin
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Corentin Noël @ 2021-10-07 12:04 UTC (permalink / raw)
To: stable
Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions,
Eric Dumazet, Xuan Zhuo
I've been experiencing crashes with 5.14-rc1 and above that do not
occur with 5.13,
here is the crash trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
[ 61.377222] RIP: 0010:skb_panic+0x43/0x45
[ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.393635] Call Trace:
[ 61.394127] <IRQ>
[ 61.394488] skb_put.cold+0x10/0x10
[ 61.395095] page_to_skb+0xf7/0x410
[ 61.395689] receive_buf+0x81/0x1660
[ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
[ 61.397180] ? napi_gro_flush+0x97/0xe0
[ 61.397896] ? detach_buf_split+0x67/0x120
[ 61.398573] virtnet_poll+0x2cf/0x420
[ 61.399197] __napi_poll+0x25/0x150
[ 61.399764] net_rx_action+0x22f/0x280
[ 61.400394] __do_softirq+0xba/0x257
[ 61.401012] irq_exit_rcu+0x8e/0xb0
[ 61.401618] common_interrupt+0x7b/0xa0
[ 61.402270] </IRQ>
[ 61.402620] asm_common_interrupt+0x1e/0x40
[ 61.403302] RIP: 0010:default_idle+0xb/0x10
[ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
[ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 61.414183] ? mwait_idle+0x70/0x70
[ 61.414805] ? mwait_idle+0x70/0x70
[ 61.415592] default_idle_call+0x2a/0xa0
[ 61.416216] do_idle+0x1e8/0x250
[ 61.416722] cpu_startup_entry+0x14/0x20
[ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
[ 61.418144] Modules linked in:
[ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
[ 61.419399] RIP: 0010:skb_panic+0x43/0x45
[ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt
[ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Here is my kernel config:
https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
here is the decoded trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
[ 61.377222] RIP: skb_panic+0x43/0x45
[ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
25: e8 6a 43 fb ff callq 0xfffffffffffb4394
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.393635] Call Trace:
[ 61.394127] <IRQ>
[ 61.394488] skb_put.cold+0x10/0x10
[ 61.395095] page_to_skb+0xf7/0x410
[ 61.395689] receive_buf+0x81/0x1660
[ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
[ 61.397180] ? napi_gro_flush+0x97/0xe0
[ 61.397896] ? detach_buf_split+0x67/0x120
[ 61.398573] virtnet_poll+0x2cf/0x420
[ 61.399197] __napi_poll+0x25/0x150
[ 61.399764] net_rx_action+0x22f/0x280
[ 61.400394] __do_softirq+0xba/0x257
[ 61.401012] irq_exit_rcu+0x8e/0xb0
[ 61.401618] common_interrupt+0x7b/0xa0
[ 61.402270] </IRQ>
[ 61.402620] asm_common_interrupt+0x1e/0x40
[ 61.403302] RIP: default_idle+0xb/0x10
[ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
All code
========
0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax
7: f0 80 60 02 df lock andb $0xdf,0x2(%rax)
c: c3 retq
d: 0f ae f0 mfence
10: 0f ae 38 clflush (%rax)
13: 0f ae f0 mfence
16: eb b9 jmp 0xffffffffffffffd1
18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
1f: eb 07 jmp 0x28
21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07
28: fb sti
29: f4 hlt
2a:* c3 retq <-- trapping
instruction
2b: cc int3
2c: cc int3
2d: cc int3
2e: cc int3
2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
# 0x77a42f67
36: 89 d2 mov %edx,%edx
38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
0x10ca20f
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: c3 retq
1: cc int3
2: cc int3
3: cc int3
4: cc int3
5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
# 0x77a42f3d
c: 89 d2 mov %edx,%edx
e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
0x10ca1e5
15: 48 rex.W
[ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 61.414183] ? mwait_idle+0x70/0x70
[ 61.414805] ? mwait_idle+0x70/0x70
[ 61.415592] default_idle_call+0x2a/0xa0
[ 61.416216] do_idle+0x1e8/0x250
[ 61.416722] cpu_startup_entry+0x14/0x20
[ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
[ 61.418144] Modules linked in:
[ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
[ 61.419399] RIP: skb_panic+0x43/0x45
[ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
25: e8 6a 43 fb ff callq 0xfffffffffffb4394
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
Regards,
Corentin
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 12:04 Corentin Noël @ 2021-10-07 13:10 ` Michael S. Tsirkin 2021-10-07 13:51 ` Eric Dumazet 2021-10-07 15:06 ` Xuan Zhuo 2021-10-08 7:59 ` Thorsten Leemhuis 2 siblings, 1 reply; 33+ messages in thread From: Michael S. Tsirkin @ 2021-10-07 13:10 UTC (permalink / raw) To: Corentin Noël Cc: stable, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > I've been experiencing crashes with 5.14-rc1 and above that do not > occur with 5.13, > > here is the crash trace: > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45 > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.393635] Call Trace: > [ 61.394127] <IRQ> > [ 61.394488] skb_put.cold+0x10/0x10 > [ 61.395095] page_to_skb+0xf7/0x410 > [ 61.395689] receive_buf+0x81/0x1660 > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > [ 61.397896] ? detach_buf_split+0x67/0x120 > [ 61.398573] virtnet_poll+0x2cf/0x420 > [ 61.399197] __napi_poll+0x25/0x150 > [ 61.399764] net_rx_action+0x22f/0x280 > [ 61.400394] __do_softirq+0xba/0x257 > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > [ 61.401618] common_interrupt+0x7b/0xa0 > [ 61.402270] </IRQ> > [ 61.402620] asm_common_interrupt+0x1e/0x40 > [ 61.403302] RIP: 0010:default_idle+0xb/0x10 > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > ffff8a5febd56f80 > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > ffff8a5febd5dd00 > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > 0000000000000000 > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > 0000000000000000 > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > [ 61.414183] ? mwait_idle+0x70/0x70 > [ 61.414805] ? mwait_idle+0x70/0x70 > [ 61.415592] default_idle_call+0x2a/0xa0 > [ 61.416216] do_idle+0x1e8/0x250 > [ 61.416722] cpu_startup_entry+0x14/0x20 > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > [ 61.418144] Modules linked in: > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45 > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > Here is my kernel config: > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config > > > here is the decoded trace: > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 61.377222] RIP: skb_panic+0x43/0x45 > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.393635] Call Trace: > [ 61.394127] <IRQ> > [ 61.394488] skb_put.cold+0x10/0x10 > [ 61.395095] page_to_skb+0xf7/0x410 > [ 61.395689] receive_buf+0x81/0x1660 > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > [ 61.397896] ? detach_buf_split+0x67/0x120 > [ 61.398573] virtnet_poll+0x2cf/0x420 > [ 61.399197] __napi_poll+0x25/0x150 > [ 61.399764] net_rx_action+0x22f/0x280 > [ 61.400394] __do_softirq+0xba/0x257 > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > [ 61.401618] common_interrupt+0x7b/0xa0 > [ 61.402270] </IRQ> > [ 61.402620] asm_common_interrupt+0x1e/0x40 > [ 61.403302] RIP: default_idle+0xb/0x10 > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > All code > ======== > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax) > c: c3 retq > d: 0f ae f0 mfence > 10: 0f ae 38 clflush (%rax) > 13: 0f ae f0 mfence > 16: eb b9 jmp 0xffffffffffffffd1 > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) > 1f: eb 07 jmp 0x28 > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07 > 28: fb sti > 29: f4 hlt > 2a:* c3 retq <-- trapping > instruction > 2b: cc int3 > 2c: cc int3 > 2d: cc int3 > 2e: cc int3 > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > # 0x77a42f67 > 36: 89 d2 mov %edx,%edx > 38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > 0x10ca20f > 3f: 48 rex.W > > Code starting with the faulting instruction > =========================================== > 0: c3 retq > 1: cc int3 > 2: cc int3 > 3: cc int3 > 4: cc int3 > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > # 0x77a42f3d > c: 89 d2 mov %edx,%edx > e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > 0x10ca1e5 > 15: 48 rex.W > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > ffff8a5febd56f80 > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > ffff8a5febd5dd00 > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > 0000000000000000 > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > 0000000000000000 > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > [ 61.414183] ? mwait_idle+0x70/0x70 > [ 61.414805] ? mwait_idle+0x70/0x70 > [ 61.415592] default_idle_call+0x2a/0xa0 > [ 61.416216] do_idle+0x1e8/0x250 > [ 61.416722] cpu_startup_entry+0x14/0x20 > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > [ 61.418144] Modules linked in: > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > [ 61.419399] RIP: skb_panic+0x43/0x45 > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > > Regards, > Corentin Don't see anything obvious.. could be a net stack change. Any chance of a bisect? -- MST ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 13:10 ` Michael S. Tsirkin @ 2021-10-07 13:51 ` Eric Dumazet 2021-10-07 14:02 ` Corentin Noël 0 siblings, 1 reply; 33+ messages in thread From: Eric Dumazet @ 2021-10-07 13:51 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Corentin Noël, linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > > I've been experiencing crashes with 5.14-rc1 and above that do not > > occur with 5.13, What about 5.14 ? 5.14-rc1 has many bugs we do not want to spend time rediscovering them... > > > > here is the crash trace: > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45 > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > 0000000000370ee0 > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 61.393635] Call Trace: > > [ 61.394127] <IRQ> > > [ 61.394488] skb_put.cold+0x10/0x10 > > [ 61.395095] page_to_skb+0xf7/0x410 > > [ 61.395689] receive_buf+0x81/0x1660 > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > [ 61.399197] __napi_poll+0x25/0x150 > > [ 61.399764] net_rx_action+0x22f/0x280 > > [ 61.400394] __do_softirq+0xba/0x257 > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > [ 61.401618] common_interrupt+0x7b/0xa0 > > [ 61.402270] </IRQ> > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > [ 61.403302] RIP: 0010:default_idle+0xb/0x10 > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > ffff8a5febd56f80 > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > ffff8a5febd5dd00 > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > 0000000000000000 > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > 0000000000000000 > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > 0000000000000000 > > [ 61.414183] ? mwait_idle+0x70/0x70 > > [ 61.414805] ? mwait_idle+0x70/0x70 > > [ 61.415592] default_idle_call+0x2a/0xa0 > > [ 61.416216] do_idle+0x1e8/0x250 > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > [ 61.418144] Modules linked in: > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45 > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > 0000000000370ee0 > > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt > > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000 > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > Here is my kernel config: > > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config > > > > > > here is the decoded trace: > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 61.377222] RIP: skb_panic+0x43/0x45 > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > 2a:* 0f 0b ud2 <-- trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > 0000000000370ee0 > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 61.393635] Call Trace: > > [ 61.394127] <IRQ> > > [ 61.394488] skb_put.cold+0x10/0x10 > > [ 61.395095] page_to_skb+0xf7/0x410 > > [ 61.395689] receive_buf+0x81/0x1660 > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > [ 61.399197] __napi_poll+0x25/0x150 > > [ 61.399764] net_rx_action+0x22f/0x280 > > [ 61.400394] __do_softirq+0xba/0x257 > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > [ 61.401618] common_interrupt+0x7b/0xa0 > > [ 61.402270] </IRQ> > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > [ 61.403302] RIP: default_idle+0xb/0x10 > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > > All code > > ======== > > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax > > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax) > > c: c3 retq > > d: 0f ae f0 mfence > > 10: 0f ae 38 clflush (%rax) > > 13: 0f ae f0 mfence > > 16: eb b9 jmp 0xffffffffffffffd1 > > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) > > 1f: eb 07 jmp 0x28 > > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07 > > 28: fb sti > > 29: f4 hlt > > 2a:* c3 retq <-- trapping > > instruction > > 2b: cc int3 > > 2c: cc int3 > > 2d: cc int3 > > 2e: cc int3 > > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > # 0x77a42f67 > > 36: 89 d2 mov %edx,%edx > > 38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > > 0x10ca20f > > 3f: 48 rex.W > > > > Code starting with the faulting instruction > > =========================================== > > 0: c3 retq > > 1: cc int3 > > 2: cc int3 > > 3: cc int3 > > 4: cc int3 > > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > # 0x77a42f3d > > c: 89 d2 mov %edx,%edx > > e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > > 0x10ca1e5 > > 15: 48 rex.W > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > ffff8a5febd56f80 > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > ffff8a5febd5dd00 > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > 0000000000000000 > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > 0000000000000000 > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > 0000000000000000 > > [ 61.414183] ? mwait_idle+0x70/0x70 > > [ 61.414805] ? mwait_idle+0x70/0x70 > > [ 61.415592] default_idle_call+0x2a/0xa0 > > [ 61.416216] do_idle+0x1e8/0x250 > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > [ 61.418144] Modules linked in: > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > [ 61.419399] RIP: skb_panic+0x43/0x45 > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > 2a:* 0f 0b ud2 <-- trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > > > Regards, > > Corentin > > Don't see anything obvious.. could be a net stack change. > Any chance of a bisect? > > -- > MST > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 13:51 ` Eric Dumazet @ 2021-10-07 14:02 ` Corentin Noël 2021-10-07 14:13 ` Greg KH 0 siblings, 1 reply; 33+ messages in thread From: Corentin Noël @ 2021-10-07 14:02 UTC (permalink / raw) To: Eric Dumazet, Michael S. Tsirkin Cc: linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit : > On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > > > I've been experiencing crashes with 5.14-rc1 and above that do > > > not > > > occur with 5.13, > > What about 5.14 ? > > 5.14-rc1 has many bugs we do not want to spend time rediscovering > them... > I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could reproduce the crash on them all. > > > here is the crash trace: > > > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 > > > len:3762 > > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45 > > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.386825] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > > 0000000000370ee0 > > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 61.393635] Call Trace: > > > [ 61.394127] <IRQ> > > > [ 61.394488] skb_put.cold+0x10/0x10 > > > [ 61.395095] page_to_skb+0xf7/0x410 > > > [ 61.395689] receive_buf+0x81/0x1660 > > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > > [ 61.399197] __napi_poll+0x25/0x150 > > > [ 61.399764] net_rx_action+0x22f/0x280 > > > [ 61.400394] __do_softirq+0xba/0x257 > > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > > [ 61.401618] common_interrupt+0x7b/0xa0 > > > [ 61.402270] </IRQ> > > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > > [ 61.403302] RIP: 0010:default_idle+0xb/0x10 > > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae > > > f0 0f > > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 > > > 00 fb > > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c > > > 01 48 > > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > > ffff8a5febd56f80 > > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > > ffff8a5febd5dd00 > > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > > 0000000000000000 > > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > > 0000000000000000 > > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > > 0000000000000000 > > > [ 61.414183] ? mwait_idle+0x70/0x70 > > > [ 61.414805] ? mwait_idle+0x70/0x70 > > > [ 61.415592] default_idle_call+0x2a/0xa0 > > > [ 61.416216] do_idle+0x1e8/0x250 > > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > > [ 61.418144] Modules linked in: > > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45 > > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.429799] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > > 0000000000370ee0 > > > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 61.435799] Kernel panic - not syncing: Fatal exception in > > > interrupt > > > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > Here is my kernel config: > > > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config > > > > > > > > > here is the decoded trace: > > > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 > > > len:3762 > > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 61.377222] RIP: skb_panic+0x43/0x45 > > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.386825] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > > 0000000000370ee0 > > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 61.393635] Call Trace: > > > [ 61.394127] <IRQ> > > > [ 61.394488] skb_put.cold+0x10/0x10 > > > [ 61.395095] page_to_skb+0xf7/0x410 > > > [ 61.395689] receive_buf+0x81/0x1660 > > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > > [ 61.399197] __napi_poll+0x25/0x150 > > > [ 61.399764] net_rx_action+0x22f/0x280 > > > [ 61.400394] __do_softirq+0xba/0x257 > > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > > [ 61.401618] common_interrupt+0x7b/0xa0 > > > [ 61.402270] </IRQ> > > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > > [ 61.403302] RIP: default_idle+0xb/0x10 > > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae > > > f0 0f > > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 > > > 00 fb > > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c > > > 01 48 > > > All code > > > ======== > > > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax > > > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax) > > > c: c3 retq > > > d: 0f ae f0 mfence > > > 10: 0f ae 38 clflush (%rax) > > > 13: 0f ae f0 mfence > > > 16: eb b9 jmp 0xffffffffffffffd1 > > > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) > > > 1f: eb 07 jmp 0x28 > > > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # > > > 0x443f07 > > > 28: fb sti > > > 29: f4 hlt > > > 2a:* c3 retq <-- > > > trapping > > > instruction > > > 2b: cc int3 > > > 2c: cc int3 > > > 2d: cc int3 > > > 2e: cc int3 > > > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > > # 0x77a42f67 > > > 36: 89 d2 mov %edx,%edx > > > 38: 48 8b 05 d0 a1 0c > > > 01 mov 0x10ca1d0(%rip),%rax # > > > 0x10ca20f > > > 3f: 48 rex.W > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: c3 retq > > > 1: cc int3 > > > 2: cc int3 > > > 3: cc int3 > > > 4: cc int3 > > > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > > # 0x77a42f3d > > > c: 89 d2 mov %edx,%edx > > > e: 48 8b 05 d0 a1 0c > > > 01 mov 0x10ca1d0(%rip),%rax # > > > 0x10ca1e5 > > > 15: 48 rex.W > > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > > ffff8a5febd56f80 > > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > > ffff8a5febd5dd00 > > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > > 0000000000000000 > > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > > 0000000000000000 > > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > > 0000000000000000 > > > [ 61.414183] ? mwait_idle+0x70/0x70 > > > [ 61.414805] ? mwait_idle+0x70/0x70 > > > [ 61.415592] default_idle_call+0x2a/0xa0 > > > [ 61.416216] do_idle+0x1e8/0x250 > > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > > [ 61.418144] Modules linked in: > > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > > [ 61.419399] RIP: skb_panic+0x43/0x45 > > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.429799] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > > > > Regards, > > > Corentin > > > > Don't see anything obvious.. could be a net stack change. > > Any chance of a bisect? > > > > -- > > MST > > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 14:02 ` Corentin Noël @ 2021-10-07 14:13 ` Greg KH 0 siblings, 0 replies; 33+ messages in thread From: Greg KH @ 2021-10-07 14:13 UTC (permalink / raw) To: Corentin Noël Cc: Eric Dumazet, Michael S. Tsirkin, linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo On Thu, Oct 07, 2021 at 04:02:10PM +0200, Corentin Noël wrote: > Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit : > > On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> > > wrote: > > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > > > > I've been experiencing crashes with 5.14-rc1 and above that do > > > > not > > > > occur with 5.13, > > > > What about 5.14 ? > > > > 5.14-rc1 has many bugs we do not want to spend time rediscovering > > them... > > > > I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could > reproduce the crash on them all. Great, any chance you can use 'git bisect' to find the offending commit? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 12:04 Corentin Noël 2021-10-07 13:10 ` Michael S. Tsirkin @ 2021-10-07 15:06 ` Xuan Zhuo 2021-10-07 15:25 ` Greg KH 2021-10-08 7:59 ` Thorsten Leemhuis 2 siblings, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-10-07 15:06 UTC (permalink / raw) To: Corentin Noël Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > I've been experiencing crashes with 5.14-rc1 and above that do not > occur with 5.13, I should have fixed this problem before. I don't know why, I just looked at the latest net code, and this commit seems to be lost. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode Can you test this patch again? Thanks. > > here is the crash trace: > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45 > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.393635] Call Trace: > [ 61.394127] <IRQ> > [ 61.394488] skb_put.cold+0x10/0x10 > [ 61.395095] page_to_skb+0xf7/0x410 > [ 61.395689] receive_buf+0x81/0x1660 > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > [ 61.397896] ? detach_buf_split+0x67/0x120 > [ 61.398573] virtnet_poll+0x2cf/0x420 > [ 61.399197] __napi_poll+0x25/0x150 > [ 61.399764] net_rx_action+0x22f/0x280 > [ 61.400394] __do_softirq+0xba/0x257 > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > [ 61.401618] common_interrupt+0x7b/0xa0 > [ 61.402270] </IRQ> > [ 61.402620] asm_common_interrupt+0x1e/0x40 > [ 61.403302] RIP: 0010:default_idle+0xb/0x10 > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > ffff8a5febd56f80 > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > ffff8a5febd5dd00 > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > 0000000000000000 > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > 0000000000000000 > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > [ 61.414183] ? mwait_idle+0x70/0x70 > [ 61.414805] ? mwait_idle+0x70/0x70 > [ 61.415592] default_idle_call+0x2a/0xa0 > [ 61.416216] do_idle+0x1e8/0x250 > [ 61.416722] cpu_startup_entry+0x14/0x20 > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > [ 61.418144] Modules linked in: > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45 > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > Here is my kernel config: > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config > > > here is the decoded trace: > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 61.377222] RIP: skb_panic+0x43/0x45 > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.393635] Call Trace: > [ 61.394127] <IRQ> > [ 61.394488] skb_put.cold+0x10/0x10 > [ 61.395095] page_to_skb+0xf7/0x410 > [ 61.395689] receive_buf+0x81/0x1660 > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > [ 61.397896] ? detach_buf_split+0x67/0x120 > [ 61.398573] virtnet_poll+0x2cf/0x420 > [ 61.399197] __napi_poll+0x25/0x150 > [ 61.399764] net_rx_action+0x22f/0x280 > [ 61.400394] __do_softirq+0xba/0x257 > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > [ 61.401618] common_interrupt+0x7b/0xa0 > [ 61.402270] </IRQ> > [ 61.402620] asm_common_interrupt+0x1e/0x40 > [ 61.403302] RIP: default_idle+0xb/0x10 > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > All code > ======== > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax) > c: c3 retq > d: 0f ae f0 mfence > 10: 0f ae 38 clflush (%rax) > 13: 0f ae f0 mfence > 16: eb b9 jmp 0xffffffffffffffd1 > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) > 1f: eb 07 jmp 0x28 > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07 > 28: fb sti > 29: f4 hlt > 2a:* c3 retq <-- trapping > instruction > 2b: cc int3 > 2c: cc int3 > 2d: cc int3 > 2e: cc int3 > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > # 0x77a42f67 > 36: 89 d2 mov %edx,%edx > 38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > 0x10ca20f > 3f: 48 rex.W > > Code starting with the faulting instruction > =========================================== > 0: c3 retq > 1: cc int3 > 2: cc int3 > 3: cc int3 > 4: cc int3 > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > # 0x77a42f3d > c: 89 d2 mov %edx,%edx > e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > 0x10ca1e5 > 15: 48 rex.W > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > ffff8a5febd56f80 > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > ffff8a5febd5dd00 > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > 0000000000000000 > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > 0000000000000000 > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > [ 61.414183] ? mwait_idle+0x70/0x70 > [ 61.414805] ? mwait_idle+0x70/0x70 > [ 61.415592] default_idle_call+0x2a/0xa0 > [ 61.416216] do_idle+0x1e8/0x250 > [ 61.416722] cpu_startup_entry+0x14/0x20 > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > [ 61.418144] Modules linked in: > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > [ 61.419399] RIP: skb_panic+0x43/0x45 > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > > Regards, > Corentin > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 15:06 ` Xuan Zhuo @ 2021-10-07 15:25 ` Greg KH 2021-10-07 16:17 ` Xuan Zhuo 0 siblings, 1 reply; 33+ messages in thread From: Greg KH @ 2021-10-07 15:25 UTC (permalink / raw) To: Xuan Zhuo Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > I've been experiencing crashes with 5.14-rc1 and above that do not > > occur with 5.13, > > I should have fixed this problem before. I don't know why, I just looked at the > latest net code, and this commit seems to be lost. > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > Can you test this patch again? That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had it in it, right? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 15:25 ` Greg KH @ 2021-10-07 16:17 ` Xuan Zhuo 2021-10-08 8:06 ` Greg KH 0 siblings, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-10-07 16:17 UTC (permalink / raw) To: Greg KH Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > occur with 5.13, > > > > I should have fixed this problem before. I don't know why, I just looked at the > > latest net code, and this commit seems to be lost. > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > Can you test this patch again? > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > it in it, right? > Yes, it may be lost due to conflicts during a certain merge. Thanks. > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 16:17 ` Xuan Zhuo @ 2021-10-08 8:06 ` Greg KH 2021-10-08 10:02 ` Michael S. Tsirkin 2021-10-08 16:27 ` Xuan Zhuo 0 siblings, 2 replies; 33+ messages in thread From: Greg KH @ 2021-10-08 8:06 UTC (permalink / raw) To: Xuan Zhuo Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > occur with 5.13, > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > latest net code, and this commit seems to be lost. > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > Can you test this patch again? > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > it in it, right? > > > > Yes, it may be lost due to conflicts during a certain merge. Really? I tried to apply that again to 5.14 and it did not work. So I do not understand what to do here, can you try to explain it better? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-08 8:06 ` Greg KH @ 2021-10-08 10:02 ` Michael S. Tsirkin 2021-10-08 12:21 ` Corentin Noël 2021-10-08 16:27 ` Xuan Zhuo 1 sibling, 1 reply; 33+ messages in thread From: Michael S. Tsirkin @ 2021-10-08 10:02 UTC (permalink / raw) To: Greg KH Cc: Xuan Zhuo, Corentin Noël, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote: > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > > occur with 5.13, > > > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > > latest net code, and this commit seems to be lost. > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > > > Can you test this patch again? > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > > it in it, right? > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > Really? I tried to apply that again to 5.14 and it did not work. So I > do not understand what to do here, can you try to explain it better? > > thanks, > > greg k-h Hmm, something like the following perhaps then? Corentin would you like to try this? Warning: untested. diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 096c2ac6b7a6..18dd9f6d107d 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi, * add_recvbuf_mergeable() + get_mergeable_buf_len() */ truesize = headroom ? PAGE_SIZE : truesize; - tailroom = truesize - len - headroom; + tailroom = truesize - headroom; buf = p - headroom; len -= hdr_len; offset += hdr_padded_len; p += hdr_padded_len; + tailroom -= hdr_padded_len + len; shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); ^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-08 10:02 ` Michael S. Tsirkin @ 2021-10-08 12:21 ` Corentin Noël 0 siblings, 0 replies; 33+ messages in thread From: Corentin Noël @ 2021-10-08 12:21 UTC (permalink / raw) To: Michael S. Tsirkin, Greg KH Cc: Xuan Zhuo, Jason Wang, virtualization, regressions, Eric Dumazet, stable Le vendredi 08 octobre 2021 à 06:02 -0400, Michael S. Tsirkin a écrit : > On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote: > > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH < > > > gregkh@linuxfoundation.org> wrote: > > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël < > > > > > corentin.noel@collabora.com> wrote: > > > > > > I've been experiencing crashes with 5.14-rc1 and above that > > > > > > do not > > > > > > occur with 5.13, > > > > > > > > > > I should have fixed this problem before. I don't know why, I > > > > > just looked at the > > > > > latest net code, and this commit seems to be lost. > > > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix > > > > > for skb_over_panic inside big mode > > > > > > > > > > Can you test this patch again? > > > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should > > > > have had > > > > it in it, right? > > > > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > > > Really? I tried to apply that again to 5.14 and it did not > > work. So I > > do not understand what to do here, can you try to explain it > > better? > > > > thanks, > > > > greg k-h > > Hmm, something like the following perhaps then? > Corentin would you like to try this? > Warning: untested. > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index 096c2ac6b7a6..18dd9f6d107d 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct > virtnet_info *vi, > * add_recvbuf_mergeable() + get_mergeable_buf_len() > */ > truesize = headroom ? PAGE_SIZE : truesize; > - tailroom = truesize - len - headroom; > + tailroom = truesize - headroom; > buf = p - headroom; > > len -= hdr_len; > offset += hdr_padded_len; > p += hdr_padded_len; > + tailroom -= hdr_padded_len + len; > > shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); > > Thank you for the patch, I started bisecting the issue but your patch actually makes it work again. Regards, Corentin ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-08 8:06 ` Greg KH 2021-10-08 10:02 ` Michael S. Tsirkin @ 2021-10-08 16:27 ` Xuan Zhuo 2021-10-09 5:19 ` Greg KH 1 sibling, 1 reply; 33+ messages in thread From: Xuan Zhuo @ 2021-10-08 16:27 UTC (permalink / raw) To: Greg KH Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > > occur with 5.13, > > > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > > latest net code, and this commit seems to be lost. > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > > > Can you test this patch again? > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > > it in it, right? > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > Really? I tried to apply that again to 5.14 and it did not work. So I > do not understand what to do here, can you try to explain it better? I took a look, and there is actually another missing patch: A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode A is replaced by another patch: commit c32325b8fdf2f979befb9fd5587918c0d5412db3 Author: Jakub Kicinski <kuba@kernel.org> Date: Mon Aug 2 10:57:29 2021 -0700 virtio-net: realign page_to_skb() after merges We ended up merging two versions of the same patch set: commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr") commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address") into net, and commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr") commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address") into net-next. Redo the merge from commit 126285651b7f ("Merge ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that the most recent code remains. Acked-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Jakub Kicinski <kuba@kernel.org> Acked-by: Jason Wang <jasowang@redhat.com> Signed-off-by: David S. Miller <davem@davemloft.net> So after this patch, patch B can be applied normally. So on the latest net branch, only lost 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode Thanks. > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-08 16:27 ` Xuan Zhuo @ 2021-10-09 5:19 ` Greg KH 2021-10-09 9:31 ` Xuan Zhuo 0 siblings, 1 reply; 33+ messages in thread From: Greg KH @ 2021-10-09 5:19 UTC (permalink / raw) To: Xuan Zhuo Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Sat, Oct 09, 2021 at 12:27:08AM +0800, Xuan Zhuo wrote: > On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > > > occur with 5.13, > > > > > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > > > latest net code, and this commit seems to be lost. > > > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > > > > > Can you test this patch again? > > > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > > > it in it, right? > > > > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > > > Really? I tried to apply that again to 5.14 and it did not work. So I > > do not understand what to do here, can you try to explain it better? > > I took a look, and there is actually another missing patch: > > A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr > B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > A is replaced by another patch: > > commit c32325b8fdf2f979befb9fd5587918c0d5412db3 > Author: Jakub Kicinski <kuba@kernel.org> > Date: Mon Aug 2 10:57:29 2021 -0700 > > virtio-net: realign page_to_skb() after merges > > We ended up merging two versions of the same patch set: > > commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr") > commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address") > > into net, and > > commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr") > commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address") > > into net-next. Redo the merge from commit 126285651b7f ("Merge > ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that > the most recent code remains. > > Acked-by: Michael S. Tsirkin <mst@redhat.com> > Signed-off-by: Jakub Kicinski <kuba@kernel.org> > Acked-by: Jason Wang <jasowang@redhat.com> > Signed-off-by: David S. Miller <davem@davemloft.net> > > So after this patch, patch B can be applied normally. > > So on the latest net branch, only lost > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode Again, I do not know what to do here, can you submit the needed fix to the networking developers so this gets fixed? thanks, greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-09 5:19 ` Greg KH @ 2021-10-09 9:31 ` Xuan Zhuo 0 siblings, 0 replies; 33+ messages in thread From: Xuan Zhuo @ 2021-10-09 9:31 UTC (permalink / raw) To: Greg KH Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Sat, 9 Oct 2021 07:19:39 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > On Sat, Oct 09, 2021 at 12:27:08AM +0800, Xuan Zhuo wrote: > > On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > > > > occur with 5.13, > > > > > > > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > > > > latest net code, and this commit seems to be lost. > > > > > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > > > > > > > Can you test this patch again? > > > > > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > > > > it in it, right? > > > > > > > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > > > > > Really? I tried to apply that again to 5.14 and it did not work. So I > > > do not understand what to do here, can you try to explain it better? > > > > I took a look, and there is actually another missing patch: > > > > A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr > > B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > A is replaced by another patch: > > > > commit c32325b8fdf2f979befb9fd5587918c0d5412db3 > > Author: Jakub Kicinski <kuba@kernel.org> > > Date: Mon Aug 2 10:57:29 2021 -0700 > > > > virtio-net: realign page_to_skb() after merges > > > > We ended up merging two versions of the same patch set: > > > > commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr") > > commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address") > > > > into net, and > > > > commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr") > > commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address") > > > > into net-next. Redo the merge from commit 126285651b7f ("Merge > > ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that > > the most recent code remains. > > > > Acked-by: Michael S. Tsirkin <mst@redhat.com> > > Signed-off-by: Jakub Kicinski <kuba@kernel.org> > > Acked-by: Jason Wang <jasowang@redhat.com> > > Signed-off-by: David S. Miller <davem@davemloft.net> > > > > So after this patch, patch B can be applied normally. > > > > So on the latest net branch, only lost > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > Again, I do not know what to do here, can you submit the needed fix to > the networking developers so this gets fixed? Michael has already submitted the patch. https://lore.kernel.org/netdev/20211009091604.84141-1-mst@redhat.com/T/#u Thanks. > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 12:04 Corentin Noël 2021-10-07 13:10 ` Michael S. Tsirkin 2021-10-07 15:06 ` Xuan Zhuo @ 2021-10-08 7:59 ` Thorsten Leemhuis 2021-10-12 12:44 ` Thorsten Leemhuis 2 siblings, 1 reply; 33+ messages in thread From: Thorsten Leemhuis @ 2021-10-08 7:59 UTC (permalink / raw) To: regressions On 07.10.21 14:04, Corentin Noël wrote: > I've been experiencing crashes with 5.14-rc1 and above that do not > occur with 5.13, Feel free to ignore this message. I write it to make regzbot track above issue. Regzbot is the regression tracking bot I'm working on. It's still in the early stages and this is still one of the first few regression I make it track to get started and things tested in the field. That also why I'm sending the mail just to the regressions list (it will do its full magic nevertheless). For details see: https://linux-regtracking.leemhuis.info/post/inital-regzbot-running/ https://linux-regtracking.leemhuis.info/post/regzbot-approach/ #regzbot ^introduced v5.13..v5.14-rc1 #regzbot link https://lore.kernel.org/regressions/1633623446.6192446-1-xuanzhuo@linux.alibaba.com/ Looks like a fix for this somehow fall through the cracks #regzbot ignore-activitiy BTW, in case anyone is reading this: the regzbot's webinterface that gives insights into what is tracked can be found here: https://linux-regtracking.leemhuis.info/regzbot/mainline.html Ciao, Thorsten ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-08 7:59 ` Thorsten Leemhuis @ 2021-10-12 12:44 ` Thorsten Leemhuis 0 siblings, 0 replies; 33+ messages in thread From: Thorsten Leemhuis @ 2021-10-12 12:44 UTC (permalink / raw) To: regressions On 08.10.21 09:59, Thorsten Leemhuis wrote: > On 07.10.21 14:04, Corentin Noël wrote: >> I've been experiencing crashes with 5.14-rc1 and above that do not >> occur with 5.13, > > Feel free to ignore this message. I write it to make regzbot track above > issue. Regzbot is the regression tracking bot I'm working on. It's still > in the early stages and this is still one of the first few regression I > make it track to get started and things tested in the field. That also > why I'm sending the mail just to the regressions list (it will do its > full magic nevertheless). For details see: > https://linux-regtracking.leemhuis.info/post/inital-regzbot-running/ > https://linux-regtracking.leemhuis.info/post/regzbot-approach/ > > #regzbot ^introduced v5.13..v5.14-rc1 > #regzbot link > https://lore.kernel.org/regressions/1633623446.6192446-1-xuanzhuo@linux.alibaba.com/ > Looks like a fix for this somehow fall through the cracks > #regzbot ignore-activitiy > > BTW, in case anyone is reading this: the regzbot's webinterface that > gives insights into what is tracked can be found here: > > https://linux-regtracking.leemhuis.info/regzbot/mainline.html > > Ciao, Thorsten #regzbot fixed-by: 732b74d647048668f0f8dc0c848f0746c69e2e2f ^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2021-10-12 12:44 UTC | newest] Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-06-01 16:06 virtio-net: kernel panic in virtio_net.c Corentin Noël 2021-06-01 17:07 ` Greg KH 2021-06-01 17:09 ` Corentin Noël 2021-06-01 17:47 ` Eric Dumazet 2021-06-02 2:01 ` Xuan Zhuo 2021-06-02 17:56 ` Corentin Noël 2021-06-02 17:54 ` Corentin Noël 2021-06-03 2:44 ` Xuan Zhuo 2021-06-03 8:57 ` Corentin Noël 2021-06-08 12:17 ` Greg KH 2021-06-09 1:48 ` Xuan Zhuo 2021-06-09 4:50 ` Greg KH 2021-06-09 6:08 ` Xuan Zhuo 2021-06-09 6:24 ` Greg KH 2021-06-09 7:51 ` Xuan Zhuo 2021-06-09 8:03 ` Greg KH 2021-06-09 8:08 ` Xuan Zhuo 2021-10-07 12:04 Corentin Noël 2021-10-07 13:10 ` Michael S. Tsirkin 2021-10-07 13:51 ` Eric Dumazet 2021-10-07 14:02 ` Corentin Noël 2021-10-07 14:13 ` Greg KH 2021-10-07 15:06 ` Xuan Zhuo 2021-10-07 15:25 ` Greg KH 2021-10-07 16:17 ` Xuan Zhuo 2021-10-08 8:06 ` Greg KH 2021-10-08 10:02 ` Michael S. Tsirkin 2021-10-08 12:21 ` Corentin Noël 2021-10-08 16:27 ` Xuan Zhuo 2021-10-09 5:19 ` Greg KH 2021-10-09 9:31 ` Xuan Zhuo 2021-10-08 7:59 ` Thorsten Leemhuis 2021-10-12 12:44 ` Thorsten Leemhuis
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).