* [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] @ 2022-10-03 7:16 Bruno Goncalves 2022-10-16 11:37 ` Thorsten Leemhuis 2022-10-16 18:08 ` Randy Dunlap 0 siblings, 2 replies; 6+ messages in thread From: Bruno Goncalves @ 2022-10-03 7:16 UTC (permalink / raw) To: LKML; +Cc: CKI Project Hello, recently we are hitting the following issue when booting using kernel with debug options enabled: [ 126.821761] CPU: 1 PID: 296 Comm: cryptomgr_test Not tainted 6.0.0 #1 [ 126.824021] Hardware name: Red Hat KVM, BIOS 1.15.0-1.el9 04/01/2014 [ 126.825721] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] [ 126.827045] Code: df 11 c4 41 31 ef fb 48 83 c2 08 0f 84 eb 01 00 00 48 89 d0 48 c1 e0 04 48 81 c7 80 00 00 00 48 29 c7 0f 10 06 66 41 0f ef c7 <0f> 28 0f c4 63 71 44 f0 01 c4 e3 71 44 d0 10 c4 63 71 44 e0 00 c4 [ 126.831535] RSP: 0018:ffff88810ed779d8 EFLAGS: 00010282 [ 126.832957] RAX: 0000000000000030 RBX: 0000000000000030 RCX: ffff88810ede19e8 [ 126.834907] RDX: 0000000000000003 RSI: ffff8881094b8000 RDI: ffff88811483c408 [ 126.836554] RBP: ffff8881094b8000 R08: ffffffffc0300835 R09: ffff88810ede19e8 [ 126.838529] R10: ffffed1021dbc33f R11: 0000000000000000 R12: 0000000000000030 [ 126.840526] R13: 0000000000000003 R14: ffff88810ede19e8 R15: ffff88810ede19e0 [ 126.842702] FS: 0000000000000000(0000) GS:ffff888193d00000(0000) knlGS:0000000000000000 [ 126.844574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 126.846796] CR2: 0000564f48e04178 CR3: 000000010be88002 CR4: 0000000000370ee0 [ 126.848934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 126.850913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 126.852879] Call Trace: [ 126.853703] <TASK> [ 126.854436] polyval_x86_update+0x102/0x1e0 [polyval_clmulni] [ 126.856092] test_shash_vec_cfg+0x523/0x9d0 [ 126.857360] ? alg_test_aead+0x170/0x170 [ 126.858682] ? pointer+0x6a0/0x6a0 [ 126.859752] ? ___slab_alloc+0x910/0xf70 [ 126.860950] ? __alg_test_hash.constprop.0+0x200/0x430 [ 126.862140] ? __kmalloc+0x64/0x380 [ 126.863233] ? sprintf+0xb3/0xe0 [ 126.864159] ? rcu_read_lock_sched_held+0x3f/0x80 [ 126.865560] ? trace_kmalloc+0x33/0xf0 [ 126.866716] __alg_test_hash.constprop.0+0x2e7/0x430 [ 126.867995] ? test_ahash_vec_cfg+0xca0/0xca0 [ 126.869389] ? lockdep_hardirqs_on_prepare+0x230/0x230 [ 126.870906] ? lockdep_hardirqs_on+0x7d/0x100 [ 126.872221] ? alg_test_hash+0x8e/0x130 [ 126.873398] alg_test+0x60d/0x7d0 [ 126.874463] ? alloc_cipher_test_sglists+0xb0/0xb0 [ 126.875867] ? lock_release+0x233/0x470 [ 126.876726] ? __kthread_parkme+0x65/0xf0 [ 126.877654] ? reacquire_held_locks+0x270/0x270 [ 126.878984] ? mark_held_locks+0x24/0x90 [ 126.879985] ? preempt_count_sub+0xb7/0x100 [ 126.881260] ? crypto_acomp_scomp_free_ctx+0x70/0x70 [ 126.882649] cryptomgr_test+0x36/0x60 [ 126.883786] kthread+0x17e/0x1b0 [ 126.884564] ? kthread_complete_and_exit+0x20/0x20 [ 126.885975] ret_from_fork+0x22/0x30 [ 126.887086] </TASK> [ 126.887858] Modules linked in: polyval_clmulni(+) polyval_generic virtio_blk(+) ghash_clmulni_intel floppy virtio_console serio_raw ata_generic pata_acpi qemu_fw_cfg full console log: https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/02/redhat:656405029/build_x86_64_redhat:656405029_x86_64_debug/tests/4/results_0001/console.log/console.log test logs: https://datawarehouse.cki-project.org/kcidb/tests/5344697 cki issue tracker: https://datawarehouse.cki-project.org/issue/1626 kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/build%20x86_64%20debug/3116060967/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.config kernel tarball: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/publish%20x86_64%20debug/3116060972/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.tar.gz We didn't bisect, but the first commit we tested that we hit the problem is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=46452d3786a82bd732ba73fb308ae5cbe4e1e591 The last commit we tested and didn't hit the problem is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a1375562c0a87f0fa2eaf3e8ce15824696d4170a Thanks, Bruno Goncalves ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] 2022-10-03 7:16 [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] Bruno Goncalves @ 2022-10-16 11:37 ` Thorsten Leemhuis 2022-10-16 18:08 ` Randy Dunlap 1 sibling, 0 replies; 6+ messages in thread From: Thorsten Leemhuis @ 2022-10-16 11:37 UTC (permalink / raw) To: Herbert Xu Cc: CKI Project, Bruno Goncalves, LKML, David S. Miller, regressions Hi, this is your Linux kernel regression tracker. CCing the regression mailing list, as it should be in the loop for all regressions, as explained here: https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html On 03.10.22 09:16, Bruno Goncalves wrote: > > recently we are hitting the following issue when booting using kernel > with debug options enabled: Bruno, FYI, reports sent just to LKML without CCing any actual people (or at least subsystem specific mailing list, but that often is not enough either) are most likely to be ignored, because nearly nobody reads LKML fully these days. That's why I added the crypto maintainers and the crypto list. > [ 126.821761] CPU: 1 PID: 296 Comm: cryptomgr_test Not tainted 6.0.0 #1 > [ 126.824021] Hardware name: Red Hat KVM, BIOS 1.15.0-1.el9 04/01/2014 > [ 126.825721] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] FWIW, I noticed a report that looked someone similar: https://bugzilla.kernel.org/show_bug.cgi?id=216587 Then I searched lore, which brought me to this mail and made me send this reply to get the people that should deal with this into the loop, which hopefully should get things rolling now. For the rest of this mail: [TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.] > [ 126.827045] Code: df 11 c4 41 31 ef fb 48 83 c2 08 0f 84 eb 01 00 > 00 48 89 d0 48 c1 e0 04 48 81 c7 80 00 00 00 48 29 c7 0f 10 06 66 41 > 0f ef c7 <0f> 28 0f c4 63 71 44 f0 01 c4 e3 71 44 d0 10 c4 63 71 44 e0 > 00 c4 > [ 126.831535] RSP: 0018:ffff88810ed779d8 EFLAGS: 00010282 > [ 126.832957] RAX: 0000000000000030 RBX: 0000000000000030 RCX: ffff88810ede19e8 > [ 126.834907] RDX: 0000000000000003 RSI: ffff8881094b8000 RDI: ffff88811483c408 > [ 126.836554] RBP: ffff8881094b8000 R08: ffffffffc0300835 R09: ffff88810ede19e8 > [ 126.838529] R10: ffffed1021dbc33f R11: 0000000000000000 R12: 0000000000000030 > [ 126.840526] R13: 0000000000000003 R14: ffff88810ede19e8 R15: ffff88810ede19e0 > [ 126.842702] FS: 0000000000000000(0000) GS:ffff888193d00000(0000) > knlGS:0000000000000000 > [ 126.844574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 126.846796] CR2: 0000564f48e04178 CR3: 000000010be88002 CR4: 0000000000370ee0 > [ 126.848934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 126.850913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 126.852879] Call Trace: > [ 126.853703] <TASK> > [ 126.854436] polyval_x86_update+0x102/0x1e0 [polyval_clmulni] > [ 126.856092] test_shash_vec_cfg+0x523/0x9d0 > [ 126.857360] ? alg_test_aead+0x170/0x170 > [ 126.858682] ? pointer+0x6a0/0x6a0 > [ 126.859752] ? ___slab_alloc+0x910/0xf70 > [ 126.860950] ? __alg_test_hash.constprop.0+0x200/0x430 > [ 126.862140] ? __kmalloc+0x64/0x380 > [ 126.863233] ? sprintf+0xb3/0xe0 > [ 126.864159] ? rcu_read_lock_sched_held+0x3f/0x80 > [ 126.865560] ? trace_kmalloc+0x33/0xf0 > [ 126.866716] __alg_test_hash.constprop.0+0x2e7/0x430 > [ 126.867995] ? test_ahash_vec_cfg+0xca0/0xca0 > [ 126.869389] ? lockdep_hardirqs_on_prepare+0x230/0x230 > [ 126.870906] ? lockdep_hardirqs_on+0x7d/0x100 > [ 126.872221] ? alg_test_hash+0x8e/0x130 > [ 126.873398] alg_test+0x60d/0x7d0 > [ 126.874463] ? alloc_cipher_test_sglists+0xb0/0xb0 > [ 126.875867] ? lock_release+0x233/0x470 > [ 126.876726] ? __kthread_parkme+0x65/0xf0 > [ 126.877654] ? reacquire_held_locks+0x270/0x270 > [ 126.878984] ? mark_held_locks+0x24/0x90 > [ 126.879985] ? preempt_count_sub+0xb7/0x100 > [ 126.881260] ? crypto_acomp_scomp_free_ctx+0x70/0x70 > [ 126.882649] cryptomgr_test+0x36/0x60 > [ 126.883786] kthread+0x17e/0x1b0 > [ 126.884564] ? kthread_complete_and_exit+0x20/0x20 > [ 126.885975] ret_from_fork+0x22/0x30 > [ 126.887086] </TASK> > [ 126.887858] Modules linked in: polyval_clmulni(+) polyval_generic > virtio_blk(+) ghash_clmulni_intel floppy virtio_console serio_raw > ata_generic pata_acpi qemu_fw_cfg > > > full console log: > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/02/redhat:656405029/build_x86_64_redhat:656405029_x86_64_debug/tests/4/results_0001/console.log/console.log > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/5344697 > cki issue tracker: https://datawarehouse.cki-project.org/issue/1626 > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/build%20x86_64%20debug/3116060967/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.config > kernel tarball: > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/publish%20x86_64%20debug/3116060972/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.tar.gz > > We didn't bisect, but the first commit we tested that we hit the > problem is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=46452d3786a82bd732ba73fb308ae5cbe4e1e591 > > The last commit we tested and didn't hit the problem is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a1375562c0a87f0fa2eaf3e8ce15824696d4170a Thanks for the report. To be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot: #regzbot introduced a1375562c0a87..46452d3786a82 ^ https://bugzilla.kernel.org/show_bug.cgi?id=216587 #regzbot title crypto: kernel panic after "RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni]" #regzbot ignore-activity This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here: https://linux-regtracking.leemhuis.info/tracked-regression/ Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replies to), as explained for in the Linux kernel's documentation; above webpage explains why this is important for tracked regressions. Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] 2022-10-03 7:16 [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] Bruno Goncalves 2022-10-16 11:37 ` Thorsten Leemhuis @ 2022-10-16 18:08 ` Randy Dunlap 2022-10-17 8:32 ` Bruno Goncalves 1 sibling, 1 reply; 6+ messages in thread From: Randy Dunlap @ 2022-10-16 18:08 UTC (permalink / raw) To: Bruno Goncalves, LKML; +Cc: CKI Project, Nathan Huckleberry, Herbert Xu [adding maintainers] Bruno- is this still an issue? On 10/3/22 00:16, Bruno Goncalves wrote: > Hello, > > recently we are hitting the following issue when booting using kernel > with debug options enabled: > > [ 126.821761] CPU: 1 PID: 296 Comm: cryptomgr_test Not tainted 6.0.0 #1 > [ 126.824021] Hardware name: Red Hat KVM, BIOS 1.15.0-1.el9 04/01/2014 > [ 126.825721] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] > [ 126.827045] Code: df 11 c4 41 31 ef fb 48 83 c2 08 0f 84 eb 01 00 > 00 48 89 d0 48 c1 e0 04 48 81 c7 80 00 00 00 48 29 c7 0f 10 06 66 41 > 0f ef c7 <0f> 28 0f c4 63 71 44 f0 01 c4 e3 71 44 d0 10 c4 63 71 44 e0 > 00 c4 This is in 'partial_stride' handling at the 'movaps' instruction: 11: 48 89 d0 mov %rdx,%rax 14: 48 c1 e0 04 shl $0x4,%rax 18: 48 81 c7 80 00 00 00 add $0x80,%rdi 1f: 48 29 c7 sub %rax,%rdi 22: 0f 10 06 movups (%rsi),%xmm0 25: 66 41 0f ef c7 pxor %xmm15,%xmm0 2a:* 0f 28 0f movaps (%rdi),%xmm1 <-- trapping instruction > [ 126.831535] RSP: 0018:ffff88810ed779d8 EFLAGS: 00010282 > [ 126.832957] RAX: 0000000000000030 RBX: 0000000000000030 RCX: ffff88810ede19e8 > [ 126.834907] RDX: 0000000000000003 RSI: ffff8881094b8000 RDI: ffff88811483c408 > [ 126.836554] RBP: ffff8881094b8000 R08: ffffffffc0300835 R09: ffff88810ede19e8 > [ 126.838529] R10: ffffed1021dbc33f R11: 0000000000000000 R12: 0000000000000030 > [ 126.840526] R13: 0000000000000003 R14: ffff88810ede19e8 R15: ffff88810ede19e0 > [ 126.842702] FS: 0000000000000000(0000) GS:ffff888193d00000(0000) > knlGS:0000000000000000 > [ 126.844574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 126.846796] CR2: 0000564f48e04178 CR3: 000000010be88002 CR4: 0000000000370ee0 > [ 126.848934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 126.850913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 126.852879] Call Trace: > [ 126.853703] <TASK> > [ 126.854436] polyval_x86_update+0x102/0x1e0 [polyval_clmulni] > [ 126.856092] test_shash_vec_cfg+0x523/0x9d0 > [ 126.857360] ? alg_test_aead+0x170/0x170 > [ 126.858682] ? pointer+0x6a0/0x6a0 > [ 126.859752] ? ___slab_alloc+0x910/0xf70 > [ 126.860950] ? __alg_test_hash.constprop.0+0x200/0x430 > [ 126.862140] ? __kmalloc+0x64/0x380 > [ 126.863233] ? sprintf+0xb3/0xe0 > [ 126.864159] ? rcu_read_lock_sched_held+0x3f/0x80 > [ 126.865560] ? trace_kmalloc+0x33/0xf0 > [ 126.866716] __alg_test_hash.constprop.0+0x2e7/0x430 > [ 126.867995] ? test_ahash_vec_cfg+0xca0/0xca0 > [ 126.869389] ? lockdep_hardirqs_on_prepare+0x230/0x230 > [ 126.870906] ? lockdep_hardirqs_on+0x7d/0x100 > [ 126.872221] ? alg_test_hash+0x8e/0x130 > [ 126.873398] alg_test+0x60d/0x7d0 > [ 126.874463] ? alloc_cipher_test_sglists+0xb0/0xb0 > [ 126.875867] ? lock_release+0x233/0x470 > [ 126.876726] ? __kthread_parkme+0x65/0xf0 > [ 126.877654] ? reacquire_held_locks+0x270/0x270 > [ 126.878984] ? mark_held_locks+0x24/0x90 > [ 126.879985] ? preempt_count_sub+0xb7/0x100 > [ 126.881260] ? crypto_acomp_scomp_free_ctx+0x70/0x70 > [ 126.882649] cryptomgr_test+0x36/0x60 > [ 126.883786] kthread+0x17e/0x1b0 > [ 126.884564] ? kthread_complete_and_exit+0x20/0x20 > [ 126.885975] ret_from_fork+0x22/0x30 > [ 126.887086] </TASK> > [ 126.887858] Modules linked in: polyval_clmulni(+) polyval_generic > virtio_blk(+) ghash_clmulni_intel floppy virtio_console serio_raw > ata_generic pata_acpi qemu_fw_cfg > > > full console log: > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/02/redhat:656405029/build_x86_64_redhat:656405029_x86_64_debug/tests/4/results_0001/console.log/console.log > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/5344697 > cki issue tracker: https://datawarehouse.cki-project.org/issue/1626 > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/build%20x86_64%20debug/3116060967/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.config > kernel tarball: > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/publish%20x86_64%20debug/3116060972/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.tar.gz > > We didn't bisect, but the first commit we tested that we hit the > problem is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=46452d3786a82bd732ba73fb308ae5cbe4e1e591 > > The last commit we tested and didn't hit the problem is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a1375562c0a87f0fa2eaf3e8ce15824696d4170a > > Thanks, > Bruno Goncalves > -- ~Randy ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] 2022-10-16 18:08 ` Randy Dunlap @ 2022-10-17 8:32 ` Bruno Goncalves 2022-10-17 19:59 ` Nathan Huckleberry 0 siblings, 1 reply; 6+ messages in thread From: Bruno Goncalves @ 2022-10-17 8:32 UTC (permalink / raw) To: Randy Dunlap; +Cc: LKML, CKI Project, Nathan Huckleberry, Herbert Xu On Sun, 16 Oct 2022 at 20:09, Randy Dunlap <rdunlap@infradead.org> wrote: > > [adding maintainers] > > Bruno- is this still an issue? Yes, we continue to reproduce it. Here is an example with 6.1-rc1 (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9abf2313adc1ca1b6180c508c25f22f9395cc780) console log: https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/17/redhat:668163532/build_x86_64_redhat:668163532_x86_64_debug/tests/1/results_0001/console.log/console.log tarball: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/668163532/publish%20x86_64%20debug/3180162139/artifacts/kernel-mainline.kernel.org-redhat_668163532_x86_64_debug.tar.gz kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/668163532/build%20x86_64%20debug/3180162134/artifacts/kernel-mainline.kernel.org-redhat_668163532_x86_64_debug.config Bruno > > > On 10/3/22 00:16, Bruno Goncalves wrote: > > Hello, > > > > recently we are hitting the following issue when booting using kernel > > with debug options enabled: > > > > [ 126.821761] CPU: 1 PID: 296 Comm: cryptomgr_test Not tainted 6.0.0 #1 > > [ 126.824021] Hardware name: Red Hat KVM, BIOS 1.15.0-1.el9 04/01/2014 > > [ 126.825721] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] > > [ 126.827045] Code: df 11 c4 41 31 ef fb 48 83 c2 08 0f 84 eb 01 00 > > 00 48 89 d0 48 c1 e0 04 48 81 c7 80 00 00 00 48 29 c7 0f 10 06 66 41 > > 0f ef c7 <0f> 28 0f c4 63 71 44 f0 01 c4 e3 71 44 d0 10 c4 63 71 44 e0 > > 00 c4 > > This is in 'partial_stride' handling at the 'movaps' instruction: > > 11: 48 89 d0 mov %rdx,%rax > 14: 48 c1 e0 04 shl $0x4,%rax > 18: 48 81 c7 80 00 00 00 add $0x80,%rdi > 1f: 48 29 c7 sub %rax,%rdi > 22: 0f 10 06 movups (%rsi),%xmm0 > 25: 66 41 0f ef c7 pxor %xmm15,%xmm0 > 2a:* 0f 28 0f movaps (%rdi),%xmm1 <-- trapping instruction > > > [ 126.831535] RSP: 0018:ffff88810ed779d8 EFLAGS: 00010282 > > [ 126.832957] RAX: 0000000000000030 RBX: 0000000000000030 RCX: ffff88810ede19e8 > > [ 126.834907] RDX: 0000000000000003 RSI: ffff8881094b8000 RDI: ffff88811483c408 > > [ 126.836554] RBP: ffff8881094b8000 R08: ffffffffc0300835 R09: ffff88810ede19e8 > > [ 126.838529] R10: ffffed1021dbc33f R11: 0000000000000000 R12: 0000000000000030 > > [ 126.840526] R13: 0000000000000003 R14: ffff88810ede19e8 R15: ffff88810ede19e0 > > [ 126.842702] FS: 0000000000000000(0000) GS:ffff888193d00000(0000) > > knlGS:0000000000000000 > > [ 126.844574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 126.846796] CR2: 0000564f48e04178 CR3: 000000010be88002 CR4: 0000000000370ee0 > > [ 126.848934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > [ 126.850913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > [ 126.852879] Call Trace: > > [ 126.853703] <TASK> > > [ 126.854436] polyval_x86_update+0x102/0x1e0 [polyval_clmulni] > > [ 126.856092] test_shash_vec_cfg+0x523/0x9d0 > > [ 126.857360] ? alg_test_aead+0x170/0x170 > > [ 126.858682] ? pointer+0x6a0/0x6a0 > > [ 126.859752] ? ___slab_alloc+0x910/0xf70 > > [ 126.860950] ? __alg_test_hash.constprop.0+0x200/0x430 > > [ 126.862140] ? __kmalloc+0x64/0x380 > > [ 126.863233] ? sprintf+0xb3/0xe0 > > [ 126.864159] ? rcu_read_lock_sched_held+0x3f/0x80 > > [ 126.865560] ? trace_kmalloc+0x33/0xf0 > > [ 126.866716] __alg_test_hash.constprop.0+0x2e7/0x430 > > [ 126.867995] ? test_ahash_vec_cfg+0xca0/0xca0 > > [ 126.869389] ? lockdep_hardirqs_on_prepare+0x230/0x230 > > [ 126.870906] ? lockdep_hardirqs_on+0x7d/0x100 > > [ 126.872221] ? alg_test_hash+0x8e/0x130 > > [ 126.873398] alg_test+0x60d/0x7d0 > > [ 126.874463] ? alloc_cipher_test_sglists+0xb0/0xb0 > > [ 126.875867] ? lock_release+0x233/0x470 > > [ 126.876726] ? __kthread_parkme+0x65/0xf0 > > [ 126.877654] ? reacquire_held_locks+0x270/0x270 > > [ 126.878984] ? mark_held_locks+0x24/0x90 > > [ 126.879985] ? preempt_count_sub+0xb7/0x100 > > [ 126.881260] ? crypto_acomp_scomp_free_ctx+0x70/0x70 > > [ 126.882649] cryptomgr_test+0x36/0x60 > > [ 126.883786] kthread+0x17e/0x1b0 > > [ 126.884564] ? kthread_complete_and_exit+0x20/0x20 > > [ 126.885975] ret_from_fork+0x22/0x30 > > [ 126.887086] </TASK> > > [ 126.887858] Modules linked in: polyval_clmulni(+) polyval_generic > > virtio_blk(+) ghash_clmulni_intel floppy virtio_console serio_raw > > ata_generic pata_acpi qemu_fw_cfg > > > > > > full console log: > > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/02/redhat:656405029/build_x86_64_redhat:656405029_x86_64_debug/tests/4/results_0001/console.log/console.log > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/5344697 > > cki issue tracker: https://datawarehouse.cki-project.org/issue/1626 > > > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/build%20x86_64%20debug/3116060967/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.config > > kernel tarball: > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/publish%20x86_64%20debug/3116060972/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.tar.gz > > > > We didn't bisect, but the first commit we tested that we hit the > > problem is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=46452d3786a82bd732ba73fb308ae5cbe4e1e591 > > > > The last commit we tested and didn't hit the problem is > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a1375562c0a87f0fa2eaf3e8ce15824696d4170a > > > > Thanks, > > Bruno Goncalves > > > > -- > ~Randy > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] 2022-10-17 8:32 ` Bruno Goncalves @ 2022-10-17 19:59 ` Nathan Huckleberry 2022-10-17 22:29 ` Nathan Huckleberry 0 siblings, 1 reply; 6+ messages in thread From: Nathan Huckleberry @ 2022-10-17 19:59 UTC (permalink / raw) To: Bruno Goncalves; +Cc: Randy Dunlap, LKML, CKI Project, Herbert Xu, Eric Biggers Hello, On Mon, Oct 17, 2022 at 1:33 AM Bruno Goncalves <bgoncalv@redhat.com> wrote: > > On Sun, 16 Oct 2022 at 20:09, Randy Dunlap <rdunlap@infradead.org> wrote: > > > > [adding maintainers] > > > > Bruno- is this still an issue? > > Yes, we continue to reproduce it. > > Here is an example with 6.1-rc1 > (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9abf2313adc1ca1b6180c508c25f22f9395cc780) > > console log: https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/17/redhat:668163532/build_x86_64_redhat:668163532_x86_64_debug/tests/1/results_0001/console.log/console.log > tarball: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/668163532/publish%20x86_64%20debug/3180162139/artifacts/kernel-mainline.kernel.org-redhat_668163532_x86_64_debug.tar.gz > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/668163532/build%20x86_64%20debug/3180162134/artifacts/kernel-mainline.kernel.org-redhat_668163532_x86_64_debug.config > > > Bruno > > > > > > On 10/3/22 00:16, Bruno Goncalves wrote: > > > Hello, > > > > > > recently we are hitting the following issue when booting using kernel > > > with debug options enabled: > > > > > > [ 126.821761] CPU: 1 PID: 296 Comm: cryptomgr_test Not tainted 6.0.0 #1 > > > [ 126.824021] Hardware name: Red Hat KVM, BIOS 1.15.0-1.el9 04/01/2014 > > > [ 126.825721] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] > > > [ 126.827045] Code: df 11 c4 41 31 ef fb 48 83 c2 08 0f 84 eb 01 00 > > > 00 48 89 d0 48 c1 e0 04 48 81 c7 80 00 00 00 48 29 c7 0f 10 06 66 41 > > > 0f ef c7 <0f> 28 0f c4 63 71 44 f0 01 c4 e3 71 44 d0 10 c4 63 71 44 e0 > > > 00 c4 > > > > This is in 'partial_stride' handling at the 'movaps' instruction: > > > > 11: 48 89 d0 mov %rdx,%rax > > 14: 48 c1 e0 04 shl $0x4,%rax > > 18: 48 81 c7 80 00 00 00 add $0x80,%rdi > > 1f: 48 29 c7 sub %rax,%rdi > > 22: 0f 10 06 movups (%rsi),%xmm0 > > 25: 66 41 0f ef c7 pxor %xmm15,%xmm0 > > 2a:* 0f 28 0f movaps (%rdi),%xmm1 <-- trapping instruction It looks like the movaps here should be a movups. The keys are not guaranteed to be 16-byte aligned. I'll submit a patch at some point today. > > > > > [ 126.831535] RSP: 0018:ffff88810ed779d8 EFLAGS: 00010282 > > > [ 126.832957] RAX: 0000000000000030 RBX: 0000000000000030 RCX: ffff88810ede19e8 > > > [ 126.834907] RDX: 0000000000000003 RSI: ffff8881094b8000 RDI: ffff88811483c408 > > > [ 126.836554] RBP: ffff8881094b8000 R08: ffffffffc0300835 R09: ffff88810ede19e8 > > > [ 126.838529] R10: ffffed1021dbc33f R11: 0000000000000000 R12: 0000000000000030 > > > [ 126.840526] R13: 0000000000000003 R14: ffff88810ede19e8 R15: ffff88810ede19e0 > > > [ 126.842702] FS: 0000000000000000(0000) GS:ffff888193d00000(0000) > > > knlGS:0000000000000000 > > > [ 126.844574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 126.846796] CR2: 0000564f48e04178 CR3: 000000010be88002 CR4: 0000000000370ee0 > > > [ 126.848934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > [ 126.850913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > [ 126.852879] Call Trace: > > > [ 126.853703] <TASK> > > > [ 126.854436] polyval_x86_update+0x102/0x1e0 [polyval_clmulni] > > > [ 126.856092] test_shash_vec_cfg+0x523/0x9d0 > > > [ 126.857360] ? alg_test_aead+0x170/0x170 > > > [ 126.858682] ? pointer+0x6a0/0x6a0 > > > [ 126.859752] ? ___slab_alloc+0x910/0xf70 > > > [ 126.860950] ? __alg_test_hash.constprop.0+0x200/0x430 > > > [ 126.862140] ? __kmalloc+0x64/0x380 > > > [ 126.863233] ? sprintf+0xb3/0xe0 > > > [ 126.864159] ? rcu_read_lock_sched_held+0x3f/0x80 > > > [ 126.865560] ? trace_kmalloc+0x33/0xf0 > > > [ 126.866716] __alg_test_hash.constprop.0+0x2e7/0x430 > > > [ 126.867995] ? test_ahash_vec_cfg+0xca0/0xca0 > > > [ 126.869389] ? lockdep_hardirqs_on_prepare+0x230/0x230 > > > [ 126.870906] ? lockdep_hardirqs_on+0x7d/0x100 > > > [ 126.872221] ? alg_test_hash+0x8e/0x130 > > > [ 126.873398] alg_test+0x60d/0x7d0 > > > [ 126.874463] ? alloc_cipher_test_sglists+0xb0/0xb0 > > > [ 126.875867] ? lock_release+0x233/0x470 > > > [ 126.876726] ? __kthread_parkme+0x65/0xf0 > > > [ 126.877654] ? reacquire_held_locks+0x270/0x270 > > > [ 126.878984] ? mark_held_locks+0x24/0x90 > > > [ 126.879985] ? preempt_count_sub+0xb7/0x100 > > > [ 126.881260] ? crypto_acomp_scomp_free_ctx+0x70/0x70 > > > [ 126.882649] cryptomgr_test+0x36/0x60 > > > [ 126.883786] kthread+0x17e/0x1b0 > > > [ 126.884564] ? kthread_complete_and_exit+0x20/0x20 > > > [ 126.885975] ret_from_fork+0x22/0x30 > > > [ 126.887086] </TASK> > > > [ 126.887858] Modules linked in: polyval_clmulni(+) polyval_generic > > > virtio_blk(+) ghash_clmulni_intel floppy virtio_console serio_raw > > > ata_generic pata_acpi qemu_fw_cfg > > > > > > > > > full console log: > > > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/02/redhat:656405029/build_x86_64_redhat:656405029_x86_64_debug/tests/4/results_0001/console.log/console.log > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/5344697 > > > cki issue tracker: https://datawarehouse.cki-project.org/issue/1626 > > > > > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/build%20x86_64%20debug/3116060967/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.config > > > kernel tarball: > > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/publish%20x86_64%20debug/3116060972/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.tar.gz > > > > > > We didn't bisect, but the first commit we tested that we hit the > > > problem is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=46452d3786a82bd732ba73fb308ae5cbe4e1e591 > > > > > > The last commit we tested and didn't hit the problem is > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a1375562c0a87f0fa2eaf3e8ce15824696d4170a > > > > > > Thanks, > > > Bruno Goncalves > > > > > > > -- > > ~Randy > > > Thanks, Huck ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] 2022-10-17 19:59 ` Nathan Huckleberry @ 2022-10-17 22:29 ` Nathan Huckleberry 0 siblings, 0 replies; 6+ messages in thread From: Nathan Huckleberry @ 2022-10-17 22:29 UTC (permalink / raw) To: Bruno Goncalves; +Cc: Randy Dunlap, LKML, CKI Project, Herbert Xu, Eric Biggers I submitted a patch to fix this. See: https://lore.kernel.org/all/20221017222620.715153-1-nhuck@google.com/T/#u Thanks, Huck On Mon, Oct 17, 2022 at 12:59 PM Nathan Huckleberry <nhuck@google.com> wrote: > > Hello, > On Mon, Oct 17, 2022 at 1:33 AM Bruno Goncalves <bgoncalv@redhat.com> wrote: > > > > On Sun, 16 Oct 2022 at 20:09, Randy Dunlap <rdunlap@infradead.org> wrote: > > > > > > [adding maintainers] > > > > > > Bruno- is this still an issue? > > > > Yes, we continue to reproduce it. > > > > Here is an example with 6.1-rc1 > > (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9abf2313adc1ca1b6180c508c25f22f9395cc780) > > > > console log: https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/17/redhat:668163532/build_x86_64_redhat:668163532_x86_64_debug/tests/1/results_0001/console.log/console.log > > tarball: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/668163532/publish%20x86_64%20debug/3180162139/artifacts/kernel-mainline.kernel.org-redhat_668163532_x86_64_debug.tar.gz > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/668163532/build%20x86_64%20debug/3180162134/artifacts/kernel-mainline.kernel.org-redhat_668163532_x86_64_debug.config > > > > > > Bruno > > > > > > > > > On 10/3/22 00:16, Bruno Goncalves wrote: > > > > Hello, > > > > > > > > recently we are hitting the following issue when booting using kernel > > > > with debug options enabled: > > > > > > > > [ 126.821761] CPU: 1 PID: 296 Comm: cryptomgr_test Not tainted 6.0.0 #1 > > > > [ 126.824021] Hardware name: Red Hat KVM, BIOS 1.15.0-1.el9 04/01/2014 > > > > [ 126.825721] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] > > > > [ 126.827045] Code: df 11 c4 41 31 ef fb 48 83 c2 08 0f 84 eb 01 00 > > > > 00 48 89 d0 48 c1 e0 04 48 81 c7 80 00 00 00 48 29 c7 0f 10 06 66 41 > > > > 0f ef c7 <0f> 28 0f c4 63 71 44 f0 01 c4 e3 71 44 d0 10 c4 63 71 44 e0 > > > > 00 c4 > > > > > > This is in 'partial_stride' handling at the 'movaps' instruction: > > > > > > 11: 48 89 d0 mov %rdx,%rax > > > 14: 48 c1 e0 04 shl $0x4,%rax > > > 18: 48 81 c7 80 00 00 00 add $0x80,%rdi > > > 1f: 48 29 c7 sub %rax,%rdi > > > 22: 0f 10 06 movups (%rsi),%xmm0 > > > 25: 66 41 0f ef c7 pxor %xmm15,%xmm0 > > > 2a:* 0f 28 0f movaps (%rdi),%xmm1 <-- trapping instruction > > It looks like the movaps here should be a movups. The keys are not > guaranteed to be 16-byte aligned. > > I'll submit a patch at some point today. > > > > > > > > [ 126.831535] RSP: 0018:ffff88810ed779d8 EFLAGS: 00010282 > > > > [ 126.832957] RAX: 0000000000000030 RBX: 0000000000000030 RCX: ffff88810ede19e8 > > > > [ 126.834907] RDX: 0000000000000003 RSI: ffff8881094b8000 RDI: ffff88811483c408 > > > > [ 126.836554] RBP: ffff8881094b8000 R08: ffffffffc0300835 R09: ffff88810ede19e8 > > > > [ 126.838529] R10: ffffed1021dbc33f R11: 0000000000000000 R12: 0000000000000030 > > > > [ 126.840526] R13: 0000000000000003 R14: ffff88810ede19e8 R15: ffff88810ede19e0 > > > > [ 126.842702] FS: 0000000000000000(0000) GS:ffff888193d00000(0000) > > > > knlGS:0000000000000000 > > > > [ 126.844574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 126.846796] CR2: 0000564f48e04178 CR3: 000000010be88002 CR4: 0000000000370ee0 > > > > [ 126.848934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > > [ 126.850913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > > [ 126.852879] Call Trace: > > > > [ 126.853703] <TASK> > > > > [ 126.854436] polyval_x86_update+0x102/0x1e0 [polyval_clmulni] > > > > [ 126.856092] test_shash_vec_cfg+0x523/0x9d0 > > > > [ 126.857360] ? alg_test_aead+0x170/0x170 > > > > [ 126.858682] ? pointer+0x6a0/0x6a0 > > > > [ 126.859752] ? ___slab_alloc+0x910/0xf70 > > > > [ 126.860950] ? __alg_test_hash.constprop.0+0x200/0x430 > > > > [ 126.862140] ? __kmalloc+0x64/0x380 > > > > [ 126.863233] ? sprintf+0xb3/0xe0 > > > > [ 126.864159] ? rcu_read_lock_sched_held+0x3f/0x80 > > > > [ 126.865560] ? trace_kmalloc+0x33/0xf0 > > > > [ 126.866716] __alg_test_hash.constprop.0+0x2e7/0x430 > > > > [ 126.867995] ? test_ahash_vec_cfg+0xca0/0xca0 > > > > [ 126.869389] ? lockdep_hardirqs_on_prepare+0x230/0x230 > > > > [ 126.870906] ? lockdep_hardirqs_on+0x7d/0x100 > > > > [ 126.872221] ? alg_test_hash+0x8e/0x130 > > > > [ 126.873398] alg_test+0x60d/0x7d0 > > > > [ 126.874463] ? alloc_cipher_test_sglists+0xb0/0xb0 > > > > [ 126.875867] ? lock_release+0x233/0x470 > > > > [ 126.876726] ? __kthread_parkme+0x65/0xf0 > > > > [ 126.877654] ? reacquire_held_locks+0x270/0x270 > > > > [ 126.878984] ? mark_held_locks+0x24/0x90 > > > > [ 126.879985] ? preempt_count_sub+0xb7/0x100 > > > > [ 126.881260] ? crypto_acomp_scomp_free_ctx+0x70/0x70 > > > > [ 126.882649] cryptomgr_test+0x36/0x60 > > > > [ 126.883786] kthread+0x17e/0x1b0 > > > > [ 126.884564] ? kthread_complete_and_exit+0x20/0x20 > > > > [ 126.885975] ret_from_fork+0x22/0x30 > > > > [ 126.887086] </TASK> > > > > [ 126.887858] Modules linked in: polyval_clmulni(+) polyval_generic > > > > virtio_blk(+) ghash_clmulni_intel floppy virtio_console serio_raw > > > > ata_generic pata_acpi qemu_fw_cfg > > > > > > > > > > > > full console log: > > > > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/10/02/redhat:656405029/build_x86_64_redhat:656405029_x86_64_debug/tests/4/results_0001/console.log/console.log > > > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/5344697 > > > > cki issue tracker: https://datawarehouse.cki-project.org/issue/1626 > > > > > > > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/build%20x86_64%20debug/3116060967/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.config > > > > kernel tarball: > > > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/656405029/publish%20x86_64%20debug/3116060972/artifacts/kernel-mainline.kernel.org-redhat_656405029_x86_64_debug.tar.gz > > > > > > > > We didn't bisect, but the first commit we tested that we hit the > > > > problem is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=46452d3786a82bd732ba73fb308ae5cbe4e1e591 > > > > > > > > The last commit we tested and didn't hit the problem is > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a1375562c0a87f0fa2eaf3e8ce15824696d4170a > > > > > > > > Thanks, > > > > Bruno Goncalves > > > > > > > > > > -- > > > ~Randy > > > > > > Thanks, > Huck ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-10-17 22:31 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-10-03 7:16 [6.0.0] RIP: 0010:clmul_polyval_update+0x3e9/0x5d0 [polyval_clmulni] Bruno Goncalves 2022-10-16 11:37 ` Thorsten Leemhuis 2022-10-16 18:08 ` Randy Dunlap 2022-10-17 8:32 ` Bruno Goncalves 2022-10-17 19:59 ` Nathan Huckleberry 2022-10-17 22:29 ` Nathan Huckleberry
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).