* kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) @ 2022-11-23 8:48 Bruno Goncalves 2022-11-23 13:46 ` Jens Axboe 0 siblings, 1 reply; 9+ messages in thread From: Bruno Goncalves @ 2022-11-23 8:48 UTC (permalink / raw) To: linux-block; +Cc: Jens Axboe, CKI Project Hello, We recently started to hit the following panic when testing the block tree (for-next branch). [ 5076.172749] list_add corruption. prev->next should be next (ffff91cd6f7fa568), but was ffff91c991ca6670. (prev=ffff91c991ca6670). [ 5076.173863] ------------[ cut here ]------------ [ 5076.174853] kernel BUG at lib/list_debug.c:30! [ 5076.175523] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 5076.175853] CPU: 15 PID: 16415 Comm: kworker/15:13 Tainted: G I 6.1.0-rc6 #1 [ 5076.176799] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 [ 5076.177198] Workqueue: cgwb_release cgwb_release_workfn [ 5076.177497] RIP: 0010:__list_add_valid.cold+0x3a/0x5b [ 5076.177788] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 c7 c7 [ 5076.179173] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 [ 5076.179472] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 [ 5076.180241] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff [ 5076.181069] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 [ 5076.182209] R10: 0000000000000003 R11: ffff91cd7ff42fe8 R12: ffff91cd6f7fa568 [ 5076.183002] R13: ffff91c991ca6670 R14: ffff91c991ca6670 R15: ffff91cd6f7f1440 [ 5076.183902] FS: 0000000000000000(0000) GS:ffff91cd6f7c0000(0000) knlGS:0000000000000000 [ 5076.184377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5076.185084] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 [ 5076.185945] Call Trace: [ 5076.186110] <TASK> [ 5076.186916] insert_work+0x46/0xc0 [ 5076.187533] __queue_work+0x1d4/0x460 [ 5076.187788] queue_work_on+0x37/0x40 [ 5076.187993] blkcg_unpin_online+0x1ad/0x1b0 [ 5076.188244] cgwb_release_workfn+0x6a/0x200 [ 5076.188464] process_one_work+0x1c7/0x380 [ 5076.188675] worker_thread+0x4d/0x380 [ 5076.188881] ? rescuer_thread+0x380/0x380 [ 5076.189089] kthread+0xe9/0x110 [ 5076.189716] ? kthread_complete_and_exit+0x20/0x20 [ 5076.190407] ret_from_fork+0x22/0x30 [ 5076.190677] </TASK> [ 5076.190816] Modules linked in: nvme nvme_core nvme_common loop tls rfkill intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp sunrpc kvm_intel kvm iTCO_wdt iapl intel_cstate intel_uncore pcspkr lpc_ich ipmi_ssif hpilo tg3 acpi_ipmi ioatdma ipmi_si ipmi_devintf dca ipmi_msghandler acpi_power_meter fuse zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 serio_raw hpsa mgag200 scsi_transport_sas [last unloaded: scsi_debug] [ 5076.293315] ---[ end trace 0000000000000000 ]--- [ 5076.295226] RIP: 0010:__list_add_valid.cold+0x3a/0x5b [ 5076.295587] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 c7 c7 [ 5076.296921] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 [ 5076.297239] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 [ 5076.297983] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff [ 5076.298768] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 [ 5076.299525] R10: 0000S: 0000000000000000(0000) GS:ffff91cd6f7c0000(0000) knlGS:0000000000000000 [ 5076.700351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5076.701046] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 [ 5076ernel panic - not syncing: Fatal exception [ 5077.924713] Shutting down cpus with NMI [ 5077.924986] Kernel Offset: 0x2b000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 5077.927946] ---[ end Kernel panic - not syncing: Fatal exception ]--- It seems to happen often during different tests. full console.log: https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/11/21/redhat:700955106/build_x86_64_redhat:700955106_x86_64/tests/1/results_0001/console.log/console.log kernel tarball: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/publish%20x86_64/3356091217/artifacts/kernel-block-redhat_700955106_x86_64.tar.gz kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/build%20x86_64/3356091207/artifacts/kernel-block-redhat_700955106_x86_64.config test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 We didn't bisect, but the first commit we hit the problem was "f65d92c600fe6eecdbd6e7fab7893c9c094dfcbf (io_uring-6.1-2022-11-18-2180-gf65d92c600fe)" and the last one where we didn't hit the problem was "40fa774af7fd04d06014ac74947c351649b6f64f (io_uring-6.1-2022-11-11-1843-g40fa774af7fd)" test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 cki issue tracker: https://datawarehouse.cki-project.org/issue/1732 Thank you, Bruno Goncalves ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-23 8:48 kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) Bruno Goncalves @ 2022-11-23 13:46 ` Jens Axboe 2022-11-24 14:57 ` Bruno Goncalves 0 siblings, 1 reply; 9+ messages in thread From: Jens Axboe @ 2022-11-23 13:46 UTC (permalink / raw) To: Bruno Goncalves, linux-block; +Cc: CKI Project On 11/23/22 1:48 AM, Bruno Goncalves wrote: > Hello, > > We recently started to hit the following panic when testing the block > tree (for-next branch). > > [ 5076.172749] list_add corruption. prev->next should be next > (ffff91cd6f7fa568), but was ffff91c991ca6670. (prev=ffff91c991ca6670). > [ 5076.173863] ------------[ cut here ]------------ > [ 5076.174853] kernel BUG at lib/list_debug.c:30! > [ 5076.175523] invalid opcode: 0000 [#1] PREEMPT SMP PTI > [ 5076.175853] CPU: 15 PID: 16415 Comm: kworker/15:13 Tainted: G > I 6.1.0-rc6 #1 > [ 5076.176799] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 > [ 5076.177198] Workqueue: cgwb_release cgwb_release_workfn > [ 5076.177497] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > [ 5076.177788] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > c7 c7 > [ 5076.179173] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > [ 5076.179472] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > [ 5076.180241] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > [ 5076.181069] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > [ 5076.182209] R10: 0000000000000003 R11: ffff91cd7ff42fe8 R12: ffff91cd6f7fa568 > [ 5076.183002] R13: ffff91c991ca6670 R14: ffff91c991ca6670 R15: ffff91cd6f7f1440 > [ 5076.183902] FS: 0000000000000000(0000) GS:ffff91cd6f7c0000(0000) > knlGS:0000000000000000 > [ 5076.184377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 5076.185084] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > [ 5076.185945] Call Trace: > [ 5076.186110] <TASK> > [ 5076.186916] insert_work+0x46/0xc0 > [ 5076.187533] __queue_work+0x1d4/0x460 > [ 5076.187788] queue_work_on+0x37/0x40 > [ 5076.187993] blkcg_unpin_online+0x1ad/0x1b0 > [ 5076.188244] cgwb_release_workfn+0x6a/0x200 > [ 5076.188464] process_one_work+0x1c7/0x380 > [ 5076.188675] worker_thread+0x4d/0x380 > [ 5076.188881] ? rescuer_thread+0x380/0x380 > [ 5076.189089] kthread+0xe9/0x110 > [ 5076.189716] ? kthread_complete_and_exit+0x20/0x20 > [ 5076.190407] ret_from_fork+0x22/0x30 > [ 5076.190677] </TASK> > [ 5076.190816] Modules linked in: nvme nvme_core nvme_common loop tls > rfkill intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal > intel_powerclamp coretemp sunrpc kvm_intel kvm iTCO_wdt iapl > intel_cstate intel_uncore pcspkr lpc_ich ipmi_ssif hpilo tg3 acpi_ipmi > ioatdma ipmi_si ipmi_devintf dca ipmi_msghandler acpi_power_meter fuse > zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni > polyval_generic ghash_clmulni_intel sha512_ssse3 serio_raw hpsa > mgag200 scsi_transport_sas [last unloaded: scsi_debug] > [ 5076.293315] ---[ end trace 0000000000000000 ]--- > [ 5076.295226] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > [ 5076.295587] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > c7 c7 > [ 5076.296921] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > [ 5076.297239] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > [ 5076.297983] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > [ 5076.298768] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > [ 5076.299525] R10: 0000S: 0000000000000000(0000) > GS:ffff91cd6f7c0000(0000) knlGS:0000000000000000 > [ 5076.700351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 5076.701046] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > [ 5076ernel panic - not syncing: Fatal exception > [ 5077.924713] Shutting down cpus with NMI > [ 5077.924986] Kernel Offset: 0x2b000000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > [ 5077.927946] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > It seems to happen often during different tests. > > full console.log: > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/11/21/redhat:700955106/build_x86_64_redhat:700955106_x86_64/tests/1/results_0001/console.log/console.log > > kernel tarball: > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/publish%20x86_64/3356091217/artifacts/kernel-block-redhat_700955106_x86_64.tar.gz > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/build%20x86_64/3356091207/artifacts/kernel-block-redhat_700955106_x86_64.config > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > We didn't bisect, but the first commit we hit the problem was > "f65d92c600fe6eecdbd6e7fab7893c9c094dfcbf > (io_uring-6.1-2022-11-18-2180-gf65d92c600fe)" and the last one where > we didn't hit the problem was > "40fa774af7fd04d06014ac74947c351649b6f64f > (io_uring-6.1-2022-11-11-1843-g40fa774af7fd)" > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > cki issue tracker: https://datawarehouse.cki-project.org/issue/1732 Please just try and clone for-6.2/block from the block tree and bisect it? -- Jens Axboe ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-23 13:46 ` Jens Axboe @ 2022-11-24 14:57 ` Bruno Goncalves 2022-11-25 8:38 ` Yi Zhang 0 siblings, 1 reply; 9+ messages in thread From: Bruno Goncalves @ 2022-11-24 14:57 UTC (permalink / raw) To: Jens Axboe; +Cc: linux-block, CKI Project On Wed, 23 Nov 2022 at 14:46, Jens Axboe <axboe@kernel.dk> wrote: > > On 11/23/22 1:48 AM, Bruno Goncalves wrote: > > Hello, > > > > We recently started to hit the following panic when testing the block > > tree (for-next branch). > > > > [ 5076.172749] list_add corruption. prev->next should be next > > (ffff91cd6f7fa568), but was ffff91c991ca6670. (prev=ffff91c991ca6670). > > [ 5076.173863] ------------[ cut here ]------------ > > [ 5076.174853] kernel BUG at lib/list_debug.c:30! > > [ 5076.175523] invalid opcode: 0000 [#1] PREEMPT SMP PTI > > [ 5076.175853] CPU: 15 PID: 16415 Comm: kworker/15:13 Tainted: G > > I 6.1.0-rc6 #1 > > [ 5076.176799] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 > > [ 5076.177198] Workqueue: cgwb_release cgwb_release_workfn > > [ 5076.177497] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > [ 5076.177788] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > c7 c7 > > [ 5076.179173] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > [ 5076.179472] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > [ 5076.180241] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > [ 5076.181069] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > [ 5076.182209] R10: 0000000000000003 R11: ffff91cd7ff42fe8 R12: ffff91cd6f7fa568 > > [ 5076.183002] R13: ffff91c991ca6670 R14: ffff91c991ca6670 R15: ffff91cd6f7f1440 > > [ 5076.183902] FS: 0000000000000000(0000) GS:ffff91cd6f7c0000(0000) > > knlGS:0000000000000000 > > [ 5076.184377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 5076.185084] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > [ 5076.185945] Call Trace: > > [ 5076.186110] <TASK> > > [ 5076.186916] insert_work+0x46/0xc0 > > [ 5076.187533] __queue_work+0x1d4/0x460 > > [ 5076.187788] queue_work_on+0x37/0x40 > > [ 5076.187993] blkcg_unpin_online+0x1ad/0x1b0 > > [ 5076.188244] cgwb_release_workfn+0x6a/0x200 > > [ 5076.188464] process_one_work+0x1c7/0x380 > > [ 5076.188675] worker_thread+0x4d/0x380 > > [ 5076.188881] ? rescuer_thread+0x380/0x380 > > [ 5076.189089] kthread+0xe9/0x110 > > [ 5076.189716] ? kthread_complete_and_exit+0x20/0x20 > > [ 5076.190407] ret_from_fork+0x22/0x30 > > [ 5076.190677] </TASK> > > [ 5076.190816] Modules linked in: nvme nvme_core nvme_common loop tls > > rfkill intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal > > intel_powerclamp coretemp sunrpc kvm_intel kvm iTCO_wdt iapl > > intel_cstate intel_uncore pcspkr lpc_ich ipmi_ssif hpilo tg3 acpi_ipmi > > ioatdma ipmi_si ipmi_devintf dca ipmi_msghandler acpi_power_meter fuse > > zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni > > polyval_generic ghash_clmulni_intel sha512_ssse3 serio_raw hpsa > > mgag200 scsi_transport_sas [last unloaded: scsi_debug] > > [ 5076.293315] ---[ end trace 0000000000000000 ]--- > > [ 5076.295226] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > [ 5076.295587] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > c7 c7 > > [ 5076.296921] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > [ 5076.297239] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > [ 5076.297983] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > [ 5076.298768] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > [ 5076.299525] R10: 0000S: 0000000000000000(0000) > > GS:ffff91cd6f7c0000(0000) knlGS:0000000000000000 > > [ 5076.700351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 5076.701046] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > [ 5076ernel panic - not syncing: Fatal exception > > [ 5077.924713] Shutting down cpus with NMI > > [ 5077.924986] Kernel Offset: 0x2b000000 from 0xffffffff81000000 > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > [ 5077.927946] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > > It seems to happen often during different tests. > > > > full console.log: > > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/11/21/redhat:700955106/build_x86_64_redhat:700955106_x86_64/tests/1/results_0001/console.log/console.log > > > > kernel tarball: > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/publish%20x86_64/3356091217/artifacts/kernel-block-redhat_700955106_x86_64.tar.gz > > > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/build%20x86_64/3356091207/artifacts/kernel-block-redhat_700955106_x86_64.config > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > > > We didn't bisect, but the first commit we hit the problem was > > "f65d92c600fe6eecdbd6e7fab7893c9c094dfcbf > > (io_uring-6.1-2022-11-18-2180-gf65d92c600fe)" and the last one where > > we didn't hit the problem was > > "40fa774af7fd04d06014ac74947c351649b6f64f > > (io_uring-6.1-2022-11-11-1843-g40fa774af7fd)" > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > cki issue tracker: https://datawarehouse.cki-project.org/issue/1732 > > Please just try and clone for-6.2/block from the block tree and bisect > it? > Hi, I've tried with commit 93c68cc46a070775cc6675e3543dd909eb9f6c9e (drbd: use consistent license), but I was not able to hit the panic with it. Bruno > -- > Jens Axboe > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-24 14:57 ` Bruno Goncalves @ 2022-11-25 8:38 ` Yi Zhang 2022-11-26 14:29 ` [bisected]kernel " Yi Zhang 0 siblings, 1 reply; 9+ messages in thread From: Yi Zhang @ 2022-11-25 8:38 UTC (permalink / raw) To: Bruno Goncalves; +Cc: Jens Axboe, linux-block, CKI Project I reproduced this issue even when system boot with the latest linux-block/for-next, will try to bisect it later. 43f3ae1898c9 (HEAD -> for-next, origin/for-next) Merge branch 'for-6.2/writeback' into for-next d6798bc243fa writeback: Add asserts for adding freed inode to lists [ 24.183829] list_add corruption. prev->next should be next (ffff9a1d9f337f68), but was ffff9a1a02119e70. (prev=ffff9a1a02119e70). [ 24.195478] ------------[ cut here ]------------ [ 24.200088] kernel BUG at lib/list_debug.c:30! [ 24.204532] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 24.209751] CPU: 4 PID: 167 Comm: kworker/4:1 Not tainted 6.1.0-rc6+ #1 [ 24.216365] Hardware name: Dell Inc. PowerEdge R6515/07PXPY, BIOS 2.8.5 08/18/2022 [ 24.223930] Workqueue: cgwb_release cgwb_release_workfn [ 24.229157] RIP: 0010:__list_add_valid.cold+0x3a/0x5b [ 24.234208] Code: f2 4c 89 c1 48 89 fe 48 c7 c7 20 23 65 a8 e8 d2 a2 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 c8 22 65 a8 e8 bb a2 fe ff <0f> 0b 4c 89 c1 48 c7 c7 70 22 65 a8 e8 aa a2 fe ff 0f 0b 48 c7 c7 [ 24.252953] RSP: 0018:ffffb035407e7da8 EFLAGS: 00010046 [ 24.258172] RAX: 0000000000000075 RBX: ffff9a1a02119e68 RCX: 0000000000000000 [ 24.265303] RDX: 0000000000000000 RSI: ffff9a1d9f31f840 RDI: ffff9a1d9f31f840 [ 24.272428] RBP: ffff9a1d9f337f00 R08: 0000000000000000 R09: 00000000ffff7fff [ 24.279560] R10: ffffb035407e7c50 R11: ffffffffa8be75e8 R12: ffff9a1d9f337f68 [ 24.286683] R13: ffff9a1a02119e70 R14: ffff9a1a02119e70 R15: ffff9a1d9f330340 [ 24.293808] FS: 0000000000000000(0000) GS:ffff9a1d9f300000(0000) knlGS:0000000000000000 [ 24.301894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 24.307641] CR2: 000055b5f1f28050 CR3: 0000000104e38000 CR4: 0000000000350ee0 [ 24.314772] Call Trace: [ 24.314774] <TASK> [ 24.314774] insert_work+0x46/0xc0 [ 24.314780] __queue_work+0x1d5/0x380 [ 24.326376] queue_work_on+0x24/0x30 [ 24.329955] blkcg_unpin_online+0x1b5/0x1c0 [ 24.334143] cgwb_release_workfn+0x6a/0x200 [ 24.338327] process_one_work+0x1e5/0x3b0 [ 24.342342] ? rescuer_thread+0x390/0x390 [ 24.346352] worker_thread+0x50/0x3a0 [ 24.350019] ? rescuer_thread+0x390/0x390 [ 24.354030] kthread+0xd9/0x100 [ 24.357177] ? kthread_complete_and_exit+0x20/0x20 [ 24.361970] ret_from_fork+0x22/0x30 [ 24.365550] </TASK> [ 24.367742] Modules linked in: sunrpc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd ipmi_ssif kvm_amd kvm mgag200 ledtrig_audio rfkill video i2c_algo_bit drm_shmem_helper dcdbas drm_kms_helper irqbypass dell_smbios rapl dell_wmi_descriptor wmi_bmof pcspkr syscopyarea acpi_ipmi sysfillrect sysimgblt fb_sys_fops ipmi_si ipmi_devintf ptdma i2c_piix4 k10temp ipmi_msghandler acpi_power_meter vfat fat acpi_cpufreq drm fuse xfs libcrc32c sd_mod sg ahci crct10dif_pclmul crc32_pclmul libahci crc32c_intel ghash_clmulni_intel mpt3sas nvme tg3 libata nvme_core ccp raid_class nvme_common t10_pi sp5100_tco scsi_transport_sas wmi dm_mirror dm_region_hash dm_log dm_mod [ 24.426475] ---[ end trace 0000000000000000 ]--- [ 24.505278] RIP: 0010:__list_add_valid.cold+0x3a/0x5b [ 24.510331] Code: f2 4c 89 c1 48 89 fe 48 c7 c7 20 23 65 a8 e8 d2 a2 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 c8 22 65 a8 e8 bb a2 fe ff <0f> 0b 4c 89 c1 48 c7 c7 70 22 65 a8 e8 aa a2 fe ff 0f 0b 48 c7 c7 [ 24.510332] RSP: 0018:ffffb035407e7da8 EFLAGS: 00010046 [ 24.510333] RAX: 0000000000000075 RBX: ffff9a1a02119e68 RCX: 0000000000000000 [ 24.510334] RDX: 0000000000000000 RSI: ffff9a1d9f31f840 RDI: ffff9a1d9f31f840 [ 24.510335] RBP: ffff9a1d9f337f00 R08: 0000000000000000 R09: 00000000ffff7fff [ 24.510337] R10: ffffb035407e7c50 R11: ffffffffa8be75e8 R12: ffff9a1d9f337f68 [ 24.562805] R13: ffff9a1a02119e70 R14: ffff9a1a02119e70 R15: ffff9a1d9f330340 [ 24.569929] FS: 0000000000000000(0000) GS:ffff9a1d9f300000(0000) knlGS:0000000000000000 [ 24.578017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 24.578018] CR2: 000055b5f1f28050 CR3: 0000000104e38000 CR4: 0000000000350ee0 [ 24.578019] Kernel panic - not syncing: Fatal exception [ 24.578653] Kernel Offset: 0x26200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 24.682013] ---[ end Kernel panic - not syncing: Fatal exception ]--- [ 24.339396] r[-- MARK -- Fri Nov 25 06:25:00 2022] On Thu, Nov 24, 2022 at 11:00 PM Bruno Goncalves <bgoncalv@redhat.com> wrote: > > On Wed, 23 Nov 2022 at 14:46, Jens Axboe <axboe@kernel.dk> wrote: > > > > On 11/23/22 1:48 AM, Bruno Goncalves wrote: > > > Hello, > > > > > > We recently started to hit the following panic when testing the block > > > tree (for-next branch). > > > > > > [ 5076.172749] list_add corruption. prev->next should be next > > > (ffff91cd6f7fa568), but was ffff91c991ca6670. (prev=ffff91c991ca6670). > > > [ 5076.173863] ------------[ cut here ]------------ > > > [ 5076.174853] kernel BUG at lib/list_debug.c:30! > > > [ 5076.175523] invalid opcode: 0000 [#1] PREEMPT SMP PTI > > > [ 5076.175853] CPU: 15 PID: 16415 Comm: kworker/15:13 Tainted: G > > > I 6.1.0-rc6 #1 > > > [ 5076.176799] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 > > > [ 5076.177198] Workqueue: cgwb_release cgwb_release_workfn > > > [ 5076.177497] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > > [ 5076.177788] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > > c7 c7 > > > [ 5076.179173] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > > [ 5076.179472] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > > [ 5076.180241] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > > [ 5076.181069] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > > [ 5076.182209] R10: 0000000000000003 R11: ffff91cd7ff42fe8 R12: ffff91cd6f7fa568 > > > [ 5076.183002] R13: ffff91c991ca6670 R14: ffff91c991ca6670 R15: ffff91cd6f7f1440 > > > [ 5076.183902] FS: 0000000000000000(0000) GS:ffff91cd6f7c0000(0000) > > > knlGS:0000000000000000 > > > [ 5076.184377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 5076.185084] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > > [ 5076.185945] Call Trace: > > > [ 5076.186110] <TASK> > > > [ 5076.186916] insert_work+0x46/0xc0 > > > [ 5076.187533] __queue_work+0x1d4/0x460 > > > [ 5076.187788] queue_work_on+0x37/0x40 > > > [ 5076.187993] blkcg_unpin_online+0x1ad/0x1b0 > > > [ 5076.188244] cgwb_release_workfn+0x6a/0x200 > > > [ 5076.188464] process_one_work+0x1c7/0x380 > > > [ 5076.188675] worker_thread+0x4d/0x380 > > > [ 5076.188881] ? rescuer_thread+0x380/0x380 > > > [ 5076.189089] kthread+0xe9/0x110 > > > [ 5076.189716] ? kthread_complete_and_exit+0x20/0x20 > > > [ 5076.190407] ret_from_fork+0x22/0x30 > > > [ 5076.190677] </TASK> > > > [ 5076.190816] Modules linked in: nvme nvme_core nvme_common loop tls > > > rfkill intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal > > > intel_powerclamp coretemp sunrpc kvm_intel kvm iTCO_wdt iapl > > > intel_cstate intel_uncore pcspkr lpc_ich ipmi_ssif hpilo tg3 acpi_ipmi > > > ioatdma ipmi_si ipmi_devintf dca ipmi_msghandler acpi_power_meter fuse > > > zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni > > > polyval_generic ghash_clmulni_intel sha512_ssse3 serio_raw hpsa > > > mgag200 scsi_transport_sas [last unloaded: scsi_debug] > > > [ 5076.293315] ---[ end trace 0000000000000000 ]--- > > > [ 5076.295226] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > > [ 5076.295587] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > > c7 c7 > > > [ 5076.296921] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > > [ 5076.297239] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > > [ 5076.297983] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > > [ 5076.298768] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > > [ 5076.299525] R10: 0000S: 0000000000000000(0000) > > > GS:ffff91cd6f7c0000(0000) knlGS:0000000000000000 > > > [ 5076.700351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 5076.701046] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > > [ 5076ernel panic - not syncing: Fatal exception > > > [ 5077.924713] Shutting down cpus with NMI > > > [ 5077.924986] Kernel Offset: 0x2b000000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > [ 5077.927946] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > > > > It seems to happen often during different tests. > > > > > > full console.log: > > > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/11/21/redhat:700955106/build_x86_64_redhat:700955106_x86_64/tests/1/results_0001/console.log/console.log > > > > > > kernel tarball: > > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/publish%20x86_64/3356091217/artifacts/kernel-block-redhat_700955106_x86_64.tar.gz > > > > > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/build%20x86_64/3356091207/artifacts/kernel-block-redhat_700955106_x86_64.config > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > > > > > We didn't bisect, but the first commit we hit the problem was > > > "f65d92c600fe6eecdbd6e7fab7893c9c094dfcbf > > > (io_uring-6.1-2022-11-18-2180-gf65d92c600fe)" and the last one where > > > we didn't hit the problem was > > > "40fa774af7fd04d06014ac74947c351649b6f64f > > > (io_uring-6.1-2022-11-11-1843-g40fa774af7fd)" > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > > cki issue tracker: https://datawarehouse.cki-project.org/issue/1732 > > > > Please just try and clone for-6.2/block from the block tree and bisect > > it? > > > > Hi, > I've tried with commit 93c68cc46a070775cc6675e3543dd909eb9f6c9e (drbd: > use consistent license), but I was not able to hit the panic with it. > > > Bruno > > > -- > > Jens Axboe > > > > > -- Best Regards, Yi Zhang ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisected]kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-25 8:38 ` Yi Zhang @ 2022-11-26 14:29 ` Yi Zhang 2022-11-26 15:53 ` Jens Axboe 2022-11-28 18:55 ` Bart Van Assche 0 siblings, 2 replies; 9+ messages in thread From: Yi Zhang @ 2022-11-26 14:29 UTC (permalink / raw) To: Jens Axboe, Waiman Long; +Cc: linux-block, CKI Project, Bruno Goncalves Hi Jens Sorry for the delay as I couldn't reproduce it with the original for-6.2/block branch. Finally, I rebased the for-6.2/block branch on 6.1-rc6 and was able to bisect it： 951d1e94801f95a3fc1c75ff342431c9f519dd14 is the first bad commit commit 951d1e94801f95a3fc1c75ff342431c9f519dd14 Author: Waiman Long <longman@redhat.com> Date: Fri Nov 4 20:59:02 2022 -0400 blk-cgroup: Flush stats at blkgs destruction path As noted by Michal, the blkg_iostat_set's in the lockless list hold reference to blkg's to protect against their removal. Those blkg's hold reference to blkcg. When a cgroup is being destroyed, cgroup_rstat_flush() is only called at css_release_work_fn() which is called when the blkcg reference count reaches 0. This circular dependency will prevent blkcg from being freed until some other events cause cgroup_rstat_flush() to be called to flush out the pending blkcg stats. To prevent this delayed blkcg removal, add a new cgroup_rstat_css_flush() function to flush stats for a given css and cpu and call it at the blkgs destruction path, blkcg_destroy_blkgs(), whenever there are still some pending stats to be flushed. This will ensure that blkcg reference count can reach 0 ASAP. Signed-off-by: Waiman Long <longman@redhat.com> Acked-by: Tejun Heo <tj@kernel.org> Link: https://lore.kernel.org/r/20221105005902.407297-4-longman@redhat.com Signed-off-by: Jens Axboe <axboe@kernel.dk> On Fri, Nov 25, 2022 at 4:38 PM Yi Zhang <yi.zhang@redhat.com> wrote: > > I reproduced this issue even when system boot with the latest > linux-block/for-next, will try to bisect it later. > > 43f3ae1898c9 (HEAD -> for-next, origin/for-next) Merge branch > 'for-6.2/writeback' into for-next > d6798bc243fa writeback: Add asserts for adding freed inode to lists > > [ 24.183829] list_add corruption. prev->next should be next > (ffff9a1d9f337f68), but was ffff9a1a02119e70. (prev=ffff9a1a02119e70). > [ 24.195478] ------------[ cut here ]------------ > [ 24.200088] kernel BUG at lib/list_debug.c:30! > [ 24.204532] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI > [ 24.209751] CPU: 4 PID: 167 Comm: kworker/4:1 Not tainted 6.1.0-rc6+ #1 > [ 24.216365] Hardware name: Dell Inc. PowerEdge R6515/07PXPY, BIOS > 2.8.5 08/18/2022 > [ 24.223930] Workqueue: cgwb_release cgwb_release_workfn > [ 24.229157] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > [ 24.234208] Code: f2 4c 89 c1 48 89 fe 48 c7 c7 20 23 65 a8 e8 d2 > a2 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 c8 22 65 a8 e8 bb > a2 fe ff <0f> 0b 4c 89 c1 48 c7 c7 70 22 65 a8 e8 aa a2 fe ff 0f 0b 48 > c7 c7 > [ 24.252953] RSP: 0018:ffffb035407e7da8 EFLAGS: 00010046 > [ 24.258172] RAX: 0000000000000075 RBX: ffff9a1a02119e68 RCX: 0000000000000000 > [ 24.265303] RDX: 0000000000000000 RSI: ffff9a1d9f31f840 RDI: ffff9a1d9f31f840 > [ 24.272428] RBP: ffff9a1d9f337f00 R08: 0000000000000000 R09: 00000000ffff7fff > [ 24.279560] R10: ffffb035407e7c50 R11: ffffffffa8be75e8 R12: ffff9a1d9f337f68 > [ 24.286683] R13: ffff9a1a02119e70 R14: ffff9a1a02119e70 R15: ffff9a1d9f330340 > [ 24.293808] FS: 0000000000000000(0000) GS:ffff9a1d9f300000(0000) > knlGS:0000000000000000 > [ 24.301894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 24.307641] CR2: 000055b5f1f28050 CR3: 0000000104e38000 CR4: 0000000000350ee0 > [ 24.314772] Call Trace: > [ 24.314774] <TASK> > [ 24.314774] insert_work+0x46/0xc0 > [ 24.314780] __queue_work+0x1d5/0x380 > [ 24.326376] queue_work_on+0x24/0x30 > [ 24.329955] blkcg_unpin_online+0x1b5/0x1c0 > [ 24.334143] cgwb_release_workfn+0x6a/0x200 > [ 24.338327] process_one_work+0x1e5/0x3b0 > [ 24.342342] ? rescuer_thread+0x390/0x390 > [ 24.346352] worker_thread+0x50/0x3a0 > [ 24.350019] ? rescuer_thread+0x390/0x390 > [ 24.354030] kthread+0xd9/0x100 > [ 24.357177] ? kthread_complete_and_exit+0x20/0x20 > [ 24.361970] ret_from_fork+0x22/0x30 > [ 24.365550] </TASK> > [ 24.367742] Modules linked in: sunrpc intel_rapl_msr > intel_rapl_common amd64_edac edac_mce_amd ipmi_ssif kvm_amd kvm > mgag200 ledtrig_audio rfkill video i2c_algo_bit drm_shmem_helper > dcdbas drm_kms_helper irqbypass dell_smbios rapl dell_wmi_descriptor > wmi_bmof pcspkr syscopyarea acpi_ipmi sysfillrect sysimgblt > fb_sys_fops ipmi_si ipmi_devintf ptdma i2c_piix4 k10temp > ipmi_msghandler acpi_power_meter vfat fat acpi_cpufreq drm fuse xfs > libcrc32c sd_mod sg ahci crct10dif_pclmul crc32_pclmul libahci > crc32c_intel ghash_clmulni_intel mpt3sas nvme tg3 libata nvme_core ccp > raid_class nvme_common t10_pi sp5100_tco scsi_transport_sas wmi > dm_mirror dm_region_hash dm_log dm_mod > [ 24.426475] ---[ end trace 0000000000000000 ]--- > [ 24.505278] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > [ 24.510331] Code: f2 4c 89 c1 48 89 fe 48 c7 c7 20 23 65 a8 e8 d2 > a2 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 c8 22 65 a8 e8 bb > a2 fe ff <0f> 0b 4c 89 c1 48 c7 c7 70 22 65 a8 e8 aa a2 fe ff 0f 0b 48 > c7 c7 > [ 24.510332] RSP: 0018:ffffb035407e7da8 EFLAGS: 00010046 > [ 24.510333] RAX: 0000000000000075 RBX: ffff9a1a02119e68 RCX: 0000000000000000 > [ 24.510334] RDX: 0000000000000000 RSI: ffff9a1d9f31f840 RDI: ffff9a1d9f31f840 > [ 24.510335] RBP: ffff9a1d9f337f00 R08: 0000000000000000 R09: 00000000ffff7fff > [ 24.510337] R10: ffffb035407e7c50 R11: ffffffffa8be75e8 R12: ffff9a1d9f337f68 > [ 24.562805] R13: ffff9a1a02119e70 R14: ffff9a1a02119e70 R15: ffff9a1d9f330340 > [ 24.569929] FS: 0000000000000000(0000) GS:ffff9a1d9f300000(0000) > knlGS:0000000000000000 > [ 24.578017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 24.578018] CR2: 000055b5f1f28050 CR3: 0000000104e38000 CR4: 0000000000350ee0 > [ 24.578019] Kernel panic - not syncing: Fatal exception > [ 24.578653] Kernel Offset: 0x26200000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > [ 24.682013] ---[ end Kernel panic - not syncing: Fatal exception ]--- > [ 24.339396] r[-- MARK -- Fri Nov 25 06:25:00 2022] > > > On Thu, Nov 24, 2022 at 11:00 PM Bruno Goncalves <bgoncalv@redhat.com> wrote: > > > > On Wed, 23 Nov 2022 at 14:46, Jens Axboe <axboe@kernel.dk> wrote: > > > > > > On 11/23/22 1:48 AM, Bruno Goncalves wrote: > > > > Hello, > > > > > > > > We recently started to hit the following panic when testing the block > > > > tree (for-next branch). > > > > > > > > [ 5076.172749] list_add corruption. prev->next should be next > > > > (ffff91cd6f7fa568), but was ffff91c991ca6670. (prev=ffff91c991ca6670). > > > > [ 5076.173863] ------------[ cut here ]------------ > > > > [ 5076.174853] kernel BUG at lib/list_debug.c:30! > > > > [ 5076.175523] invalid opcode: 0000 [#1] PREEMPT SMP PTI > > > > [ 5076.175853] CPU: 15 PID: 16415 Comm: kworker/15:13 Tainted: G > > > > I 6.1.0-rc6 #1 > > > > [ 5076.176799] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 > > > > [ 5076.177198] Workqueue: cgwb_release cgwb_release_workfn > > > > [ 5076.177497] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > > > [ 5076.177788] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > > > c7 c7 > > > > [ 5076.179173] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > > > [ 5076.179472] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > > > [ 5076.180241] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > > > [ 5076.181069] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > > > [ 5076.182209] R10: 0000000000000003 R11: ffff91cd7ff42fe8 R12: ffff91cd6f7fa568 > > > > [ 5076.183002] R13: ffff91c991ca6670 R14: ffff91c991ca6670 R15: ffff91cd6f7f1440 > > > > [ 5076.183902] FS: 0000000000000000(0000) GS:ffff91cd6f7c0000(0000) > > > > knlGS:0000000000000000 > > > > [ 5076.184377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 5076.185084] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > > > [ 5076.185945] Call Trace: > > > > [ 5076.186110] <TASK> > > > > [ 5076.186916] insert_work+0x46/0xc0 > > > > [ 5076.187533] __queue_work+0x1d4/0x460 > > > > [ 5076.187788] queue_work_on+0x37/0x40 > > > > [ 5076.187993] blkcg_unpin_online+0x1ad/0x1b0 > > > > [ 5076.188244] cgwb_release_workfn+0x6a/0x200 > > > > [ 5076.188464] process_one_work+0x1c7/0x380 > > > > [ 5076.188675] worker_thread+0x4d/0x380 > > > > [ 5076.188881] ? rescuer_thread+0x380/0x380 > > > > [ 5076.189089] kthread+0xe9/0x110 > > > > [ 5076.189716] ? kthread_complete_and_exit+0x20/0x20 > > > > [ 5076.190407] ret_from_fork+0x22/0x30 > > > > [ 5076.190677] </TASK> > > > > [ 5076.190816] Modules linked in: nvme nvme_core nvme_common loop tls > > > > rfkill intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal > > > > intel_powerclamp coretemp sunrpc kvm_intel kvm iTCO_wdt iapl > > > > intel_cstate intel_uncore pcspkr lpc_ich ipmi_ssif hpilo tg3 acpi_ipmi > > > > ioatdma ipmi_si ipmi_devintf dca ipmi_msghandler acpi_power_meter fuse > > > > zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni > > > > polyval_generic ghash_clmulni_intel sha512_ssse3 serio_raw hpsa > > > > mgag200 scsi_transport_sas [last unloaded: scsi_debug] > > > > [ 5076.293315] ---[ end trace 0000000000000000 ]--- > > > > [ 5076.295226] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > > > [ 5076.295587] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > > > c7 c7 > > > > [ 5076.296921] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > > > [ 5076.297239] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > > > [ 5076.297983] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > > > [ 5076.298768] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > > > [ 5076.299525] R10: 0000S: 0000000000000000(0000) > > > > GS:ffff91cd6f7c0000(0000) knlGS:0000000000000000 > > > > [ 5076.700351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 5076.701046] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > > > [ 5076ernel panic - not syncing: Fatal exception > > > > [ 5077.924713] Shutting down cpus with NMI > > > > [ 5077.924986] Kernel Offset: 0x2b000000 from 0xffffffff81000000 > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > [ 5077.927946] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > > > > > > It seems to happen often during different tests. > > > > > > > > full console.log: > > > > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/11/21/redhat:700955106/build_x86_64_redhat:700955106_x86_64/tests/1/results_0001/console.log/console.log > > > > > > > > kernel tarball: > > > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/publish%20x86_64/3356091217/artifacts/kernel-block-redhat_700955106_x86_64.tar.gz > > > > > > > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/build%20x86_64/3356091207/artifacts/kernel-block-redhat_700955106_x86_64.config > > > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > > > > > > > We didn't bisect, but the first commit we hit the problem was > > > > "f65d92c600fe6eecdbd6e7fab7893c9c094dfcbf > > > > (io_uring-6.1-2022-11-18-2180-gf65d92c600fe)" and the last one where > > > > we didn't hit the problem was > > > > "40fa774af7fd04d06014ac74947c351649b6f64f > > > > (io_uring-6.1-2022-11-11-1843-g40fa774af7fd)" > > > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > > > cki issue tracker: https://datawarehouse.cki-project.org/issue/1732 > > > > > > Please just try and clone for-6.2/block from the block tree and bisect > > > it? > > > > > > > Hi, > > I've tried with commit 93c68cc46a070775cc6675e3543dd909eb9f6c9e (drbd: > > use consistent license), but I was not able to hit the panic with it. > > > > > > Bruno > > > > > -- > > > Jens Axboe > > > > > > > > > > > -- > Best Regards, > Yi Zhang -- Best Regards, Yi Zhang ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisected]kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-26 14:29 ` [bisected]kernel " Yi Zhang @ 2022-11-26 15:53 ` Jens Axboe 2022-11-26 22:54 ` Waiman Long 2022-11-28 18:55 ` Bart Van Assche 1 sibling, 1 reply; 9+ messages in thread From: Jens Axboe @ 2022-11-26 15:53 UTC (permalink / raw) To: Yi Zhang, Waiman Long; +Cc: linux-block, CKI Project, Bruno Goncalves On 11/26/22 7:29 AM, Yi Zhang wrote: > Hi Jens > Sorry for the delay as I couldn't reproduce it with the original > for-6.2/block branch. > Finally, I rebased the for-6.2/block branch on 6.1-rc6 and was able to > bisect it： > > > 951d1e94801f95a3fc1c75ff342431c9f519dd14 is the first bad commit > commit 951d1e94801f95a3fc1c75ff342431c9f519dd14 > Author: Waiman Long <longman@redhat.com> > Date: Fri Nov 4 20:59:02 2022 -0400 > > blk-cgroup: Flush stats at blkgs destruction path > > As noted by Michal, the blkg_iostat_set's in the lockless list > hold reference to blkg's to protect against their removal. Those > blkg's hold reference to blkcg. When a cgroup is being destroyed, > cgroup_rstat_flush() is only called at css_release_work_fn() which is > called when the blkcg reference count reaches 0. This circular dependency > will prevent blkcg from being freed until some other events cause > cgroup_rstat_flush() to be called to flush out the pending blkcg stats. > > To prevent this delayed blkcg removal, add a new cgroup_rstat_css_flush() > function to flush stats for a given css and cpu and call it at the blkgs > destruction path, blkcg_destroy_blkgs(), whenever there are still some > pending stats to be flushed. This will ensure that blkcg reference > count can reach 0 ASAP. > > Signed-off-by: Waiman Long <longman@redhat.com> > Acked-by: Tejun Heo <tj@kernel.org> > Link: https://lore.kernel.org/r/20221105005902.407297-4-longman@redhat.com > Signed-off-by: Jens Axboe <axboe@kernel.dk> Waiman, let me know if you have an idea what is going on here and can send in a fix, or if I need to revert this one. From looking at the lists of commits after these reports came in, I did suspect this commit. But I don't know enough about this area to render an opinion on a fix without spending more time on it. -- Jens Axboe ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisected]kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-26 15:53 ` Jens Axboe @ 2022-11-26 22:54 ` Waiman Long 2022-11-27 4:13 ` Waiman Long 0 siblings, 1 reply; 9+ messages in thread From: Waiman Long @ 2022-11-26 22:54 UTC (permalink / raw) To: Jens Axboe, Yi Zhang; +Cc: linux-block, CKI Project, Bruno Goncalves On 11/26/22 10:53, Jens Axboe wrote: > On 11/26/22 7:29 AM, Yi Zhang wrote: >> Hi Jens >> Sorry for the delay as I couldn't reproduce it with the original >> for-6.2/block branch. >> Finally, I rebased the for-6.2/block branch on 6.1-rc6 and was able to >> bisect it： >> >> >> 951d1e94801f95a3fc1c75ff342431c9f519dd14 is the first bad commit >> commit 951d1e94801f95a3fc1c75ff342431c9f519dd14 >> Author: Waiman Long <longman@redhat.com> >> Date: Fri Nov 4 20:59:02 2022 -0400 >> >> blk-cgroup: Flush stats at blkgs destruction path >> >> As noted by Michal, the blkg_iostat_set's in the lockless list >> hold reference to blkg's to protect against their removal. Those >> blkg's hold reference to blkcg. When a cgroup is being destroyed, >> cgroup_rstat_flush() is only called at css_release_work_fn() which is >> called when the blkcg reference count reaches 0. This circular dependency >> will prevent blkcg from being freed until some other events cause >> cgroup_rstat_flush() to be called to flush out the pending blkcg stats. >> >> To prevent this delayed blkcg removal, add a new cgroup_rstat_css_flush() >> function to flush stats for a given css and cpu and call it at the blkgs >> destruction path, blkcg_destroy_blkgs(), whenever there are still some >> pending stats to be flushed. This will ensure that blkcg reference >> count can reach 0 ASAP. >> >> Signed-off-by: Waiman Long <longman@redhat.com> >> Acked-by: Tejun Heo <tj@kernel.org> >> Link: https://lore.kernel.org/r/20221105005902.407297-4-longman@redhat.com >> Signed-off-by: Jens Axboe <axboe@kernel.dk> > Waiman, let me know if you have an idea what is going on here and can > send in a fix, or if I need to revert this one. From looking at the > lists of commits after these reports came in, I did suspect this > commit. But I don't know enough about this area to render an opinion > on a fix without spending more time on it. > Sure. I will take a closer look at that. Will let you know my investigation result ASAP. Thanks, Longman ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisected]kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-26 22:54 ` Waiman Long @ 2022-11-27 4:13 ` Waiman Long 0 siblings, 0 replies; 9+ messages in thread From: Waiman Long @ 2022-11-27 4:13 UTC (permalink / raw) To: Jens Axboe, Yi Zhang; +Cc: linux-block, CKI Project, Bruno Goncalves On 11/26/22 17:54, Waiman Long wrote: > > On 11/26/22 10:53, Jens Axboe wrote: >> On 11/26/22 7:29 AM, Yi Zhang wrote: >>> Hi Jens >>> Sorry for the delay as I couldn't reproduce it with the original >>> for-6.2/block branch. >>> Finally, I rebased the for-6.2/block branch on 6.1-rc6 and was able to >>> bisect it： >>> >>> >>> 951d1e94801f95a3fc1c75ff342431c9f519dd14 is the first bad commit >>> commit 951d1e94801f95a3fc1c75ff342431c9f519dd14 >>> Author: Waiman Long <longman@redhat.com> >>> Date: Fri Nov 4 20:59:02 2022 -0400 >>> >>> blk-cgroup: Flush stats at blkgs destruction path >>> >>> As noted by Michal, the blkg_iostat_set's in the lockless list >>> hold reference to blkg's to protect against their removal. Those >>> blkg's hold reference to blkcg. When a cgroup is being destroyed, >>> cgroup_rstat_flush() is only called at css_release_work_fn() >>> which is >>> called when the blkcg reference count reaches 0. This circular >>> dependency >>> will prevent blkcg from being freed until some other events cause >>> cgroup_rstat_flush() to be called to flush out the pending >>> blkcg stats. >>> >>> To prevent this delayed blkcg removal, add a new >>> cgroup_rstat_css_flush() >>> function to flush stats for a given css and cpu and call it at >>> the blkgs >>> destruction path, blkcg_destroy_blkgs(), whenever there are >>> still some >>> pending stats to be flushed. This will ensure that blkcg reference >>> count can reach 0 ASAP. >>> >>> Signed-off-by: Waiman Long <longman@redhat.com> >>> Acked-by: Tejun Heo <tj@kernel.org> >>> Link: >>> https://lore.kernel.org/r/20221105005902.407297-4-longman@redhat.com >>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >> Waiman, let me know if you have an idea what is going on here and can >> send in a fix, or if I need to revert this one. From looking at the >> lists of commits after these reports came in, I did suspect this >> commit. But I don't know enough about this area to render an opinion >> on a fix without spending more time on it. >> > Sure. I will take a closer look at that. Will let you know my > investigation result ASAP. > Thanks Yi for allowing me to access the system that can reproduce the bug. I found out that the panic problem is fixed by moving the rstat flushing before the destruction of blkgs in blkcg_destroy_blkgs(). I will post another patch later to fix that bug. However, I want to spend a bit more time to see if I can figure out what cause the panic in the first place. Cheers, Longman ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisected]kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) 2022-11-26 14:29 ` [bisected]kernel " Yi Zhang 2022-11-26 15:53 ` Jens Axboe @ 2022-11-28 18:55 ` Bart Van Assche 1 sibling, 0 replies; 9+ messages in thread From: Bart Van Assche @ 2022-11-28 18:55 UTC (permalink / raw) To: Yi Zhang, Jens Axboe, Waiman Long Cc: linux-block, CKI Project, Bruno Goncalves On 11/26/22 06:29, Yi Zhang wrote: > Finally, I rebased the for-6.2/block branch on 6.1-rc6 and was able to > bisect it： > > > 951d1e94801f95a3fc1c75ff342431c9f519dd14 is the first bad commit > commit 951d1e94801f95a3fc1c75ff342431c9f519dd14 > Author: Waiman Long <longman@redhat.com> > Date: Fri Nov 4 20:59:02 2022 -0400 > > blk-cgroup: Flush stats at blkgs destruction path > > As noted by Michal, the blkg_iostat_set's in the lockless list > hold reference to blkg's to protect against their removal. Those > blkg's hold reference to blkcg. When a cgroup is being destroyed, > cgroup_rstat_flush() is only called at css_release_work_fn() which is > called when the blkcg reference count reaches 0. This circular dependency > will prevent blkcg from being freed until some other events cause > cgroup_rstat_flush() to be called to flush out the pending blkcg stats. > > To prevent this delayed blkcg removal, add a new cgroup_rstat_css_flush() > function to flush stats for a given css and cpu and call it at the blkgs > destruction path, blkcg_destroy_blkgs(), whenever there are still some > pending stats to be flushed. This will ensure that blkcg reference > count can reach 0 ASAP. > > Signed-off-by: Waiman Long <longman@redhat.com> > Acked-by: Tejun Heo <tj@kernel.org> > Link: https://lore.kernel.org/r/20221105005902.407297-4-longman@redhat.com > Signed-off-by: Jens Axboe <axboe@kernel.dk> I can confirm this report. If I revert patch "blk-cgroup: Flush stats at blkgs destruction path" on top of the block/for-next branch from last Wednesday then test block/027 passes. Test block/027 fails systematically with an unmodified block/for-next branch. Bart. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-11-28 18:55 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-23 8:48 kernel BUG at lib/list_debug.c:30! (list_add corruption. prev->next should be nex) Bruno Goncalves 2022-11-23 13:46 ` Jens Axboe 2022-11-24 14:57 ` Bruno Goncalves 2022-11-25 8:38 ` Yi Zhang 2022-11-26 14:29 ` [bisected]kernel " Yi Zhang 2022-11-26 15:53 ` Jens Axboe 2022-11-26 22:54 ` Waiman Long 2022-11-27 4:13 ` Waiman Long 2022-11-28 18:55 ` Bart Van Assche
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.