* IOZONE fails intermittently while toggling the interface
@ 2020-02-07 14:00 Dakshaja Uppalapati
2020-02-07 16:29 ` Bart Van Assche
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Dakshaja Uppalapati @ 2020-02-07 14:00 UTC (permalink / raw)
To: target-devel
[-- Attachment #1: Type: text/plain, Size: 950 bytes --]
Hi Bart,
While testing iser with kernel 5.5, I see attached soft lockups.
Logged in and ran iozone on mounted disks. While the traffic is running, toggled the initiators interface down to up for 13 secs.
I see the attached hung task warnings in the target dmesg.
# iozone -a -I -+d -g 32m
The same test with kernel-5.4 runs fine. I have suspected the following three commits which are all correlated, so removed these commits from 5.5 kernel, rebuilt the kernel and ran the same test then no issue is seen. Iozone is running successfully.
80647a89eaf3f2: scsi: target: core: Release SPC-2 reservations when closing a session
e9d3009cb936b: scsi: target: iscsi: Wait for all commands to finish before freeing a session
04060db41178c: scsi: RDMA/isert: Fix a recently introduced regression related to logout
I am trying to root cause the issue. Please do suggest me if you want me to check some thing specific.
Thanks,
Dakshaja
[-- Attachment #2: dmesg_trace.txt --]
[-- Type: text/plain, Size: 271812 bytes --]
[ 2331.349796] ------------[ cut here ]------------
[ 2331.366501] list_del corruption. prev->next should be ffff94d4b4532a30, but was ffff94d4b45331e0
[ 2331.387557] WARNING: CPU: 4 PID: 653 at lib/list_debug.c:53 __list_del_entry_valid+0x7c/0xa0
[ 2331.408271] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2331.603010] CPU: 4 PID: 653 Comm: kworker/4:1H Kdump: loaded Tainted: G W 5.5.0 #34
[ 2331.627078] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2331.648887] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[ 2331.669744] RIP: 0010:__list_del_entry_valid+0x7c/0xa0
[ 2331.690233] Code: 31 c0 c3 48 89 fe 31 c0 48 c7 c7 80 d1 94 97 e8 ea a7 bf ff 0f 0b 31 c0 c3 48 89 fe 31 c0 48 c7 c7 40 d1 94 97 e8 d4 a7 bf ff <0f> 0b 31 c0 c3 48 89 fe 31 c0 48 c7 c7 08 d1 94 97 e8 be a7 bf ff
[ 2331.741354] RSP: 0018:ffffa8a880563da0 EFLAGS: 00010286
[ 2331.763018] RAX: 0000000000000000 RBX: ffff94d4b4532730 RCX: 0000000000000000
[ 2331.786550] RDX: ffff94d570a26700 RSI: ffff94d57061a0f8 RDI: ffff94d57061a0f8
[ 2331.810085] RBP: ffff94d4b4532d30 R08: 0000000000000000 R09: 0000000000000000
[ 2331.833597] R10: 0000000000000001 R11: 0000000000aaaaaa R12: ffff94d5651fa000
[ 2331.857135] R13: ffff94d4b4bb8570 R14: 0000000000000001 R15: ffff94d4b4532a30
[ 2331.880504] FS: 0000000000000000(0000) GS:ffff94d570600000(0000) knlGS:0000000000000000
[ 2331.904928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2331.927048] CR2: 00007ff478c22000 CR3: 00000003a067c000 CR4: 00000000000006e0
[ 2331.950701] Call Trace:
[ 2331.969451] isert_put_cmd+0x80/0x260 [ib_isert]
[ 2331.990473] __ib_process_cq+0x7a/0xc0 [ib_core]
[ 2332.011452] ib_cq_poll_work+0x26/0x80 [ib_core]
[ 2332.032258] process_one_work+0x200/0x570
[ 2332.052409] ? process_one_work+0x17a/0x570
[ 2332.072666] worker_thread+0x4c/0x3f0
[ 2332.092259] kthread+0x103/0x140
[ 2332.111344] ? process_one_work+0x570/0x570
[ 2332.131360] ? kthread_bind+0x10/0x10
[ 2332.150690] ret_from_fork+0x3a/0x50
[ 2332.169878] irq event stamp: 661057429
[ 2332.189162] hardirqs last enabled at (661057428): [<ffffffff96925fd4>] console_unlock+0x3f4/0x5e0
[ 2332.214043] hardirqs last disabled at (661057429): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2332.239700] softirqs last enabled at (661057394): [<ffffffffc0ce3688>] isert_put_cmd+0x218/0x260 [ib_isert]
[ 2332.265635] softirqs last disabled at (661057408): [<ffffffffc0ce34dc>] isert_put_cmd+0x6c/0x260 [ib_isert]
[ 2332.291363] ---[ end trace 23790484fcb46095 ]---
[ 2346.313224] iSCSI Login timeout on Network Portal 106.1.1.16:3260
[ 2357.529878] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2357.553338] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2357.777288] irq event stamp: 414993294
[ 2357.798074] hardirqs last enabled at (414993293): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2357.824626] hardirqs last disabled at (414993294): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2357.851226] softirqs last enabled at (414993292): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2357.878846] softirqs last disabled at (414993290): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2357.905970] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W 5.5.0 #34
[ 2357.931080] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2357.953910] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 2357.975000] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 2358.027064] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 2358.051008] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 2358.074427] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 2358.097682] RBP: ffffffffc0e6c0fe R08: 0000000000000000 R09: 0000000000000000
[ 2358.120892] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2358.144046] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb85e0 R15: ffff94d4b4533690
[ 2358.167250] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2358.191621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2358.213482] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2358.236735] Call Trace:
[ 2358.255125] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2358.276896] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2358.298351] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2358.320777] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2358.344664] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2358.367311] ? remove_wait_queue+0x60/0x60
[ 2358.387877] kthread+0x103/0x140
[ 2358.407538] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2358.430674] ? kthread_bind+0x10/0x10
[ 2358.451017] ret_from_fork+0x3a/0x50
[ 2373.448449] iSCSI Login timeout on Network Portal 106.1.1.16:3260
[ 2373.471810] rx_data returned -512, expecting 48.
[ 2373.476667] iSCSI Login negotiation failed.
[ 2385.529115] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2385.553354] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2385.783433] irq event stamp: 849926535
[ 2385.805060] hardirqs last enabled at (849926533): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2385.832440] hardirqs last disabled at (849926535): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2385.859854] softirqs last enabled at (849926532): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2385.888328] softirqs last disabled at (849926534): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2385.916316] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2385.942237] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2385.965786] RIP: 0010:lock_acquire+0x39/0x1c0
[ 2385.987010] Code: 25 40 8e 01 00 55 53 48 83 ec 30 8b 82 bc 0c 00 00 48 89 7c 24 20 85 c0 0f 85 ad 00 00 00 89 f5 41 89 cd 45 89 c6 4d 89 cf 9c <58> 66 66 90 66 90 48 89 44 24 28 fa 66 66 90 66 66 90 c7 82 bc 0c
[ 2386.040829] RSP: 0018:ffffa8a885bafd20 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2386.065936] RAX: 0000000000000000 RBX: ffff94d4b4bb8650 RCX: 0000000000000000
[ 2386.090551] RDX: ffff94d550e60000 RSI: 0000000000000000 RDI: ffff94d4b4bb8668
[ 2386.115122] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 2386.139344] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2386.163128] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2386.186705] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2386.211223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2386.233485] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2386.257333] Call Trace:
[ 2386.276393] ? __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2386.299082] ? _raw_spin_lock_bh+0x3d/0x50
[ 2386.319722] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2386.342154] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2386.364313] ? iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2386.386056] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2386.408712] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2386.432701] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2386.455429] ? remove_wait_queue+0x60/0x60
[ 2386.475732] ? kthread+0x103/0x140
[ 2386.495136] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2386.517725] ? kthread_bind+0x10/0x10
[ 2386.537511] ? ret_from_fork+0x3a/0x50
[ 2397.312796] rcu: INFO: rcu_sched self-detected stall on CPU
[ 2397.334519] rcu: 0-....: (63028 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=16066
[ 2397.361013] (t=65022 jiffies g=82081 q=13121)
[ 2397.381933] NMI backtrace for cpu 0
[ 2397.401765] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2397.427327] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2397.450619] Call Trace:
[ 2397.469297] <IRQ>
[ 2397.487047] dump_stack+0x82/0xb3
[ 2397.505712] nmi_cpu_backtrace+0x90/0xa0
[ 2397.524614] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2397.543618] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2397.562988] rcu_dump_cpu_stacks+0x97/0xc5
[ 2397.580971] rcu_sched_clock_irq+0x6ec/0x980
[ 2397.598734] ? rcu_read_lock_sched_held+0x52/0x90
[ 2397.616708] ? tick_sched_do_timer+0x70/0x70
[ 2397.633990] update_process_times+0x24/0x50
[ 2397.651099] tick_sched_handle+0x25/0x60
[ 2397.667766] tick_sched_timer+0x37/0x70
[ 2397.684365] __hrtimer_run_queues+0xe8/0x450
[ 2397.701083] hrtimer_interrupt+0x122/0x270
[ 2397.717852] smp_apic_timer_interrupt+0x86/0x240
[ 2397.734637] apic_timer_interrupt+0xf/0x20
[ 2397.750513] </IRQ>
[ 2397.763871] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 2397.780125] Code: fa 66 66 90 66 66 90 65 01 35 7c 0a 77 69 65 8b 05 75 0a 77 69 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 8d cc 06 00 48 89 df 57 9d <66> 66 90 66 90 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 2397.822619] RSP: 0018:ffffa8a885bafd68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2397.842267] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 2397.861593] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 2397.880900] RBP: ffff94d4b45337a8 R08: 0000000000000001 R09: 0000000000000000
[ 2397.900073] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2397.919146] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2397.938152] _raw_spin_lock_bh+0x1c/0x50
[ 2397.953795] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2397.971904] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2397.989525] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2398.006767] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2398.025169] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2398.044886] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2398.063304] ? remove_wait_queue+0x60/0x60
[ 2398.079489] kthread+0x103/0x140
[ 2398.094622] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2398.113087] ? kthread_bind+0x10/0x10
[ 2398.128783] ret_from_fork+0x3a/0x50
[ 2398.144301] NMI backtrace for cpu 0
[ 2398.144302] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2398.144302] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2398.144303] RIP: 0010:cfb_imageblit+0x4ba/0x4f0
[ 2398.144304] Code: 49 89 ed b9 08 00 00 00 89 04 24 eb 2e 41 0f be 55 00 44 29 c1 4d 8d 74 24 04 d3 fa 44 21 d2 41 8b 04 91 44 21 d8 89 c2 31 fa <41> 89 14 24 85 c9 75 06 49 83 c5 01 b1 08 4d 89 f4 83 ee 01 83 fe
[ 2398.144304] RSP: 0018:ffffa8a8800037a0 EFLAGS: 00000046
[ 2398.144305] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000003
[ 2398.144306] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000000
[ 2398.144306] RBP: ffff94d56c20aba4 R08: 0000000000000001 R09: ffffffff976bd6e0
[ 2398.144307] R10: 0000000000000001 R11: 0000000000aaaaaa R12: ffffa8a881d9cc50
[ 2398.144307] R13: ffff94d56c20aba6 R14: ffffa8a881d9cc54 R15: ffffa8a881d9cc00
[ 2398.144308] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2398.144308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2398.144309] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2398.144309] Call Trace:
[ 2398.144309] <IRQ>
[ 2398.144310] ? _raw_spin_unlock_irqrestore+0x50/0x52
[ 2398.144310] drm_fb_helper_cfb_imageblit+0x12/0x30 [drm_kms_helper]
[ 2398.144311] bit_putcs+0x2ec/0x500
[ 2398.144311] ? find_held_lock+0x2d/0x90
[ 2398.144312] ? soft_cursor+0x1af/0x220
[ 2398.144312] fbcon_putcs+0x11c/0x130
[ 2398.144313] ? update_attr.isra.2+0x80/0x80
[ 2398.144313] fbcon_redraw.isra.21+0x17b/0x1f0
[ 2398.144314] fbcon_scroll+0x4a5/0xda0
[ 2398.144314] con_scroll+0x1e0/0x200
[ 2398.144314] lf+0x9e/0xb0
[ 2398.144315] vt_console_print+0x248/0x3b0
[ 2398.144315] console_unlock+0x4cf/0x5e0
[ 2398.144316] ? console_unlock+0x1b2/0x5e0
[ 2398.144316] ? __down_trylock_console_sem+0x7d/0xa0
[ 2398.144317] vprintk_emit+0x13b/0x380
[ 2398.144317] ? __down_trylock_console_sem+0x7d/0xa0
[ 2398.144318] ? apic_timer_interrupt+0xf/0x20
[ 2398.144318] printk+0x58/0x6f
[ 2398.144318] show_opcodes+0x97/0xa0
[ 2398.144319] show_iret_regs+0x15/0x40
[ 2398.144319] __show_regs+0x1d/0x2c0
[ 2398.144320] ? apic_timer_interrupt+0xf/0x20
[ 2398.144320] show_trace_log_lvl+0x16e/0x400
[ 2398.144321] ? __local_bh_disable_ip+0x48/0x60
[ 2398.144321] dump_stack+0x82/0xb3
[ 2398.144322] nmi_cpu_backtrace+0x90/0xa0
[ 2398.144322] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2398.144322] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2398.144323] rcu_dump_cpu_stacks+0x97/0xc5
[ 2398.144323] rcu_sched_clock_irq+0x6ec/0x980
[ 2398.144324] ? rcu_read_lock_sched_held+0x52/0x90
[ 2398.144324] ? tick_sched_do_timer+0x70/0x70
[ 2398.144325] update_process_times+0x24/0x50
[ 2398.144325] tick_sched_handle+0x25/0x60
[ 2398.144326] tick_sched_timer+0x37/0x70
[ 2398.144326] __hrtimer_run_queues+0xe8/0x450
[ 2398.144327] hrtimer_interrupt+0x122/0x270
[ 2398.144327] smp_apic_timer_interrupt+0x86/0x240
[ 2398.144328] apic_timer_interrupt+0xf/0x20
[ 2398.144328] </IRQ>
[ 2398.144328] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 2398.144329] Code: fa 66 66 90 66 66 90 65 01 35 7c 0a 77 69 65 8b 05 75 0a 77 69 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 8d cc 06 00 48 89 df 57 9d <66> 66 90 66 90 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 2398.144330] RSP: 0018:ffffa8a885bafd68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2398.144331] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 2398.144332] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 2398.144332] RBP: ffff94d4b45337a8 R08: 0000000000000001 R09: 0000000000000000
[ 2398.144333] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2398.144333] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2398.144333] _raw_spin_lock_bh+0x1c/0x50
[ 2398.144334] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2398.144335] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2398.144335] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2398.144336] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2398.144336] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2398.144337] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2398.144337] ? remove_wait_queue+0x60/0x60
[ 2398.144338] kthread+0x103/0x140
[ 2398.144338] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2398.144339] ? kthread_bind+0x10/0x10
[ 2398.144339] ret_from_fork+0x3a/0x50
[ 2398.144340] ------------[ cut here ]------------
[ 2398.144340] igb_poll+0x0/0x740 [igb] exceeded budget in poll
[ 2398.144341] WARNING: CPU: 0 PID: 6148 at net/core/netpoll.c:153 netpoll_poll_dev+0x266/0x270
[ 2398.144341] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2398.144376] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2398.144376] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2398.144377] RIP: 0010:netpoll_poll_dev+0x266/0x270
[ 2398.144378] Code: cd fe ff ff 80 3d c9 1f e9 00 00 0f 85 9f fe ff ff 48 8b 73 28 48 c7 c7 0a 2c 9b 97 31 c0 c6 05 af 1f e9 00 01 e8 fa dd 8a ff <0f> 0b e9 7f fe ff ff 0f 1f 00 66 66 66 66 90 41 57 41 56 41 55 49
[ 2398.144378] RSP: 0018:ffffa8a880003b00 EFLAGS: 00010086
[ 2398.144379] RAX: 0000000000000000 RBX: ffff94d412b56450 RCX: 0000000000000000
[ 2398.144379] RDX: 0000000000000001 RSI: 0000000000000058 RDI: ffff94d56f01a148
[ 2398.144380] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff94d56f01a187
[ 2398.144380] R10: 0000000000000018 R11: 000000000000000f R12: ffff94d550fc4058
[ 2398.144381] R13: ffff94d545655a08 R14: 0000000000000014 R15: 00000000ffffffff
[ 2398.144381] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2398.144382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2398.144383] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2398.144383] Call Trace:
[ 2398.144383] <IRQ>
[ 2398.144384] netpoll_send_skb_on_dev+0x218/0x2d0
[ 2398.144384] netpoll_send_udp+0x2df/0x440
[ 2398.144385] write_msg+0xb5/0x100 [netconsole]
[ 2398.144385] console_unlock+0x4cf/0x5e0
[ 2398.144386] ? console_unlock+0x1b2/0x5e0
[ 2398.144386] ? __down_trylock_console_sem+0x7d/0xa0
[ 2398.144387] vprintk_emit+0x13b/0x380
[ 2398.144387] ? hrtimer_interrupt+0x122/0x270
[ 2398.144387] ? hrtimer_interrupt+0x122/0x270
[ 2398.144388] printk+0x58/0x6f
[ 2398.144388] ? hrtimer_interrupt+0x122/0x270
[ 2398.144389] show_trace_log_lvl+0x257/0x400
[ 2398.144389] ? hrtimer_interrupt+0x122/0x270
[ 2398.144390] dump_stack+0x82/0xb3
[ 2398.144390] nmi_cpu_backtrace+0x90/0xa0
[ 2398.144391] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2398.144391] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2398.144392] rcu_dump_cpu_stacks+0x97/0xc5
[ 2398.144392] rcu_sched_clock_irq+0x6ec/0x980
[ 2398.144392] ? rcu_read_lock_sched_held+0x52/0x90
[ 2398.144393] ? tick_sched_do_timer+0x70/0x70
[ 2398.144393] update_process_times+0x24/0x50
[ 2398.144394] tick_sched_handle+0x25/0x60
[ 2398.144394] tick_sched_timer+0x37/0x70
[ 2398.144395] __hrtimer_run_queues+0xe8/0x450
[ 2398.144395] hrtimer_interrupt+0x122/0x270
[ 2398.144396] smp_apic_timer_interrupt+0x86/0x240
[ 2398.144396] apic_timer_interrupt+0xf/0x20
[ 2398.144397] </IRQ>
[ 2398.144397] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 2398.144398] Code: fa 66 66 90 66 66 90 65 01 35 7c 0a 77 69 65 8b 05 75 0a 77 69 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 8d cc 06 00 48 89 df 57 9d <66> 66 90 66 90 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 2398.144398] RSP: 0018:ffffa8a885bafd68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2398.144399] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 2398.144400] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 2398.144400] RBP: ffff94d4b45337a8 R08: 0000000000000001 R09: 0000000000000000
[ 2398.144401] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2398.144401] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2398.144402] _raw_spin_lock_bh+0x1c/0x50
[ 2398.144402] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2398.144403] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2398.144403] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2398.144404] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2398.144404] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2398.144405] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2398.144405] ? remove_wait_queue+0x60/0x60
[ 2398.144406] kthread+0x103/0x140
[ 2398.144406] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2398.144407] ? kthread_bind+0x10/0x10
[ 2398.144407] ret_from_fork+0x3a/0x50
[ 2398.144408] irq event stamp: 1022549451
[ 2398.144408] hardirqs last enabled at (1022549449): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 2398.144409] hardirqs last disabled at (1022549451): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2398.144410] softirqs last enabled at (1022549446): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2398.144410] softirqs last disabled at (1022549450): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2398.144411] ---[ end trace 23790484fcb46096 ]---
[ 2425.528032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2425.546651] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2425.739048] irq event stamp: 1417476805
[ 2425.757557] hardirqs last enabled at (1417476803): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2425.782149] hardirqs last disabled at (1417476805): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2425.807227] softirqs last enabled at (1417476802): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2425.833808] softirqs last disabled at (1417476804): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2425.876915] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2425.902098] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2425.925187] RIP: 0010:lock_release+0x165/0x2a0
[ 2425.946114] Code: 00 00 00 00 00 57 9d 66 66 90 66 90 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9 00 00 00 48 83 c4 10 5b 5d 41 5c 41 5d <41> 5e 41 5f c3 65 8b 05 ef e2 6f 69 89 c0 48 0f a3 05 25 05 5a 01
[ 2425.999569] RSP: 0018:ffffa8a885bafd80 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 2426.024477] RAX: 0000000000000000 RBX: ffff94d4b4533850 RCX: ffffa8a885bafd54
[ 2426.048664] RDX: 0000000000000000 RSI: ffff94d4b4533868 RDI: 0000000000000246
[ 2426.072623] RBP: ffff94d4b45339a0 R08: 0000000000000000 R09: 0000000000000000
[ 2426.096541] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2426.120567] R13: ffff94d4b4bb8000 R14: ffff94d550e60cc0 R15: 0000000000000001
[ 2426.144628] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2426.169878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2426.192742] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2426.217070] Call Trace:
[ 2426.236568] _raw_spin_unlock_bh+0x17/0x30
[ 2426.257731] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2426.280628] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2426.303207] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2426.326664] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2426.351413] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2426.374848] ? remove_wait_queue+0x60/0x60
[ 2426.396236] kthread+0x103/0x140
[ 2426.416621] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2426.440395] ? kthread_bind+0x10/0x10
[ 2426.461444] ret_from_fork+0x3a/0x50
[ 2453.527274] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 2453.552105] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2453.786779] irq event stamp: 1852094964
[ 2453.809063] hardirqs last enabled at (1852094963): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2453.837007] hardirqs last disabled at (1852094964): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2453.865060] softirqs last enabled at (1852094962): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2453.894159] softirqs last disabled at (1852094960): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2453.922777] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2453.949067] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2453.973031] RIP: 0010:__init_waitqueue_head+0x11/0x50
[ 2453.995237] Code: 37 48 c7 47 10 00 ed 8f 96 48 89 47 18 48 89 47 20 c3 0f 1f 80 00 00 00 00 66 66 66 66 90 41 54 49 89 d4 48 c7 c2 70 15 92 98 <55> 48 89 f5 48 c7 c6 b2 e6 8f 97 53 48 89 fb e8 3b 9f 01 00 48 8d
[ 2454.049572] RSP: 0018:ffffa8a885bafd58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2454.075033] RAX: 0000000000000000 RBX: ffff94d4b45339a0 RCX: 0000000000000000
[ 2454.099764] RDX: ffffffff98921570 RSI: ffffffffc0c4767c RDI: ffffa8a885bafd80
[ 2454.124350] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
[ 2454.148856] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffffffffc0c6d460
[ 2454.173274] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2454.197305] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2454.222043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2454.244259] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2454.267945] Call Trace:
[ 2454.286912] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 2454.310024] ? lockdep_hardirqs_on+0x122/0x1b0
[ 2454.331205] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2454.354003] ? __local_bh_enable_ip+0x80/0xf0
[ 2454.375036] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2454.397631] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 2454.419601] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2454.442409] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2454.466650] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2454.489593] ? remove_wait_queue+0x60/0x60
[ 2454.510370] kthread+0x103/0x140
[ 2454.530160] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2454.553265] ? kthread_bind+0x10/0x10
[ 2454.573412] ret_from_fork+0x3a/0x50
[ 2459.974176] INFO: task iscsi_np:4183 blocked for more than 122 seconds.
[ 2459.981267] Tainted: G W L 5.5.0 #34
[ 2459.986774] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2459.994897] iscsi_np D 0 4183 2 0x80004084
[ 2460.000623] Call Trace:
[ 2460.003343] ? __schedule+0x38c/0x990
[ 2460.007252] schedule+0x5f/0xd0
[ 2460.010626] schedule_timeout+0x1e4/0x310
[ 2460.014874] ? find_held_lock+0x2d/0x90
[ 2460.018945] ? wait_for_completion+0x126/0x190
[ 2460.023615] wait_for_completion+0x12e/0x190
[ 2460.028136] ? wake_up_q+0xa0/0xa0
[ 2460.031803] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 2460.039517] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 2460.045747] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 2460.052694] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 2460.059439] kthread+0x103/0x140
[ 2460.062880] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 2460.069881] ? kthread_bind+0x10/0x10
[ 2460.073747] ret_from_fork+0x3a/0x50
[ 2460.077538]
[ 2460.077538] Showing all locks held in the system:
[ 2460.084257] 1 lock held by khungtaskd/55:
[ 2460.088446] #0: ffffffff97c85280 (rcu_read_lock){....}, at: debug_show_all_locks+0xe/0x1a0
[ 2460.096998] 2 locks held by agetty/1628:
[ 2460.101113] #0: ffff94d5628d6090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2460.109645] #1: ffffa8a88101b2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2460.118378] 2 locks held by agetty/1631:
[ 2460.122479] #0: ffff94d56ab95090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2460.131039] #1: ffffa8a8811fb2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2460.139756] 2 locks held by bash/3758:
[ 2460.143681] #0: ffff94d569f8f890 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2460.152219] #1: ffffa8a8802872e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2460.160939]
[ 2460.162603] =============================================
[ 2460.162603]
[ 2485.526406] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2485.546901] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2485.750520] irq event stamp: 2348916499
[ 2485.769976] hardirqs last enabled at (2348916497): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2485.795552] hardirqs last disabled at (2348916499): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2485.821426] softirqs last enabled at (2348916496): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2485.848616] softirqs last disabled at (2348916498): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2485.892144] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2485.917418] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2485.940617] RIP: 0010:iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2485.964451] Code: 00 e9 ff fe ff ff b8 ff ff ff ff e9 2b ff ff ff 90 66 66 66 66 90 55 48 8d af 18 01 00 00 53 48 89 fb 48 89 ef e8 f7 89 32 d6 <8b> 03 a8 01 74 29 83 c8 02 48 89 ef 89 03 e8 f4 8a 32 d6 48 8d bb
[ 2486.018280] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 2486.043741] RAX: ffff94d550e60000 RBX: ffff94d4b4533690 RCX: 0000000000000000
[ 2486.068922] RDX: 0000000000000001 RSI: 0000000096612835 RDI: ffff94d4b45337a8
[ 2486.094158] RBP: ffff94d4b45337a8 R08: 00000000fffffff0 R09: 0000000000000000
[ 2486.119300] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2486.144370] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2486.169449] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2486.195653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2486.219433] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2486.244693] Call Trace:
[ 2486.265045] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2486.288986] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2486.312679] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2486.337358] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2486.363401] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2486.388014] ? remove_wait_queue+0x60/0x60
[ 2486.410330] kthread+0x103/0x140
[ 2486.431518] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2486.455978] ? kthread_bind+0x10/0x10
[ 2486.477379] ret_from_fork+0x3a/0x50
[ 2513.525649] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2513.550594] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2513.786669] irq event stamp: 2782780619
[ 2513.809512] hardirqs last enabled at (2782780617): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2513.838093] hardirqs last disabled at (2782780619): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2513.866732] softirqs last enabled at (2782780616): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2513.896442] softirqs last disabled at (2782780618): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2513.925689] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2513.952594] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2513.977025] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 2513.998960] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 2514.054006] RSP: 0018:ffffa8a885bafd28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2514.079819] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 2514.105185] RDX: 0000000000000001 RSI: 000000003d7869c8 RDI: 0000000000000246
[ 2514.130188] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 2514.154907] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2514.179354] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2514.203655] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2514.228673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2514.250977] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2514.274660] Call Trace:
[ 2514.293413] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2514.315805] _raw_spin_lock_bh+0x3d/0x50
[ 2514.336165] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2514.358580] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2514.380885] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2514.402665] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2514.425293] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2514.449171] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2514.471710] ? remove_wait_queue+0x60/0x60
[ 2514.492064] kthread+0x103/0x140
[ 2514.511439] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2514.534206] ? kthread_bind+0x10/0x10
[ 2514.554029] ret_from_fork+0x3a/0x50
[ 2541.524891] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2541.548356] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2541.774819] irq event stamp: 3215859132
[ 2541.796265] hardirqs last enabled at (3215859131): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2541.823448] hardirqs last disabled at (3215859132): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2541.850563] softirqs last enabled at (3215859130): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2541.878743] softirqs last disabled at (3215859128): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2541.923695] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2541.949426] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2541.973005] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 2541.994801] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 2542.048646] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 2542.073728] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 2542.098363] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 2542.122598] RBP: ffffffffc0e6c0b9 R08: 0000000000000000 R09: 0000000000000000
[ 2542.146415] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2542.170085] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2542.193710] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2542.218489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2542.241023] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2542.265223] Call Trace:
[ 2542.284679] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2542.307554] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2542.329995] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2542.353276] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2542.377801] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2542.401043] ? remove_wait_queue+0x60/0x60
[ 2542.422079] kthread+0x103/0x140
[ 2542.442130] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2542.465586] ? kthread_bind+0x10/0x10
[ 2542.486141] ret_from_fork+0x3a/0x50
[ 2569.524133] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 2569.548362] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2569.781909] irq event stamp: 3649430970
[ 2569.804603] hardirqs last enabled at (3649430969): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2569.833013] hardirqs last disabled at (3649430970): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2569.861467] softirqs last enabled at (3649430968): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2569.890970] softirqs last disabled at (3649430966): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2569.919988] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2569.946700] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2569.970919] RIP: 0010:iscsit_close_connection+0x503/0x910 [iscsi_target_mod]
[ 2569.995515] Code: 89 ef e8 60 69 31 d6 48 8b 54 24 28 48 8b 02 48 39 ea 4c 8d aa 00 fd ff ff 49 89 d6 4c 8d b8 00 fd ff ff 75 0b e9 59 fc ff ff <4d> 89 fd 49 89 c7 4c 89 f7 e8 9f 20 e3 d5 84 c0 74 15 49 8b b5 00
[ 2570.050091] RSP: 0018:ffffa8a885bafe10 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
[ 2570.075730] RAX: ffff94d4b4533690 RBX: ffff94d4b4bb8000 RCX: 0000000000000000
[ 2570.100923] RDX: ffff94d4b4533ae0 RSI: 0000000000000003 RDI: ffff94d4b4533ae0
[ 2570.125788] RBP: ffffa8a885bafe38 R08: 0000000000000000 R09: 0000000000000000
[ 2570.150451] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffff94d4b4bbb800
[ 2570.174985] R13: ffff94d4b4533690 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2570.199451] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2570.224745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2570.247340] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2570.271329] Call Trace:
[ 2570.290403] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2570.314902] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2570.338236] ? remove_wait_queue+0x60/0x60
[ 2570.359434] kthread+0x103/0x140
[ 2570.379636] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2570.403092] ? kthread_bind+0x10/0x10
[ 2570.423583] ret_from_fork+0x3a/0x50
[ 2582.850839] INFO: task iscsi_np:4183 blocked for more than 245 seconds.
[ 2582.857725] Tainted: G W L 5.5.0 #34
[ 2582.863192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2582.871244] iscsi_np D 0 4183 2 0x80004084
[ 2582.876977] Call Trace:
[ 2582.879679] ? __schedule+0x38c/0x990
[ 2582.883597] schedule+0x5f/0xd0
[ 2582.887001] schedule_timeout+0x1e4/0x310
[ 2582.891248] ? find_held_lock+0x2d/0x90
[ 2582.895329] ? wait_for_completion+0x126/0x190
[ 2582.900023] wait_for_completion+0x12e/0x190
[ 2582.904526] ? wake_up_q+0xa0/0xa0
[ 2582.908185] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 2582.915921] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 2582.922165] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 2582.929122] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 2582.935907] kthread+0x103/0x140
[ 2582.939377] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 2582.946389] ? kthread_bind+0x10/0x10
[ 2582.950275] ret_from_fork+0x3a/0x50
[ 2582.954078]
[ 2582.954078] Showing all locks held in the system:
[ 2582.960882] 1 lock held by khungtaskd/55:
[ 2582.965102] #0: ffffffff97c85280 (rcu_read_lock){....}, at: debug_show_all_locks+0xe/0x1a0
[ 2582.973699] 2 locks held by agetty/1628:
[ 2582.977834] #0: ffff94d5628d6090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2582.986388] #1: ffffa8a88101b2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2582.995137] 2 locks held by agetty/1631:
[ 2582.999269] #0: ffff94d56ab95090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2583.007827] #1: ffffa8a8811fb2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2583.016553] 2 locks held by bash/3758:
[ 2583.020502] #0: ffff94d569f8f890 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2583.029073] #1: ffffa8a8802872e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2583.037807] 1 lock held by sosreport/6414:
[ 2583.042086] #0: ffff94d40ee920a0 (&p->lock){+.+.}, at: seq_read+0x3c/0x3e0
[ 2583.049250]
[ 2583.050937] =============================================
[ 2583.050937]
[ 2592.310517] rcu: INFO: rcu_sched self-detected stall on CPU
[ 2592.329471] rcu: 0-....: (248478 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=64200
[ 2592.353108] (t=260022 jiffies g=82081 q=61043)
[ 2592.371225] NMI backtrace for cpu 0
[ 2592.388109] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2592.410628] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2592.431063] Call Trace:
[ 2592.447157] <IRQ>
[ 2592.462614] dump_stack+0x82/0xb3
[ 2592.479412] nmi_cpu_backtrace+0x90/0xa0
[ 2592.496712] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2592.514436] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2592.532623] rcu_dump_cpu_stacks+0x97/0xc5
[ 2592.549646] rcu_sched_clock_irq+0x6ec/0x980
[ 2592.566811] ? rcu_read_lock_sched_held+0x52/0x90
[ 2592.584477] ? tick_sched_do_timer+0x70/0x70
[ 2592.601718] update_process_times+0x24/0x50
[ 2592.618928] tick_sched_handle+0x25/0x60
[ 2592.635872] tick_sched_timer+0x37/0x70
[ 2592.652668] __hrtimer_run_queues+0xe8/0x450
[ 2592.669906] hrtimer_interrupt+0x122/0x270
[ 2592.686959] smp_apic_timer_interrupt+0x86/0x240
[ 2592.704420] apic_timer_interrupt+0xf/0x20
[ 2592.721251] </IRQ>
[ 2592.735898] RIP: 0010:iscsit_increment_maxcmdsn+0x0/0x50 [iscsi_target_mod]
[ 2592.755714] Code: 74 0c 48 8b 87 f8 01 00 00 48 83 e8 30 c3 8b 77 20 31 c0 48 c7 c7 50 f1 e7 c0 e8 7f 7d ac d5 31 c0 c3 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 48 b8 ff 00 00 00 00 00 ff 00 48 85 47 10 75 17 c6
[ 2592.801165] RSP: 0018:ffffa8a885bafe08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 2592.822215] RAX: ffff94d4b4533990 RBX: ffff94d4b4bb8000 RCX: dead000000000122
[ 2592.842996] RDX: ffff94d4b4533990 RSI: ffff94d4b4bbb800 RDI: ffff94d4b4533690
[ 2592.863769] RBP: ffffa8a885bafe38 R08: 0000000000000000 R09: 0000000000000000
[ 2592.884609] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffff94d4b4bbb800
[ 2592.905478] R13: ffff94d4b4533690 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2592.926224] ? target_put_sess_cmd+0x40/0x50 [target_core_mod]
[ 2592.945529] iscsit_close_connection+0x54b/0x910 [iscsi_target_mod]
[ 2592.965279] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2592.986158] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2593.005597] ? remove_wait_queue+0x60/0x60
[ 2593.022752] kthread+0x103/0x140
[ 2593.038810] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2593.058082] ? kthread_bind+0x10/0x10
[ 2593.074332] ret_from_fork+0x3a/0x50
[ 2593.090301] NMI backtrace for cpu 0
[ 2593.090302] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2593.090302] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2593.090303] RIP: 0010:cfb_imageblit+0x4ba/0x4f0
[ 2593.090303] Code: 49 89 ed b9 08 00 00 00 89 04 24 eb 2e 41 0f be 55 00 44 29 c1 4d 8d 74 24 04 d3 fa 44 21 d2 41 8b 04 91 44 21 d8 89 c2 31 fa <41> 89 14 24 85 c9 75 06 49 83 c5 01 b1 08 4d 89 f4 83 ee 01 83 fe
[ 2593.090304] RSP: 0018:ffffa8a880003868 EFLAGS: 00000046
[ 2593.090305] RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000000000004
[ 2593.090305] RDX: 0000000000000000 RSI: 000000000000002c RDI: 0000000000000000
[ 2593.090306] RBP: ffff94d56c20ab86 R08: 0000000000000001 R09: ffffffff976bd6e0
[ 2593.090306] R10: 0000000000000001 R11: 0000000000aaaaaa R12: ffffa8a881eea22c
[ 2593.090307] R13: ffff94d56c20ab87 R14: ffffa8a881eea230 R15: ffffa8a881eea200
[ 2593.090307] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2593.090308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2593.090308] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2593.090309] Call Trace:
[ 2593.090309] <IRQ>
[ 2593.090310] ? _raw_spin_unlock_irqrestore+0x50/0x52
[ 2593.090310] drm_fb_helper_cfb_imageblit+0x12/0x30 [drm_kms_helper]
[ 2593.090311] bit_putcs+0x2ec/0x500
[ 2593.090311] ? find_held_lock+0x2d/0x90
[ 2593.090311] ? soft_cursor+0x1af/0x220
[ 2593.090312] fbcon_putcs+0x11c/0x130
[ 2593.090312] ? update_attr.isra.2+0x80/0x80
[ 2593.090313] fbcon_redraw.isra.21+0x17b/0x1f0
[ 2593.090313] fbcon_scroll+0x4a5/0xda0
[ 2593.090314] con_scroll+0x1e0/0x200
[ 2593.090314] lf+0x9e/0xb0
[ 2593.090315] vt_console_print+0x248/0x3b0
[ 2593.090315] console_unlock+0x4cf/0x5e0
[ 2593.090315] ? console_unlock+0x1b2/0x5e0
[ 2593.090316] ? __down_trylock_console_sem+0x7d/0xa0
[ 2593.090317] vprintk_emit+0x13b/0x380
[ 2593.090317] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2593.090317] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2593.090318] printk+0x58/0x6f
[ 2593.090318] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2593.090319] show_trace_log_lvl+0x19b/0x400
[ 2593.090319] ? nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2593.090320] dump_stack+0x82/0xb3
[ 2593.090320] nmi_cpu_backtrace+0x90/0xa0
[ 2593.090320] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2593.090321] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2593.090321] rcu_dump_cpu_stacks+0x97/0xc5
[ 2593.090322] rcu_sched_clock_irq+0x6ec/0x980
[ 2593.090322] ? rcu_read_lock_sched_held+0x52/0x90
[ 2593.090323] ? tick_sched_do_timer+0x70/0x70
[ 2593.090323] update_process_times+0x24/0x50
[ 2593.090324] tick_sched_handle+0x25/0x60
[ 2593.090324] tick_sched_timer+0x37/0x70
[ 2593.090325] __hrtimer_run_queues+0xe8/0x450
[ 2593.090325] hrtimer_interrupt+0x122/0x270
[ 2593.090326] smp_apic_timer_interrupt+0x86/0x240
[ 2593.090326] apic_timer_interrupt+0xf/0x20
[ 2593.090326] </IRQ>
[ 2593.090327] RIP: 0010:iscsit_increment_maxcmdsn+0x0/0x50 [iscsi_target_mod]
[ 2593.090328] Code: 74 0c 48 8b 87 f8 01 00 00 48 83 e8 30 c3 8b 77 20 31 c0 48 c7 c7 50 f1 e7 c0 e8 7f 7d ac d5 31 c0 c3 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 48 b8 ff 00 00 00 00 00 ff 00 48 85 47 10 75 17 c6
[ 2593.090328] RSP: 0018:ffffa8a885bafe08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 2593.090329] RAX: ffff94d4b4533990 RBX: ffff94d4b4bb8000 RCX: dead000000000122
[ 2593.090330] RDX: ffff94d4b4533990 RSI: ffff94d4b4bbb800 RDI: ffff94d4b4533690
[ 2593.090330] RBP: ffffa8a885bafe38 R08: 0000000000000000 R09: 0000000000000000
[ 2593.090331] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffff94d4b4bbb800
[ 2593.090331] R13: ffff94d4b4533690 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2593.090332] ? target_put_sess_cmd+0x40/0x50 [target_core_mod]
[ 2593.090332] iscsit_close_connection+0x54b/0x910 [iscsi_target_mod]
[ 2593.090333] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2593.090333] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2593.090334] ? remove_wait_queue+0x60/0x60
[ 2593.090334] kthread+0x103/0x140
[ 2593.090335] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2593.090335] ? kthread_bind+0x10/0x10
[ 2593.090336] ret_from_fork+0x3a/0x50
[ 2617.522833] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [iscsi_ttx:6148]
[ 2617.540087] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2617.722533] irq event stamp: 79251060
[ 2617.739777] hardirqs last enabled at (79251059): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2617.763173] hardirqs last disabled at (79251060): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2617.786979] softirqs last enabled at (79251058): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2617.812216] softirqs last disabled at (79251056): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2617.837323] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2617.860710] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2617.882051] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 2617.901777] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 2617.951674] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 2617.975185] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 2617.998373] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 2618.021601] RBP: ffffffffc0e6c0f1 R08: 0000000000000000 R09: 0000000000000000
[ 2618.044883] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2618.068275] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb8650 R15: ffff94d4b4533690
[ 2618.091766] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2618.116497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2618.138924] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2618.162943] Call Trace:
[ 2618.182225] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2618.204949] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2618.227025] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2618.249707] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2618.273592] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2618.296129] ? remove_wait_queue+0x60/0x60
[ 2618.316579] kthread+0x103/0x140
[ 2618.336100] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2618.358997] ? kthread_bind+0x10/0x10
[ 2618.379065] ret_from_fork+0x3a/0x50
[ 2645.522076] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2645.545826] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2645.776110] irq event stamp: 514882333
[ 2645.798601] hardirqs last enabled at (514882331): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2645.826777] hardirqs last disabled at (514882333): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2645.855058] softirqs last enabled at (514882330): [<ffffffffc0e6c0c1>] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2645.884386] softirqs last disabled at (514882332): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2645.913287] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2645.940042] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2645.964279] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 2645.986660] Code: fa 66 66 90 66 66 90 65 01 35 7c 0a 77 69 65 8b 05 75 0a 77 69 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 8d cc 06 00 48 89 df 57 9d <66> 66 90 66 90 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 2646.040850] RSP: 0018:ffffa8a885bafd80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2646.066276] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 2646.091197] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 2646.116122] RBP: ffff94d4b45339a0 R08: 0000000000000001 R09: 0000000000000000
[ 2646.140745] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2646.165212] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb85e0 R15: ffff94d4b4533690
[ 2646.189642] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2646.214848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2646.237375] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2646.261256] Call Trace:
[ 2646.280195] _raw_spin_lock_bh+0x1c/0x50
[ 2646.300736] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2646.323154] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2646.345325] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2646.368381] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2646.392663] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2646.415540] ? remove_wait_queue+0x60/0x60
[ 2646.436204] kthread+0x103/0x140
[ 2646.455798] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2646.478774] ? kthread_bind+0x10/0x10
[ 2646.498836] ret_from_fork+0x3a/0x50
[ 2673.521319] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2673.545113] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2673.774793] irq event stamp: 948535530
[ 2673.796925] hardirqs last enabled at (948535529): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2673.824726] hardirqs last disabled at (948535530): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2673.852606] softirqs last enabled at (948535528): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2673.881571] softirqs last disabled at (948535526): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2673.910089] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2673.936376] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2673.960139] RIP: 0010:rcu_read_lock_sched_held+0xe/0x90
[ 2673.982351] Code: 00 00 c3 48 89 3c 24 e8 a0 3f 00 00 89 c2 31 c0 84 d2 75 e1 48 8b 3c 24 eb d3 66 90 66 66 66 66 90 48 83 ec 10 48 8d 7c 24 07 <65> 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 e8 6d ff ff ff 84
[ 2674.036022] RSP: 0018:ffffa8a885bafcd8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 2674.061159] RAX: 0000000000000001 RBX: ffffffffc0c6d460 RCX: ffff94d571c2d000
[ 2674.085937] RDX: 0000000000000008 RSI: 0000000000000008 RDI: ffffa8a885bafcdf
[ 2674.110411] RBP: ffffffffc0c6d460 R08: 0000000000000000 R09: 0000000000000000
[ 2674.134744] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2674.158992] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2674.182856] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2674.207499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2674.229621] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2674.253209] Call Trace:
[ 2674.272099] ? cpumask_next+0x17/0x20
[ 2674.292205] ? __is_kernel_percpu_address+0x39/0xa0
[ 2674.313567] module_assert_mutex_or_preempt+0x14/0x40
[ 2674.335076] __module_address+0x28/0xf0
[ 2674.355165] is_module_address+0x11/0x20
[ 2674.375178] static_obj+0x31/0x50
[ 2674.394374] lockdep_init_map+0x79/0x1f0
[ 2674.414076] __init_waitqueue_head+0x36/0x50
[ 2674.434170] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 2674.456474] ? lockdep_hardirqs_on+0x122/0x1b0
[ 2674.476815] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 2674.498132] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2674.520296] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2674.543755] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2674.565869] ? remove_wait_queue+0x60/0x60
[ 2674.585884] kthread+0x103/0x140
[ 2674.604886] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2674.627254] ? kthread_bind+0x10/0x10
[ 2674.646838] ret_from_fork+0x3a/0x50
[ 2701.520561] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2701.543490] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2701.756206] irq event stamp: 1379740796
[ 2701.775682] hardirqs last enabled at (1379740795): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2701.801178] hardirqs last disabled at (1379740796): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2701.826927] softirqs last enabled at (1379740794): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2701.854243] softirqs last disabled at (1379740792): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2701.881118] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2701.905949] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2701.928576] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 2701.949200] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 2701.999892] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 2702.023507] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 2702.046844] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 2702.070092] RBP: ffffffffc0e6c0fe R08: 0000000000000000 R09: 0000000000000000
[ 2702.093306] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2702.116561] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb85e0 R15: ffff94d4b4533690
[ 2702.139749] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2702.163977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2702.185771] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2702.209011] Call Trace:
[ 2702.227511] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2702.249551] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2702.271226] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2702.293862] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2702.317912] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2702.340756] ? remove_wait_queue+0x60/0x60
[ 2702.361521] kthread+0x103/0x140
[ 2702.381342] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2702.404529] ? kthread_bind+0x10/0x10
[ 2702.424784] ret_from_fork+0x3a/0x50
[ 2705.727507] INFO: task iscsi_np:4183 blocked for more than 368 seconds.
[ 2705.734402] Tainted: G W L 5.5.0 #34
[ 2705.739870] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2705.747958] iscsi_np D 0 4183 2 0x80004084
[ 2705.753698] Call Trace:
[ 2705.756386] ? __schedule+0x38c/0x990
[ 2705.760291] schedule+0x5f/0xd0
[ 2705.763673] schedule_timeout+0x1e4/0x310
[ 2705.767922] ? find_held_lock+0x2d/0x90
[ 2705.771992] ? wait_for_completion+0x126/0x190
[ 2705.776678] wait_for_completion+0x12e/0x190
[ 2705.781177] ? wake_up_q+0xa0/0xa0
[ 2705.784836] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 2705.792568] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 2705.798820] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 2705.805745] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 2705.812511] kthread+0x103/0x140
[ 2705.815960] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 2705.822946] ? kthread_bind+0x10/0x10
[ 2705.826819] ret_from_fork+0x3a/0x50
[ 2705.830618] INFO: task sosreport:6414 blocked for more than 122 seconds.
[ 2705.837543] Tainted: G W L 5.5.0 #34
[ 2705.842985] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2705.851024] sosreport D 0 6414 6413 0x00000080
[ 2705.856720] Call Trace:
[ 2705.859383] ? __schedule+0x38c/0x990
[ 2705.863247] schedule+0x5f/0xd0
[ 2705.866589] schedule_timeout+0x1e4/0x310
[ 2705.870778] ? find_held_lock+0x2d/0x90
[ 2705.874817] ? mark_held_locks+0x55/0x70
[ 2705.878924] wait_for_completion+0x12e/0x190
[ 2705.883362] ? wake_up_q+0xa0/0xa0
[ 2705.886946] __wait_rcu_gp+0xfd/0x140
[ 2705.890784] synchronize_rcu.part.81+0x3e/0x70
[ 2705.895411] ? __call_rcu+0x590/0x590
[ 2705.899238] ? __bpf_trace_rcu_utilization+0x10/0x10
[ 2705.904375] rcu_sync_enter+0x54/0xf0
[ 2705.908196] ? rcu_read_lock_sched_held+0x52/0x90
[ 2705.913070] percpu_down_write+0x15/0xe0
[ 2705.917162] ? seq_read+0x327/0x3e0
[ 2705.920814] locks_start+0x28/0x50
[ 2705.924376] seq_read+0x1da/0x3e0
[ 2705.927857] proc_reg_read+0x3c/0x70
[ 2705.931601] vfs_read+0xac/0x160
[ 2705.934995] ksys_read+0x5c/0xd0
[ 2705.938395] do_syscall_64+0x60/0x270
[ 2705.942237] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 2705.947457] RIP: 0033:0x7fc60f980c00
[ 2705.951186] Code: Bad RIP value.
[ 2705.954577] RSP: 002b:00007fff4e546148 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 2705.962315] RAX: ffffffffffffffda RBX: 000000000128c9e0 RCX: 00007fc60f980c00
[ 2705.969629] RDX: 0000000000004000 RSI: 0000000001290494 RDI: 000000000000000b
[ 2705.976934] RBP: 00007fc60fc543a0 R08: ffffffffffffffff R09: 0000000000000000
[ 2705.984238] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 2705.991553] R13: 0000000000000d70 R14: 00007fc60fc53858 R15: 0000000001290494
[ 2705.998858]
[ 2705.998858] Showing all locks held in the system:
[ 2706.005345] 1 lock held by khungtaskd/55:
[ 2706.009537] #0: ffffffff97c85280 (rcu_read_lock){....}, at: debug_show_all_locks+0xe/0x1a0
[ 2706.018067] 2 locks held by agetty/1628:
[ 2706.022155] #0: ffff94d5628d6090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2706.030676] #1: ffffa8a88101b2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2706.039372] 2 locks held by agetty/1631:
[ 2706.043471] #0: ffff94d56ab95090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2706.052016] #1: ffffa8a8811fb2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2706.060722] 2 locks held by bash/3758:
[ 2706.064637] #0: ffff94d569f8f890 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2706.073166] #1: ffffa8a8802872e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2706.081871] 1 lock held by iscsi_ttx/6148:
[ 2706.086156] 1 lock held by sosreport/6414:
[ 2706.090419] #0: ffff94d40ee920a0 (&p->lock){+.+.}, at: seq_read+0x3c/0x3e0
[ 2706.097560]
[ 2706.099214] =============================================
[ 2706.099214]
[ 2733.519695] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2733.538759] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2733.734593] irq event stamp: 1878332466
[ 2733.753417] hardirqs last enabled at (1878332465): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2733.778336] hardirqs last disabled at (1878332466): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2733.803643] softirqs last enabled at (1878332464): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2733.830456] softirqs last disabled at (1878332462): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2733.857260] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2733.882272] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2733.905214] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 2733.926574] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 2733.979846] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 2734.005014] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 2734.029829] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 2734.054661] RBP: ffffffffc0e6c0fe R08: 0000000000000000 R09: 0000000000000000
[ 2734.079315] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2734.103885] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb85e0 R15: ffff94d4b4533690
[ 2734.128500] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2734.154284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2734.177739] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2734.202772] Call Trace:
[ 2734.222878] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 2734.246535] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2734.269860] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2734.293956] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2734.319259] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2734.343131] ? remove_wait_queue+0x60/0x60
[ 2734.364741] kthread+0x103/0x140
[ 2734.385211] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2734.408936] ? kthread_bind+0x10/0x10
[ 2734.429674] ret_from_fork+0x3a/0x50
[ 2761.518938] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2761.543128] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2761.775193] irq event stamp: 2312852781
[ 2761.797890] hardirqs last enabled at (2312852779): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2761.826305] hardirqs last disabled at (2312852781): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2761.854782] softirqs last enabled at (2312852778): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2761.884305] softirqs last disabled at (2312852780): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2761.932031] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2761.958867] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2761.983194] RIP: 0010:lock_release+0x17/0x2a0
[ 2762.005082] Code: ea 5c 8e 97 31 c0 e8 38 c2 f8 ff 0f 0b e9 e8 fe ff ff 90 41 57 65 48 8b 14 25 40 8e 01 00 41 56 41 55 41 54 55 53 48 83 ec 10 <44> 8b 82 bc 0c 00 00 65 48 8b 04 25 28 00 00 00 48 89 44 24 08 31
[ 2762.060009] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 2762.085805] RAX: ffff94d4b4533898 RBX: ffff94d4b4533850 RCX: 0000000000000000
[ 2762.111213] RDX: ffff94d550e60000 RSI: ffffffffc0e6c0c1 RDI: ffff94d4b4533868
[ 2762.136362] RBP: ffff94d4b45339a0 R08: 00000000fffffff0 R09: 0000000000000000
[ 2762.161408] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2762.186344] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2762.210983] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2762.236429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2762.259373] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2762.283748] Call Trace:
[ 2762.303369] _raw_spin_unlock_bh+0x17/0x30
[ 2762.324735] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2762.347859] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2762.370587] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2762.394124] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2762.418926] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2762.442341] ? remove_wait_queue+0x60/0x60
[ 2762.463524] kthread+0x103/0x140
[ 2762.483724] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2762.507283] ? kthread_bind+0x10/0x10
[ 2762.527976] ret_from_fork+0x3a/0x50
[ 2787.308240] rcu: INFO: rcu_sched self-detected stall on CPU
[ 2787.330717] rcu: 0-....: (435632 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=112757
[ 2787.358006] (t=455029 jiffies g=82081 q=74555)
[ 2787.379848] NMI backtrace for cpu 0
[ 2787.400520] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2787.426827] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2787.451087] Call Trace:
[ 2787.470896] <IRQ>
[ 2787.489852] dump_stack+0x82/0xb3
[ 2787.509723] nmi_cpu_backtrace+0x90/0xa0
[ 2787.529852] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2787.550027] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2787.570618] rcu_dump_cpu_stacks+0x97/0xc5
[ 2787.589830] rcu_sched_clock_irq+0x6ec/0x980
[ 2787.608830] ? rcu_read_lock_sched_held+0x52/0x90
[ 2787.627851] ? tick_sched_do_timer+0x70/0x70
[ 2787.646232] update_process_times+0x24/0x50
[ 2787.664268] tick_sched_handle+0x25/0x60
[ 2787.681934] tick_sched_timer+0x37/0x70
[ 2787.699287] __hrtimer_run_queues+0xe8/0x450
[ 2787.716959] hrtimer_interrupt+0x122/0x270
[ 2787.734043] smp_apic_timer_interrupt+0x86/0x240
[ 2787.751482] apic_timer_interrupt+0xf/0x20
[ 2787.768197] </IRQ>
[ 2787.782683] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 2787.799122] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 2787.842879] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2787.862943] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 2787.882587] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 2787.902186] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 2787.921723] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2787.941107] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2787.960278] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2787.978786] _raw_spin_lock_bh+0x3d/0x50
[ 2787.994564] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2788.013037] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2788.031468] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2788.049283] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2788.066723] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2788.085056] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2788.104659] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2788.122923] ? remove_wait_queue+0x60/0x60
[ 2788.138985] kthread+0x103/0x140
[ 2788.154158] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2788.172688] ? kthread_bind+0x10/0x10
[ 2788.188263] ret_from_fork+0x3a/0x50
[ 2788.203770] NMI backtrace for cpu 0
[ 2788.203770] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2788.203771] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2788.203771] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 2788.203772] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 d1 98 6f 69 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 b4 98 6f 69 89 43 08 65 48 8b 04 25
[ 2788.203773] RSP: 0018:ffffa8a8800037d0 EFLAGS: 00000046
[ 2788.203774] RAX: 0000000000000000 RBX: ffff94d56f284930 RCX: 0000000000000000
[ 2788.203774] RDX: 0000000000000001 RSI: 00000000c92926a5 RDI: ffff94d56f284930
[ 2788.203775] RBP: 0000000000000082 R08: 00000000691b6730 R09: 0000000000000001
[ 2788.203775] R10: 0000000000000000 R11: 0000000000aaaaaa R12: 0000000000000200
[ 2788.203776] R13: ffff94d56f284930 R14: 0000000000000010 R15: 0000000000000058
[ 2788.203776] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2788.203777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2788.203777] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2788.203778] Call Trace:
[ 2788.203778] <IRQ>
[ 2788.203779] _raw_spin_lock_irqsave+0x54/0x60
[ 2788.203779] ? drm_fb_helper_dirty.isra.9+0x51/0xf0 [drm_kms_helper]
[ 2788.203780] drm_fb_helper_dirty.isra.9+0x51/0xf0 [drm_kms_helper]
[ 2788.203780] bit_putcs+0x2ec/0x500
[ 2788.203780] ? find_held_lock+0x2d/0x90
[ 2788.203781] ? soft_cursor+0x1af/0x220
[ 2788.203781] fbcon_putcs+0x11c/0x130
[ 2788.203782] ? update_attr.isra.2+0x80/0x80
[ 2788.203782] fbcon_redraw.isra.21+0x17b/0x1f0
[ 2788.203783] fbcon_scroll+0x4a5/0xda0
[ 2788.203783] con_scroll+0x1e0/0x200
[ 2788.203783] lf+0x9e/0xb0
[ 2788.203784] vt_console_print+0x248/0x3b0
[ 2788.203784] console_unlock+0x4cf/0x5e0
[ 2788.203785] ? console_unlock+0x1b2/0x5e0
[ 2788.203785] ? __down_trylock_console_sem+0x7d/0xa0
[ 2788.203786] vprintk_emit+0x13b/0x380
[ 2788.203786] ? __down_trylock_console_sem+0x7d/0xa0
[ 2788.203787] ? apic_timer_interrupt+0xf/0x20
[ 2788.203787] printk+0x58/0x6f
[ 2788.203788] show_opcodes+0x97/0xa0
[ 2788.203788] show_iret_regs+0x15/0x40
[ 2788.203788] __show_regs+0x1d/0x2c0
[ 2788.203789] ? apic_timer_interrupt+0xf/0x20
[ 2788.203789] show_trace_log_lvl+0x16e/0x400
[ 2788.203790] ? lock_acquire+0xd5/0x1c0
[ 2788.203790] dump_stack+0x82/0xb3
[ 2788.203791] nmi_cpu_backtrace+0x90/0xa0
[ 2788.203791] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2788.203792] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2788.203792] rcu_dump_cpu_stacks+0x97/0xc5
[ 2788.203793] rcu_sched_clock_irq+0x6ec/0x980
[ 2788.203793] ? rcu_read_lock_sched_held+0x52/0x90
[ 2788.203793] ? tick_sched_do_timer+0x70/0x70
[ 2788.203794] update_process_times+0x24/0x50
[ 2788.203794] tick_sched_handle+0x25/0x60
[ 2788.203795] tick_sched_timer+0x37/0x70
[ 2788.203795] __hrtimer_run_queues+0xe8/0x450
[ 2788.203796] hrtimer_interrupt+0x122/0x270
[ 2788.203796] smp_apic_timer_interrupt+0x86/0x240
[ 2788.203797] apic_timer_interrupt+0xf/0x20
[ 2788.203797] </IRQ>
[ 2788.203798] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 2788.203798] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 2788.203799] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2788.203800] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 2788.203800] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 2788.203801] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 2788.203801] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2788.203802] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2788.203802] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2788.203803] _raw_spin_lock_bh+0x3d/0x50
[ 2788.203803] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2788.203804] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2788.203804] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2788.203805] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2788.203805] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2788.203806] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2788.203806] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2788.203807] ? remove_wait_queue+0x60/0x60
[ 2788.203807] kthread+0x103/0x140
[ 2788.203808] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2788.203808] ? kthread_bind+0x10/0x10
[ 2788.203809] ret_from_fork+0x3a/0x50
[ 2813.517530] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 2813.534845] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2813.717906] irq event stamp: 3096402481
[ 2813.735363] hardirqs last enabled at (3096402479): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2813.758926] hardirqs last disabled at (3096402481): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2813.782901] softirqs last enabled at (3096402478): [<ffffffffc0e6c0c1>] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2813.808311] softirqs last disabled at (3096402480): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2813.833731] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2813.857304] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2813.878780] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 2813.898062] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 2813.948453] RSP: 0018:ffffa8a885bafd28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2813.972245] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 2813.995728] RDX: 0000000000000001 RSI: 0000000002c1e137 RDI: 0000000000000246
[ 2814.019202] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 2814.042660] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2814.066120] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2814.089537] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2814.113698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2814.135152] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2814.157971] Call Trace:
[ 2814.175920] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2814.197550] _raw_spin_lock_bh+0x3d/0x50
[ 2814.217310] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2814.239236] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2814.261127] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2814.282668] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2814.305069] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2814.328836] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2814.351249] ? remove_wait_queue+0x60/0x60
[ 2814.371506] kthread+0x103/0x140
[ 2814.390788] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2814.413489] ? kthread_bind+0x10/0x10
[ 2814.433279] ret_from_fork+0x3a/0x50
[ 2828.604178] INFO: task iscsi_np:4183 blocked for more than 491 seconds.
[ 2828.611060] Tainted: G W L 5.5.0 #34
[ 2828.616509] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2828.624582] iscsi_np D 0 4183 2 0x80004084
[ 2828.630303] Call Trace:
[ 2828.632996] ? __schedule+0x38c/0x990
[ 2828.636889] schedule+0x5f/0xd0
[ 2828.640265] schedule_timeout+0x1e4/0x310
[ 2828.644501] ? find_held_lock+0x2d/0x90
[ 2828.648561] ? wait_for_completion+0x126/0x190
[ 2828.653247] wait_for_completion+0x12e/0x190
[ 2828.657745] ? wake_up_q+0xa0/0xa0
[ 2828.661383] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 2828.669094] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 2828.675333] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 2828.682257] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 2828.689010] kthread+0x103/0x140
[ 2828.692440] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 2828.699444] ? kthread_bind+0x10/0x10
[ 2828.703318] ret_from_fork+0x3a/0x50
[ 2828.707101] INFO: task sosreport:6414 blocked for more than 245 seconds.
[ 2828.714000] Tainted: G W L 5.5.0 #34
[ 2828.719429] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2828.727452] sosreport D 0 6414 6413 0x00000080
[ 2828.733142] Call Trace:
[ 2828.735779] ? __schedule+0x38c/0x990
[ 2828.739631] schedule+0x5f/0xd0
[ 2828.742955] schedule_timeout+0x1e4/0x310
[ 2828.747153] ? find_held_lock+0x2d/0x90
[ 2828.751189] ? mark_held_locks+0x55/0x70
[ 2828.755295] wait_for_completion+0x12e/0x190
[ 2828.759744] ? wake_up_q+0xa0/0xa0
[ 2828.763317] __wait_rcu_gp+0xfd/0x140
[ 2828.767160] synchronize_rcu.part.81+0x3e/0x70
[ 2828.771790] ? __call_rcu+0x590/0x590
[ 2828.775615] ? __bpf_trace_rcu_utilization+0x10/0x10
[ 2828.780743] rcu_sync_enter+0x54/0xf0
[ 2828.784569] ? rcu_read_lock_sched_held+0x52/0x90
[ 2828.789430] percpu_down_write+0x15/0xe0
[ 2828.793524] ? seq_read+0x327/0x3e0
[ 2828.797194] locks_start+0x28/0x50
[ 2828.800754] seq_read+0x1da/0x3e0
[ 2828.804261] proc_reg_read+0x3c/0x70
[ 2828.808005] vfs_read+0xac/0x160
[ 2828.811396] ksys_read+0x5c/0xd0
[ 2828.814788] do_syscall_64+0x60/0x270
[ 2828.818607] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 2828.823825] RIP: 0033:0x7fc60f980c00
[ 2828.827558] Code: Bad RIP value.
[ 2828.830939] RSP: 002b:00007fff4e546148 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 2828.838690] RAX: ffffffffffffffda RBX: 000000000128c9e0 RCX: 00007fc60f980c00
[ 2828.845987] RDX: 0000000000004000 RSI: 0000000001290494 RDI: 000000000000000b
[ 2828.853299] RBP: 00007fc60fc543a0 R08: ffffffffffffffff R09: 0000000000000000
[ 2828.860609] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 2828.867915] R13: 0000000000000d70 R14: 00007fc60fc53858 R15: 0000000001290494
[ 2828.875245]
[ 2828.875245] Showing all locks held in the system:
[ 2828.881734] 1 lock held by khungtaskd/55:
[ 2828.885905] #0: ffffffff97c85280 (rcu_read_lock){....}, at: debug_show_all_locks+0xe/0x1a0
[ 2828.894446] 2 locks held by agetty/1628:
[ 2828.898529] #0: ffff94d5628d6090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2828.907047] #1: ffffa8a88101b2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2828.915751] 2 locks held by agetty/1631:
[ 2828.919860] #0: ffff94d56ab95090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2828.928392] #1: ffffa8a8811fb2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2828.937101] 2 locks held by bash/3758:
[ 2828.941034] #0: ffff94d569f8f890 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2828.949577] #1: ffffa8a8802872e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2828.958293] 1 lock held by sosreport/6414:
[ 2828.962555] #0: ffff94d40ee920a0 (&p->lock){+.+.}, at: seq_read+0x3c/0x3e0
[ 2828.969686]
[ 2828.971345] =============================================
[ 2828.971345]
[ 2853.516449] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2853.535479] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2853.731630] irq event stamp: 3723586220
[ 2853.750546] hardirqs last enabled at (3723586219): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2853.775524] hardirqs last disabled at (3723586220): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2853.800932] softirqs last enabled at (3723586218): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2853.827773] softirqs last disabled at (3723586216): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2853.854515] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2853.879400] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2853.902244] RIP: 0010:debug_check_no_locks_freed+0x8b/0x130
[ 2853.924148] Code: 48 8b 53 10 4c 8d 42 20 4c 39 c5 73 ea 49 39 d6 76 e5 e8 c8 e8 37 00 85 c0 74 0a 8b 05 6e a3 5a 01 85 c0 74 13 4c 89 ef 57 9d <66> 66 90 66 90 5b 5d 41 5c 41 5d 41 5e c3 48 c7 c7 81 e7 8f 97 31
[ 2853.977295] RSP: 0018:ffffa8a885bafcf8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 2854.002460] RAX: 0000000000000000 RBX: ffffa8a885bafd80 RCX: 0000000000000000
[ 2854.027277] RDX: 0000000000000001 RSI: 0000000000000038 RDI: 0000000000000202
[ 2854.052048] RBP: ffffffff978fe6b2 R08: 0000000000000000 R09: 0000000000000000
[ 2854.076833] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffff94d550e60000
[ 2854.101497] R13: 0000000000000202 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2854.126054] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2854.151718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2854.175043] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2854.199973] Call Trace:
[ 2854.220107] __raw_spin_lock_init+0x1c/0x50
[ 2854.241925] __init_waitqueue_head+0x25/0x50
[ 2854.263845] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 2854.288012] ? lockdep_hardirqs_on+0x122/0x1b0
[ 2854.310080] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2854.333556] ? __local_bh_enable_ip+0x80/0xf0
[ 2854.355254] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2854.378481] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 2854.401021] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2854.424320] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2854.448870] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2854.471942] ? remove_wait_queue+0x60/0x60
[ 2854.492722] kthread+0x103/0x140
[ 2854.512405] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2854.535347] ? kthread_bind+0x10/0x10
[ 2854.555319] ret_from_fork+0x3a/0x50
[ 2881.515691] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2881.539356] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2881.762218] irq event stamp: 4156025386
[ 2881.782723] hardirqs last enabled at (4156025385): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2881.808946] hardirqs last disabled at (4156025386): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2881.835380] softirqs last enabled at (4156025384): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2881.863013] softirqs last disabled at (4156025382): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2881.890451] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2881.915794] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2881.939168] RIP: 0010:target_put_sess_cmd+0x19/0x50 [target_core_mod]
[ 2881.962114] Code: d6 e9 aa fe ff ff 90 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8d 97 40 01 00 00 b8 ff ff ff ff f0 0f c1 87 40 01 00 00 <83> f8 01 74 07 85 c0 7e 11 31 c0 c3 48 89 d7 e8 23 fb ff ff b8 01
[ 2882.014881] RSP: 0018:ffffa8a885bafe08 EFLAGS: 00000287 ORIG_RAX: ffffffffffffff13
[ 2882.039293] RAX: 00000000c0000000 RBX: ffff94d4b4bb8000 RCX: 0000000000000000
[ 2882.063024] RDX: ffff94d4b4533ae0 RSI: 0000000000000006 RDI: ffff94d4b45339a0
[ 2882.086629] RBP: ffffa8a885bafe38 R08: 0000000000000000 R09: 0000000000000000
[ 2882.110071] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffff94d4b4bbb800
[ 2882.133642] R13: ffff94d4b4533690 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 2882.157274] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2882.182054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2882.204501] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2882.228393] Call Trace:
[ 2882.247329] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2882.270281] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2882.294465] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2882.317415] ? remove_wait_queue+0x60/0x60
[ 2882.338311] kthread+0x103/0x140
[ 2882.358180] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2882.381425] ? kthread_bind+0x10/0x10
[ 2882.401734] ret_from_fork+0x3a/0x50
[ 2909.514933] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2909.539045] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2909.771408] irq event stamp: 295797379
[ 2909.794521] hardirqs last enabled at (295797377): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2909.823565] hardirqs last disabled at (295797379): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2909.852655] softirqs last enabled at (295797376): [<ffffffffc0e6c0c1>] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2909.882781] softirqs last disabled at (295797378): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2909.912493] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2909.940074] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2909.965188] RIP: 0010:lock_acquire+0x39/0x1c0
[ 2909.987643] Code: 25 40 8e 01 00 55 53 48 83 ec 30 8b 82 bc 0c 00 00 48 89 7c 24 20 85 c0 0f 85 ad 00 00 00 89 f5 41 89 cd 45 89 c6 4d 89 cf 9c <58> 66 66 90 66 90 48 89 44 24 28 fa 66 66 90 66 66 90 c7 82 bc 0c
[ 2910.043107] RSP: 0018:ffffa8a885bafd20 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2910.068945] RAX: 0000000000000000 RBX: ffff94d4b4bb85e0 RCX: 0000000000000000
[ 2910.094216] RDX: ffff94d550e60000 RSI: 0000000000000000 RDI: ffff94d4b4bb85f8
[ 2910.119505] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 2910.144654] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2910.169846] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2910.194648] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2910.220398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2910.243699] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2910.268228] Call Trace:
[ 2910.287544] ? __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2910.310383] ? _raw_spin_lock_bh+0x3d/0x50
[ 2910.331250] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2910.353918] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 2910.376401] ? iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2910.398580] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2910.421695] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2910.446028] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2910.468975] ? remove_wait_queue+0x60/0x60
[ 2910.489517] ? kthread+0x103/0x140
[ 2910.509169] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2910.521946] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 0-... } 68611 jiffies s: 121 root: 0x1/.
[ 2910.531970] ? kthread_bind+0x10/0x10
[ 2910.531974] ? ret_from_fork+0x3a/0x50
[ 2910.550727] rcu: blocking rcu_node structures:
[ 2910.555418] Task dump for CPU 0:
[ 2910.558873] iscsi_ttx R running task 0 6148 2 0x8000408c
[ 2910.566162] Call Trace:
[ 2910.568863] ? mark_held_locks+0x55/0x70
[ 2910.573041] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 2910.577897] ? lockdep_hardirqs_on+0x6e/0x1b0
[ 2910.582488] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 2910.587337] ? __lock_acquire+0x24b/0x1080
[ 2910.591665] ? module_assert_mutex_or_preempt+0x14/0x40
[ 2910.597142] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2910.603889] ? find_held_lock+0x2d/0x90
[ 2910.607955] ? __local_bh_enable_ip+0x47/0xf0
[ 2910.612527] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2910.618660] ? iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 2910.624450] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2910.631119] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2910.639028] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2910.645601] ? remove_wait_queue+0x60/0x60
[ 2910.649929] ? kthread+0x103/0x140
[ 2910.653532] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2910.660179] ? kthread_bind+0x10/0x10
[ 2910.664045] ? ret_from_fork+0x3a/0x50
[ 2937.514176] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 2937.533644] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2937.729551] irq event stamp: 729029174
[ 2937.747906] hardirqs last enabled at (729029173): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 2937.772256] hardirqs last disabled at (729029174): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2937.796993] softirqs last enabled at (729029172): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2937.823085] softirqs last disabled at (729029170): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 2937.849242] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2937.873475] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2937.895579] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 2937.915934] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 2937.966667] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 2937.990513] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 2938.013980] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 2938.037101] RBP: ffffffffc0e6c0f1 R08: 0000000000000000 R09: 0000000000000000
[ 2938.059861] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 2938.082723] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb8650 R15: ffff94d4b4533690
[ 2938.105641] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2938.129734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2938.151421] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2938.174605] Call Trace:
[ 2938.193048] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2938.215025] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2938.236702] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2938.259337] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2938.283351] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2938.306153] ? remove_wait_queue+0x60/0x60
[ 2938.326766] kthread+0x103/0x140
[ 2938.346335] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2938.369383] ? kthread_bind+0x10/0x10
[ 2938.389533] ret_from_fork+0x3a/0x50
[ 2951.480857] INFO: task iscsi_np:4183 blocked for more than 614 seconds.
[ 2951.487750] Tainted: G W L 5.5.0 #34
[ 2951.493224] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2951.501302] iscsi_np D 0 4183 2 0x80004084
[ 2951.507033] Call Trace:
[ 2951.509725] ? __schedule+0x38c/0x990
[ 2951.513631] schedule+0x5f/0xd0
[ 2951.517015] schedule_timeout+0x1e4/0x310
[ 2951.521265] ? find_held_lock+0x2d/0x90
[ 2951.525342] ? wait_for_completion+0x126/0x190
[ 2951.530025] wait_for_completion+0x12e/0x190
[ 2951.534521] ? wake_up_q+0xa0/0xa0
[ 2951.538164] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 2951.545890] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 2951.552132] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 2951.559073] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 2951.565835] kthread+0x103/0x140
[ 2951.569284] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 2951.576293] ? kthread_bind+0x10/0x10
[ 2951.580159] ret_from_fork+0x3a/0x50
[ 2951.583961] INFO: task sosreport:6414 blocked for more than 368 seconds.
[ 2951.590919] Tainted: G W L 5.5.0 #34
[ 2951.596369] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2951.604421] sosreport D 0 6414 6413 0x00000080
[ 2951.610121] Call Trace:
[ 2951.612785] ? __schedule+0x38c/0x990
[ 2951.616657] schedule+0x5f/0xd0
[ 2951.619995] schedule_timeout+0x1e4/0x310
[ 2951.624187] ? find_held_lock+0x2d/0x90
[ 2951.628218] ? mark_held_locks+0x55/0x70
[ 2951.632334] wait_for_completion+0x12e/0x190
[ 2951.636784] ? wake_up_q+0xa0/0xa0
[ 2951.640368] __wait_rcu_gp+0xfd/0x140
[ 2951.644210] synchronize_rcu.part.81+0x3e/0x70
[ 2951.648836] ? __call_rcu+0x590/0x590
[ 2951.652674] ? __bpf_trace_rcu_utilization+0x10/0x10
[ 2951.657827] rcu_sync_enter+0x54/0xf0
[ 2951.661669] ? rcu_read_lock_sched_held+0x52/0x90
[ 2951.666547] percpu_down_write+0x15/0xe0
[ 2951.670632] ? seq_read+0x327/0x3e0
[ 2951.674287] locks_start+0x28/0x50
[ 2951.677854] seq_read+0x1da/0x3e0
[ 2951.681337] proc_reg_read+0x3c/0x70
[ 2951.685073] vfs_read+0xac/0x160
[ 2951.688462] ksys_read+0x5c/0xd0
[ 2951.691862] do_syscall_64+0x60/0x270
[ 2951.695675] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 2951.700899] RIP: 0033:0x7fc60f980c00
[ 2951.704629] Code: Bad RIP value.
[ 2951.708011] RSP: 002b:00007fff4e546148 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 2951.715759] RAX: ffffffffffffffda RBX: 000000000128c9e0 RCX: 00007fc60f980c00
[ 2951.723069] RDX: 0000000000004000 RSI: 0000000001290494 RDI: 000000000000000b
[ 2951.730368] RBP: 00007fc60fc543a0 R08: ffffffffffffffff R09: 0000000000000000
[ 2951.737673] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 2951.744980] R13: 0000000000000d70 R14: 00007fc60fc53858 R15: 0000000001290494
[ 2951.752292]
[ 2951.752292] Showing all locks held in the system:
[ 2951.758791] 1 lock held by khungtaskd/55:
[ 2951.762975] #0: ffffffff97c85280 (rcu_read_lock){....}, at: debug_show_all_locks+0xe/0x1a0
[ 2951.771498] 1 lock held by systemd-logind/1474:
[ 2951.776189] #0: ffffffff97d0fd78 (rcu_state.exp_mutex){+.+.}, at: synchronize_rcu_expedited+0x23c/0x3f0
[ 2951.785869] 2 locks held by agetty/1628:
[ 2951.789971] #0: ffff94d5628d6090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2951.798496] #1: ffffa8a88101b2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2951.807204] 2 locks held by agetty/1631:
[ 2951.811295] #0: ffff94d56ab95090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2951.819819] #1: ffffa8a8811fb2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2951.828527] 2 locks held by bash/3758:
[ 2951.832452] #0: ffff94d569f8f890 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 2951.840976] #1: ffffa8a8802872e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 2951.849689] 2 locks held by kworker/1:1/5887:
[ 2951.854232] #0: ffff94d2c7c35728 ((wq_completion)rcu_gp){+.+.}, at: process_one_work+0x17a/0x570
[ 2951.863276] #1: ffffa8a8850d7e60 ((work_completion)(&rew.rew_work)){+.+.}, at: process_one_work+0x17a/0x570
[ 2951.873276] 1 lock held by sosreport/6414:
[ 2951.877552] #0: ffff94d40ee920a0 (&p->lock){+.+.}, at: seq_read+0x3c/0x3e0
[ 2951.884689]
[ 2951.886355] =============================================
[ 2951.886355]
[ 2977.513093] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 2977.533563] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 2977.739918] irq event stamp: 1356505063
[ 2977.760112] hardirqs last enabled at (1356505061): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 2977.787122] hardirqs last disabled at (1356505063): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 2977.813962] softirqs last enabled at (1356505058): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 2977.842312] softirqs last disabled at (1356505062): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2977.888814] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2977.915689] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2977.940434] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 2977.962944] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 2978.019180] RSP: 0018:ffffa8a885bafd10 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2978.045730] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 2978.071901] RDX: 0000000000000001 RSI: 0000000096612835 RDI: 0000000000000246
[ 2978.098051] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 2978.124123] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2978.150051] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2978.175903] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2978.202852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2978.227289] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2978.253138] Call Trace:
[ 2978.273933] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2978.298755] _raw_spin_lock_bh+0x3d/0x50
[ 2978.320858] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2978.345542] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 2978.369924] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 2978.393638] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2978.416891] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2978.440942] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2978.466170] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2978.489985] ? remove_wait_queue+0x60/0x60
[ 2978.511510] kthread+0x103/0x140
[ 2978.531893] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2978.555497] ? kthread_bind+0x10/0x10
[ 2978.576006] ret_from_fork+0x3a/0x50
[ 2982.305964] rcu: INFO: rcu_sched self-detected stall on CPU
[ 2982.328258] rcu: 0-....: (622564 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=161290
[ 2982.355402] (t=650032 jiffies g=82081 q=85360)
[ 2982.377079] NMI backtrace for cpu 0
[ 2982.397472] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2982.423299] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2982.446609] Call Trace:
[ 2982.465290] <IRQ>
[ 2982.483045] dump_stack+0x82/0xb3
[ 2982.501752] nmi_cpu_backtrace+0x90/0xa0
[ 2982.520720] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2982.539848] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2982.559213] rcu_dump_cpu_stacks+0x97/0xc5
[ 2982.577376] rcu_sched_clock_irq+0x6ec/0x980
[ 2982.595412] ? rcu_read_lock_sched_held+0x52/0x90
[ 2982.613813] ? tick_sched_do_timer+0x70/0x70
[ 2982.631654] update_process_times+0x24/0x50
[ 2982.649261] tick_sched_handle+0x25/0x60
[ 2982.666188] tick_sched_timer+0x37/0x70
[ 2982.682719] __hrtimer_run_queues+0xe8/0x450
[ 2982.699525] hrtimer_interrupt+0x122/0x270
[ 2982.716030] smp_apic_timer_interrupt+0x86/0x240
[ 2982.732729] apic_timer_interrupt+0xf/0x20
[ 2982.748467] </IRQ>
[ 2982.761898] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 2982.777594] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 2982.820747] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2982.840914] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 2982.860765] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 2982.880562] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 2982.900142] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2982.919527] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2982.938742] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2982.957349] _raw_spin_lock_bh+0x3d/0x50
[ 2982.973249] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2982.991858] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2983.010299] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2983.028117] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2983.045561] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2983.063922] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2983.083542] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2983.101783] ? remove_wait_queue+0x60/0x60
[ 2983.117854] kthread+0x103/0x140
[ 2983.133009] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2983.151527] ? kthread_bind+0x10/0x10
[ 2983.167128] ret_from_fork+0x3a/0x50
[ 2983.182624] NMI backtrace for cpu 0
[ 2983.182624] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 2983.182625] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 2983.182625] RIP: 0010:cfb_imageblit+0x4ba/0x4f0
[ 2983.182626] Code: 49 89 ed b9 08 00 00 00 89 04 24 eb 2e 41 0f be 55 00 44 29 c1 4d 8d 74 24 04 d3 fa 44 21 d2 41 8b 04 91 44 21 d8 89 c2 31 fa <41> 89 14 24 85 c9 75 06 49 83 c5 01 b1 08 4d 89 f4 83 ee 01 83 fe
[ 2983.182627] RSP: 0018:ffffa8a880003868 EFLAGS: 00000046
[ 2983.182627] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000003
[ 2983.182628] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000000000
[ 2983.182629] RBP: ffff94d56c20ae3c R08: 0000000000000001 R09: ffffffff976bd6e0
[ 2983.182629] R10: 0000000000000001 R11: 0000000000aaaaaa R12: ffffa8a881dcc890
[ 2983.182630] R13: ffff94d56c20ae57 R14: ffffa8a881dcc894 R15: ffffa8a881dcc520
[ 2983.182630] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 2983.182631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2983.182631] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 2983.182631] Call Trace:
[ 2983.182632] <IRQ>
[ 2983.182632] ? _raw_spin_unlock_irqrestore+0x50/0x52
[ 2983.182633] drm_fb_helper_cfb_imageblit+0x12/0x30 [drm_kms_helper]
[ 2983.182633] bit_putcs+0x2ec/0x500
[ 2983.182634] ? find_held_lock+0x2d/0x90
[ 2983.182634] ? soft_cursor+0x1af/0x220
[ 2983.182635] fbcon_putcs+0x11c/0x130
[ 2983.182635] ? update_attr.isra.2+0x80/0x80
[ 2983.182635] fbcon_redraw.isra.21+0x17b/0x1f0
[ 2983.182636] fbcon_scroll+0x4a5/0xda0
[ 2983.182636] con_scroll+0x1e0/0x200
[ 2983.182637] lf+0x9e/0xb0
[ 2983.182637] vt_console_print+0x248/0x3b0
[ 2983.182638] console_unlock+0x4cf/0x5e0
[ 2983.182638] ? console_unlock+0x1b2/0x5e0
[ 2983.182639] ? __down_trylock_console_sem+0x7d/0xa0
[ 2983.182639] vprintk_emit+0x13b/0x380
[ 2983.182639] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2983.182640] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2983.182640] printk+0x58/0x6f
[ 2983.182641] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2983.182641] show_trace_log_lvl+0x19b/0x400
[ 2983.182642] ? nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2983.182642] dump_stack+0x82/0xb3
[ 2983.182643] nmi_cpu_backtrace+0x90/0xa0
[ 2983.182643] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 2983.182644] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 2983.182644] rcu_dump_cpu_stacks+0x97/0xc5
[ 2983.182644] rcu_sched_clock_irq+0x6ec/0x980
[ 2983.182645] ? rcu_read_lock_sched_held+0x52/0x90
[ 2983.182645] ? tick_sched_do_timer+0x70/0x70
[ 2983.182646] update_process_times+0x24/0x50
[ 2983.182646] tick_sched_handle+0x25/0x60
[ 2983.182647] tick_sched_timer+0x37/0x70
[ 2983.182647] __hrtimer_run_queues+0xe8/0x450
[ 2983.182648] hrtimer_interrupt+0x122/0x270
[ 2983.182648] smp_apic_timer_interrupt+0x86/0x240
[ 2983.182649] apic_timer_interrupt+0xf/0x20
[ 2983.182649] </IRQ>
[ 2983.182649] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 2983.182650] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 2983.182651] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2983.182652] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 2983.182652] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 2983.182653] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 2983.182653] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 2983.182654] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 2983.182655] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2983.182655] _raw_spin_lock_bh+0x3d/0x50
[ 2983.182656] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2983.182656] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 2983.182656] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 2983.182657] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 2983.182658] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 2983.182658] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 2983.182659] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 2983.182659] ? remove_wait_queue+0x60/0x60
[ 2983.182660] kthread+0x103/0x140
[ 2983.182660] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 2983.182661] ? kthread_bind+0x10/0x10
[ 2983.182661] ret_from_fork+0x3a/0x50
[ 3009.512227] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3009.529702] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3009.713592] irq event stamp: 1819586670
[ 3009.731035] hardirqs last enabled at (1819586669): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3009.755312] hardirqs last disabled at (1819586670): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3009.779378] softirqs last enabled at (1819586666): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3009.804879] softirqs last disabled at (1819586664): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3009.830336] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3009.853855] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3009.875278] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 3009.895780] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 65 96 78 ff 48 89 ef e8 9d e2 78 ff f6 c7 02 74 19 e8 73 a0 84 ff 48 89 df 57 9d <66> 66 90 66 90 65 ff 0d 4a e5 e8 68 5b 5d c3 48 89 df 57 9d 66 66
[ 3009.946113] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 3009.969869] RAX: ffff94d550e60000 RBX: 0000000000000296 RCX: 0000000000000000
[ 3009.993336] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: 0000000000000296
[ 3010.016721] RBP: ffff94d4b4533aa8 R08: 0000000000000000 R09: 0000000000000000
[ 3010.040048] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3010.063482] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3010.086943] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3010.111171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3010.132704] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3010.155629] Call Trace:
[ 3010.173683] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 3010.196051] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3010.217482] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3010.239978] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3010.263941] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3010.286549] ? remove_wait_queue+0x60/0x60
[ 3010.306997] kthread+0x103/0x140
[ 3010.326438] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3010.349223] ? kthread_bind+0x10/0x10
[ 3010.369120] ret_from_fork+0x3a/0x50
[ 3037.511470] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3037.535147] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3037.764796] irq event stamp: 2255356306
[ 3037.787305] hardirqs last enabled at (2255356305): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3037.815503] hardirqs last disabled at (2255356306): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3037.843747] softirqs last enabled at (2255356304): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 3037.873056] softirqs last disabled at (2255356302): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3037.901958] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3037.928650] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3037.952853] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3037.975033] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3038.029057] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 3038.054366] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3038.079213] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3038.104073] RBP: ffffffffc0e6c0fe R08: 0000000000000000 R09: 0000000000000000
[ 3038.128561] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3038.152922] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb85e0 R15: ffff94d4b4533690
[ 3038.177274] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3038.202426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3038.224818] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3038.248618] Call Trace:
[ 3038.267517] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 3038.290038] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3038.312258] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3038.335417] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3038.359997] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3038.383203] ? remove_wait_queue+0x60/0x60
[ 3038.404153] kthread+0x103/0x140
[ 3038.424028] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3038.447230] ? kthread_bind+0x10/0x10
[ 3038.467515] ret_from_fork+0x3a/0x50
[ 3065.510713] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3065.534687] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3065.765978] irq event stamp: 2688999525
[ 3065.788643] hardirqs last enabled at (2688999523): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3065.817008] hardirqs last disabled at (2688999525): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3065.845400] softirqs last enabled at (2688999522): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 3065.874890] softirqs last disabled at (2688999524): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3065.903929] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3065.930675] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3065.954887] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3065.976483] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3066.030700] RSP: 0018:ffffa8a885bafd28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3066.056119] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3066.081010] RDX: 0000000000000001 RSI: 000000003d7869c8 RDI: 0000000000000246
[ 3066.105892] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3066.130339] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3066.154602] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3066.178724] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3066.203545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3066.225684] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3066.249223] Call Trace:
[ 3066.267835] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3066.290194] _raw_spin_lock_bh+0x3d/0x50
[ 3066.310542] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3066.333028] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3066.355322] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3066.377134] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3066.399781] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3066.423692] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3066.446246] ? remove_wait_queue+0x60/0x60
[ 3066.466641] kthread+0x103/0x140
[ 3066.486025] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3066.508818] ? kthread_bind+0x10/0x10
[ 3066.528683] ret_from_fork+0x3a/0x50
[ 3074.357521] INFO: task systemd-logind:1474 blocked for more than 122 seconds.
[ 3074.364933] Tainted: G W L 5.5.0 #34
[ 3074.370393] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3074.378444] systemd-logind D 0 1474 1 0x00000084
[ 3074.384173] Call Trace:
[ 3074.386868] ? __schedule+0x38c/0x990
[ 3074.390757] schedule+0x5f/0xd0
[ 3074.394126] synchronize_rcu_expedited+0x3b0/0x3f0
[ 3074.399162] ? remove_wait_queue+0x60/0x60
[ 3074.403501] ? cond_synchronize_rcu+0x20/0x20
[ 3074.408101] namespace_unlock+0xc8/0x160
[ 3074.412248] ksys_umount+0x2b9/0x530
[ 3074.416036] ? __x64_sys_umount+0x12/0x20
[ 3074.420258] ? syscall_trace_enter+0x1f8/0x350
[ 3074.424908] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3074.429740] __x64_sys_umount+0x12/0x20
[ 3074.433776] do_syscall_64+0x60/0x270
[ 3074.437670] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 3074.442913] RIP: 0033:0x7f41dceedbd7
[ 3074.446691] Code: Bad RIP value.
[ 3074.450109] RSP: 002b:00007ffd0d625958 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[ 3074.457905] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f41dceedbd7
[ 3074.465254] RDX: 000000000000d3f0 RSI: 0000000000000002 RDI: 000055ccf3314cb0
[ 3074.472607] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000008030
[ 3074.479925] R10: 0000000000000076 R11: 0000000000000246 R12: 000055ccf3312890
[ 3074.487241] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
[ 3074.494582] INFO: task iscsi_np:4183 blocked for more than 737 seconds.
[ 3074.501365] Tainted: G W L 5.5.0 #34
[ 3074.506792] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3074.514800] iscsi_np D 0 4183 2 0x80004084
[ 3074.520469] Call Trace:
[ 3074.523100] ? __schedule+0x38c/0x990
[ 3074.526932] schedule+0x5f/0xd0
[ 3074.530246] schedule_timeout+0x1e4/0x310
[ 3074.534425] ? find_held_lock+0x2d/0x90
[ 3074.538433] ? wait_for_completion+0x126/0x190
[ 3074.543058] wait_for_completion+0x12e/0x190
[ 3074.547504] ? wake_up_q+0xa0/0xa0
[ 3074.551108] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 3074.558763] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 3074.564967] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 3074.571843] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 3074.578570] kthread+0x103/0x140
[ 3074.581966] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 3074.588920] ? kthread_bind+0x10/0x10
[ 3074.592757] ret_from_fork+0x3a/0x50
[ 3074.596535]
[ 3074.596535] Showing all locks held in the system:
[ 3074.603020] 1 lock held by khungtaskd/55:
[ 3074.607202] #0: ffffffff97c85280 (rcu_read_lock){....}, at: debug_show_all_locks+0xe/0x1a0
[ 3074.615739] 1 lock held by systemd-logind/1474:
[ 3074.620428] #0: ffffffff97d0fd78 (rcu_state.exp_mutex){+.+.}, at: synchronize_rcu_expedited+0x23c/0x3f0
[ 3074.630091] 2 locks held by agetty/1628:
[ 3074.634179] #0: ffff94d5628d6090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 3074.642704] #1: ffffa8a88101b2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 3074.651400] 2 locks held by agetty/1631:
[ 3074.655495] #0: ffff94d56ab95090 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 3074.664012] #1: ffffa8a8811fb2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 3074.672728] 2 locks held by bash/3758:
[ 3074.676649] #0: ffff94d569f8f890 (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x24/0x50
[ 3074.685179] #1: ffffa8a8802872e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0xd8/0x970
[ 3074.693871] 2 locks held by kworker/1:1/5887:
[ 3074.698397] #0: ffff94d2c7c35728 ((wq_completion)rcu_gp){+.+.}, at: process_one_work+0x17a/0x570
[ 3074.707452] #1: ffffa8a8850d7e60 ((work_completion)(&rew.rew_work)){+.+.}, at: process_one_work+0x17a/0x570
[ 3074.717505] 1 lock held by sosreport/6414:
[ 3074.721779] #0: ffff94d40ee920a0 (&p->lock){+.+.}, at: seq_read+0x3c/0x3e0
[ 3074.728924]
[ 3074.730599] =============================================
[ 3074.730599]
[ 3101.509738] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3101.531023] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3101.742524] irq event stamp: 3250415979
[ 3101.762991] hardirqs last enabled at (3250415977): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3101.789556] hardirqs last disabled at (3250415979): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3101.816623] softirqs last enabled at (3250415976): [<ffffffffc0e6c0c1>] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3101.845177] softirqs last disabled at (3250415978): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3101.873688] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3101.900352] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3101.924916] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3101.947097] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3102.002792] RSP: 0018:ffffa8a885bafd28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3102.029056] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3102.054866] RDX: 0000000000000001 RSI: 0000000002c1e137 RDI: 0000000000000246
[ 3102.080546] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3102.106079] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3102.131596] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3102.157129] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3102.183761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3102.208045] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3102.233733] Call Trace:
[ 3102.254449] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3102.278562] _raw_spin_lock_bh+0x3d/0x50
[ 3102.300460] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3102.324219] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3102.347717] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3102.370809] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3102.394650] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3102.419699] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3102.443344] ? remove_wait_queue+0x60/0x60
[ 3102.464693] kthread+0x103/0x140
[ 3102.484956] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3102.508432] ? kthread_bind+0x10/0x10
[ 3102.528864] ret_from_fork+0x3a/0x50
[ 3107.124609] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 0-... } 265219 jiffies s: 121 root: 0x1/.
[ 3107.135520] rcu: blocking rcu_node structures:
[ 3107.140205] Task dump for CPU 0:
[ 3107.143678] iscsi_ttx R running task 0 6148 2 0x8000408c
[ 3107.150975] Call Trace:
[ 3107.153684] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3107.158545] ? mark_held_locks+0x55/0x70
[ 3107.162714] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3107.167575] ? __lock_acquire+0x24b/0x1080
[ 3107.171922] ? find_held_lock+0x2d/0x90
[ 3107.176020] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3107.182794] ? _raw_spin_lock_bh+0x3d/0x50
[ 3107.187109] ? _raw_spin_lock_bh+0x3d/0x50
[ 3107.191428] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3107.197564] ? iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3107.203355] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3107.210015] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3107.217915] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3107.224474] ? remove_wait_queue+0x60/0x60
[ 3107.228787] ? kthread+0x103/0x140
[ 3107.232417] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3107.239063] ? kthread_bind+0x10/0x10
[ 3107.242935] ? ret_from_fork+0x3a/0x50
[ 3129.508980] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3129.528620] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3129.726050] irq event stamp: 3683115933
[ 3129.744788] hardirqs last enabled at (3683115931): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3129.769477] hardirqs last disabled at (3683115933): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3129.794515] softirqs last enabled at (3683115930): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 3129.820911] softirqs last disabled at (3683115932): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3129.847186] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3129.871463] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3129.893562] RIP: 0010:lock_acquire+0x39/0x1c0
[ 3129.913333] Code: 25 40 8e 01 00 55 53 48 83 ec 30 8b 82 bc 0c 00 00 48 89 7c 24 20 85 c0 0f 85 ad 00 00 00 89 f5 41 89 cd 45 89 c6 4d 89 cf 9c <58> 66 66 90 66 90 48 89 44 24 28 fa 66 66 90 66 66 90 c7 82 bc 0c
[ 3129.964241] RSP: 0018:ffffa8a885bafd20 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3129.988044] RAX: 0000000000000000 RBX: ffff94d4b4bb8650 RCX: 0000000000000000
[ 3130.011539] RDX: ffff94d550e60000 RSI: 0000000000000000 RDI: ffff94d4b4bb8668
[ 3130.035024] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 3130.058432] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3130.081840] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3130.104880] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3130.128717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3130.150226] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3130.173189] Call Trace:
[ 3130.191319] ? __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 3130.213140] ? _raw_spin_lock_bh+0x3d/0x50
[ 3130.233119] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3130.255052] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3130.276845] ? iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3130.298389] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3130.320818] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3130.344536] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3130.367162] ? remove_wait_queue+0x60/0x60
[ 3130.387486] ? kthread+0x103/0x140
[ 3130.406973] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3130.429649] ? kthread_bind+0x10/0x10
[ 3130.449410] ? ret_from_fork+0x3a/0x50
[ 3157.508222] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3157.531637] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3157.758001] irq event stamp: 4117013998
[ 3157.779331] hardirqs last enabled at (4117013997): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3157.806359] hardirqs last disabled at (4117013998): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3157.833447] softirqs last enabled at (4117013996): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3157.861595] softirqs last disabled at (4117013994): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3157.889273] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3157.914755] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3157.937904] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3157.959396] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3158.012486] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 3158.037211] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3158.061478] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3158.085589] RBP: ffffffffc0e6c0f1 R08: 0000000000000000 R09: 0000000000000000
[ 3158.109296] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3158.132632] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb8650 R15: ffff94d4b4533690
[ 3158.155826] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3158.180079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3158.201927] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3158.225361] Call Trace:
[ 3158.244050] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3158.266309] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3158.288190] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3158.310875] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3158.334846] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3158.357457] ? remove_wait_queue+0x60/0x60
[ 3158.377904] kthread+0x103/0x140
[ 3158.397345] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3158.420198] ? kthread_bind+0x10/0x10
[ 3158.440377] ret_from_fork+0x3a/0x50
[ 3177.303687] rcu: INFO: rcu_sched self-detected stall on CPU
[ 3177.325783] rcu: 0-....: (809615 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=209828
[ 3177.352616] (t=845035 jiffies g=82081 q=94981)
[ 3177.373899] NMI backtrace for cpu 0
[ 3177.393995] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3177.419859] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3177.443527] Call Trace:
[ 3177.462836] <IRQ>
[ 3177.481654] dump_stack+0x82/0xb3
[ 3177.501595] nmi_cpu_backtrace+0x90/0xa0
[ 3177.521815] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 3177.542104] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 3177.562740] rcu_dump_cpu_stacks+0x97/0xc5
[ 3177.582024] rcu_sched_clock_irq+0x6ec/0x980
[ 3177.601142] ? rcu_read_lock_sched_held+0x52/0x90
[ 3177.620308] ? tick_sched_do_timer+0x70/0x70
[ 3177.638642] update_process_times+0x24/0x50
[ 3177.656479] tick_sched_handle+0x25/0x60
[ 3177.673823] tick_sched_timer+0x37/0x70
[ 3177.690756] __hrtimer_run_queues+0xe8/0x450
[ 3177.708082] hrtimer_interrupt+0x122/0x270
[ 3177.725071] smp_apic_timer_interrupt+0x86/0x240
[ 3177.742706] apic_timer_interrupt+0xf/0x20
[ 3177.759475] </IRQ>
[ 3177.773936] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 3177.791970] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 65 96 78 ff 48 89 ef e8 9d e2 78 ff f6 c7 02 74 19 e8 73 a0 84 ff 48 89 df 57 9d <66> 66 90 66 90 65 ff 0d 4a e5 e8 68 5b 5d c3 48 89 df 57 9d 66 66
[ 3177.836026] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 3177.856324] RAX: ffff94d550e60000 RBX: 0000000000000296 RCX: 0000000000000000
[ 3177.876000] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: 0000000000000296
[ 3177.895613] RBP: ffff94d4b4533aa8 R08: 0000000000000000 R09: 0000000000000000
[ 3177.915198] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3177.934808] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3177.954314] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 3177.973040] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3177.990613] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3178.009025] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3178.028727] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3178.047128] ? remove_wait_queue+0x60/0x60
[ 3178.063337] kthread+0x103/0x140
[ 3178.078594] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3178.097168] ? kthread_bind+0x10/0x10
[ 3178.112772] ret_from_fork+0x3a/0x50
[ 3178.128184] NMI backtrace for cpu 0
[ 3178.128185] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3178.128185] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3178.128186] RIP: 0010:cfb_imageblit+0x4ba/0x4f0
[ 3178.128187] Code: 49 89 ed b9 08 00 00 00 89 04 24 eb 2e 41 0f be 55 00 44 29 c1 4d 8d 74 24 04 d3 fa 44 21 d2 41 8b 04 91 44 21 d8 89 c2 31 fa <41> 89 14 24 85 c9 75 06 49 83 c5 01 b1 08 4d 89 f4 83 ee 01 83 fe
[ 3178.128187] RSP: 0018:ffffa8a880003808 EFLAGS: 00000046
[ 3178.128188] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000003
[ 3178.128189] RDX: 0000000000000000 RSI: 0000000000000053 RDI: 0000000000000000
[ 3178.128189] RBP: ffff94d56c20bdb4 R08: 0000000000000001 R09: ffffffff976bd6e0
[ 3178.128190] R10: 0000000000000001 R11: 0000000000aaaaaa R12: ffffa8a881cec390
[ 3178.128190] R13: ffff94d56c20bdc0 R14: ffffa8a881cec394 R15: ffffa8a881cec200
[ 3178.128191] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3178.128191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3178.128192] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3178.128192] Call Trace:
[ 3178.128192] <IRQ>
[ 3178.128193] ? _raw_spin_unlock_irqrestore+0x50/0x52
[ 3178.128193] drm_fb_helper_cfb_imageblit+0x12/0x30 [drm_kms_helper]
[ 3178.128194] bit_putcs+0x2ec/0x500
[ 3178.128194] ? find_held_lock+0x2d/0x90
[ 3178.128195] ? soft_cursor+0x1af/0x220
[ 3178.128195] fbcon_putcs+0x11c/0x130
[ 3178.128196] ? update_attr.isra.2+0x80/0x80
[ 3178.128196] fbcon_redraw.isra.21+0x17b/0x1f0
[ 3178.128197] fbcon_scroll+0x4a5/0xda0
[ 3178.128197] con_scroll+0x1e0/0x200
[ 3178.128197] lf+0x9e/0xb0
[ 3178.128198] vt_console_print+0x248/0x3b0
[ 3178.128198] console_unlock+0x4cf/0x5e0
[ 3178.128199] ? console_unlock+0x1b2/0x5e0
[ 3178.128199] ? __down_trylock_console_sem+0x7d/0xa0
[ 3178.128200] vprintk_emit+0x13b/0x380
[ 3178.128200] ? apic_timer_interrupt+0xf/0x20
[ 3178.128201] printk+0x58/0x6f
[ 3178.128201] __show_regs+0x39/0x2c0
[ 3178.128201] ? apic_timer_interrupt+0xf/0x20
[ 3178.128202] show_trace_log_lvl+0x16e/0x400
[ 3178.128202] ? _raw_spin_unlock_irqrestore+0x32/0x52
[ 3178.128203] dump_stack+0x82/0xb3
[ 3178.128203] nmi_cpu_backtrace+0x90/0xa0
[ 3178.128204] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 3178.128204] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 3178.128205] rcu_dump_cpu_stacks+0x97/0xc5
[ 3178.128205] rcu_sched_clock_irq+0x6ec/0x980
[ 3178.128206] ? rcu_read_lock_sched_held+0x52/0x90
[ 3178.128206] ? tick_sched_do_timer+0x70/0x70
[ 3178.128207] update_process_times+0x24/0x50
[ 3178.128207] tick_sched_handle+0x25/0x60
[ 3178.128208] tick_sched_timer+0x37/0x70
[ 3178.128208] __hrtimer_run_queues+0xe8/0x450
[ 3178.128209] hrtimer_interrupt+0x122/0x270
[ 3178.128209] smp_apic_timer_interrupt+0x86/0x240
[ 3178.128209] apic_timer_interrupt+0xf/0x20
[ 3178.128210] </IRQ>
[ 3178.128210] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 3178.128211] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 65 96 78 ff 48 89 ef e8 9d e2 78 ff f6 c7 02 74 19 e8 73 a0 84 ff 48 89 df 57 9d <66> 66 90 66 90 65 ff 0d 4a e5 e8 68 5b 5d c3 48 89 df 57 9d 66 66
[ 3178.128212] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 3178.128213] RAX: ffff94d550e60000 RBX: 0000000000000296 RCX: 0000000000000000
[ 3178.128213] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: 0000000000000296
[ 3178.128214] RBP: ffff94d4b4533aa8 R08: 0000000000000000 R09: 0000000000000000
[ 3178.128214] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3178.128215] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3178.128215] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 3178.128216] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3178.128216] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3178.128217] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3178.128217] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3178.128218] ? remove_wait_queue+0x60/0x60
[ 3178.128218] kthread+0x103/0x140
[ 3178.128219] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3178.128219] ? kthread_bind+0x10/0x10
[ 3178.128220] ret_from_fork+0x3a/0x50
[ 3201.507031] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [iscsi_ttx:6148]
[ 3201.524223] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3201.705727] irq event stamp: 481194410
[ 3201.722883] hardirqs last enabled at (481194409): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3201.746134] hardirqs last disabled at (481194410): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3201.769831] softirqs last enabled at (481194408): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3201.794935] softirqs last disabled at (481194406): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3201.835109] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3201.858852] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3201.880513] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3201.900602] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3201.951263] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 3201.975118] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3201.998695] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3202.022349] RBP: ffffffffc0e6c0b9 R08: 0000000000000000 R09: 0000000000000000
[ 3202.046040] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3202.069781] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3202.093601] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3202.118652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3202.141435] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3202.165742] Call Trace:
[ 3202.184947] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3202.207236] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3202.229047] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3202.251746] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3202.275928] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3202.298945] ? remove_wait_queue+0x60/0x60
[ 3202.319885] kthread+0x103/0x140
[ 3202.339932] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3202.363305] ? kthread_bind+0x10/0x10
[ 3202.383784] ret_from_fork+0x3a/0x50
[ 3229.506274] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3229.530453] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3229.764224] irq event stamp: 916387472
[ 3229.786869] hardirqs last enabled at (916387471): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3229.815232] hardirqs last disabled at (916387472): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3229.843633] softirqs last enabled at (916387470): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3229.873128] softirqs last disabled at (916387468): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3229.920749] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3229.947643] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3229.972201] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3229.994876] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3230.050339] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3230.076372] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3230.101696] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3230.126779] RBP: ffffffffc0e6c0b9 R08: 0000000000000000 R09: 0000000000000000
[ 3230.151675] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3230.176485] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3230.200980] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3230.226275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3230.249033] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3230.273268] Call Trace:
[ 3230.292758] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3230.315736] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3230.338407] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3230.361970] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3230.386756] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3230.410126] ? remove_wait_queue+0x60/0x60
[ 3230.431304] kthread+0x103/0x140
[ 3230.451410] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3230.474888] ? kthread_bind+0x10/0x10
[ 3230.495480] ret_from_fork+0x3a/0x50
[ 3257.505517] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3257.529778] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3257.763573] irq event stamp: 1349522865
[ 3257.786270] hardirqs last enabled at (1349522863): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3257.815376] hardirqs last disabled at (1349522865): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3257.843913] softirqs last enabled at (1349522860): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3257.873548] softirqs last disabled at (1349522864): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3257.921305] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3257.948213] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3257.972803] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3257.994873] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3258.050375] RSP: 0018:ffffa8a885bafd10 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3258.076408] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3258.101702] RDX: 0000000000000001 RSI: 0000000096612835 RDI: 0000000000000246
[ 3258.126797] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3258.151659] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3258.176370] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3258.200636] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3258.225632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3258.248095] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3258.272004] Call Trace:
[ 3258.291160] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3258.314629] _raw_spin_lock_bh+0x3d/0x50
[ 3258.335522] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3258.359089] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3258.382377] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3258.405045] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3258.427307] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3258.450394] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3258.474842] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3258.497948] ? remove_wait_queue+0x60/0x60
[ 3258.518895] kthread+0x103/0x140
[ 3258.538819] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3258.562065] ? kthread_bind+0x10/0x10
[ 3258.582373] ret_from_fork+0x3a/0x50
[ 3285.504759] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3285.528884] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3285.757374] irq event stamp: 1781625214
[ 3285.778727] hardirqs last enabled at (1781625213): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3285.805761] hardirqs last disabled at (1781625214): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3285.832826] softirqs last enabled at (1781625212): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3285.861082] softirqs last disabled at (1781625210): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3285.889010] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3285.914890] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3285.938526] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3285.960404] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3286.013596] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 3286.038381] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3286.062757] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3286.086730] RBP: ffffffffc0e6c0f1 R08: 0000000000000000 R09: 0000000000000000
[ 3286.110258] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3286.133664] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb8650 R15: ffff94d4b4533690
[ 3286.157028] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3286.181571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3286.203857] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3286.227782] Call Trace:
[ 3286.246993] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3286.269615] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3286.291805] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3286.314832] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3286.339095] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3286.362031] ? remove_wait_queue+0x60/0x60
[ 3286.382755] kthread+0x103/0x140
[ 3286.402482] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3286.425582] ? kthread_bind+0x10/0x10
[ 3286.445753] ret_from_fork+0x3a/0x50
[ 3303.727290] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 0-... } 461827 jiffies s: 121 root: 0x1/.
[ 3303.738187] rcu: blocking rcu_node structures:
[ 3303.742876] Task dump for CPU 0:
[ 3303.746352] iscsi_ttx R running task 0 6148 2 0x8000408c
[ 3303.753656] Call Trace:
[ 3303.756367] ? mark_held_locks+0x55/0x70
[ 3303.760539] ? retint_kernel+0x10/0x10
[ 3303.764537] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3303.769414] ? __lock_acquire+0x24b/0x1080
[ 3303.773748] ? __lock_acquire+0x24b/0x1080
[ 3303.778076] ? __lock_acquire+0x24b/0x1080
[ 3303.782415] ? is_module_address+0x11/0x20
[ 3303.786759] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3303.793518] ? find_held_lock+0x2d/0x90
[ 3303.797585] ? __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3303.803739] ? __local_bh_enable_ip+0x80/0xf0
[ 3303.808346] ? iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3303.814150] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3303.820812] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3303.828692] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3303.835289] ? remove_wait_queue+0x60/0x60
[ 3303.839597] ? kthread+0x103/0x140
[ 3303.843207] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3303.849871] ? kthread_bind+0x10/0x10
[ 3303.853762] ? ret_from_fork+0x3a/0x50
[ 3313.504002] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3313.523988] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3313.720481] irq event stamp: 2215613263
[ 3313.738952] hardirqs last enabled at (2215613261): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3313.763362] hardirqs last disabled at (2215613263): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3313.788049] softirqs last enabled at (2215613260): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3313.814109] softirqs last disabled at (2215613262): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3313.855823] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3313.880111] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3313.902163] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 3313.922130] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 d1 98 6f 69 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 b4 98 6f 69 89 43 08 65 48 8b 04 25
[ 3313.973224] RSP: 0018:ffffa8a885bafd70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3313.997154] RAX: 0000000000000000 RBX: ffff94d4b4533850 RCX: 0000000000000000
[ 3314.020808] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: ffff94d4b4533850
[ 3314.044137] RBP: ffff94d4b4533690 R08: 00000000fffffff0 R09: 0000000000000000
[ 3314.067216] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: ffff94d4b4533898
[ 3314.090383] R13: ffff94d4b4533850 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3314.113460] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3314.137690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3314.159583] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3314.183115] Call Trace:
[ 3314.201865] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3314.224822] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3314.247385] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3314.269815] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3314.293250] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3314.317956] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3314.341363] ? remove_wait_queue+0x60/0x60
[ 3314.362591] kthread+0x103/0x140
[ 3314.382846] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3314.406523] ? kthread_bind+0x10/0x10
[ 3314.427288] ret_from_fork+0x3a/0x50
[ 3341.503244] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3341.527627] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3341.762540] irq event stamp: 2650020504
[ 3341.785153] hardirqs last enabled at (2650020503): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3341.813461] hardirqs last disabled at (2650020504): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3341.841875] softirqs last enabled at (2650020502): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3341.871312] softirqs last disabled at (2650020500): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3341.900244] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3341.926868] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3341.951197] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3341.973672] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3342.028761] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 3342.054565] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3342.079708] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3342.104601] RBP: ffffffffc0e6c0f1 R08: 0000000000000000 R09: 0000000000000000
[ 3342.129263] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3342.153860] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb8650 R15: ffff94d4b4533690
[ 3342.178132] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3342.203206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3342.225769] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3342.249783] Call Trace:
[ 3342.268971] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3342.291648] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3342.313908] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3342.337149] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3342.361586] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3342.384625] ? remove_wait_queue+0x60/0x60
[ 3342.405447] kthread+0x103/0x140
[ 3342.425201] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3342.448311] ? kthread_bind+0x10/0x10
[ 3342.468481] ret_from_fork+0x3a/0x50
[ 3369.502486] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3369.526395] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3369.757023] irq event stamp: 3083651909
[ 3369.779574] hardirqs last enabled at (3083651907): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3369.808545] hardirqs last disabled at (3083651909): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3369.836939] softirqs last enabled at (3083651904): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3369.866420] softirqs last disabled at (3083651908): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3369.913927] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3369.940707] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3369.965002] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3369.986787] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3370.041546] RSP: 0018:ffffa8a885bafd10 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3370.067204] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3370.092664] RDX: 0000000000000001 RSI: 0000000096612835 RDI: 0000000000000246
[ 3370.117749] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3370.142614] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3370.167317] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3370.191599] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3370.216599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3370.239055] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3370.262938] Call Trace:
[ 3370.282082] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3370.305520] _raw_spin_lock_bh+0x3d/0x50
[ 3370.326393] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3370.349943] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3370.373222] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3370.395851] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3370.418064] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3370.441134] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3370.465490] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3370.488576] ? remove_wait_queue+0x60/0x60
[ 3370.509470] kthread+0x103/0x140
[ 3370.529344] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3370.552533] ? kthread_bind+0x10/0x10
[ 3370.572801] ret_from_fork+0x3a/0x50
[ 3372.301411] rcu: INFO: rcu_sched self-detected stall on CPU
[ 3372.323650] rcu: 0-....: (995636 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=258380
[ 3372.350698] (t=1040038 jiffies g=82081 q=103411)
[ 3372.372534] NMI backtrace for cpu 0
[ 3372.392941] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3372.418735] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3372.442040] Call Trace:
[ 3372.460725] <IRQ>
[ 3372.478497] dump_stack+0x82/0xb3
[ 3372.497211] nmi_cpu_backtrace+0x90/0xa0
[ 3372.516167] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 3372.535161] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 3372.554511] rcu_dump_cpu_stacks+0x97/0xc5
[ 3372.572740] rcu_sched_clock_irq+0x6ec/0x980
[ 3372.590767] ? rcu_read_lock_sched_held+0x52/0x90
[ 3372.609146] ? tick_sched_do_timer+0x70/0x70
[ 3372.626916] update_process_times+0x24/0x50
[ 3372.644444] tick_sched_handle+0x25/0x60
[ 3372.661305] tick_sched_timer+0x37/0x70
[ 3372.677795] __hrtimer_run_queues+0xe8/0x450
[ 3372.694569] hrtimer_interrupt+0x122/0x270
[ 3372.711003] smp_apic_timer_interrupt+0x86/0x240
[ 3372.727638] apic_timer_interrupt+0xf/0x20
[ 3372.743319] </IRQ>
[ 3372.756696] RIP: 0010:lock_is_held_type+0xb0/0xd0
[ 3372.772729] Code: b6 47 22 31 d2 83 e0 03 41 39 c6 0f 94 c2 eb 02 31 d2 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 5b 5d 41 5c 41 5d 41 5e 89 d0 41 5f c3 ba 01 00 00
[ 3372.815791] RSP: 0018:ffffa8a885bafca0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3372.835941] RAX: ffff94d550e60000 RBX: 0000000000000000 RCX: 0000000000000002
[ 3372.855777] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000000000246
[ 3372.875521] RBP: ffff94d550e60000 R08: 0000000000000000 R09: 0000000000000000
[ 3372.895117] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffffffff97c85240
[ 3372.914651] R13: 0000000000000246 R14: 00000000ffffffff R15: ffff94d4b4533690
[ 3372.934112] rcu_read_lock_sched_held+0x52/0x90
[ 3372.950834] ? cpumask_next+0x17/0x20
[ 3372.966531] module_assert_mutex_or_preempt+0x14/0x40
[ 3372.983648] __module_address+0x28/0xf0
[ 3372.999413] is_module_address+0x11/0x20
[ 3373.015157] static_obj+0x31/0x50
[ 3373.030173] lockdep_init_map+0x79/0x1f0
[ 3373.045750] __init_waitqueue_head+0x36/0x50
[ 3373.061575] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 3373.079536] ? lockdep_hardirqs_on+0x122/0x1b0
[ 3373.095512] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3373.112549] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3373.130469] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3373.149679] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3373.167704] ? remove_wait_queue+0x60/0x60
[ 3373.183536] kthread+0x103/0x140
[ 3373.198210] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3373.216044] ? kthread_bind+0x10/0x10
[ 3373.231050] ret_from_fork+0x3a/0x50
[ 3373.245748] NMI backtrace for cpu 0
[ 3373.245748] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3373.245749] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3373.245749] RIP: 0010:io_serial_in+0x14/0x20
[ 3373.245750] Code: 00 00 d3 e6 48 63 f6 48 03 77 40 8b 06 c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 0f b6 8f e9 00 00 00 d3 e6 03 77 38 89 f2 ec <0f> b6 c0 c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 0f b6 8f e9 00
[ 3373.245751] RSP: 0018:ffffa8a880003b48 EFLAGS: 00000002
[ 3373.245751] RAX: ffffffff96db9400 RBX: ffffffff9b537ee0 RCX: 0000000000000000
[ 3373.245752] RDX: 00000000000003fd RSI: 00000000000003fd RDI: ffffffff9b537ee0
[ 3373.245753] RBP: 0000000000002700 R08: 0000000000000000 R09: 0000000000000001
[ 3373.245753] R10: 0000000000000000 R11: ffffa8a880003aa0 R12: 0000000000000020
[ 3373.245754] R13: ffffffff9936498e R14: ffffffff99364960 R15: 000000000000002e
[ 3373.245754] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3373.245755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3373.245755] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3373.245755] Call Trace:
[ 3373.245756] <IRQ>
[ 3373.245756] wait_for_xmitr+0x40/0xb0
[ 3373.245757] serial8250_console_putchar+0x16/0x30
[ 3373.245757] ? wait_for_xmitr+0xb0/0xb0
[ 3373.245758] uart_console_write+0x2e/0x70
[ 3373.245758] serial8250_console_write+0xd7/0x2e0
[ 3373.245759] ? console_unlock+0x1b2/0x5e0
[ 3373.245759] console_unlock+0x4cf/0x5e0
[ 3373.245760] ? console_unlock+0x1b2/0x5e0
[ 3373.245760] ? __down_trylock_console_sem+0x7d/0xa0
[ 3373.245761] vprintk_emit+0x13b/0x380
[ 3373.245761] ? rcu_dump_cpu_stacks+0x97/0xc5
[ 3373.245762] ? rcu_dump_cpu_stacks+0x97/0xc5
[ 3373.245762] printk+0x58/0x6f
[ 3373.245762] ? rcu_dump_cpu_stacks+0x97/0xc5
[ 3373.245763] show_trace_log_lvl+0x257/0x400
[ 3373.245763] ? rcu_dump_cpu_stacks+0x97/0xc5
[ 3373.245764] dump_stack+0x82/0xb3
[ 3373.245764] nmi_cpu_backtrace+0x90/0xa0
[ 3373.245765] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 3373.245765] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 3373.245766] rcu_dump_cpu_stacks+0x97/0xc5
[ 3373.245766] rcu_sched_clock_irq+0x6ec/0x980
[ 3373.245767] ? rcu_read_lock_sched_held+0x52/0x90
[ 3373.245767] ? tick_sched_do_timer+0x70/0x70
[ 3373.245767] update_process_times+0x24/0x50
[ 3373.245768] tick_sched_handle+0x25/0x60
[ 3373.245768] tick_sched_timer+0x37/0x70
[ 3373.245769] __hrtimer_run_queues+0xe8/0x450
[ 3373.245769] hrtimer_interrupt+0x122/0x270
[ 3373.245770] smp_apic_timer_interrupt+0x86/0x240
[ 3373.245770] apic_timer_interrupt+0xf/0x20
[ 3373.245771] </IRQ>
[ 3373.245771] RIP: 0010:lock_is_held_type+0xb0/0xd0
[ 3373.245772] Code: b6 47 22 31 d2 83 e0 03 41 39 c6 0f 94 c2 eb 02 31 d2 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 5b 5d 41 5c 41 5d 41 5e 89 d0 41 5f c3 ba 01 00 00
[ 3373.245773] RSP: 0018:ffffa8a885bafca0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3373.245774] RAX: ffff94d550e60000 RBX: 0000000000000000 RCX: 0000000000000002
[ 3373.245774] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000000000246
[ 3373.245775] RBP: ffff94d550e60000 R08: 0000000000000000 R09: 0000000000000000
[ 3373.245775] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: ffffffff97c85240
[ 3373.245776] R13: 0000000000000246 R14: 00000000ffffffff R15: ffff94d4b4533690
[ 3373.245776] rcu_read_lock_sched_held+0x52/0x90
[ 3373.245776] ? cpumask_next+0x17/0x20
[ 3373.245777] module_assert_mutex_or_preempt+0x14/0x40
[ 3373.245777] __module_address+0x28/0xf0
[ 3373.245778] is_module_address+0x11/0x20
[ 3373.245778] static_obj+0x31/0x50
[ 3373.245779] lockdep_init_map+0x79/0x1f0
[ 3373.245779] __init_waitqueue_head+0x36/0x50
[ 3373.245780] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 3373.245780] ? lockdep_hardirqs_on+0x122/0x1b0
[ 3373.245781] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3373.245781] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3373.245782] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3373.245782] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3373.245783] ? remove_wait_queue+0x60/0x60
[ 3373.245783] kthread+0x103/0x140
[ 3373.245784] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3373.245784] ? kthread_bind+0x10/0x10
[ 3373.245785] ret_from_fork+0x3a/0x50
[ 3397.501729] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3397.519274] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3397.703104] irq event stamp: 3480889226
[ 3397.720545] hardirqs last enabled at (3480889225): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3397.744755] hardirqs last disabled at (3480889226): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3397.768817] softirqs last enabled at (3480889222): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3397.794348] softirqs last disabled at (3480889220): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3397.819897] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3397.843526] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3397.865076] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 3397.885643] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 65 96 78 ff 48 89 ef e8 9d e2 78 ff f6 c7 02 74 19 e8 73 a0 84 ff 48 89 df 57 9d <66> 66 90 66 90 65 ff 0d 4a e5 e8 68 5b 5d c3 48 89 df 57 9d 66 66
[ 3397.935988] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 3397.959731] RAX: ffff94d550e60000 RBX: 0000000000000296 RCX: 0000000000000000
[ 3397.982753] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: 0000000000000296
[ 3398.005278] RBP: ffff94d4b4533aa8 R08: 0000000000000000 R09: 0000000000000000
[ 3398.027585] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3398.049793] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3398.072044] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3398.095470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3398.116526] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3398.139172] Call Trace:
[ 3398.156914] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 3398.178982] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3398.200040] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3398.222081] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3398.245579] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3398.267899] ? remove_wait_queue+0x60/0x60
[ 3398.288129] kthread+0x103/0x140
[ 3398.307487] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3398.330200] ? kthread_bind+0x10/0x10
[ 3398.350018] ret_from_fork+0x3a/0x50
[ 3425.500971] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3425.524632] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3425.754024] irq event stamp: 3916309299
[ 3425.776498] hardirqs last enabled at (3916309297): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3425.805359] hardirqs last disabled at (3916309299): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3425.833666] softirqs last enabled at (3916309294): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3425.863027] softirqs last disabled at (3916309298): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3425.910339] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3425.937027] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3425.961213] RIP: 0010:do_raw_spin_lock+0x4c/0xa0
[ 3425.983211] Code: c2 74 66 8b 53 08 65 8b 05 d1 98 6f 69 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 0f 94 c2 84 d2 74 27 65 8b 05 b4 98 6f 69 <89> 43 08 65 48 8b 04 25 40 8e 01 00 48 89 43 10 5b c3 48 c7 c6 58
[ 3426.037716] RSP: 0018:ffffa8a885bafd80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3426.063277] RAX: 0000000000000000 RBX: ffff94d4b45337a8 RCX: 0000000000000000
[ 3426.088603] RDX: 0000000000000001 RSI: 0000000096612835 RDI: ffff94d4b45337a8
[ 3426.113625] RBP: ffff94d4b45337a8 R08: 00000000fffffff0 R09: 0000000000000000
[ 3426.138503] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3426.163332] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3426.187835] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3426.213131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3426.235920] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3426.260181] Call Trace:
[ 3426.279730] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3426.303421] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3426.326641] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3426.349554] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3426.373301] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3426.398263] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3426.421885] ? remove_wait_queue+0x60/0x60
[ 3426.443272] kthread+0x103/0x140
[ 3426.463617] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3426.487356] ? kthread_bind+0x10/0x10
[ 3426.508183] ret_from_fork+0x3a/0x50
[ 3453.500213] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3453.524653] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3453.759655] irq event stamp: 54157291
[ 3453.782059] hardirqs last enabled at (54157289): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3453.810167] hardirqs last disabled at (54157291): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3453.838373] softirqs last enabled at (54157288): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3453.867650] softirqs last disabled at (54157290): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3453.914950] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3453.941858] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3453.966385] RIP: 0010:iscsit_free_r2ts_from_list+0x2e/0x70 [iscsi_target_mod]
[ 3453.991473] Code: 90 41 55 4c 8d af c0 01 00 00 41 54 55 48 89 fd 4c 89 ef 4c 8d a5 08 02 00 00 53 e8 5c ea 31 d6 48 8b 95 08 02 00 00 48 8b 02 <49> 39 d4 48 8d 7a e0 48 8d 58 e0 75 08 eb 1f 48 89 df 48 89 d3 48
[ 3454.047006] RSP: 0018:ffffa8a885bafd80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3454.072814] RAX: ffff94d4b4533898 RBX: ffff94d4b4533690 RCX: 0000000000000000
[ 3454.098122] RDX: ffff94d4b4533898 RSI: 0000000031e50c30 RDI: ffff94d4b4533850
[ 3454.123276] RBP: ffff94d4b4533690 R08: 00000000fffffff0 R09: 0000000000000000
[ 3454.148327] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: ffff94d4b4533898
[ 3454.173124] R13: ffff94d4b4533850 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3454.197492] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3454.222882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3454.245814] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3454.270281] Call Trace:
[ 3454.289933] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3454.313079] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3454.335861] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3454.359435] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3454.384227] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3454.407684] ? remove_wait_queue+0x60/0x60
[ 3454.428902] kthread+0x103/0x140
[ 3454.449110] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3454.472659] ? kthread_bind+0x10/0x10
[ 3454.493353] ret_from_fork+0x3a/0x50
[ 3481.499456] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3481.523824] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3481.758241] irq event stamp: 487256119
[ 3481.780972] hardirqs last enabled at (487256117): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3481.809416] hardirqs last disabled at (487256119): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3481.837892] softirqs last enabled at (487256116): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3481.867464] softirqs last disabled at (487256118): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3481.915221] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3481.942198] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3481.966840] RIP: 0010:lock_release+0xf/0x2a0
[ 3481.988886] Code: c6 6d 06 90 97 48 c7 c7 ea 5c 8e 97 31 c0 e8 38 c2 f8 ff 0f 0b e9 e8 fe ff ff 90 41 57 65 48 8b 14 25 40 8e 01 00 41 56 41 55 <41> 54 55 53 48 83 ec 10 44 8b 82 bc 0c 00 00 65 48 8b 04 25 28 00
[ 3482.044525] RSP: 0018:ffffa8a885bafd78 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3482.070664] RAX: 0000000000000000 RBX: ffff94d4b45337a8 RCX: 0000000000000000
[ 3482.096085] RDX: ffff94d550e60000 RSI: ffffffffc0e6c0b9 RDI: ffff94d4b45337c0
[ 3482.121312] RBP: ffff94d4b45339a0 R08: 00000000fffffff0 R09: 0000000000000000
[ 3482.146442] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3482.171468] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3482.196172] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3482.221714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3482.244746] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3482.269230] Call Trace:
[ 3482.288919] _raw_spin_unlock_bh+0x17/0x30
[ 3482.310298] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3482.333403] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3482.356167] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3482.379729] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3482.404542] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3482.427978] ? remove_wait_queue+0x60/0x60
[ 3482.449173] kthread+0x103/0x140
[ 3482.469380] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3482.492959] ? kthread_bind+0x10/0x10
[ 3482.513665] ret_from_fork+0x3a/0x50
[ 3500.329969] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 0-... } 658435 jiffies s: 121 root: 0x1/.
[ 3500.340902] rcu: blocking rcu_node structures:
[ 3500.345598] Task dump for CPU 0:
[ 3500.349080] iscsi_ttx R running task 0 6148 2 0x8000408c
[ 3500.356374] Call Trace:
[ 3500.359092] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3500.363971] ? mark_held_locks+0x55/0x70
[ 3500.368147] ? lockdep_hardirqs_on+0x6e/0x1b0
[ 3500.372764] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3500.377626] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3500.382475] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3500.387333] ? lockdep_hardirqs_on+0x6e/0x1b0
[ 3500.391933] ? retint_kernel+0x10/0x10
[ 3500.395929] ? find_held_lock+0x2d/0x90
[ 3500.400014] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3500.406758] ? find_held_lock+0x2d/0x90
[ 3500.410830] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3500.417585] ? lockdep_hardirqs_on+0x122/0x1b0
[ 3500.422266] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3500.428928] ? __local_bh_enable_ip+0xdb/0xf0
[ 3500.433501] ? __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 3500.439648] ? iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3500.445443] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3500.452105] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3500.459983] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3500.466543] ? remove_wait_queue+0x60/0x60
[ 3500.470842] ? kthread+0x103/0x140
[ 3500.474434] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3500.481067] ? kthread_bind+0x10/0x10
[ 3500.484916] ? ret_from_fork+0x3a/0x50
[ 3509.498698] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3509.518303] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3509.713246] irq event stamp: 919836520
[ 3509.731555] hardirqs last enabled at (919836519): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3509.755834] hardirqs last disabled at (919836520): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3509.780392] softirqs last enabled at (919836518): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3509.806328] softirqs last disabled at (919836516): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3509.847908] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3509.872229] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3509.894457] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3509.914709] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3509.965071] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3509.988655] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3510.011827] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3510.034996] RBP: ffffffffc0e6c0b9 R08: 0000000000000000 R09: 0000000000000000
[ 3510.058138] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3510.081344] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3510.104603] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3510.129228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3510.151664] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3510.175702] Call Trace:
[ 3510.194973] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3510.217838] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3510.240286] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3510.263616] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3510.288210] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3510.311469] ? remove_wait_queue+0x60/0x60
[ 3510.332602] kthread+0x103/0x140
[ 3510.352712] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3510.376225] ? kthread_bind+0x10/0x10
[ 3510.396852] ret_from_fork+0x3a/0x50
[ 3537.497940] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3537.522252] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3537.756409] irq event stamp: 1354405450
[ 3537.779165] hardirqs last enabled at (1354405449): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3537.808310] hardirqs last disabled at (1354405450): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3537.836892] softirqs last enabled at (1354405446): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3537.866576] softirqs last disabled at (1354405444): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3537.895775] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3537.922630] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3537.947022] RIP: 0010:refcount_warn_saturate+0x84/0x100
[ 3537.969841] Code: 0b c3 80 3d 93 83 1e 01 00 75 ca 48 c7 c7 40 c3 94 97 31 c0 c6 05 81 83 1e 01 01 e8 56 42 c0 ff 0f 0b c3 80 3d 74 83 1e 01 00 <75> a9 48 c7 c7 e8 c2 94 97 31 c0 c6 05 62 83 1e 01 01 e8 35 42 c0
[ 3538.024747] RSP: 0018:ffffa8a885bafd60 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3538.050527] RAX: 00000000c0000000 RBX: ffff94d4b45339a0 RCX: 0000000000000000
[ 3538.075815] RDX: ffff94d550e60000 RSI: 0000000000000003 RDI: ffff94d4b4533ae0
[ 3538.100793] RBP: ffff94d4b4533aa8 R08: 0000000000000000 R09: 0000000000000000
[ 3538.125580] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3538.150186] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3538.174695] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3538.199941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3538.222482] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3538.246421] Call Trace:
[ 3538.265454] transport_generic_free_cmd+0x155/0x160 [target_core_mod]
[ 3538.288761] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3538.311108] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3538.334399] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3538.359027] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3538.382204] ? remove_wait_queue+0x60/0x60
[ 3538.403133] kthread+0x103/0x140
[ 3538.423017] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3538.446226] ? kthread_bind+0x10/0x10
[ 3538.466520] ret_from_fork+0x3a/0x50
[ 3565.497183] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3565.521230] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3565.753244] irq event stamp: 1787766347
[ 3565.775977] hardirqs last enabled at (1787766345): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3565.804399] hardirqs last disabled at (1787766347): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3565.832883] softirqs last enabled at (1787766344): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3565.862433] softirqs last disabled at (1787766346): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3565.910228] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3565.937095] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3565.961468] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3565.983379] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3566.038223] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3566.063959] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3566.089435] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 3566.114584] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3566.139470] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3566.164192] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3566.188542] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3566.213529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3566.236005] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3566.259932] Call Trace:
[ 3566.279001] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3566.302458] _raw_spin_lock_bh+0x3d/0x50
[ 3566.323278] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3566.346920] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3566.370359] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3566.393101] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3566.415406] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3566.438543] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3566.463030] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3566.486162] ? remove_wait_queue+0x60/0x60
[ 3566.507120] kthread+0x103/0x140
[ 3566.527019] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3566.550257] ? kthread_bind+0x10/0x10
[ 3566.570581] ret_from_fork+0x3a/0x50
[ 3567.299134] rcu: INFO: rcu_sched self-detected stall on CPU
[ 3567.321417] rcu: 0-....: (1181539 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=306901
[ 3567.348636] (t=1235041 jiffies g=82081 q=112197)
[ 3567.370519] NMI backtrace for cpu 0
[ 3567.390995] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3567.416856] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3567.440269] Call Trace:
[ 3567.459010] <IRQ>
[ 3567.476835] dump_stack+0x82/0xb3
[ 3567.495596] nmi_cpu_backtrace+0x90/0xa0
[ 3567.514611] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 3567.533659] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 3567.553062] rcu_dump_cpu_stacks+0x97/0xc5
[ 3567.571267] rcu_sched_clock_irq+0x6ec/0x980
[ 3567.589397] ? rcu_read_lock_sched_held+0x52/0x90
[ 3567.607884] ? tick_sched_do_timer+0x70/0x70
[ 3567.625782] update_process_times+0x24/0x50
[ 3567.643471] tick_sched_handle+0x25/0x60
[ 3567.660502] tick_sched_timer+0x37/0x70
[ 3567.677159] __hrtimer_run_queues+0xe8/0x450
[ 3567.694084] hrtimer_interrupt+0x122/0x270
[ 3567.710688] smp_apic_timer_interrupt+0x86/0x240
[ 3567.727447] apic_timer_interrupt+0xf/0x20
[ 3567.743271] </IRQ>
[ 3567.756794] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3567.772575] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3567.815871] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3567.836100] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3567.856025] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 3567.876033] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3567.895674] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3567.915148] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3567.934416] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3567.953086] _raw_spin_lock_bh+0x3d/0x50
[ 3567.969004] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3567.987745] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3568.006240] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3568.024124] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3568.041634] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3568.060031] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3568.079698] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3568.097988] ? remove_wait_queue+0x60/0x60
[ 3568.114098] kthread+0x103/0x140
[ 3568.129289] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3568.147830] ? kthread_bind+0x10/0x10
[ 3568.163451] ret_from_fork+0x3a/0x50
[ 3568.179006] NMI backtrace for cpu 0
[ 3568.179007] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3568.179008] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3568.179008] RIP: 0010:cfb_imageblit+0x4be/0x4f0
[ 3568.179009] Code: 08 00 00 00 89 04 24 eb 2e 41 0f be 55 00 44 29 c1 4d 8d 74 24 04 d3 fa 44 21 d2 41 8b 04 91 44 21 d8 89 c2 31 fa 41 89 14 24 <85> c9 75 06 49 83 c5 01 b1 08 4d 89 f4 83 ee 01 83 fe ff 75 ca 83
[ 3568.179009] RSP: 0018:ffffa8a880003808 EFLAGS: 00000006
[ 3568.179010] RAX: 0000000000aaaaaa RBX: 000000000000000a RCX: 0000000000000006
[ 3568.179011] RDX: 0000000000aaaaaa RSI: 0000000000000026 RDI: 0000000000000000
[ 3568.179012] RBP: ffff94d56c20aa79 R08: 0000000000000001 R09: ffffffff976bd6e0
[ 3568.179012] R10: 0000000000000001 R11: 0000000000aaaaaa R12: ffffa8a881d35104
[ 3568.179013] R13: ffff94d56c20aa79 R14: ffffa8a881d35108 R15: ffffa8a881d35100
[ 3568.179013] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3568.179014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3568.179014] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3568.179015] Call Trace:
[ 3568.179015] <IRQ>
[ 3568.179015] ? _raw_spin_unlock_irqrestore+0x50/0x52
[ 3568.179016] drm_fb_helper_cfb_imageblit+0x12/0x30 [drm_kms_helper]
[ 3568.179016] bit_putcs+0x2ec/0x500
[ 3568.179017] ? find_held_lock+0x2d/0x90
[ 3568.179017] ? soft_cursor+0x1af/0x220
[ 3568.179018] fbcon_putcs+0x11c/0x130
[ 3568.179018] ? update_attr.isra.2+0x80/0x80
[ 3568.179019] fbcon_redraw.isra.21+0x17b/0x1f0
[ 3568.179019] fbcon_scroll+0x4a5/0xda0
[ 3568.179019] con_scroll+0x1e0/0x200
[ 3568.179020] lf+0x9e/0xb0
[ 3568.179020] vt_console_print+0x248/0x3b0
[ 3568.179021] console_unlock+0x4cf/0x5e0
[ 3568.179021] ? console_unlock+0x1b2/0x5e0
[ 3568.179022] ? __down_trylock_console_sem+0x7d/0xa0
[ 3568.179022] vprintk_emit+0x13b/0x380
[ 3568.179023] ? apic_timer_interrupt+0xf/0x20
[ 3568.179023] printk+0x58/0x6f
[ 3568.179023] __show_regs+0x6d/0x2c0
[ 3568.179024] ? apic_timer_interrupt+0xf/0x20
[ 3568.179024] show_trace_log_lvl+0x16e/0x400
[ 3568.179025] ? lock_acquire+0xd5/0x1c0
[ 3568.179025] dump_stack+0x82/0xb3
[ 3568.179026] nmi_cpu_backtrace+0x90/0xa0
[ 3568.179026] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 3568.179027] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 3568.179027] rcu_dump_cpu_stacks+0x97/0xc5
[ 3568.179027] rcu_sched_clock_irq+0x6ec/0x980
[ 3568.179028] ? rcu_read_lock_sched_held+0x52/0x90
[ 3568.179028] ? tick_sched_do_timer+0x70/0x70
[ 3568.179029] update_process_times+0x24/0x50
[ 3568.179029] tick_sched_handle+0x25/0x60
[ 3568.179030] tick_sched_timer+0x37/0x70
[ 3568.179030] __hrtimer_run_queues+0xe8/0x450
[ 3568.179031] hrtimer_interrupt+0x122/0x270
[ 3568.179031] smp_apic_timer_interrupt+0x86/0x240
[ 3568.179032] apic_timer_interrupt+0xf/0x20
[ 3568.179032] </IRQ>
[ 3568.179032] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3568.179033] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3568.179034] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3568.179035] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3568.179035] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 3568.179036] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3568.179036] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3568.179037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3568.179037] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3568.179038] _raw_spin_lock_bh+0x3d/0x50
[ 3568.179038] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3568.179039] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3568.179039] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3568.179040] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3568.179040] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3568.179041] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3568.179041] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3568.179042] ? remove_wait_queue+0x60/0x60
[ 3568.179042] kthread+0x103/0x140
[ 3568.179043] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3568.179043] ? kthread_bind+0x10/0x10
[ 3568.179044] ret_from_fork+0x3a/0x50
[ 3593.496425] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3593.513844] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3593.697105] irq event stamp: 2186059063
[ 3593.714547] hardirqs last enabled at (2186059061): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3593.738169] hardirqs last disabled at (2186059063): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3593.762178] softirqs last enabled at (2186059060): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3593.787625] softirqs last disabled at (2186059062): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3593.828416] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3593.852417] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3593.874342] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3593.894043] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3593.945352] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3593.969558] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3593.993430] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 3594.017304] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3594.041169] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3594.065043] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3594.088497] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3594.112659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3594.134306] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3594.157461] Call Trace:
[ 3594.175855] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3594.198710] _raw_spin_lock_bh+0x3d/0x50
[ 3594.219010] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3594.242259] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3594.265301] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3594.287728] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3594.309706] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3594.332630] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3594.356859] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3594.379797] ? remove_wait_queue+0x60/0x60
[ 3594.400632] kthread+0x103/0x140
[ 3594.420447] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3594.443615] ? kthread_bind+0x10/0x10
[ 3594.463851] ret_from_fork+0x3a/0x50
[ 3621.495667] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3621.519757] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3621.748431] irq event stamp: 2619591534
[ 3621.769781] hardirqs last enabled at (2619591533): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3621.797528] hardirqs last disabled at (2619591534): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3621.824667] softirqs last enabled at (2619591530): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3621.853016] softirqs last disabled at (2619591528): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3621.881009] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3621.907030] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3621.930770] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 3621.953402] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 65 96 78 ff 48 89 ef e8 9d e2 78 ff f6 c7 02 74 19 e8 73 a0 84 ff 48 89 df 57 9d <66> 66 90 66 90 65 ff 0d 4a e5 e8 68 5b 5d c3 48 89 df 57 9d 66 66
[ 3622.006846] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 3622.031784] RAX: ffff94d550e60000 RBX: 0000000000000296 RCX: 0000000000000000
[ 3622.056290] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: 0000000000000296
[ 3622.080337] RBP: ffff94d4b4533aa8 R08: 0000000000000000 R09: 0000000000000000
[ 3622.103914] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3622.127332] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3622.150620] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3622.175081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3622.197258] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3622.221067] Call Trace:
[ 3622.240140] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 3622.263412] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3622.285518] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3622.308493] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3622.332712] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3622.355600] ? remove_wait_queue+0x60/0x60
[ 3622.376275] kthread+0x103/0x140
[ 3622.395914] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3622.418913] ? kthread_bind+0x10/0x10
[ 3622.438970] ret_from_fork+0x3a/0x50
[ 3649.494910] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3649.518727] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3649.748557] irq event stamp: 3053768190
[ 3649.771016] hardirqs last enabled at (3053768189): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3649.799177] hardirqs last disabled at (3053768190): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3649.827397] softirqs last enabled at (3053768188): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3649.856688] softirqs last disabled at (3053768186): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3649.903802] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3649.930413] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3649.954540] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3649.976826] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3650.031085] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 3650.056522] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3650.081739] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3650.106579] RBP: ffffffffc0e6c0b9 R08: 0000000000000000 R09: 0000000000000000
[ 3650.131236] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3650.155820] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3650.180072] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3650.205120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3650.227669] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3650.251683] Call Trace:
[ 3650.271001] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3650.293898] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3650.316455] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3650.340008] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3650.364809] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3650.388231] ? remove_wait_queue+0x60/0x60
[ 3650.409438] kthread+0x103/0x140
[ 3650.429575] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3650.453064] ? kthread_bind+0x10/0x10
[ 3650.473643] ret_from_fork+0x3a/0x50
[ 3677.494152] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3677.518394] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3677.751997] irq event stamp: 3487234139
[ 3677.774680] hardirqs last enabled at (3487234137): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3677.803052] hardirqs last disabled at (3487234139): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3677.831493] softirqs last enabled at (3487234136): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3677.861024] softirqs last disabled at (3487234138): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3677.908713] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3677.935530] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3677.960045] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3677.982038] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3678.037436] RSP: 0018:ffffa8a885bafd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3678.063451] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3678.088767] RDX: 0000000000000001 RSI: 0000000031e50c30 RDI: 0000000000000246
[ 3678.113797] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3678.138649] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3678.163315] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3678.187576] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3678.212548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3678.234990] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3678.258892] Call Trace:
[ 3678.278042] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3678.301599] _raw_spin_lock_bh+0x3d/0x50
[ 3678.322522] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3678.346195] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3678.369590] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3678.392317] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3678.414586] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3678.437698] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3678.462131] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3678.485299] ? remove_wait_queue+0x60/0x60
[ 3678.506275] kthread+0x103/0x140
[ 3678.526217] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3678.549488] ? kthread_bind+0x10/0x10
[ 3678.569819] ret_from_fork+0x3a/0x50
[ 3696.932651] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 0-... } 855043 jiffies s: 121 root: 0x1/.
[ 3696.943578] rcu: blocking rcu_node structures:
[ 3696.948278] Task dump for CPU 0:
[ 3696.951763] iscsi_ttx R running task 0 6148 2 0x8000408c
[ 3696.959081] Call Trace:
[ 3696.961792] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3696.966661] ? mark_held_locks+0x55/0x70
[ 3696.970837] ? mark_held_locks+0x55/0x70
[ 3696.974994] ? lockdep_hardirqs_on+0x6e/0x1b0
[ 3696.979583] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 3696.984452] ? __lock_acquire+0x24b/0x1080
[ 3696.988778] ? __lock_acquire+0x24b/0x1080
[ 3696.993117] ? transport_generic_free_cmd+0xfc/0x160 [target_core_mod]
[ 3696.999897] ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3697.006056] ? __local_bh_enable_ip+0x80/0xf0
[ 3697.010662] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3697.016808] ? __local_bh_enable_ip+0x47/0xf0
[ 3697.021377] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3697.027526] ? iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3697.033320] ? iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3697.039990] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3697.047865] ? iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3697.054413] ? remove_wait_queue+0x60/0x60
[ 3697.058701] ? kthread+0x103/0x140
[ 3697.062311] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3697.068931] ? kthread_bind+0x10/0x10
[ 3697.072789] ? ret_from_fork+0x3a/0x50
[ 3705.493395] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3705.513085] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3705.710135] irq event stamp: 3919434961
[ 3705.728575] hardirqs last enabled at (3919434959): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3705.752997] hardirqs last disabled at (3919434961): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3705.777819] softirqs last enabled at (3919434958): [<ffffffffc0e6c0c1>] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3705.803989] softirqs last disabled at (3919434960): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3705.829963] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3705.853970] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3705.875783] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 3705.896064] Code: fa 66 66 90 66 66 90 65 01 35 7c 0a 77 69 65 8b 05 75 0a 77 69 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 8d cc 06 00 48 89 df 57 9d <66> 66 90 66 90 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 3705.946597] RSP: 0018:ffffa8a885bafd80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3705.970365] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 3705.993408] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 3706.016090] RBP: ffff94d4b45339a0 R08: 0000000000000001 R09: 0000000000000000
[ 3706.038785] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3706.061406] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb85e0 R15: ffff94d4b4533690
[ 3706.084021] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3706.107788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3706.129256] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3706.152450] Call Trace:
[ 3706.170898] _raw_spin_lock_bh+0x1c/0x50
[ 3706.190858] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3706.212706] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3706.234485] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3706.257136] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3706.281081] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3706.303746] ? remove_wait_queue+0x60/0x60
[ 3706.324270] kthread+0x103/0x140
[ 3706.343797] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3706.366726] ? kthread_bind+0x10/0x10
[ 3706.386753] ret_from_fork+0x3a/0x50
[ 3733.492637] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3733.516380] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3733.746063] irq event stamp: 59837300
[ 3733.768080] hardirqs last enabled at (59837299): [<ffffffff9718a8ad>] _raw_spin_unlock_irqrestore+0x2d/0x52
[ 3733.796482] hardirqs last disabled at (59837300): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3733.824308] softirqs last enabled at (59837296): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3733.853232] softirqs last disabled at (59837294): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3733.881682] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3733.907976] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3733.931796] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 3733.954392] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 65 96 78 ff 48 89 ef e8 9d e2 78 ff f6 c7 02 74 19 e8 73 a0 84 ff 48 89 df 57 9d <66> 66 90 66 90 65 ff 0d 4a e5 e8 68 5b 5d c3 48 89 df 57 9d 66 66
[ 3734.008168] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 3734.033386] RAX: ffff94d550e60000 RBX: 0000000000000296 RCX: 0000000000000000
[ 3734.058227] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: 0000000000000296
[ 3734.082640] RBP: ffff94d4b4533aa8 R08: 0000000000000000 R09: 0000000000000000
[ 3734.106811] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3734.130927] R13: 0000000000000001 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3734.154681] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3734.179198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3734.201205] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3734.224685] Call Trace:
[ 3734.243444] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 3734.266508] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 3734.288529] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3734.311442] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3734.335570] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3734.358299] ? remove_wait_queue+0x60/0x60
[ 3734.378844] kthread+0x103/0x140
[ 3734.398296] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3734.421227] ? kthread_bind+0x10/0x10
[ 3734.441289] ret_from_fork+0x3a/0x50
[ 3761.491879] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3761.515679] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3761.745547] irq event stamp: 493858040
[ 3761.767923] hardirqs last enabled at (493858039): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3761.796013] hardirqs last disabled at (493858040): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3761.824154] softirqs last enabled at (493858038): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3761.853346] softirqs last disabled at (493858036): [<ffffffffc0e61669>] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 3761.900502] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3761.927136] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3761.951280] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3761.973577] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3762.027906] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3762.053372] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3762.078567] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3762.103472] RBP: ffffffffc0e6c0b9 R08: 0000000000000000 R09: 0000000000000000
[ 3762.128175] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3762.152787] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3762.177109] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3762.202188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3762.224762] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3762.248844] Call Trace:
[ 3762.268195] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3762.291135] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3762.298860] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3762.313728] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3762.336932] rcu: 0-....: (1367699 ticks this GP) idle=826/1/0x4000000000000002 softirq=48932/48932 fqs=355435
[ 3762.360481] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3762.388186] (detected by 2, t=1430023 jiffies, g=82081, q=120171)
[ 3762.413305] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3762.437093] Sending NMI from CPU 2 to CPUs 0:
[ 3762.460901] ? remove_wait_queue+0x60/0x60
[ 3762.483800] NMI backtrace for cpu 0
[ 3762.483801] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3762.483801] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3762.483802] RIP: 0010:__lock_acquire+0x836/0x1080
[ 3762.483803] Code: 50 7e 5a 01 45 85 c9 0f 84 e3 03 00 00 49 bd eb 83 b5 80 46 86 c8 61 31 c0 4c 0f af 2c 24 49 c1 ed 31 4a 8b 14 ed 20 d0 fe 98 <4a> 8d 1c ed 20 d0 fe 98 48 8d 4a f8 48 85 d2 48 0f 45 c1 eb 04 48
[ 3762.483803] RSP: 0018:ffffa8a880003c28 EFLAGS: 00000806
[ 3762.483804] RAX: 0000000000000000 RBX: 00000000650b0e7f RCX: 0000000010c3d267
[ 3762.483805] RDX: ffffffff98923028 RSI: 00000000606d6c44 RDI: 000000000566623b
[ 3762.483805] RBP: ffff94d550e60000 R08: 0000000031e29840 R09: 0000000000000001
[ 3762.483806] R10: 0000000000000000 R11: ffffa8a880003b80 R12: ffff94d550e60ce8
[ 3762.483806] R13: 0000000000001596 R14: ffffffff97c71058 R15: 0000000000000001
[ 3762.483807] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3762.483807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3762.483808] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3762.483808] Call Trace:
[ 3762.483809] <IRQ>
[ 3762.483809] lock_acquire+0xbd/0x1c0
[ 3762.483809] ? console_unlock+0x1d8/0x5e0
[ 3762.483810] _raw_spin_lock+0x35/0x50
[ 3762.483810] ? console_unlock+0x1d8/0x5e0
[ 3762.483811] console_unlock+0x1d8/0x5e0
[ 3762.483811] vprintk_emit+0x13b/0x380
[ 3762.483812] ? vprintk_emit+0x2c0/0x380
[ 3762.483812] ? remove_wait_queue+0x60/0x60
[ 3762.483813] ? remove_wait_queue+0x60/0x60
[ 3762.483813] printk+0x58/0x6f
[ 3762.483813] ? remove_wait_queue+0x60/0x60
[ 3762.483814] show_trace_log_lvl+0x19b/0x400
[ 3762.483814] ? kthread+0x103/0x140
[ 3762.483815] watchdog_timer_fn+0x20c/0x270
[ 3762.483815] ? lockup_detector_update_enable+0x50/0x50
[ 3762.483816] __hrtimer_run_queues+0xe8/0x450
[ 3762.483816] hrtimer_interrupt+0x122/0x270
[ 3762.483817] smp_apic_timer_interrupt+0x86/0x240
[ 3762.483817] apic_timer_interrupt+0xf/0x20
[ 3762.483818] </IRQ>
[ 3762.483818] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3762.483819] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3762.483820] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3762.483821] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3762.483821] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3762.483822] RBP: ffffffffc0e6c0b9 R08: 0000000000000000 R09: 0000000000000000
[ 3762.483822] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3762.483823] R13: ffff94d4b4bb8000 R14: ffff94d4b4533990 R15: ffff94d4b4533690
[ 3762.483823] ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3762.483824] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3762.483824] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3762.483825] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3762.483825] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3762.483826] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3762.483826] ? remove_wait_queue+0x60/0x60
[ 3762.483827] kthread+0x103/0x140
[ 3762.483827] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3762.483828] ? kthread_bind+0x10/0x10
[ 3762.483828] ret_from_fork+0x3a/0x50
[ 3763.590012] kthread+0x103/0x140
[ 3763.603232] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3763.619904] ? kthread_bind+0x10/0x10
[ 3763.633659] ret_from_fork+0x3a/0x50
[ 3789.491122] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3789.508532] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3789.691212] irq event stamp: 909023527
[ 3789.708560] hardirqs last enabled at (909023525): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3789.732053] hardirqs last disabled at (909023527): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3789.756158] softirqs last enabled at (909023524): [<ffffffffc0e6c0c1>] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3789.781745] softirqs last disabled at (909023526): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3789.807286] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3789.831113] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3789.852866] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 3789.872404] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 f3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 3789.923163] RSP: 0018:ffffa8a885bafd28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3789.947079] RAX: ffff94d550e60000 RBX: 0000000000000246 RCX: 0000000000000000
[ 3789.970659] RDX: 0000000000000001 RSI: 0000000002c1e137 RDI: 0000000000000246
[ 3789.994221] RBP: 0000000000000000 R08: 00000000fffffff0 R09: 0000000000000000
[ 3790.017793] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: 0000000000000000
[ 3790.041327] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 3790.064466] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3790.088304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3790.109639] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3790.132441] Call Trace:
[ 3790.150517] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3790.172254] _raw_spin_lock_bh+0x3d/0x50
[ 3790.192085] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3790.213896] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3790.235502] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3790.256835] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3790.279129] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3790.302751] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3790.325085] ? remove_wait_queue+0x60/0x60
[ 3790.345195] kthread+0x103/0x140
[ 3790.364280] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3790.386877] ? kthread_bind+0x10/0x10
[ 3790.406599] ret_from_fork+0x3a/0x50
[ 3817.490364] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_ttx:6148]
[ 3817.513954] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3817.741602] irq event stamp: 1343581230
[ 3817.763096] hardirqs last enabled at (1343581229): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3817.790311] hardirqs last disabled at (1343581230): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3817.817578] softirqs last enabled at (1343581228): [<ffffffffc0e6c0f1>] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3817.845930] softirqs last disabled at (1343581226): [<ffffffffc0e6bf91>] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 3817.873750] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3817.899398] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3817.922707] RIP: 0010:__local_bh_enable_ip+0x84/0xf0
[ 3817.944353] Code: 02 00 00 74 57 65 01 1d a2 02 77 69 65 8b 05 9b 02 77 69 a9 00 ff 1f 00 74 55 65 ff 0d 8d 02 77 69 e8 a0 bd 12 00 fb 66 66 90 <66> 66 90 5b 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d ed b5 5d 01
[ 3817.997651] RSP: 0018:ffffa8a885bafd90 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 3818.022477] RAX: ffff94d550e60000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 3818.046801] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3818.070981] RBP: ffffffffc0e6c0f1 R08: 0000000000000000 R09: 0000000000000000
[ 3818.094767] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3818.118173] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb8650 R15: ffff94d4b4533690
[ 3818.141480] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3818.165859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3818.187952] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3818.211679] Call Trace:
[ 3818.230665] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 3818.253217] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3818.275375] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3818.298371] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3818.322596] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3818.345466] ? remove_wait_queue+0x60/0x60
[ 3818.366132] kthread+0x103/0x140
[ 3818.385810] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3818.408876] ? kthread_bind+0x10/0x10
[ 3818.429044] ret_from_fork+0x3a/0x50
[ 3845.489607] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3845.513510] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3845.744271] irq event stamp: 1778037030
[ 3845.766867] hardirqs last enabled at (1778037029): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3845.795180] hardirqs last disabled at (1778037030): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3845.823521] softirqs last enabled at (1778037028): [<ffffffffc0e6c0fe>] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 3845.852968] softirqs last disabled at (1778037026): [<ffffffffc0e6bf19>] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 3845.881939] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3845.908692] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3845.932988] RIP: 0010:__iscsit_free_cmd+0x112/0x2c0 [iscsi_target_mod]
[ 3845.956877] Code: 00 00 00 48 8b 45 10 48 8d 7d 10 4c 39 ff 48 8d 50 f0 75 de 4c 89 f7 e8 fc e1 31 d6 8b 83 d8 00 00 00 85 c0 0f 85 8a 01 00 00 <4d> 8d b5 50 06 00 00 4c 89 f7 e8 cf e0 31 d6 8b 83 dc 00 00 00 85
[ 3846.011089] RSP: 0018:ffffa8a885bafda8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 3846.036510] RAX: ffff94d550e60000 RBX: ffff94d4b4533690 RCX: 0000000000000000
[ 3846.061511] RDX: ffff94d550e60000 RSI: 0000000000000006 RDI: ffff94d550e60000
[ 3846.086562] RBP: ffff94d4b45339a0 R08: 0000000000000000 R09: 0000000000000000
[ 3846.111299] R10: 0000000000000001 R11: ffffa8a885bafd70 R12: 0000000000000001
[ 3846.135862] R13: ffff94d4b4bb8000 R14: ffff94d4b4bb85e0 R15: ffff94d4b4533690
[ 3846.160369] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3846.185712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3846.208345] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3846.232360] Call Trace:
[ 3846.251450] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 3846.273720] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3846.296981] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3846.321596] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3846.344901] ? remove_wait_queue+0x60/0x60
[ 3846.365912] kthread+0x103/0x140
[ 3846.385837] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3846.409114] ? kthread_bind+0x10/0x10
[ 3846.429406] ret_from_fork+0x3a/0x50
[ 3873.488849] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_ttx:6148]
[ 3873.512861] Modules linked in: netconsole target_core_user uio target_core_pscsi target_core_file target_core_iblock macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib bnxt_en rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod bnxt_re iw_cxgb4 libcxgb rdma_cm iw_cm ib_cm mlx5_ib mlx5_core mlxfw pci_hyperv_intf ext4 mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en iTCO_wdt gpio_ich iTCO_vendor_support ppdev mlx4_core coretemp kvm irqbypass pcspkr i2c_i801 sg parport_pc cxgb4 lpc_ich parport i5400_edac pata_acpi i5k_amb sch_fq_codel ip_tables xfs libcrc32c nouveau sr_mod sd_mod cdrom ata_generic video mxm_wmi wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb ttm ahci libahci ptp drm pps_core dca firewire_ohci i2c_algo_bit ata_piix firewire_core libata serio_raw i2c_core crc_itu_t
[ 3873.744778] irq event stamp: 2211944235
[ 3873.767778] hardirqs last enabled at (2211944233): [<ffffffff968a8b80>] __local_bh_enable_ip+0x80/0xf0
[ 3873.796482] hardirqs last disabled at (2211944235): [<ffffffff96803977>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 3873.825242] softirqs last enabled at (2211944232): [<ffffffffc0e6c0b9>] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 3873.855076] softirqs last disabled at (2211944234): [<ffffffffc0e6b604>] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 3873.903390] CPU: 0 PID: 6148 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.5.0 #34
[ 3873.930562] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[ 3873.955232] RIP: 0010:lock_release+0x142/0x2a0
[ 3873.977325] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <66> 66 90 66 90 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 3874.032324] RSP: 0018:ffffa8a885bafd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3874.058150] RAX: ffff94d550e60000 RBX: ffff94d550e60000 RCX: ffffa8a885bafd54
[ 3874.083510] RDX: 0000000000000000 RSI: ffff94d4b4533868 RDI: 0000000000000246
[ 3874.108880] RBP: ffff94d4b4533868 R08: 0000000000000000 R09: 0000000000000000
[ 3874.133923] R10: 0000000000000000 R11: ffffa8a885bafd70 R12: ffffffffc0e6c0c1
[ 3874.158893] R13: 0000000000000246 R14: ffff94d550e60cc0 R15: 0000000000000001
[ 3874.183749] FS: 0000000000000000(0000) GS:ffff94d56ee00000(0000) knlGS:0000000000000000
[ 3874.209367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3874.232304] CR2: 00007fe7f507e000 CR3: 000000024ef2c000 CR4: 00000000000006f0
[ 3874.256616] Call Trace:
[ 3874.275997] _raw_spin_unlock_bh+0x17/0x30
[ 3874.297129] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 3874.320027] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 3874.342603] iscsit_close_connection+0x558/0x910 [iscsi_target_mod]
[ 3874.366093] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 3874.390785] iscsi_target_tx_thread+0x170/0x210 [iscsi_target_mod]
[ 3874.414085] ? remove_wait_queue+0x60/0x60
[ 3874.435155] kthread+0x103/0x140
[ 3874.455163] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 3874.478588] ? kthread_bind+0x10/0x10
[ 3874.499114] ret_from_fork+0x3a/0x50
[-- Attachment #3: iscsi.conf --]
[-- Type: application/octet-stream, Size: 21183 bytes --]
{
"fabric_modules": [],
"storage_objects": [
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram10",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "c8a2fa9b-7cb8-4247-810b-cdbe195fc110"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram9",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "cb62b9fc-de23-4634-827d-042e2a46c593"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram8",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "bdbdd38b-a47b-4d7d-bb4f-784f829446bc"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram7",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "3c7f28b0-bf23-4167-9b5c-2bc0cfa9456c"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram6",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "bd195f97-a442-4260-9ac8-066a0e8e07cd"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram5",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "12dd3a65-6cd2-4988-8214-18300b425673"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram4",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "551dffba-3095-4175-8892-f4c96db35b94"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram3",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "e14f0341-d301-4ed9-979b-33f081c7c058"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram2",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "c2fca632-69ad-4a42-af08-245858cb6e22"
},
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_pr": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 0,
"optimal_sectors": 4294967288,
"pi_prot_format": 0,
"pi_prot_type": 0,
"pi_prot_verify": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"name": "ram1",
"plugin": "ramdisk",
"size": 52428800,
"wwn": "6e715bd5-b0e1-4756-9dac-538b232d87e9"
}
],
"targets": [
{
"fabric": "iscsi",
"tpgs": [
{
"attributes": {
"authentication": 0,
"cache_dynamic_acls": 1,
"default_cmdsn_depth": 64,
"default_erl": 0,
"demo_mode_discovery": 1,
"demo_mode_write_protect": 0,
"fabric_prot_type": 0,
"generate_node_acls": 1,
"login_keys_workaround": 1,
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0,
"t10_pi": 0,
"tpg_enabled_sendtargets": 1
},
"enable": true,
"luns": [
{
"alias": "317c593828",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 9,
"storage_object": "/backstores/ramdisk/ram10"
},
{
"alias": "79d2192ad8",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 8,
"storage_object": "/backstores/ramdisk/ram9"
},
{
"alias": "fa74ab57a9",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 7,
"storage_object": "/backstores/ramdisk/ram8"
},
{
"alias": "cf35d62218",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 6,
"storage_object": "/backstores/ramdisk/ram7"
},
{
"alias": "2240974159",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 5,
"storage_object": "/backstores/ramdisk/ram6"
},
{
"alias": "e17e6a789c",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 4,
"storage_object": "/backstores/ramdisk/ram5"
},
{
"alias": "51fb8e0b54",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 3,
"storage_object": "/backstores/ramdisk/ram4"
},
{
"alias": "2624cec56e",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 2,
"storage_object": "/backstores/ramdisk/ram3"
},
{
"alias": "59385fa6d7",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 1,
"storage_object": "/backstores/ramdisk/ram2"
},
{
"alias": "d192b80c7b",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 0,
"storage_object": "/backstores/ramdisk/ram1"
}
],
"node_acls": [],
"parameters": {
"AuthMethod": "CHAP,None",
"DataDigest": "CRC32C,None",
"DataPDUInOrder": "Yes",
"DataSequenceInOrder": "Yes",
"DefaultTime2Retain": "20",
"DefaultTime2Wait": "2",
"ErrorRecoveryLevel": "0",
"FirstBurstLength": "65536",
"HeaderDigest": "CRC32C,None",
"IFMarkInt": "Reject",
"IFMarker": "No",
"ImmediateData": "Yes",
"InitialR2T": "Yes",
"MaxBurstLength": "262144",
"MaxConnections": "1",
"MaxOutstandingR2T": "1",
"MaxRecvDataSegmentLength": "8192",
"MaxXmitDataSegmentLength": "262144",
"OFMarkInt": "Reject",
"OFMarker": "No",
"TargetAlias": "LIO Target"
},
"portals": [
{
"ip_address": "106.1.1.16",
"iser": true,
"offload": false,
"port": 3260
}
],
"tag": 1
}
],
"wwn": "iqn.2017-01.org.lun1.target"
}
]
}
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: IOZONE fails intermittently while toggling the interface
2020-02-07 14:00 IOZONE fails intermittently while toggling the interface Dakshaja Uppalapati
@ 2020-02-07 16:29 ` Bart Van Assche
2020-02-09 17:29 ` Bart Van Assche
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Bart Van Assche @ 2020-02-07 16:29 UTC (permalink / raw)
To: target-devel
On 2/7/20 6:00 AM, Dakshaja Uppalapati wrote:
> While testing iser with kernel 5.5, I see attached soft lockups.
> Logged in and ran iozone on mounted disks. While the traffic is running, toggled the initiators interface down to up for 13 secs.
> I see the attached hung task warnings in the target dmesg.
> # iozone -a -I -+d -g 32m
>
> The same test with kernel-5.4 runs fine. I have suspected the following three commits which are all correlated, so removed these commits from 5.5 kernel, rebuilt the kernel and ran the same test then no issue is seen. Iozone is running successfully.
>
> 80647a89eaf3f2: scsi: target: core: Release SPC-2 reservations when closing a session
> e9d3009cb936b: scsi: target: iscsi: Wait for all commands to finish before freeing a session
> 04060db41178c: scsi: RDMA/isert: Fix a recently introduced regression related to logout
>
> I am trying to root cause the issue. Please do suggest me if you want me to check some thing specific.
Hi Dakshaja,
Thanks for having included a call trace; that's very useful. I will try
to reproduce this issue this weekend and will keep you posted.
Bart.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: IOZONE fails intermittently while toggling the interface
2020-02-07 14:00 IOZONE fails intermittently while toggling the interface Dakshaja Uppalapati
2020-02-07 16:29 ` Bart Van Assche
@ 2020-02-09 17:29 ` Bart Van Assche
2020-02-10 14:31 ` Dakshaja Uppalapati
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Bart Van Assche @ 2020-02-09 17:29 UTC (permalink / raw)
To: target-devel
[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]
On 2020-02-07 06:00, Dakshaja Uppalapati wrote:
> While testing iser with kernel 5.5, I see attached soft lockups.
> Logged in and ran iozone on mounted disks. While the traffic is running, toggled the initiators interface down to up for 13 secs.
> I see the attached hung task warnings in the target dmesg.
> # iozone -a -I -+d -g 32m
>
> The same test with kernel-5.4 runs fine. I have suspected the following three commits which are all correlated, so removed these commits from 5.5 kernel, rebuilt the kernel and ran the same test then no issue is seen. Iozone is running successfully.
>
> 80647a89eaf3f2: scsi: target: core: Release SPC-2 reservations when closing a session
> e9d3009cb936b: scsi: target: iscsi: Wait for all commands to finish before freeing a session
> 04060db41178c: scsi: RDMA/isert: Fix a recently introduced regression related to logout
>
> I am trying to root cause the issue. Please do suggest me if you want me to check some thing specific.
Does the attached patch help?
Thanks,
Bart.
[-- Attachment #2: 0001-RDMA-isert-Really-fix-logout.patch --]
[-- Type: text/x-patch, Size: 4216 bytes --]
From 855af6a1c94f8319867ac944d2e0d97c2c06742c Mon Sep 17 00:00:00 2001
From: Bart Van Assche <bvanassche@acm.org>
Date: Fri, 7 Feb 2020 19:57:35 -0800
Subject: [PATCH] RDMA/isert: Really fix logout
Commit 04060db41178 moved the isert_put_unsol_pending_cmds() call from
before target_wait_for_sess_cmds() to after that call. However, a comment
above isert_put_unsol_pending_cmds() is as follows: "We might still have
commands that are waiting for unsolicited dataouts messages. We must put
the extra reference on those before blocking on the
target_wait_for_session_cmds()". Make sure that
isert_put_unsol_pending_cmds() is again called before
target_wait_for_sess_cmds().
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Rahul Kundu <rahul.kundu@chelsio.com>
Cc: Mike Marciniszyn <mike.marciniszyn@intel.com>
Reported-by: Dakshaja Uppalapati <dakshaja@chelsio.com>
Fixes: 04060db41178 ("scsi: RDMA/isert: Fix a recently introduced regression related to logout")
---
drivers/infiniband/ulp/isert/ib_isert.c | 15 ++++++++++++---
drivers/target/iscsi/iscsi_target.c | 3 +++
include/target/iscsi/iscsi_transport.h | 1 +
3 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index b273e421e910..65111025cd87 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -2582,7 +2582,7 @@ isert_wait4logout(struct isert_conn *isert_conn)
*
* We might still have commands that are waiting for unsolicited
* dataouts messages. We must put the extra reference on those
- * before blocking on the target_wait_for_session_cmds
+ * before blocking on the target_wait_for_sess_cmds().
*/
static void
isert_put_unsol_pending_cmds(struct iscsi_conn *conn)
@@ -2610,11 +2610,11 @@ isert_put_unsol_pending_cmds(struct iscsi_conn *conn)
}
}
-static void isert_wait_conn(struct iscsi_conn *conn)
+static void isert_close_conn(struct iscsi_conn *conn)
{
struct isert_conn *isert_conn = conn->context;
- isert_info("Starting conn %p\n", isert_conn);
+ isert_info("Closing conn %p\n", isert_conn);
mutex_lock(&isert_conn->mutex);
isert_conn_terminate(isert_conn);
@@ -2622,6 +2622,14 @@ static void isert_wait_conn(struct iscsi_conn *conn)
ib_drain_qp(isert_conn->qp);
isert_put_unsol_pending_cmds(conn);
+}
+
+static void isert_wait_conn(struct iscsi_conn *conn)
+{
+ struct isert_conn *isert_conn = conn->context;
+
+ isert_info("Waiting for logout on conn %p\n", isert_conn);
+
isert_wait4logout(isert_conn);
queue_work(isert_release_wq, &isert_conn->release_work);
@@ -2653,6 +2661,7 @@ static struct iscsit_transport iser_target_transport = {
.iscsit_setup_np = isert_setup_np,
.iscsit_accept_np = isert_accept_np,
.iscsit_free_np = isert_free_np,
+ .iscsit_close_conn = isert_close_conn,
.iscsit_wait_conn = isert_wait_conn,
.iscsit_free_conn = isert_free_conn,
.iscsit_get_login_rx = isert_get_login_rx,
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index b94ed4e30770..0e6449c87924 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -4226,6 +4226,9 @@ int iscsit_close_connection(
atomic_set(&conn->connection_reinstatement, 1);
spin_unlock_bh(&conn->state_lock);
+ if (conn->conn_transport->iscsit_close_conn)
+ conn->conn_transport->iscsit_close_conn(conn);
+
/*
* If any other processes are accessing this connection pointer we
* must wait until they have completed.
diff --git a/include/target/iscsi/iscsi_transport.h b/include/target/iscsi/iscsi_transport.h
index 75bee29fd7dd..f5a370b9e5b3 100644
--- a/include/target/iscsi/iscsi_transport.h
+++ b/include/target/iscsi/iscsi_transport.h
@@ -14,6 +14,7 @@ struct iscsit_transport {
int (*iscsit_setup_np)(struct iscsi_np *, struct sockaddr_storage *);
int (*iscsit_accept_np)(struct iscsi_np *, struct iscsi_conn *);
void (*iscsit_free_np)(struct iscsi_np *);
+ void (*iscsit_close_conn)(struct iscsi_conn *);
void (*iscsit_wait_conn)(struct iscsi_conn *);
void (*iscsit_free_conn)(struct iscsi_conn *);
int (*iscsit_get_login_rx)(struct iscsi_conn *, struct iscsi_login *);
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: IOZONE fails intermittently while toggling the interface
2020-02-07 14:00 IOZONE fails intermittently while toggling the interface Dakshaja Uppalapati
2020-02-07 16:29 ` Bart Van Assche
2020-02-09 17:29 ` Bart Van Assche
@ 2020-02-10 14:31 ` Dakshaja Uppalapati
2020-02-11 3:47 ` Bart Van Assche
2020-02-12 5:43 ` Dakshaja Uppalapati
4 siblings, 0 replies; 6+ messages in thread
From: Dakshaja Uppalapati @ 2020-02-10 14:31 UTC (permalink / raw)
To: target-devel
[-- Attachment #1: Type: text/plain, Size: 1227 bytes --]
On Sunday, February 02/09/20, 2020 at 09:29:36 -0800, Bart Van Assche wrote:
> On 2020-02-07 06:00, Dakshaja Uppalapati wrote:
> > While testing iser with kernel 5.5, I see attached soft lockups.
> > Logged in and ran iozone on mounted disks. While the traffic is running, toggled the initiators interface down to up for 13 secs.
> > I see the attached hung task warnings in the target dmesg.
> > # iozone -a -I -+d -g 32m
> >
> > The same test with kernel-5.4 runs fine. I have suspected the following three commits which are all correlated, so removed these commits from 5.5 kernel, rebuilt the kernel and ran the same test then no issue is seen. Iozone is running successfully.
> >
> > 80647a89eaf3f2: scsi: target: core: Release SPC-2 reservations when closing a session
> > e9d3009cb936b: scsi: target: iscsi: Wait for all commands to finish before freeing a session
> > 04060db41178c: scsi: RDMA/isert: Fix a recently introduced regression related to logout
> >
> > I am trying to root cause the issue. Please do suggest me if you want me to check some thing specific.
>
> Does the attached patch help?
Hi Bart,
I had run the test multiple times with the given patch, attached traces are seen.
Thanks,
Dakshaja
[-- Attachment #2: dmesg_trace2.txt --]
[-- Type: text/plain, Size: 57685 bytes --]
[ 532.046617] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[ 532.047027] cxgb4 0000:03:00.4 ens1f4: link down
[ 545.865331] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[ 547.498413] ------------[ cut here ]------------
[ 547.499049] refcount_t: underflow; use-after-free.
[ 547.499757] WARNING: CPU: 5 PID: 801 at lib/refcount.c:28 refcount_warn_saturate+0x9b/0x100
[ 547.500464] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 547.500533] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 547.512682] CPU: 5 PID: 801 Comm: kworker/5:1H Kdump: loaded Not tainted 5.6.0-rc1+ #18
[ 547.513954] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 547.515337] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[ 547.516662] RIP: 0010:refcount_warn_saturate+0x9b/0x100
[ 547.517941] Code: e9 1e 01 01 e8 d6 73 bf ff 0f 0b c3 80 3d 98 e9 1e 01 00 75 a9 48 c7 c7 b0 f9 34 97 31 c0 c6 05 86 e9 1e 01 01 e8 b5 73 bf ff <0f> 0b c3 80 3d 7a e9 1e 01 00 75 88 48 c7 c7 58 f9 34 97 31 c0 c6
[ 547.520824] RSP: 0018:ffffa2d14065fd50 EFLAGS: 00010282
[ 547.522500] RAX: 0000000000000000 RBX: ffff8d8610a22c90 RCX: 0000000000000000
[ 547.523976] RDX: ffff8d87dc0268c0 RSI: ffff8d87dbc1a0f8 RDI: ffff8d87dbc1a0f8
[ 547.525567] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001
[ 547.526989] R10: 0000000000000000 R11: ffff8d8583af86f0 R12: 0000000000000002
[ 547.528551] R13: 0000000000000000 R14: 0000000000000010 R15: ffff8d87a488bc40
[ 547.530003] FS: 0000000000000000(0000) GS:ffff8d87dbc00000(0000) knlGS:0000000000000000
[ 547.531637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 547.533166] CR2: 00007f52e7df0140 CR3: 00000002fd810002 CR4: 00000000003606e0
[ 547.534786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 547.536466] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 547.538018] Call Trace:
[ 547.539726] transport_generic_free_cmd+0x155/0x160 [target_core_mod]
[ 547.541466] __ib_process_cq+0xc0/0x280 [ib_core]
[ 547.543121] ib_cq_poll_work+0x26/0x80 [ib_core]
[ 547.544848] process_one_work+0x202/0x580
[ 547.546620] ? process_one_work+0x17c/0x580
[ 547.548361] worker_thread+0x4c/0x3f0
[ 547.550065] kthread+0x103/0x140
[ 547.551877] ? process_one_work+0x580/0x580
[ 547.553700] ? kthread_bind+0x10/0x10
[ 547.555523] ret_from_fork+0x3a/0x50
[ 547.557310] irq event stamp: 730
[ 547.559101] hardirqs last enabled at (729): [<ffffffff96b858b4>] _raw_spin_unlock_irq+0x24/0x40
[ 547.560943] hardirqs last disabled at (730): [<ffffffff96b7e44a>] __schedule+0xda/0x990
[ 547.562821] softirqs last enabled at (726): [<ffffffff969a3a24>] flush_backlog+0x134/0x150
[ 547.564703] softirqs last disabled at (722): [<ffffffff969a38f5>] flush_backlog+0x5/0x150
[ 547.566545] ---[ end trace e964c55ac82fd95e ]---
[ 563.114653] iSCSI Login timeout on Network Portal 10.0.2.2:3260
[ 572.936188] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 572.936293] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[ 572.937403] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 572.937467] ipmi_msghandler
[ 572.938545] Modules linked in:
[ 572.947394] snd_timer
[ 572.948499] target_core_user
[ 572.949546] wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 572.950618] uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm ipmi_msghandler snd_timer wmi snd soundcore
[ 572.950643] lpc_ich
[ 572.951651] irq event stamp: 0
[ 572.951655] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 572.954755] i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 572.963700] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 572.963702] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 572.964863] irq event stamp: 0
[ 572.964865] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 572.966050] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 572.966053] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G W 5.6.0-rc1+ #18
[ 572.967213] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 572.967216] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 572.970899] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 572.970903] RIP: 0010:lock_acquire+0x3a/0x1c0
[ 572.972170] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 572.972172] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G W 5.6.0-rc1+ #18
[ 572.973451] Code: 40 8e 01 00 55 53 48 83 ec 30 8b 82 bc 0c 00 00 48 89 7c 24 20 85 c0 0f 85 ad 00 00 00 89 f5 41 89 cd 45 89 c6 4d 89 cf 9c 58 <0f> 1f 44 00 00 48 89 44 24 28 fa 66 0f 1f 44 00 00 c7 82 bc 0c 00
[ 572.974678] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 572.974682] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 572.975916] RSP: 0018:ffffa2d1430cfd40 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 572.977228] Code: 74 57 65 01 1d 32 08 d7 69 65 8b 05 2b 08 d7 69 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 d7 69 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 572.978557] RAX: 0000000000000246 RBX: ffff8d8610a23aa0 RCX: 0000000000000000
[ 572.978558] RDX: ffff8d878eb1c180 RSI: 0000000000000000 RDI: ffff8d8610a23ab8
[ 572.979827] RSP: 0018:ffffa2d14308fdd0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 572.981123] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 572.981124] R10: 0000000000000000 R11: ffffa2d1430cfdb0 R12: 0000000000000000
[ 572.982452] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 572.982453] RDX: ffff8d8786724180 RSI: 0000000000000201 RDI: ffffffff962a85f0
[ 572.983742] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 572.983743] FS: 0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[ 572.985017] RBP: ffffffffc0db70fe R08: 0000000000000000 R09: 0000000000000000
[ 572.985018] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[ 572.986355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Message from syslogd@mehrangarh at Feb 10 15:06:42 ...
kernel:watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 572.986356] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[ 572.989015] R13: ffff8d808737f800 R14: ffff8d808737fde0 R15: ffff8d86111af160
[ 572.989016] FS: 0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[ 572.990426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 572.991772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 572.991773] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[ 572.993133] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 572.993134] Call Trace:
[ 572.995871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 572.997343] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 572.998648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 572.998649] Call Trace:
[ 572.999941] _raw_spin_lock_bh+0x3d/0x50
[ 573.001326] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 573.002585] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 573.003794] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 573.005047] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 573.006314] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 573.007570] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 573.008807] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 573.010048] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 573.011296] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
Message from syslogd@mehrangarh at Feb 10 15:06:42 ...
kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[ 573.012519] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 573.013714] kthread+0x103/0x140
[ 573.014900] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 573.016085] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 573.017215] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 573.018340] ? kthread_bind+0x10/0x10
[ 573.019435] kthread+0x103/0x140
[ 573.020532] ret_from_fork+0x3a/0x50
[ 573.021599] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 573.045424] ? kthread_bind+0x10/0x10
[ 573.046408] ret_from_fork+0x3a/0x50
[ 600.936308] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 600.936414] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[ 600.937243] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 600.937275] ipmi_msghandler
[ 600.938049] Modules linked in:
[ 600.944983] snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 600.945964] target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm ipmi_msghandler snd_timer
[ 600.945985] wmi
[ 600.946934] irq event stamp: 0
[ 600.946937] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 600.949897] snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 600.958443] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 600.958445] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 600.959594] irq event stamp: 0
[ 600.959596] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 600.960731] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 600.960733] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 600.961867] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 600.961869] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 600.965365] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 600.965369] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 600.966604] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 600.966606] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 600.967804] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 55 ef 78 ff 48 89 ef e8 9d 3b 79 ff f6 c7 02 74 19 e8 b3 11 85 ff 48 89 df 57 9d <0f> 1f 44 00 00 65 ff 0d 9a 31 49 69 5b 5d c3 48 89 df 57 9d 0f 1f
[ 600.968983] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 600.968986] RIP: 0010:lock_release+0xf/0x2a0
[ 600.970199] RSP: 0018:ffffa2d1430cfd90 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 600.971426] Code: c6 d0 23 30 97 48 c7 c7 a2 65 2e 97 31 c0 e8 08 b0 f8 ff 0f 0b e9 e8 fe ff ff 90 41 57 65 48 8b 14 25 40 8e 01 00 41 56 41 55 <41> 54 55 53 48 83 ec 10 44 8b 82 bc 0c 00 00 65 48 8b 04 25 28 00
[ 600.972637] RAX: 0000000000000001 RBX: 0000000000000296 RCX: 0000000000000000
[ 600.972638] RDX: 0000000000000006 RSI: ffffffffc0d53e4a RDI: 0000000000000296
Message from syslogd@mehrangarh at Feb 10 15:07:10 ...
kernel:watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 600.973824] RSP: 0018:ffffa2d14308fdb8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 600.975009] RBP: ffff8d8610a23cf8 R08: 0000000000000000 R09: 0000000000000001
[ 600.975010] R10: 0000000000000000 R11: ffffa2d1430cfdb0 R12: 0000000000000001
[ 600.976228] RAX: 0000000000000000 RBX: ffff8d808737fe50 RCX: 0000000000000000
[ 600.976230] RDX: ffff8d8786724180 RSI: ffffffffc0db70f1 RDI: ffff8d808737fe68
[ 600.977458] R13: 0000000000000001 R14: ffff8d8610a23be0 R15: ffff8d8610a238e0
[ 600.977460] FS: 0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[ 600.978662] RBP: ffff8d86111af470 R08: 0000000000000001 R09: 0000000000000000
[ 600.978663] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[ 600.979845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 600.979846] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[ 600.982292] R13: ffff8d808737f800 R14: ffff8d808737fe50 R15: ffff8d86111af160
[ 600.982294] FS: 0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[ 600.983568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 600.984810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 600.984811] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[ 600.986052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Message from syslogd@mehrangarh at Feb 10 15:07:10 ...
kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[ 600.986053] Call Trace:
[ 600.988635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 600.989941] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 600.991230] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 600.991231] Call Trace:
[ 600.992544] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 600.993790] _raw_spin_unlock_bh+0x17/0x30
[ 600.995013] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 600.996240] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 600.997468] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 600.998638] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 600.999801] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 601.000947] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 601.002088] kthread+0x103/0x140
[ 601.003223] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 601.004345] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 601.005483] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 601.006603] ? kthread_bind+0x10/0x10
[ 601.006606] ret_from_fork+0x3a/0x50
[ 601.007701] kthread+0x103/0x140
[ 601.030430] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 601.031328] ? kthread_bind+0x10/0x10
[ 601.032240] ret_from_fork+0x3a/0x50
[ 605.130023] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[ 605.131611] cxgb4 0000:03:00.4 ens1f4: link down
[ 612.501078] rcu: INFO: rcu_sched self-detected stall on CPU
[ 612.501922] rcu: 13-....: (64820 ticks this GP) idle=bd6/1/0x4000000000000002 softirq=27999/27999 fqs=15343
[ 612.503598] (t=65002 jiffies g=69481 q=31952)
[ 612.504590] NMI backtrace for cpu 13
[ 612.505775] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 612.507577] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 612.509359] Call Trace:
[ 612.511146] <IRQ>
[ 612.512865] dump_stack+0x82/0xb3
[ 612.514613] nmi_cpu_backtrace+0x90/0xa0
[ 612.516343] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 612.518050] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 612.519773] rcu_dump_cpu_stacks+0x97/0xc5
[ 612.521482] rcu_sched_clock_irq+0x6ec/0x980
[ 612.523172] ? tick_sched_do_timer+0x70/0x70
[ 612.524829] update_process_times+0x24/0x50
[ 612.526495] tick_sched_handle+0x25/0x60
[ 612.528128] tick_sched_timer+0x37/0x70
[ 612.529720] __hrtimer_run_queues+0xe8/0x450
[ 612.531315] hrtimer_interrupt+0x122/0x270
[ 612.532882] ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 612.534466] smp_apic_timer_interrupt+0x86/0x240
[ 612.536019] apic_timer_interrupt+0xf/0x20
[ 612.537576] </IRQ>
[ 612.539071] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 612.540595] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c cf 69 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c cf 69 89 43 08 65 48 8b 04 25
[ 612.543756] RSP: 0018:ffffa2d14308fdd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 612.545368] RAX: 0000000000000000 RBX: ffff8d808737fde0 RCX: 0000000000000000
[ 612.547012] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8d808737fde0
[ 612.548630] RBP: ffff8d86111af470 R08: 0000000000000001 R09: 0000000000000000
[ 612.550237] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[ 612.551819] R13: ffff8d808737f800 R14: ffff8d808737fde0 R15: ffff8d86111af160
[ 612.553458] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 612.555043] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 612.556645] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 612.558229] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 612.559802] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 612.561401] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 612.562982] kthread+0x103/0x140
[ 612.564547] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 612.566140] ? kthread_bind+0x10/0x10
[ 612.567724] ret_from_fork+0x3a/0x50
[ 618.978720] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[ 628.936291] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 628.937158] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 628.937191] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 628.947559] irq event stamp: 0
[ 628.948639] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 628.949684] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
Message from syslogd@mehrangarh at Feb 10 15:07:38 ...
kernel:watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 628.950729] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 628.951764] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 628.952774] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 628.953800] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 628.954831] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 628.955845] Code: 74 57 65 01 1d 32 08 d7 69 65 8b 05 2b 08 d7 69 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 d7 69 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 628.957947] RSP: 0018:ffffa2d1430cfdd0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 628.959010] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 628.960117] RDX: ffff8d878eb1c180 RSI: 0000000000000201 RDI: ffffffff962a85f0
[ 628.961208] RBP: ffffffffc0db70f1 R08: 0000000000000000 R09: 0000000000000000
[ 628.962312] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 628.963440] R13: ffff8d8562ea2800 R14: ffff8d8562ea2e50 R15: ffff8d8610a238e0
[ 628.964550] FS: 0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[ 628.965630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 628.966710] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[ 628.967789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 628.968863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 628.969907] Call Trace:
[ 628.970964] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 628.972026] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 628.973100] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 628.974135] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 628.975201] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 628.976260] kthread+0x103/0x140
[ 628.977319] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 628.978418] ? kthread_bind+0x10/0x10
[ 628.979486] ret_from_fork+0x3a/0x50
[ 636.936393] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[ 636.937563] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 636.937605] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
Message from syslogd@mehrangarh at Feb 10 15:07:46 ...
kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[ 636.950692] irq event stamp: 0
[ 636.951792] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 636.952889] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 636.953960] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 636.955021] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 636.956046] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 636.957104] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 636.958169] RIP: 0010:cpumask_next+0x0/0x20
[ 636.959224] Code: 45 ff ff ff c6 00 00 eb be 48 83 c0 01 48 89 02 8d 41 ff 48 01 f8 80 38 22 75 9d c6 00 00 eb 98 cc cc cc cc cc cc cc cc cc cc <48> 89 f0 8b 35 bb 93 d6 00 83 c7 01 48 63 d7 48 89 c7 e8 39 66 b4
[ 636.961414] RSP: 0018:ffffa2d14308fd20 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 636.962540] RAX: 0000000000000003 RBX: ffffffffc0d8d660 RCX: ffff8d87db42d000
[ 636.963646] RDX: ffff8d87db000000 RSI: ffffffff978c6920 RDI: 0000000000000003
[ 636.964711] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000fff8
[ 636.965780] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 000000000042d000
[ 636.966837] R13: 0000000000000000 R14: ffff8d86111af460 R15: ffff8d86111af160
[ 636.967885] FS: 0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[ 636.968940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 636.969990] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[ 636.971038] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 636.972065] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 636.973064] Call Trace:
[ 636.974057] __is_kernel_percpu_address+0x39/0xa0
[ 636.975056] static_obj+0x25/0x50
[ 636.976050] lockdep_init_map+0x79/0x1f0
[ 636.977041] __init_waitqueue_head+0x36/0x50
[ 636.978074] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 636.979096] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 636.980072] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 636.981071] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 636.982070] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 636.983063] kthread+0x103/0x140
[ 636.984045] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 636.985056] ? kthread_bind+0x10/0x10
[ 636.986050] ret_from_fork+0x3a/0x50
[ 656.936314] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 656.937474] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 656.937513] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 656.949583] irq event stamp: 0
[ 656.950654] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 656.951731] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 656.952818] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 656.953887] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 656.954916] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 656.955949] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 656.956981] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 656.957988] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c cf 69 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c cf 69 89 43 08 65 48 8b 04 25
[ 656.959999] RSP: 0018:ffffa2d1430cfdc0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 656.961029] RAX: 0000000000000000 RBX: ffff8d8610a239f8 RCX: 0000000000000000
[ 656.962055] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8d8610a239f8
[ 656.963070] RBP: ffff8d8610a239f8 R08: 0000000000000001 R09: 0000000000000000
[ 656.964089] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 656.965167] R13: ffff8d8562ea2800 R14: ffff8d8610a23be0 R15: ffff8d8610a238e0
[ 656.966184] FS: 0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[ 656.967207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 656.968231] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[ 656.969271] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 656.970314] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 656.971391] Call Trace:
[ 656.972500] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 656.973582] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 656.974664] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 656.975741] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 656.976817] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 656.977892] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 656.978940] kthread+0x103/0x140
[ 656.979992] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 656.981046] ? kthread_bind+0x10/0x10
[ 656.982083] ret_from_fork+0x3a/0x50
[ 664.936435] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[ 664.937608] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 664.937646] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 664.950547] irq event stamp: 0
[ 664.951626] hardirqs last enabled at (0): [<0000000000000000>] 0x0
Message from syslogd@mehrangarh at Feb 10 15:08:14 ...
kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[ 664.952702] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 664.953783] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 664.954846] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 664.955900] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 664.956970] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 664.958041] RIP: 0010:__raw_spin_lock_init+0xa/0x50
[ 664.959103] Code: c7 c7 08 c0 36 97 31 c0 e8 e7 f7 00 00 5b 48 c7 c7 b8 b4 39 97 31 c0 e9 d8 f7 00 00 5b c3 cc cc 0f 1f 44 00 00 41 54 49 89 d4 <55> 48 89 f5 be 38 00 00 00 53 48 89 fb e8 d4 82 ff ff 48 8d 7b 18
[ 664.961224] RSP: 0018:ffffa2d14308fd78 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 664.962309] RAX: 0000000000000000 RBX: ffffa2d14308fdc0 RCX: 0000000000000000
[ 664.963388] RDX: ffffffff9831f530 RSI: ffffffff97300412 RDI: ffffa2d14308fdc0
[ 664.964490] RBP: ffffffffc0d67e09 R08: 0000000000000000 R09: 0000000000000000
[ 664.965580] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: ffffffff9831f530
[ 664.966667] R13: 0000000000000001 R14: ffff8d86111af460 R15: ffff8d86111af160
[ 664.967745] FS: 0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[ 664.968810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 664.969884] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[ 664.970957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 664.972011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 664.973058] Call Trace:
[ 664.974097] __init_waitqueue_head+0x25/0x50
[ 664.975143] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 664.976168] ? __local_bh_enable_ip+0x80/0xf0
[ 664.977188] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 664.978236] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 664.979289] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 664.980348] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 664.981479] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 664.982563] kthread+0x103/0x140
[ 664.983632] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 664.984703] ? kthread_bind+0x10/0x10
[ 664.985762] ret_from_fork+0x3a/0x50
[ 673.196146] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 6-... 13-... } 68051 jiffies s: 409 root: 0x2040/.
[ 673.199958] rcu: blocking rcu_node structures:
[ 673.201895] Task dump for CPU 6:
[ 673.203851] iscsi_trx R running task 0 12756 2 0x8000408c
[ 673.205900] Call Trace:
[ 673.207825] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 673.209691] ? retint_kernel+0x10/0x10
[ 673.211611] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 673.213445] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 673.215228] ? __is_kernel_percpu_address+0x39/0xa0
[ 673.216826] ? __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 673.218524] ? __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 673.220046] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 673.221765] ? iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 673.223383] ? iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 673.224960] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 673.226588] ? iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 673.228207] ? kthread+0x103/0x140
[ 673.229733] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 673.231326] ? kthread_bind+0x10/0x10
[ 673.232884] ? ret_from_fork+0x3a/0x50
[ 673.234527] Task dump for CPU 13:
[ 673.236003] iscsi_trx R running task 0 12748 2 0x8000408c
[ 673.237607] Call Trace:
[ 673.239116] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 673.240644] ? run_timer_softirq+0x52d/0x570
[ 673.242090] ? run_timer_softirq+0x52d/0x570
[ 673.243596] ? retint_kernel+0x10/0x10
[ 673.245020] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 673.246562] ? iscsit_free_all_datain_reqs+0x24/0xa0 [iscsi_target_mod]
[ 673.247962] ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 673.249398] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 673.250740] ? iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 673.252186] ? iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 673.253560] ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 673.255025] ? iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 673.256578] ? kthread+0x103/0x140
[ 673.257991] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 673.259536] ? kthread_bind+0x10/0x10
[ 673.260961] ? ret_from_fork+0x3a/0x50
[ 684.936368] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 684.937180] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 684.937210] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 684.946894] irq event stamp: 0
[ 684.947821] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 684.948772] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 684.949743] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 684.950709] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 684.951673] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 684.952663] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 684.953657] RIP: 0010:lock_acquire+0xdf/0x1c0
[ 684.954629] Code: 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d 0f 1f 44 00 00 48 83 c4 30 5b <5d> 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 38 d4 cf 69 89 c0 48 0f a3
[ 684.956615] RSP: 0018:ffffa2d1430cfd88 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 684.957650] RAX: ffff8d878eb1c180 RBX: ffff8d8610a239f8 RCX: 0000000000000000
[ 684.958685] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 684.959714] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 684.960732] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 684.961746] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 684.962756] FS: 0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[ 684.963776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 684.964769] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[ 684.965769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 684.966790] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 684.967829] Call Trace:
[ 684.968828] _raw_spin_lock_bh+0x3d/0x50
[ 684.969833] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 684.970863] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 684.971890] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 684.972977] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 684.974006] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 684.975041] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 684.976070] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 684.977092] kthread+0x103/0x140
[ 684.978104] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 684.979120] ? kthread_bind+0x10/0x10
[ 684.980119] ret_from_fork+0x3a/0x50
[ 692.936490] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[ 692.937632] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 692.937671] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
Message from syslogd@mehrangarh at Feb 10 15:08:42 ...
kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[ 692.949728] irq event stamp: 0
[ 692.950765] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 692.951807] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 692.952857] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 692.953898] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 692.954934] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 692.955975] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 692.957004] RIP: 0010:lock_release+0x0/0x2a0
[ 692.958019] Code: 83 3d b3 84 5b 01 00 0f 85 04 ff ff ff 48 c7 c6 d0 23 30 97 48 c7 c7 a2 65 2e 97 31 c0 e8 08 b0 f8 ff 0f 0b e9 e8 fe ff ff 90 <41> 57 65 48 8b 14 25 40 8e 01 00 41 56 41 55 41 54 55 53 48 83 ec
[ 692.960047] RSP: 0018:ffffa2d14308fdd0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 692.961073] RAX: 0000000000000000 RBX: ffff8d808737fe50 RCX: 0000000000000000
[ 692.962097] RDX: 0000000000000001 RSI: ffffffffc0db70f1 RDI: ffff8d808737fe68
[ 692.963115] RBP: ffff8d86111af470 R08: 0000000000000001 R09: 0000000000000000
[ 692.964139] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[ 692.965165] R13: ffff8d808737f800 R14: ffff8d808737fe50 R15: ffff8d86111af160
[ 692.966206] FS: 0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[ 692.967258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 692.968301] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[ 692.969351] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 692.970413] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 692.971509] Call Trace:
[ 692.972586] _raw_spin_unlock_bh+0x17/0x30
[ 692.973666] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 692.974745] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 692.975812] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 692.976886] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 692.977961] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 692.979021] kthread+0x103/0x140
[ 692.980049] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 692.981103] ? kthread_bind+0x10/0x10
[ 692.982141] ret_from_fork+0x3a/0x50
[ 712.936369] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[ 712.937563] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 712.937602] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 712.950360] irq event stamp: 0
[ 712.951416] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 712.952502] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 712.953594] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 712.954669] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 712.955735] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 712.956820] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 712.957901] RIP: 0010:_raw_spin_unlock_irqrestore+0x3e/0x52
[ 712.958967] Code: 24 10 e8 55 ef 78 ff 48 89 ef e8 9d 3b 79 ff f6 c7 02 74 19 e8 b3 11 85 ff 48 89 df 57 9d 0f 1f 44 00 00 65 ff 0d 9a 31 49 69 <5b> 5d c3 48 89 df 57 9d 0f 1f 44 00 00 e8 b0 10 85 ff eb e5 cc cc
[ 712.961098] RSP: 0018:ffffa2d1430cfd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 712.962194] RAX: 0000000000000001 RBX: 0000000000000296 RCX: 0000000000000000
[ 712.963247] RDX: 0000000000000006 RSI: ffffffffc0d53e4a RDI: 0000000000000296
[ 712.964289] RBP: ffff8d8610a23cf8 R08: 0000000000000000 R09: 0000000000000001
[ 712.965329] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 712.966364] R13: 0000000000000001 R14: ffff8d8610a23be0 R15: ffff8d8610a238e0
[ 712.967403] FS: 0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[ 712.968519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 712.969577] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[ 712.970632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 712.971700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 712.972754] Call Trace:
[ 712.973786] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 712.974854] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 712.975906] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 712.976946] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 712.977989] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 712.979022] kthread+0x103/0x140
[ 712.980064] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 712.981111] ? kthread_bind+0x10/0x10
[ 712.982154] ret_from_fork+0x3a/0x50
[ 720.936424] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[ 720.937584] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[ 720.937622] ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 720.950527] irq event stamp: 0
[ 720.951618] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 720.952701] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 720.953765] softirqs last enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[ 720.954818] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 720.955860] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 720.956914] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 720.957958] RIP: 0010:lock_release+0x142/0x2a0
[ 720.959010] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 720.961159] RSP: 0018:ffffa2d14308fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 720.962243] RAX: ffff8d8786724180 RBX: ffff8d8786724180 RCX: 0000000000000000
[ 720.963316] RDX: 0000000000000000 RSI: ffffffffc0db70f1 RDI: 0000000000000246
[ 720.964373] RBP: ffff8d808737fe68 R08: 0000000000000000 R09: 0000000000000000
[ 720.965462] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: ffffffffc0db70f1
[ 720.966542] R13: 0000000000000246 R14: ffff8d808737fe50 R15: ffff8d86111af160
[ 720.967600] FS: 0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[ 720.968680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 720.969754] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[ 720.970831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 720.971899] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 720.972954] Call Trace:
[ 720.973993] _raw_spin_unlock_bh+0x17/0x30
[ 720.975037] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 720.976089] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 720.977128] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 720.978149] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 720.979194] iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[ 720.980237] kthread+0x103/0x140
[ 720.981271] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[ 720.982317] ? kthread_bind+0x10/0x10
[ 720.983414] ret_from_fork+0x3a/0x50
[-- Attachment #3: dmesg_trace1.txt --]
[-- Type: text/plain, Size: 182833 bytes --]
[ 271.548371] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.573691] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.592535] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.593316] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.594096] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.613094] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.631967] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.650686] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.669852] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[ 271.688996] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[root@mehrangarh ~]# dmesg -w
[ 453.414110] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[ 453.414689] cxgb4 0000:03:00.4 ens1f4: link down
[ 467.162880] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[ 468.807914] ------------[ cut here ]------------
[ 468.808496] refcount_t: underflow; use-after-free.
[ 468.809181] WARNING: CPU: 12 PID: 753 at lib/refcount.c:28 refcount_warn_saturate+0x9b/0
[ 468.809910] Modules linked in: target_core_user uio target_core_pscsi target_core_file t_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_pxt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_ed_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_culni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 468.809974] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi ables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea salgo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uv
[ 468.817870] CPU: 12 PID: 753 Comm: kworker/12:1H Kdump: loaded Not tainted 5.6.0-rc1+ #1
[ 468.818615] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 468.819379] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[ 468.820144] RIP: 0010:refcount_warn_saturate+0x9b/0x100
[ 468.820880] Code: e9 1e 01 01 e8 d6 73 bf ff 0f 0b c3 80 3d 98 e9 1e 01 00 75 a9 48 c7 c c7 c7 58 f9 14 af 31 c0 c6
[ 468.822426] RSP: 0018:ffffa637004bbd50 EFLAGS: 00010282
[ 468.823236] RAX: 0000000000000000 RBX: ffff895068723440 RCX: 0000000000000000
[ 468.824060] RDX: ffff8951dea268c0 RSI: ffff8951de61a0f8 RDI: ffff8951de61a0f8
[ 468.824827] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001
[ 468.825621] R10: 0000000000000000 R11: ffff8951c491fef0 R12: 000000000000000f
[ 468.826460] R13: 0000000000000080 R14: 0000000000000010 R15: ffff8951b6a07680
[ 468.827300] FS: 0000000000000000(0000) GS:ffff8951de600000(0000) knlGS:0000000000000000
[ 468.828166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 468.829029] CR2: 0000555d41387060 CR3: 0000000008a10003 CR4: 00000000003606e0
[ 468.829870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 468.830768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 468.831636] Call Trace:
[ 468.832556] transport_generic_free_cmd+0x155/0x160 [target_core_mod]
[ 468.833477] __ib_process_cq+0xc0/0x280 [ib_core]
[ 468.834374] ib_cq_poll_work+0x26/0x80 [ib_core]
[ 468.835274] process_one_work+0x202/0x580
[ 468.836194] ? process_one_work+0x17c/0x580
[ 468.837106] worker_thread+0x4c/0x3f0
[ 468.838020] kthread+0x103/0x140
[ 468.838901] ? process_one_work+0x580/0x580
[ 468.839835] ? kthread_bind+0x10/0x10
[ 468.840781] ret_from_fork+0x3a/0x50
[ 468.841717] irq event stamp: 2036
[ 468.842642] hardirqs last enabled at (2035): [<ffffffffae9858b4>] _raw_spin_unlock_irq+
[ 468.843585] hardirqs last disabled at (2036): [<ffffffffae97e44a>] __schedule+0xda/0x990
[ 468.844556] softirqs last enabled at (2032): [<ffffffffae7a3a24>] flush_backlog+0x134/0
[ 468.845513] softirqs last disabled at (2028): [<ffffffffae7a38f5>] flush_backlog+0x5/0x1
[ 468.846456] ---[ end trace b89f796ccb2b161b ]---
[ 779.866619] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[ 779.867557] cxgb4 0000:03:00.4 ens1f4: link down
[ 793.645349] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[ 826.926701] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[ 826.927629] cxgb4 0000:03:00.4 ens1f4: link down
[ 840.735421] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[ 857.562340] iSCSI Login timeout on Network Portal 10.0.2.2:3260
[ 867.136473] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 867.137485] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 867.137547] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 867.148103] irq event stamp: 0
[ 867.149010] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 867.149971] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 867.150932] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 867.151865] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 867.152792] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W 5.6.0-rc1+ #18
[ 867.153732] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 867.154656] RIP: 0010:do_raw_spin_lock+0x5c/0xa0
[ 867.155586] Code: 47 31 c0 ba 01 00 00 00 f0 0f b1 13 0f 94 c2 84 d2 74 27 65 8b 05 0c 8c ef 51 89 43 08 65 48 8b 04 25 40 8e 01 00 48 89 43 10 <5b> c3 48 c7 c6 c8 41 10 af e8 07 03 00 00 eb a7 89 c6 48 89 df e8
[ 867.157466] RSP: 0018:ffffa6370223fd98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 867.158429] RAX: ffff8951948fa0c0 RBX: ffff89506770ce50 RCX: 0000000000000000
[ 867.159394] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff89506770ce50
[ 867.160359] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[ 867.161327] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 867.162300] R13: ffff89506770c800 R14: ffff89506770ce50 R15: ffff895065a14a90
[ 867.163257] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 867.164247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 867.165247] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 867.166259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 867.167273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 867.168285] Call Trace:
[ 867.169305] __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[ 867.170322] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 867.171337] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 867.172360] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 867.173373] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 867.174395] ? remove_wait_queue+0x60/0x60
[ 867.175408] kthread+0x103/0x140
[ 867.176400] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 867.177420] ? kthread_bind+0x10/0x10
[ 867.178437] ret_from_fork+0x3a/0x50
Message from syslogd@mehrangarh at Feb 10 15:33:58 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 895.136221] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 895.137439] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 895.137460] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 895.150359] irq event stamp: 0
[ 895.151458] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 895.152546] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 895.153623] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 895.154660] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 895.155686] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 895.156739] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 895.157795] RIP: 0010:__is_kernel_percpu_address+0x4e/0xa0
[ 895.158841] Code: 00 00 00 53 48 89 fb 48 c7 c6 20 69 6c af 89 c7 e8 a7 34 6e 00 3b 05 65 c8 44 01 73 4c 89 c2 48 89 e9 48 03 0c d5 80 59 1c af <48> 39 d9 48 89 ca 77 d5 4a 8d 0c 21 48 39 cb 73 cc 4d 85 ed 74 32
[ 895.160972] RSP: 0018:ffffa6370223fce8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 895.162023] RAX: 000000000000000d RBX: ffffffffc0ebe660 RCX: ffff8951dec00000
[ 895.163048] RDX: 000000000000000d RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 895.164088] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000e000
[ 895.165089] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 895.166101] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 895.167116] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 895.168141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 895.169158] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 895.170152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 895.171164] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 895.172139] Call Trace:
[ 895.173132] static_obj+0x25/0x50
[ 895.174103] lockdep_init_map+0x79/0x1f0
[ 895.175092] __init_waitqueue_head+0x36/0x50
[ 895.176058] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 895.177054] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 895.178046] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 895.179019] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 895.180019] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
Message from syslogd@mehrangarh at Feb 10 15:34:26 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 895.181012] ? remove_wait_queue+0x60/0x60
[ 895.182005] kthread+0x103/0x140
[ 895.182994] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 895.183987] ? kthread_bind+0x10/0x10
[ 895.184979] ret_from_fork+0x3a/0x50
[ 907.379831] rcu: INFO: rcu_sched self-detected stall on CPU
[ 907.380817] rcu: 11-....: (64908 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=16230
[ 907.382776] (t=65000 jiffies g=68413 q=33708)
[ 907.383730] NMI backtrace for cpu 11
[ 907.385490] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 907.387402] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 907.389266] Call Trace:
[ 907.391022] <IRQ>
[ 907.392690] dump_stack+0x82/0xb3
[ 907.394377] nmi_cpu_backtrace+0x90/0xa0
[ 907.395963] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 907.397530] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 907.399058] rcu_dump_cpu_stacks+0x97/0xc5
[ 907.400560] rcu_sched_clock_irq+0x6ec/0x980
[ 907.402046] ? tick_sched_do_timer+0x70/0x70
[ 907.403521] update_process_times+0x24/0x50
[ 907.404996] tick_sched_handle+0x25/0x60
[ 907.406464] tick_sched_timer+0x37/0x70
[ 907.407866] __hrtimer_run_queues+0xe8/0x450
[ 907.409292] hrtimer_interrupt+0x122/0x270
[ 907.410635] ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 907.412051] smp_apic_timer_interrupt+0x86/0x240
[ 907.413394] apic_timer_interrupt+0xf/0x20
[ 907.414659] </IRQ>
[ 907.415923] RIP: 0010:lock_release+0x142/0x2a0
[ 907.417209] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 907.419901] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 907.421333] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 907.422706] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 907.424108] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 907.425499] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 907.426863] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 907.428317] ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 907.429691] ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 907.431097] _raw_spin_unlock_bh+0x17/0x30
[ 907.432502] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 907.433888] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 907.435322] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 907.436689] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 907.438148] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 907.439511] ? remove_wait_queue+0x60/0x60
[ 907.440901] kthread+0x103/0x140
[ 907.442332] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 907.443714] ? kthread_bind+0x10/0x10
[ 907.445122] ret_from_fork+0x3a/0x50
[ 935.135942] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 935.136806] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 935.136838] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 935.146955] irq event stamp: 0
[ 935.147949] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 935.148945] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 935.150016] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 935.150950] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 935.151901] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 935.152847] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 935.153834] RIP: 0010:lock_acquire+0xd5/0x1c0
Message from syslogd@mehrangarh at Feb 10 15:35:06 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 935.154807] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 935.156848] RSP: 0018:ffffa6370223fd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 935.157878] RAX: ffff8951948fa0c0 RBX: 0000000000000246 RCX: 0000000000000000
[ 935.158913] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 935.160016] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 935.161037] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 935.162054] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 935.163040] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 935.164025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 935.165025] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 935.166016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 935.167019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 935.168018] Call Trace:
[ 935.169010] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 935.170006] _raw_spin_lock_bh+0x3d/0x50
[ 935.171016] ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 935.172037] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 935.173058] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 935.174079] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 935.175097] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 935.176115] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 935.177118] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 935.178139] ? remove_wait_queue+0x60/0x60
[ 935.179143] kthread+0x103/0x140
[ 935.180152] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 935.181156] ? kthread_bind+0x10/0x10
[ 935.182167] ret_from_fork+0x3a/0x50
[ 963.135798] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 963.136936] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 963.136962] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 963.148871] irq event stamp: 0
[ 963.149921] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 963.150979] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 963.152059] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 963.153059] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 963.154028] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 963.155023] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 963.156018] RIP: 0010:iscsit_free_r2ts_from_list+0x10/0x70 [iscsi_target_mod]
[ 963.157006] Code: d8 01 00 00 be ff ff ff ff e8 0c 8e 22 ed 85 c0 75 ab 0f 0b eb a7 0f 1f 40 00 0f 1f 44 00 00 41 55 4c 8d af c0 01 00 00 41 54 <55> 48 89 fd 4c 89 ef 4c 8d a5 08 02 00 00 53 e8 0c de a9 ed 48 8b
[ 963.158989] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 963.160009] RAX: 0000000000000000 RBX: ffff895065a14a90 RCX: 0000000000000000
[ 963.161022] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffff895065a14a90
[ 963.162043] RBP: ffff895065a14da0 R08: 0000000000000000 R09: 0000000000000000
[ 963.163084] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 963.164123] R13: ffff895065a14c50 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 963.165141] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 963.166185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 963.167227] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 963.168265] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 963.169298] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 963.170340] Call Trace:
[ 963.171379] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 963.172422] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 963.173444] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 963.174487] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 963.175520] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 963.176577] ? remove_wait_queue+0x60/0x60
[ 963.177612] kthread+0x103/0x140
[ 963.178629] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 963.179658] ? kthread_bind+0x10/0x10
[ 963.180687] ret_from_fork+0x3a/0x50
[ 984.025504] INFO: task iscsi_np:7853 blocked for more than 122 seconds.
[ 984.027401] Tainted: G W L 5.6.0-rc1+ #18
[ 984.029326] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 984.031370] iscsi_np D 0 7853 2 0x80004084
[ 984.033356] Call Trace:
[ 984.035312] ? __schedule+0x38c/0x990
[ 984.037350] schedule+0x5f/0xd0
[ 984.039211] schedule_timeout+0x1e4/0x310
[ 984.041057] ? wait_for_completion+0x49/0x190
[ 984.042900] wait_for_completion+0x12e/0x190
[ 984.044712] ? wake_up_q+0xa0/0xa0
[ 984.046268] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 984.047983] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 984.049674] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 984.051169] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 984.052806] kthread+0x103/0x140
[ 984.054412] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 984.055970] ? kthread_bind+0x10/0x10
[ 984.057541] ret_from_fork+0x3a/0x50
[ 984.059021] INFO: lockdep is turned off.
[ 991.135707] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 991.136638] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 991.136668] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 991.147124] irq event stamp: 0
[ 991.148106] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 991.149103] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 991.150110] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
Message from syslogd@mehrangarh at Feb 10 15:36:02 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 991.151077] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 991.152037] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 991.153013] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 991.153994] RIP: 0010:__is_kernel_percpu_address+0x41/0xa0
[ 991.154951] Code: 00 49 81 ec 00 00 00 00 55 48 c7 c5 00 00 00 00 53 48 89 fb 48 c7 c6 20 69 6c af 89 c7 e8 a7 34 6e 00 3b 05 65 c8 44 01 73 4c <89> c2 48 89 e9 48 03 0c d5 80 59 1c af 48 39 d9 48 89 ca 77 d5 4a
[ 991.156945] RSP: 0018:ffffa6370223fce8 EFLAGS: 00000283 ORIG_RAX: ffffffffffffff13
[ 991.157964] RAX: 000000000000000e RBX: ffffffffc0ebe660 RCX: 0000000000000000
[ 991.158965] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 991.159964] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000c000
[ 991.160981] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 991.161973] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 991.162958] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 991.163957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 991.164951] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 991.165953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 991.166959] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 991.167945] Call Trace:
[ 991.168948] static_obj+0x25/0x50
[ 991.169950] lockdep_init_map+0x79/0x1f0
[ 991.170953] __init_waitqueue_head+0x36/0x50
[ 991.171967] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 991.172982] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 991.173971] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 991.174964] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 991.175979] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 991.176990] ? remove_wait_queue+0x60/0x60
[ 991.177992] kthread+0x103/0x140
[ 991.178969] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 991.179960] ? kthread_bind+0x10/0x10
[ 991.180967] ret_from_fork+0x3a/0x50
[ 1019.135673] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1019.136829] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1019.136854] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1019.148830] irq event stamp: 0
[ 1019.149870] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1019.150923] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1019.151990] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1019.153005] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1019.154006] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1019.155002] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1019.155993] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 1019.156982] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 55 ef 78 ff 48 89 ef e8 9d 3b 79 ff f6 c7 02 74 19 e8 b3 11 85 ff 48 89 df 57 9d <0f> 1f 44 00 00 65 ff 0d 9a 31 69 51 5b 5d c3 48 89 df 57 9d 0f 1f
[ 1019.158939] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 1019.159927] RAX: 0000000000000001 RBX: 0000000000000296 RCX: 0000000000000000
[ 1019.160919] RDX: 000000000000000b RSI: ffffffffc0e84e4a RDI: 0000000000000296
[ 1019.161931] RBP: ffff895065a14ea8 R08: 0000000000000000 R09: 0000000000000001
[ 1019.162946] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1019.163963] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1019.164974] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1019.165988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1019.166997] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1019.168014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1019.169030] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1019.170041] Call Trace:
[ 1019.171064] transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 1019.172104] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1019.173139] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1019.174177] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1019.175201] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1019.176238] ? remove_wait_queue+0x60/0x60
[ 1019.177268] kthread+0x103/0x140
[ 1019.178288] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1019.179298] ? kthread_bind+0x10/0x10
[ 1019.180322] ret_from_fork+0x3a/0x50
[ 1047.135629] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1047.136825] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1047.136848] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1047.149702] irq event stamp: 0
[ 1047.150766] hardirqs last enabled at (0): [<0000000000000000>] 0x0
Message from syslogd@mehrangarh at Feb 10 15:36:58 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1047.151853] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1047.152927] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1047.153951] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1047.154970] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1047.156013] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1047.157062] RIP: 0010:lock_release+0x142/0x2a0
[ 1047.158097] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1047.160204] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1047.161261] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1047.162302] RDX: 0000000000000000 RSI: ffffffffc0ee80fe RDI: 0000000000000246
[ 1047.163336] RBP: ffff89506770cdf8 R08: 0000000000000000 R09: 0000000000000000
[ 1047.164334] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80fe
[ 1047.165339] R13: 0000000000000246 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1047.166356] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1047.167385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1047.168423] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1047.169442] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1047.170474] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1047.171469] Call Trace:
[ 1047.172487] _raw_spin_unlock_bh+0x17/0x30
[ 1047.173521] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1047.174554] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1047.175576] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1047.176606] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1047.177692] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1047.178727] ? remove_wait_queue+0x60/0x60
[ 1047.179776] kthread+0x103/0x140
[ 1047.180795] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1047.181805] ? kthread_bind+0x10/0x10
[ 1047.182825] ret_from_fork+0x3a/0x50
[ 1075.135523] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1075.136720] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1075.136744] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1075.149337] irq event stamp: 0
Message from syslogd@mehrangarh at Feb 10 15:37:26 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1075.150374] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1075.151422] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1075.152498] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1075.153528] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1075.154602] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1075.155645] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1075.156685] RIP: 0010:_raw_spin_unlock_bh+0x5/0x30
[ 1075.157713] Code: 48 89 fb 48 8d 7f 18 e8 99 f6 78 ff 48 89 df e8 e1 42 79 ff 65 ff 0d f2 38 69 51 5b c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <53> 48 8b 74 24 08 48 89 fb 48 8d 7f 18 e8 69 f6 78 ff 48 89 df e8
[ 1075.159780] RSP: 0018:ffffa6370223fda0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1075.160800] RAX: ffff895065a14c98 RBX: ffff895065a14a90 RCX: 0000000000000000
[ 1075.161832] RDX: ffff895065a14c98 RSI: 0000000000000000 RDI: ffff895065a14c50
[ 1075.162858] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[ 1075.163879] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1075.164891] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1075.165914] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1075.166931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1075.167961] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1075.168992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1075.170016] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1075.171037] Call Trace:
[ 1075.172049] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1075.173098] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1075.174112] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1075.175131] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1075.176198] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1075.177219] ? remove_wait_queue+0x60/0x60
[ 1075.178239] kthread+0x103/0x140
[ 1075.179251] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1075.180288] ? kthread_bind+0x10/0x10
[ 1075.181310] ret_from_fork+0x3a/0x50
[ 1102.382142] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1102.383188] rcu: 11-....: (259573 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=64897
[ 1102.384357] (t=260003 jiffies g=68413 q=44407)
[ 1102.385505] NMI backtrace for cpu 11
[ 1102.386852] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1102.388898] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1102.390929] Call Trace:
[ 1102.392907] <IRQ>
[ 1102.394761] dump_stack+0x82/0xb3
[ 1102.396614] nmi_cpu_backtrace+0x90/0xa0
[ 1102.398356] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1102.400051] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1102.401788] rcu_dump_cpu_stacks+0x97/0xc5
[ 1102.403461] rcu_sched_clock_irq+0x6ec/0x980
[ 1102.405023] ? tick_sched_do_timer+0x70/0x70
[ 1102.406624] update_process_times+0x24/0x50
[ 1102.408100] tick_sched_handle+0x25/0x60
[ 1102.409665] tick_sched_timer+0x37/0x70
[ 1102.411091] __hrtimer_run_queues+0xe8/0x450
[ 1102.412659] hrtimer_interrupt+0x122/0x270
[ 1102.414115] ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1102.415636] smp_apic_timer_interrupt+0x86/0x240
[ 1102.417044] apic_timer_interrupt+0xf/0x20
[ 1102.418537] </IRQ>
[ 1102.419888] RIP: 0010:__is_kernel_percpu_address+0x91/0xa0
[ 1102.421233] Code: 49 89 5d 00 48 63 05 5e 14 ea 01 48 03 1c c5 80 59 1c af b8 01 00 00 00 48 01 dd 5b 49 89 6d 00 5d 41 5c 41 5d c3 31 c0 5b 5d <41> 5c 41 5d c3 b8 01 00 00 00 eb f2 0f 1f 00 0f 1f 44 00 00 31 f6
[ 1102.424069] RSP: 0018:ffffa6370223fcf8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1102.425575] RAX: 0000000000000000 RBX: ffffffffc0ebe660 RCX: 0000000000000010
[ 1102.426993] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1102.428511] RBP: ffffffffc0ebe660 R08: 0000000000000000 R09: 0000000000008000
[ 1102.429927] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1102.431383] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1102.432805] static_obj+0x25/0x50
[ 1102.434186] lockdep_init_map+0x79/0x1f0
[ 1102.435694] __init_waitqueue_head+0x36/0x50
[ 1102.437089] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1102.438589] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1102.439977] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1102.441437] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1102.442882] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1102.444296] ? remove_wait_queue+0x60/0x60
[ 1102.445738] kthread+0x103/0x140
[ 1102.447100] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1102.448572] ? kthread_bind+0x10/0x10
[ 1102.449947] ret_from_fork+0x3a/0x50
[ 1106.905189] INFO: task iscsi_np:7853 blocked for more than 245 seconds.
[ 1106.906566] Tainted: G W L 5.6.0-rc1+ #18
[ 1106.907916] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1106.909399] iscsi_np D 0 7853 2 0x80004084
[ 1106.910762] Call Trace:
[ 1106.912196] ? __schedule+0x38c/0x990
[ 1106.913533] schedule+0x5f/0xd0
[ 1106.914900] schedule_timeout+0x1e4/0x310
[ 1106.916261] ? wait_for_completion+0x49/0x190
[ 1106.917555] wait_for_completion+0x12e/0x190
[ 1106.918923] ? wake_up_q+0xa0/0xa0
[ 1106.920314] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1106.921641] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1106.923045] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1106.924553] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1106.926007] kthread+0x103/0x140
[ 1106.927358] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1106.928857] ? kthread_bind+0x10/0x10
[ 1106.930320] ret_from_fork+0x3a/0x50
[ 1106.931678] INFO: task sosreport:8582 blocked for more than 122 seconds.
[ 1106.933131] Tainted: G W L 5.6.0-rc1+ #18
[ 1106.934651] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1106.936132] sosreport D 0 8582 8581 0x00000080
[ 1106.937680] Call Trace:
[ 1106.939173] ? __schedule+0x38c/0x990
[ 1106.940620] ? avc_has_perm_noaudit+0x37/0x250
[ 1106.942108] schedule+0x5f/0xd0
[ 1106.943618] schedule_timeout+0x1e4/0x310
[ 1106.945022] ? wait_for_completion+0x49/0x190
[ 1106.946519] wait_for_completion+0x12e/0x190
[ 1106.947908] ? wake_up_q+0xa0/0xa0
[ 1106.949165] __wait_rcu_gp+0xfd/0x140
[ 1106.950607] synchronize_rcu.part.75+0x3e/0x70
[ 1106.951963] ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1106.953335] ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1106.954621] rcu_sync_enter+0x54/0xf0
[ 1106.955974] percpu_down_write+0x15/0xe0
[ 1106.957342] ? seq_read+0x327/0x3e0
[ 1106.958591] locks_start+0x28/0x50
[ 1106.959913] seq_read+0x1da/0x3e0
[ 1106.961229] proc_reg_read+0x3c/0x70
[ 1106.962444] vfs_read+0xac/0x160
[ 1106.963763] ksys_read+0x5c/0xd0
[ 1106.965020] do_syscall_64+0x60/0x270
[ 1106.966272] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1106.967441] RIP: 0033:0x7f37d1ec3f70
[ 1106.968726] Code: Bad RIP value.
[ 1106.969974] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1106.971227] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1106.972487] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1106.973762] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1106.975005] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1106.976185] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1106.977384] INFO: lockdep is turned off.
[ 1127.135327] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1127.136074] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1127.136106] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1127.145283] irq event stamp: 0
Message from syslogd@mehrangarh at Feb 10 15:38:18 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1127.146134] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1127.146989] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1127.147875] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1127.148752] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1127.149629] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1127.150506] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1127.151389] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 1127.152213] Code: fa 66 0f 1f 44 00 00 65 01 35 0c 10 f7 51 65 8b 05 05 10 f7 51 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 cd de 06 00 48 89 df 57 9d <0f> 1f 44 00 00 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 1127.154020] RSP: 0018:ffffa6370223fd58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1127.154972] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 1127.155941] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: 0000000000000246
[ 1127.156908] RBP: ffff895065a14a90 R08: 0000000000000000 R09: 0000000000000000
[ 1127.157873] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffff895065a14c98
[ 1127.158826] R13: ffff895065a14c50 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1127.159797] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1127.160780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1127.161767] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1127.162766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1127.163768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1127.164768] Call Trace:
[ 1127.165767] _raw_spin_lock_bh+0x1c/0x50
[ 1127.166776] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 1127.167799] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1127.168823] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1127.169848] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1127.170884] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1127.171918] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1127.172944] ? remove_wait_queue+0x60/0x60
[ 1127.173965] kthread+0x103/0x140
[ 1127.174983] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1127.175993] ? kthread_bind+0x10/0x10
[ 1127.177019] ret_from_fork+0x3a/0x50
[ 1155.135221] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1155.136412] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1155.136435] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1155.148788] irq event stamp: 0
Message from syslogd@mehrangarh at Feb 10 15:38:46 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1155.149801] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1155.150826] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1155.151842] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1155.152873] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1155.153897] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1155.154928] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1155.155943] RIP: 0010:lock_release+0x142/0x2a0
[ 1155.156947] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1155.159007] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1155.160033] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1155.161053] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1155.162071] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1155.163102] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1155.164105] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1155.165108] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1155.166114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1155.167133] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1155.168157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1155.169177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1155.170201] Call Trace:
[ 1155.171287] _raw_spin_unlock_bh+0x17/0x30
[ 1155.172319] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1155.173352] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1155.174379] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1155.175412] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1155.176452] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1155.177474] ? remove_wait_queue+0x60/0x60
[ 1155.178479] kthread+0x103/0x140
[ 1155.179477] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1155.180486] ? kthread_bind+0x10/0x10
[ 1155.181507] ret_from_fork+0x3a/0x50
[ 1183.135153] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1183.136353] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1183.136377] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1183.148928] irq event stamp: 0
[ 1183.149969] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1183.151016] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
Message from syslogd@mehrangarh at Feb 10 15:39:14 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1183.152046] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1183.153057] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1183.154077] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1183.155089] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1183.156124] RIP: 0010:_find_next_bit.constprop.1+0x66/0x70
[ 1183.157196] Code: 8b 0c d7 74 04 4e 23 0c d6 4d 31 c1 75 0d 48 83 c1 40 48 39 ca 77 de 48 89 d0 c3 f3 49 0f bc c1 48 01 c8 48 39 d0 48 0f 47 c2 <c3> 66 0f 1f 84 00 00 00 00 00 45 31 c0 eb 8b 90 66 2e 0f 1f 84 00
[ 1183.159300] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000283 ORIG_RAX: ffffffffffffff13
[ 1183.160336] RAX: 0000000000000004 RBX: ffffffffc0ebe660 RCX: 0000000000000000
[ 1183.161371] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1183.162399] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000fff0
[ 1183.163426] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1183.164448] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1183.165465] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1183.166486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1183.167503] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1183.168526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1183.169524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1183.170536] Call Trace:
[ 1183.171550] cpumask_next+0x17/0x20
[ 1183.172553] __is_kernel_percpu_address+0x39/0xa0
[ 1183.173564] static_obj+0x25/0x50
[ 1183.174565] lockdep_init_map+0x79/0x1f0
[ 1183.175565] __init_waitqueue_head+0x36/0x50
[ 1183.176552] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1183.177564] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1183.178566] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1183.179569] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1183.180577] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1183.181583] ? remove_wait_queue+0x60/0x60
[ 1183.182583] kthread+0x103/0x140
[ 1183.183581] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1183.184581] ? kthread_bind+0x10/0x10
[ 1183.185563] ret_from_fork+0x3a/0x50
[ 1211.135147] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1211.136267] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1211.136286] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1211.147682] irq event stamp: 0
Message from syslogd@mehrangarh at Feb 10 15:39:42 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1211.148694] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1211.149722] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1211.150753] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1211.151746] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1211.152708] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1211.153690] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1211.154652] RIP: 0010:transport_generic_free_cmd+0x58/0x160 [target_core_mod]
[ 1211.155601] Code: 04 25 28 00 00 00 48 89 44 24 60 31 c0 e8 c0 a6 27 ed 85 ed c6 44 24 06 00 c6 44 24 07 00 0f 85 9b 00 00 00 f6 43 29 01 74 0a <f6> 43 06 02 0f 85 cd 00 00 00 80 7c 24 06 00 74 0c 48 8d 44 24 10
[ 1211.157568] RSP: 0018:ffffa6370223fd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1211.158575] RAX: 0000000000000001 RBX: ffff895065a14da0 RCX: 0000000000000000
[ 1211.159570] RDX: 000000000000000b RSI: ffffffffc0e84e4a RDI: 0000000000000296
[ 1211.160581] RBP: ffff895065a14ea8 R08: 0000000000000000 R09: 0000000000000001
[ 1211.161589] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1211.162574] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1211.163572] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1211.164579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1211.165590] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1211.166604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1211.167620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1211.168629] Call Trace:
[ 1211.169644] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1211.170667] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1211.171702] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1211.172738] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1211.173765] ? remove_wait_queue+0x60/0x60
[ 1211.174767] kthread+0x103/0x140
[ 1211.175781] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1211.176800] ? kthread_bind+0x10/0x10
[ 1211.177791] ret_from_fork+0x3a/0x50
[ 1229.784929] INFO: task iscsi_np:7853 blocked for more than 368 seconds.
[ 1229.786796] Tainted: G W L 5.6.0-rc1+ #18
[ 1229.788694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1229.790784] iscsi_np D 0 7853 2 0x80004084
[ 1229.792835] Call Trace:
[ 1229.794782] ? __schedule+0x38c/0x990
[ 1229.796717] schedule+0x5f/0xd0
[ 1229.798633] schedule_timeout+0x1e4/0x310
[ 1229.800384] ? wait_for_completion+0x49/0x190
[ 1229.802226] wait_for_completion+0x12e/0x190
[ 1229.803981] ? wake_up_q+0xa0/0xa0
[ 1229.805554] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1229.807369] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1229.808991] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1229.810578] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1229.812228] kthread+0x103/0x140
[ 1229.813771] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1229.815344] ? kthread_bind+0x10/0x10
[ 1229.816853] ret_from_fork+0x3a/0x50
[ 1229.818407] INFO: task sosreport:8582 blocked for more than 245 seconds.
[ 1229.819984] Tainted: G W L 5.6.0-rc1+ #18
[ 1229.821556] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1229.823138] sosreport D 0 8582 8581 0x00000080
[ 1229.824567] Call Trace:
[ 1229.826086] ? __schedule+0x38c/0x990
[ 1229.827527] ? avc_has_perm_noaudit+0x37/0x250
[ 1229.828984] schedule+0x5f/0xd0
[ 1229.830284] schedule_timeout+0x1e4/0x310
[ 1229.831712] ? wait_for_completion+0x49/0x190
[ 1229.833162] wait_for_completion+0x12e/0x190
[ 1229.834661] ? wake_up_q+0xa0/0xa0
[ 1229.836146] __wait_rcu_gp+0xfd/0x140
[ 1229.837500] synchronize_rcu.part.75+0x3e/0x70
[ 1229.838864] ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1229.840304] ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1229.841524] rcu_sync_enter+0x54/0xf0
[ 1229.842840] percpu_down_write+0x15/0xe0
[ 1229.844237] ? seq_read+0x327/0x3e0
[ 1229.845609] locks_start+0x28/0x50
[ 1229.846936] seq_read+0x1da/0x3e0
[ 1229.848216] proc_reg_read+0x3c/0x70
[ 1229.849636] vfs_read+0xac/0x160
[ 1229.850834] ksys_read+0x5c/0xd0
[ 1229.852130] do_syscall_64+0x60/0x270
[ 1229.853313] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1229.854604] RIP: 0033:0x7f37d1ec3f70
[ 1229.855872] Code: Bad RIP value.
[ 1229.857136] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1229.858305] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1229.859573] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1229.860752] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1229.861984] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1229.863156] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1229.864482] INFO: lockdep is turned off.
[ 1239.135142] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1239.135898] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1239.135929] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1239.145136] irq event stamp: 0
[ 1239.146022] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1239.146901] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1239.147748] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1239.148606] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1239.149481] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
Message from syslogd@mehrangarh at Feb 10 15:40:10 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1239.150373] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1239.151275] RIP: 0010:_find_next_bit.constprop.1+0x19/0x70
[ 1239.152121] Code: c0 48 01 d0 c3 48 89 f0 f3 c3 0f 1f 80 00 00 00 00 48 39 d1 73 4e 48 89 c8 48 c1 e8 06 48 85 f6 4c 8b 0c c7 74 04 4c 23 0c c6 <48> c7 c0 ff ff ff ff 4d 31 c1 48 d3 e0 48 83 e1 c0 49 21 c1 74 1b
[ 1239.153959] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1239.154889] RAX: 0000000000000000 RBX: ffffffffc0ebe660 RCX: 000000000000000e
[ 1239.155824] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1239.156784] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000ffff
[ 1239.157754] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1239.158722] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1239.159691] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1239.160651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1239.161628] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1239.162621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1239.163616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1239.164611] Call Trace:
[ 1239.165587] cpumask_next+0x17/0x20
[ 1239.166584] __is_kernel_percpu_address+0x39/0xa0
[ 1239.167589] static_obj+0x25/0x50
[ 1239.168587] lockdep_init_map+0x79/0x1f0
[ 1239.169565] __init_waitqueue_head+0x36/0x50
[ 1239.170557] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1239.171570] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1239.172570] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1239.173557] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1239.174569] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1239.175561] ? remove_wait_queue+0x60/0x60
[ 1239.176567] kthread+0x103/0x140
[ 1239.177566] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1239.178567] ? kthread_bind+0x10/0x10
[ 1239.179548] ret_from_fork+0x3a/0x50
[ 1267.135137] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1267.136257] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1267.136276] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1267.147700] irq event stamp: 0
[ 1267.148682] hardirqs last enabled at (0): [<0000000000000000>] 0x0
Message from syslogd@mehrangarh at Feb 10 15:40:38 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1267.149712] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1267.150726] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1267.151720] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1267.152703] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1267.153683] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1267.154654] RIP: 0010:lock_acquire+0x20/0x1c0
[ 1267.155612] Code: 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54 41 89 d4 65 48 8b 14 25 40 8e 01 00 55 53 48 83 ec 30 8b 82 bc 0c 00 00 <48> 89 7c 24 20 85 c0 0f 85 ad 00 00 00 89 f5 41 89 cd 45 89 c6 4d
[ 1267.157579] RSP: 0018:ffffa6370223fd28 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 1267.158586] RAX: 0000000000000000 RBX: ffff89506770cde0 RCX: 0000000000000000
[ 1267.159598] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: ffff89506770cdf8
[ 1267.160599] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[ 1267.161611] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1267.162618] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1267.163622] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1267.164638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1267.165655] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1267.166660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1267.167685] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1267.168706] Call Trace:
[ 1267.169725] _raw_spin_lock_bh+0x3d/0x50
[ 1267.170749] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1267.171783] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1267.172799] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1267.173830] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1267.174852] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1267.175903] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1267.176900] ? remove_wait_queue+0x60/0x60
[ 1267.177889] kthread+0x103/0x140
[ 1267.178880] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1267.179904] ? kthread_bind+0x10/0x10
[ 1267.180918] ret_from_fork+0x3a/0x50
[ 1295.135131] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1295.136324] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1295.136343] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1295.148740] irq event stamp: 0
[ 1295.149740] hardirqs last enabled at (0): [<0000000000000000>] 0x0
Message from syslogd@mehrangarh at Feb 10 15:41:06 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1295.150752] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1295.151771] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1295.152784] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1295.153814] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1295.154834] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1295.155858] RIP: 0010:lock_release+0x142/0x2a0
[ 1295.156882] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1295.158947] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1295.159947] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1295.160945] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1295.161939] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1295.162962] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1295.163987] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1295.165006] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1295.166031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1295.167037] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1295.168055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1295.169075] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1295.170096] Call Trace:
[ 1295.171138] _raw_spin_unlock_bh+0x17/0x30
[ 1295.172209] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1295.173281] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1295.174291] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1295.175306] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1295.176342] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1295.177371] ? remove_wait_queue+0x60/0x60
[ 1295.178376] kthread+0x103/0x140
[ 1295.179389] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1295.180413] ? kthread_bind+0x10/0x10
[ 1295.181506] ret_from_fork+0x3a/0x50
[ 1297.384853] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1297.385888] rcu: 11-....: (454193 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=113554
[ 1297.387143] (t=455006 jiffies g=68413 q=53587)
[ 1297.388237] NMI backtrace for cpu 11
[ 1297.389469] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1297.391434] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1297.393367] Call Trace:
[ 1297.395270] <IRQ>
[ 1297.397059] dump_stack+0x82/0xb3
[ 1297.398791] nmi_cpu_backtrace+0x90/0xa0
[ 1297.400554] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1297.402254] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1297.403818] rcu_dump_cpu_stacks+0x97/0xc5
[ 1297.405460] rcu_sched_clock_irq+0x6ec/0x980
[ 1297.406994] ? tick_sched_do_timer+0x70/0x70
[ 1297.408524] update_process_times+0x24/0x50
[ 1297.410024] tick_sched_handle+0x25/0x60
[ 1297.411530] tick_sched_timer+0x37/0x70
[ 1297.413023] __hrtimer_run_queues+0xe8/0x450
[ 1297.414514] hrtimer_interrupt+0x122/0x270
[ 1297.415973] ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1297.417445] smp_apic_timer_interrupt+0x86/0x240
[ 1297.418840] apic_timer_interrupt+0xf/0x20
[ 1297.420274] </IRQ>
[ 1297.421576] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1297.422893] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1297.425691] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1297.427145] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1297.428596] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1297.430067] RBP: ffffffffc0ee80fe R08: 0000000000000000 R09: 0000000000000000
[ 1297.431489] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1297.432884] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1297.434335] ? __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1297.435723] ? __local_bh_enable_ip+0x80/0xf0
[ 1297.437226] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1297.438628] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1297.440064] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1297.441526] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1297.442964] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1297.444449] ? remove_wait_queue+0x60/0x60
[ 1297.445855] kthread+0x103/0x140
[ 1297.447323] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1297.448722] ? kthread_bind+0x10/0x10
[ 1297.450222] ret_from_fork+0x3a/0x50
[ 1323.135126] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1323.135996] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1323.136022] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1323.146274] irq event stamp: 0
[ 1323.147279] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1323.148301] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1323.149331] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1323.150313] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1323.151292] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1323.152291] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1323.153301] RIP: 0010:__module_address+0x5c/0xf0
[ 1323.154297] Code: 83 e2 01 48 8d 42 04 48 8b 04 c5 c8 9a 40 af 48 85 c0 74 2f 48 8d 3c 12 48 01 d7 48 c1 e7 03 48 f7 df 48 8d 14 38 48 8b 4a e0 <48> 39 cb 72 61 8b 72 e8 48 01 f1 48 39 cb 72 0b 48 8b 40 08 48 85
[ 1323.156379] RSP: 0018:ffffa6370223fcf8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 1323.157441] RAX: ffffffffc0a5ade0 RBX: ffffffffc0ebe660 RCX: ffffffffc0a51000
[ 1323.158510] RDX: ffffffffc0a5ade0 RSI: 000000000001a000 RDI: 0000000000000000
[ 1323.159580] RBP: ffffffffc0ebe660 R08: 0000000000000522 R09: 0000000000008000
[ 1323.160646] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1323.161695] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1323.162726] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1323.163756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1323.164762] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1323.165793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1323.166804] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1323.167824] Call Trace:
[ 1323.168829] is_module_address+0x11/0x20
[ 1323.169826] static_obj+0x31/0x50
[ 1323.170813] lockdep_init_map+0x79/0x1f0
[ 1323.171798] __init_waitqueue_head+0x36/0x50
[ 1323.172788] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1323.173782] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1323.174786] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1323.175773] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1323.176781] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1323.177788] ? remove_wait_queue+0x60/0x60
[ 1323.178771] kthread+0x103/0x140
[ 1323.179751] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1323.180742] ? kthread_bind+0x10/0x10
[ 1323.181735] ret_from_fork+0x3a/0x50
[ 1351.135121] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1351.136243] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1351.136262] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1351.147911] irq event stamp: 0
[ 1351.148885] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1351.149908] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1351.150941] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1351.151930] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1351.152877] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1351.153871] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1351.154832] RIP: 0010:_find_next_bit.constprop.1+0x19/0x70
[ 1351.155793] Code: c0 48 01 d0 c3 48 89 f0 f3 c3 0f 1f 80 00 00 00 00 48 39 d1 73 4e 48 89 c8 48 c1 e8 06 48 85 f6 4c 8b 0c c7 74 04 4c 23 0c c6 <48> c7 c0 ff ff ff ff 4d 31 c1 48 d3 e0 48 83 e1 c0 49 21 c1 74 1b
[ 1351.157773] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1351.158780] RAX: 0000000000000000 RBX: ffffffffc0ebe660 RCX: 0000000000000008
[ 1351.159778] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1351.160794] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000ffff
[ 1351.161805] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1351.162812] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1351.163800] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1351.164803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1351.165794] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1351.166810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1351.167808] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1351.168804] Call Trace:
[ 1351.169799] cpumask_next+0x17/0x20
[ 1351.170787] __is_kernel_percpu_address+0x39/0xa0
[ 1351.171781] static_obj+0x25/0x50
[ 1351.172777] lockdep_init_map+0x79/0x1f0
[ 1351.173759] __init_waitqueue_head+0x36/0x50
[ 1351.174764] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1351.175772] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1351.176756] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1351.177760] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1351.178772] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1351.179761] ? remove_wait_queue+0x60/0x60
[ 1351.180761] kthread+0x103/0x140
[ 1351.181762] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1351.182764] ? kthread_bind+0x10/0x10
[ 1351.183729] ret_from_fork+0x3a/0x50
[ 1352.664902] INFO: task iscsi_np:7853 blocked for more than 491 seconds.
[ 1352.666638] Tainted: G W L 5.6.0-rc1+ #18
[ 1352.668358] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1352.670191] iscsi_np D 0 7853 2 0x80004084
[ 1352.672011] Call Trace:
[ 1352.673617] ? __schedule+0x38c/0x990
[ 1352.675307] schedule+0x5f/0xd0
[ 1352.676907] schedule_timeout+0x1e4/0x310
[ 1352.678374] ? wait_for_completion+0x49/0x190
[ 1352.679886] wait_for_completion+0x12e/0x190
[ 1352.681345] ? wake_up_q+0xa0/0xa0
[ 1352.682810] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1352.684332] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1352.685836] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1352.687433] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1352.688927] kthread+0x103/0x140
[ 1352.690354] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1352.691812] ? kthread_bind+0x10/0x10
[ 1352.693301] ret_from_fork+0x3a/0x50
[ 1352.694722] INFO: task sosreport:8582 blocked for more than 368 seconds.
[ 1352.696182] Tainted: G W L 5.6.0-rc1+ #18
[ 1352.697561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1352.698967] sosreport D 0 8582 8581 0x00000080
[ 1352.700316] Call Trace:
[ 1352.701723] ? __schedule+0x38c/0x990
[ 1352.703194] ? avc_has_perm_noaudit+0x37/0x250
[ 1352.704651] schedule+0x5f/0xd0
[ 1352.706020] schedule_timeout+0x1e4/0x310
[ 1352.707434] ? wait_for_completion+0x49/0x190
[ 1352.708787] wait_for_completion+0x12e/0x190
[ 1352.710234] ? wake_up_q+0xa0/0xa0
[ 1352.711614] __wait_rcu_gp+0xfd/0x140
[ 1352.712952] synchronize_rcu.part.75+0x3e/0x70
[ 1352.714257] ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1352.715603] ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1352.716954] rcu_sync_enter+0x54/0xf0
[ 1352.718259] percpu_down_write+0x15/0xe0
[ 1352.719660] ? seq_read+0x327/0x3e0
[ 1352.721039] locks_start+0x28/0x50
[ 1352.722319] seq_read+0x1da/0x3e0
[ 1352.723677] proc_reg_read+0x3c/0x70
[ 1352.724939] vfs_read+0xac/0x160
[ 1352.726243] ksys_read+0x5c/0xd0
[ 1352.727516] do_syscall_64+0x60/0x270
[ 1352.728778] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1352.730102] RIP: 0033:0x7f37d1ec3f70
[ 1352.731311] Code: Bad RIP value.
[ 1352.732578] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1352.733875] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1352.735194] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1352.736503] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1352.737734] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1352.739001] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1352.740269] INFO: lockdep is turned off.
[ 1379.135114] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1379.135862] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1379.135889] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1379.145062] irq event stamp: 0
[ 1379.145955] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1379.146806] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1379.147676] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1379.148552] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1379.149429] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1379.150324] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1379.151225] RIP: 0010:lock_release+0x14c/0x2a0
[ 1379.152060] Code: 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d 0f 1f 44 00 00 48 8b 44 24 08 <65> 48 33 04 25 28 00 00 00 0f 85 e9 00 00 00 48 83 c4 10 5b 5d 41
[ 1379.153879] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1379.154802] RAX: fe6fc5eb227b1400 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1379.155758] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
Message from syslogd@mehrangarh at Feb 10 15:42:30 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1379.156725] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1379.157700] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1379.158679] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1379.159640] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1379.160628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1379.161602] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1379.162604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1379.163611] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1379.164599] Call Trace:
[ 1379.165605] _raw_spin_unlock_bh+0x17/0x30
[ 1379.166604] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1379.167630] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1379.168657] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1379.169688] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1379.170713] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1379.171730] ? remove_wait_queue+0x60/0x60
[ 1379.172753] kthread+0x103/0x140
[ 1379.173756] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1379.174767] ? kthread_bind+0x10/0x10
[ 1379.175787] ret_from_fork+0x3a/0x50
[ 1407.135105] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1407.136304] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1407.136323] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1407.148898] irq event stamp: 0
[ 1407.149911] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1407.150934] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1407.151988] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1407.152994] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1407.154011] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1407.155025] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1407.156071] RIP: 0010:target_put_sess_cmd+0x19/0x50 [target_core_mod]
[ 1407.157115] Code: ed e9 aa fe ff ff 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 97 40 01 00 00 b8 ff ff ff ff f0 0f c1 87 40 01 00 00 <83> f8 01 74 07 85 c0 7e 11 31 c0 c3 48 89 d7 e8 23 fb ff ff b8 01
Message from syslogd@mehrangarh at Feb 10 15:42:58 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1407.159219] RSP: 0018:ffffa6370223fe08 EFLAGS: 00000287 ORIG_RAX: ffffffffffffff13
[ 1407.160256] RAX: 00000000c0000000 RBX: ffff89506770c800 RCX: 0000000000000000
[ 1407.161288] RDX: ffff895065a14ee0 RSI: 0000000000000201 RDI: ffff895065a14da0
[ 1407.162318] RBP: ffffa6370223fe38 R08: 0000000000000000 R09: 0000000000000000
[ 1407.163333] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffff89506770e000
[ 1407.164367] R13: ffff895065a14a90 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1407.165379] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1407.166420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1407.167451] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1407.168488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1407.169521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1407.170550] Call Trace:
[ 1407.171583] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1407.172627] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1407.173661] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1407.174709] ? remove_wait_queue+0x60/0x60
[ 1407.175751] kthread+0x103/0x140
[ 1407.176772] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1407.177814] ? kthread_bind+0x10/0x10
[ 1407.178858] ret_from_fork+0x3a/0x50
[ 1435.135096] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1435.136305] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1435.136325] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1435.149704] irq event stamp: 0
[ 1435.150793] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1435.151897] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1435.152998] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1435.154042] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1435.155043] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1435.156084] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1435.157161] RIP: 0010:find_next_bit+0x9/0x10
[ 1435.158212] Code: ee 81 e6 ff 00 00 00 49 89 34 24 41 5c c3 48 89 d8 5b 5d 41 5c c3 90 66 2e 0f 1f 84 00 00 00 00 00 48 89 d1 45 31 c0 48 89 f2 <31> f6 e9 f0 fe ff ff 48 8b 0a 48 89 0e 48 89 c8 f0 48 0f b1 3a 48
[ 1435.160373] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1435.161441] RAX: ffffffffaf6c6920 RBX: ffffffffc0ebe660 RCX: 000000000000000e
Message from syslogd@mehrangarh at Feb 10 15:43:26 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1435.162524] RDX: 0000000000000010 RSI: 0000000000000010 RDI: ffffffffaf6c6920
[ 1435.163595] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000e000
[ 1435.164646] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1435.165684] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1435.166699] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1435.167717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1435.168751] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1435.169790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1435.170820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1435.171852] Call Trace:
[ 1435.172814] cpumask_next+0x17/0x20
[ 1435.173810] __is_kernel_percpu_address+0x39/0xa0
[ 1435.174813] static_obj+0x25/0x50
[ 1435.175791] lockdep_init_map+0x79/0x1f0
[ 1435.176786] __init_waitqueue_head+0x36/0x50
[ 1435.177782] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1435.178793] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 1435.179786] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1435.180776] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1435.181787] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1435.182797] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1435.183793] ? remove_wait_queue+0x60/0x60
[ 1435.184790] kthread+0x103/0x140
[ 1435.185776] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1435.186766] ? kthread_bind+0x10/0x10
[ 1435.187717] ret_from_fork+0x3a/0x50
[ 1463.135086] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1463.136192] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1463.136214] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1463.147526] irq event stamp: 0
[ 1463.148550] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1463.149576] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1463.150602] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1463.151594] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1463.152568] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1463.153519] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1463.154488] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 1463.155428] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c ef 51 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c ef 51 89 43 08 65 48 8b 04 25
[ 1463.157402] RSP: 0018:ffffa6370223fd80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1463.158414] RAX: 0000000000000000 RBX: ffff895065a14ba8 RCX: 0000000000000000
Message from syslogd@mehrangarh at Feb 10 15:43:54 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1463.159431] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff895065a14ba8
[ 1463.160445] RBP: ffff895065a14ba8 R08: 0000000000000001 R09: 0000000000000000
[ 1463.161454] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1463.162459] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1463.163467] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1463.164475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1463.165492] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1463.166521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1463.167551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1463.168577] Call Trace:
[ 1463.169610] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1463.170653] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1463.171681] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1463.172722] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1463.173768] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1463.174839] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1463.175880] ? remove_wait_queue+0x60/0x60
[ 1463.176912] kthread+0x103/0x140
[ 1463.177936] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1463.178971] ? kthread_bind+0x10/0x10
[ 1463.179974] ret_from_fork+0x3a/0x50
[ 1475.544863] INFO: task iscsi_np:7853 blocked for more than 614 seconds.
[ 1475.546777] Tainted: G W L 5.6.0-rc1+ #18
[ 1475.548695] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1475.550755] iscsi_np D 0 7853 2 0x80004084
[ 1475.552779] Call Trace:
[ 1475.554699] ? __schedule+0x38c/0x990
[ 1475.556663] schedule+0x5f/0xd0
[ 1475.558522] schedule_timeout+0x1e4/0x310
[ 1475.560343] ? wait_for_completion+0x49/0x190
[ 1475.562164] wait_for_completion+0x12e/0x190
[ 1475.563903] ? wake_up_q+0xa0/0xa0
[ 1475.565483] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1475.567206] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1475.568704] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1475.570395] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1475.571950] kthread+0x103/0x140
[ 1475.573433] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1475.575017] ? kthread_bind+0x10/0x10
[ 1475.576548] ret_from_fork+0x3a/0x50
[ 1475.578194] INFO: task sosreport:8582 blocked for more than 491 seconds.
[ 1475.579670] Tainted: G W L 5.6.0-rc1+ #18
[ 1475.581319] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1475.582873] sosreport D 0 8582 8581 0x00000080
[ 1475.584374] Call Trace:
[ 1475.585905] ? __schedule+0x38c/0x990
[ 1475.587352] ? avc_has_perm_noaudit+0x37/0x250
[ 1475.588790] schedule+0x5f/0xd0
[ 1475.590313] schedule_timeout+0x1e4/0x310
[ 1475.591707] ? wait_for_completion+0x49/0x190
[ 1475.593253] wait_for_completion+0x12e/0x190
[ 1475.594683] ? wake_up_q+0xa0/0xa0
[ 1475.596185] __wait_rcu_gp+0xfd/0x140
[ 1475.597602] synchronize_rcu.part.75+0x3e/0x70
[ 1475.599036] ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1475.600361] ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1475.601740] rcu_sync_enter+0x54/0xf0
[ 1475.603213] percpu_down_write+0x15/0xe0
[ 1475.604583] ? seq_read+0x327/0x3e0
[ 1475.605968] locks_start+0x28/0x50
[ 1475.607319] seq_read+0x1da/0x3e0
[ 1475.608640] proc_reg_read+0x3c/0x70
[ 1475.609957] vfs_read+0xac/0x160
[ 1475.611240] ksys_read+0x5c/0xd0
[ 1475.612532] do_syscall_64+0x60/0x270
[ 1475.613786] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1475.615058] RIP: 0033:0x7f37d1ec3f70
[ 1475.616266] Code: Bad RIP value.
[ 1475.617525] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1475.618770] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1475.620177] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1475.621477] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1475.622727] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1475.624003] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1475.625272] INFO: lockdep is turned off.
[ 1491.135077] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1491.135822] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1491.135850] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1491.145026] irq event stamp: 0
[ 1491.145926] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1491.146788] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1491.147661] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1491.148519] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1491.149393] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1491.150287] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1491.151171] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1491.152002] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1491.153806] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1491.154733] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1491.155683] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
Message from syslogd@mehrangarh at Feb 10 15:44:22 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1491.156634] RBP: ffffffffc0ee80b9 R08: 0000000000000000 R09: 0000000000000000
[ 1491.157591] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1491.158567] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1491.159545] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1491.160520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1491.161514] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1491.162522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1491.163515] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1491.164507] Call Trace:
[ 1491.165518] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1491.166532] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1491.167547] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1491.168569] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1491.169613] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1491.170642] ? remove_wait_queue+0x60/0x60
[ 1491.171661] kthread+0x103/0x140
[ 1491.172671] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1491.173726] ? kthread_bind+0x10/0x10
[ 1491.174756] ret_from_fork+0x3a/0x50
[ 1492.387798] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1492.388846] rcu: 11-....: (648817 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=162211
[ 1492.390039] (t=650009 jiffies g=68413 q=59730)
[ 1492.391114] NMI backtrace for cpu 11
[ 1492.392368] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1492.394391] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1492.396382] Call Trace:
[ 1492.398317] <IRQ>
[ 1492.400218] dump_stack+0x82/0xb3
[ 1492.402032] nmi_cpu_backtrace+0x90/0xa0
[ 1492.403779] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1492.405517] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1492.407221] rcu_dump_cpu_stacks+0x97/0xc5
[ 1492.408836] rcu_sched_clock_irq+0x6ec/0x980
[ 1492.410470] ? tick_sched_do_timer+0x70/0x70
[ 1492.412031] update_process_times+0x24/0x50
[ 1492.413525] tick_sched_handle+0x25/0x60
[ 1492.415050] tick_sched_timer+0x37/0x70
[ 1492.416551] __hrtimer_run_queues+0xe8/0x450
[ 1492.418090] hrtimer_interrupt+0x122/0x270
[ 1492.419592] ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1492.421126] smp_apic_timer_interrupt+0x86/0x240
[ 1492.422552] apic_timer_interrupt+0xf/0x20
[ 1492.424002] </IRQ>
[ 1492.425395] RIP: 0010:lock_acquire+0xde/0x1c0
[ 1492.426726] Code: 41 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d 0f 1f 44 00 00 48 83 c4 30 <5b> 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 38 d4 ef 51 89 c0 48 0f
[ 1492.429577] RSP: 0018:ffffa6370223fd58 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 1492.431014] RAX: ffff8951948fa0c0 RBX: 0000000000000246 RCX: 0000000000000000
[ 1492.432456] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 1492.433890] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 1492.435334] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1492.436699] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 1492.438142] _raw_spin_lock_bh+0x3d/0x50
[ 1492.439459] ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1492.440830] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1492.442253] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1492.443622] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1492.445051] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1492.446440] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1492.447835] ? remove_wait_queue+0x60/0x60
[ 1492.449262] kthread+0x103/0x140
[ 1492.450604] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1492.452012] ? kthread_bind+0x10/0x10
[ 1492.453400] ret_from_fork+0x3a/0x50
[ 1519.135050] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1519.135886] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1519.135914] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1519.145816] irq event stamp: 0
[ 1519.146738] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1519.147678] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1519.148643] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1519.149603] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1519.150562] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1519.151562] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1519.152550] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 1519.153536] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c ef 51 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c ef 51 89 43 08 65 48 8b 04 25
[ 1519.155576] RSP: 0018:ffffa6370223fd98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1519.156620] RAX: 0000000000000000 RBX: ffff89506770cde0 RCX: 0000000000000000
[ 1519.157667] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff89506770cde0
[ 1519.158720] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[ 1519.159777] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
Message from syslogd@mehrangarh at Feb 10 15:44:50 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1519.160829] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1519.161844] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1519.162862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1519.163871] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1519.164890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1519.165911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1519.166909] Call Trace:
[ 1519.167911] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1519.168944] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1519.169974] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1519.171007] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1519.172087] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1519.173128] ? remove_wait_queue+0x60/0x60
[ 1519.174206] kthread+0x103/0x140
[ 1519.175227] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1519.176241] ? kthread_bind+0x10/0x10
[ 1519.177273] ret_from_fork+0x3a/0x50
[ 1547.135018] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1547.136213] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1547.136235] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1547.149143] irq event stamp: 0
[ 1547.150238] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1547.151327] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1547.152406] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1547.153419] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1547.154446] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1547.155479] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1547.156534] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1547.157575] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1547.159702] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1547.160772] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1547.161824] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1547.162859] RBP: ffffffffc0ee80b9 R08: 0000000000000000 R09: 0000000000000000
[ 1547.163890] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1547.164916] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1547.165950] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1547.166996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1547.168069] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1547.169112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1547.170149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1547.171175] Call Trace:
[ 1547.172203] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1547.173246] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1547.174273] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1547.175316] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1547.176365] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1547.177411] ? remove_wait_queue+0x60/0x60
[ 1547.178429] kthread+0x103/0x140
[ 1547.179474] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1547.180508] ? kthread_bind+0x10/0x10
[ 1547.181544] ret_from_fork+0x3a/0x50
[ 1575.134987] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1575.136187] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1575.136206] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1575.149191] irq event stamp: 0
[ 1575.150277] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1575.151354] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1575.152408] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1575.153441] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1575.154481] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1575.155531] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1575.156597] RIP: 0010:transport_generic_free_cmd+0x58/0x160 [target_core_mod]
[ 1575.157641] Code: 04 25 28 00 00 00 48 89 44 24 60 31 c0 e8 c0 a6 27 ed 85 ed c6 44 24 06 00 c6 44 24 07 00 0f 85 9b 00 00 00 f6 43 29 01 74 0a <f6> 43 06 02 0f 85 cd 00 00 00 80 7c 24 06 00 74 0c 48 8d 44 24 10
[ 1575.159771] RSP: 0018:ffffa6370223fd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1575.160810] RAX: 0000000000000001 RBX: ffff895065a14da0 RCX: 0000000000000000
Message from syslogd@mehrangarh at Feb 10 15:45:46 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1575.161860] RDX: 000000000000000b RSI: ffffffffc0e84e4a RDI: 0000000000000296
[ 1575.162889] RBP: ffff895065a14ea8 R08: 0000000000000000 R09: 0000000000000001
[ 1575.163918] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1575.164943] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1575.165964] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1575.167050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1575.168084] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1575.169119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1575.170145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1575.171181] Call Trace:
[ 1575.172200] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1575.173220] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1575.174252] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1575.175283] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1575.176316] ? remove_wait_queue+0x60/0x60
[ 1575.177344] kthread+0x103/0x140
[ 1575.178363] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1575.179386] ? kthread_bind+0x10/0x10
[ 1575.180384] ret_from_fork+0x3a/0x50
[ 1598.424741] INFO: task iscsi_np:7853 blocked for more than 737 seconds.
[ 1598.426598] Tainted: G W L 5.6.0-rc1+ #18
[ 1598.428502] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1598.430588] iscsi_np D 0 7853 2 0x80004084
[ 1598.432649] Call Trace:
[ 1598.434613] ? __schedule+0x38c/0x990
[ 1598.436602] schedule+0x5f/0xd0
[ 1598.438494] schedule_timeout+0x1e4/0x310
[ 1598.440363] ? wait_for_completion+0x49/0x190
[ 1598.442176] wait_for_completion+0x12e/0x190
[ 1598.443979] ? wake_up_q+0xa0/0xa0
[ 1598.445693] iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1598.447393] iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1598.448970] iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1598.450677] iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1598.452286] kthread+0x103/0x140
[ 1598.453651] ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1598.455287] ? kthread_bind+0x10/0x10
[ 1598.456822] ret_from_fork+0x3a/0x50
[ 1598.458278] INFO: lockdep is turned off.
[ 1603.134955] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1603.135870] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1603.135896] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1603.146278] irq event stamp: 0
[ 1603.147263] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1603.148229] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1603.149202] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1603.150170] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1603.151130] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1603.152105] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1603.153085] RIP: 0010:_find_next_bit.constprop.1+0x5f/0x70
[ 1603.154059] Code: c1 ea 06 48 85 f6 4e 8b 0c d7 74 04 4e 23 0c d6 4d 31 c1 75 0d 48 83 c1 40 48 39 ca 77 de 48 89 d0 c3 f3 49 0f bc c1 48 01 c8 <48> 39 d0 48 0f 47 c2 c3 66 0f 1f 84 00 00 00 00 00 45 31 c0 eb 8b
[ 1603.156051] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1603.157070] RAX: 000000000000000d RBX: ffffffffc0ebe660 RCX: 0000000000000000
[ 1603.158071] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1603.159069] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000e000
[ 1603.160067] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1603.161076] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1603.162062] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1603.163063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1603.164037] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1603.165023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1603.166008] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1603.166976] Call Trace:
[ 1603.167939] cpumask_next+0x17/0x20
[ 1603.168970] __is_kernel_percpu_address+0x39/0xa0
[ 1603.169938] static_obj+0x25/0x50
[ 1603.170970] lockdep_init_map+0x79/0x1f0
[ 1603.171926] __init_waitqueue_head+0x36/0x50
[ 1603.172927] transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1603.173889] ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 1603.174876] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1603.175863] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1603.176845] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1603.177809] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1603.178796] ? remove_wait_queue+0x60/0x60
[ 1603.179787] kthread+0x103/0x140
[ 1603.180764] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1603.181730] ? kthread_bind+0x10/0x10
[ 1603.182670] ret_from_fork+0x3a/0x50
[ 1631.134928] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1631.136029] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1631.136047] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1631.147287] irq event stamp: 0
[ 1631.148312] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1631.149344] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1631.150371] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1631.151360] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1631.152330] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1631.153279] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1631.154242] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 1631.155179] Code: fa 66 0f 1f 44 00 00 65 01 35 0c 10 f7 51 65 8b 05 05 10 f7 51 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 cd de 06 00 48 89 df 57 9d <0f> 1f 44 00 00 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 1631.157153] RSP: 0018:ffffa6370223fd58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1631.158164] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 1631.159162] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: 0000000000000246
[ 1631.160169] RBP: ffff895065a14a90 R08: 0000000000000000 R09: 0000000000000000
[ 1631.161151] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffff895065a14c98
[ 1631.162150] R13: ffff895065a14c50 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1631.163149] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1631.164165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1631.165170] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1631.166190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1631.167211] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1631.168230] Call Trace:
[ 1631.169244] _raw_spin_lock_bh+0x1c/0x50
[ 1631.170267] ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1631.171294] iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 1631.172323] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1631.173334] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1631.174362] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1631.175375] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1631.176408] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1631.177448] ? remove_wait_queue+0x60/0x60
[ 1631.178479] kthread+0x103/0x140
[ 1631.179479] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1631.180485] ? kthread_bind+0x10/0x10
[ 1631.181509] ret_from_fork+0x3a/0x50
[ 1659.134920] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1659.136098] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1659.136117] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1659.148275] irq event stamp: 0
[ 1659.149300] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1659.150358] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1659.151420] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1659.152435] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1659.153455] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1659.154477] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1659.155503] RIP: 0010:lock_release+0x142/0x2a0
[ 1659.156506] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1659.158531] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1659.159538] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1659.160569] RDX: 0000000000000000 RSI: ffffffffc0ee80f1 RDI: 0000000000000246
[ 1659.161604] RBP: ffff89506770ce68 R08: 0000000000000000 R09: 0000000000000000
[ 1659.162620] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80f1
[ 1659.163648] R13: 0000000000000246 R14: ffff89506770ce50 R15: ffff895065a14a90
[ 1659.164684] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1659.165687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1659.166705] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1659.167733] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1659.168744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1659.169740] Call Trace:
[ 1659.170760] _raw_spin_unlock_bh+0x17/0x30
[ 1659.171785] __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 1659.172787] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1659.173785] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1659.174813] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1659.175849] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1659.176876] ? remove_wait_queue+0x60/0x60
[ 1659.177893] kthread+0x103/0x140
[ 1659.178935] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1659.180009] ? kthread_bind+0x10/0x10
[ 1659.181034] ret_from_fork+0x3a/0x50
[ 1687.134913] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1687.136110] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1687.136128] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1687.148735] irq event stamp: 0
[ 1687.149772] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1687.150799] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1687.151856] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1687.152845] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1687.153868] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1687.154882] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1687.155892] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 1687.156981] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 1687.159072] RSP: 0018:ffffa6370223fd10 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1687.160111] RAX: ffff8951948fa0c0 RBX: 0000000000000246 RCX: 0000000000000000
[ 1687.161144] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 1687.162169] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 1687.163187] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1687.164184] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 1687.165189] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1687.166201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1687.167189] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1687.168180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1687.169184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1687.170165] Call Trace:
[ 1687.171166] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1687.172173] _raw_spin_lock_bh+0x3d/0x50
[ 1687.173164] ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1687.174183] iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1687.175207] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1687.176229] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1687.177230] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1687.178250] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1687.179272] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1687.180294] ? remove_wait_queue+0x60/0x60
[ 1687.181292] kthread+0x103/0x140
[ 1687.182300] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1687.183318] ? kthread_bind+0x10/0x10
[ 1687.184331] ret_from_fork+0x3a/0x50
[ 1687.390634] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1687.391599] rcu: 11-....: (843434 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=210868
[ 1687.393510] (t=845012 jiffies g=68413 q=64187)
[ 1687.394480] NMI backtrace for cpu 11
[ 1687.396158] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1687.398016] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1687.399758] Call Trace:
[ 1687.401478] <IRQ>
[ 1687.403179] dump_stack+0x82/0xb3
[ 1687.404799] nmi_cpu_backtrace+0x90/0xa0
[ 1687.406393] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1687.407982] nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1687.409462] rcu_dump_cpu_stacks+0x97/0xc5
[ 1687.411032] rcu_sched_clock_irq+0x6ec/0x980
[ 1687.412473] ? tick_sched_do_timer+0x70/0x70
[ 1687.414044] update_process_times+0x24/0x50
[ 1687.415506] tick_sched_handle+0x25/0x60
[ 1687.417045] tick_sched_timer+0x37/0x70
[ 1687.418449] __hrtimer_run_queues+0xe8/0x450
[ 1687.419899] hrtimer_interrupt+0x122/0x270
[ 1687.421276] ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1687.422601] smp_apic_timer_interrupt+0x86/0x240
[ 1687.423965] apic_timer_interrupt+0xf/0x20
[ 1687.425230] </IRQ>
[ 1687.426499] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1687.427826] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1687.430671] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1687.432171] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1687.433581] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1687.435084] RBP: ffffffffc0ee80c1 R08: 0000000000000000 R09: 0000000000000000
[ 1687.436489] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1687.438011] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1687.439433] ? __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1687.440904] ? __local_bh_enable_ip+0x80/0xf0
[ 1687.442353] __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1687.443808] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1687.445254] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1687.446679] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1687.448160] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1687.449577] ? remove_wait_queue+0x60/0x60
[ 1687.451080] kthread+0x103/0x140
[ 1687.452459] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1687.453921] ? kthread_bind+0x10/0x10
[ 1687.455316] ret_from_fork+0x3a/0x50
[ 1715.134905] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1715.135762] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1715.135786] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1715.145998] irq event stamp: 0
[ 1715.146998] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1715.148020] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1715.149053] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1715.150034] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1715.151013] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1715.152011] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1715.153021] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1715.154030] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1715.156116] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1715.157181] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1715.158249] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1715.159323] RBP: ffffffffc0ee80fe R08: 0000000000000000 R09: 0000000000000000
[ 1715.160381] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1715.161440] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1715.162485] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1715.163531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1715.164574] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1715.165621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1715.166656] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1715.167673] Call Trace:
[ 1715.168706] __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1715.169719] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1715.170754] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1715.171769] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1715.172816] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1715.173861] ? remove_wait_queue+0x60/0x60
[ 1715.174913] kthread+0x103/0x140
[ 1715.175986] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1715.177064] ? kthread_bind+0x10/0x10
[ 1715.178095] ret_from_fork+0x3a/0x50
[ 1743.134897] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1743.136093] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1743.136113] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1743.149085] irq event stamp: 0
[ 1743.150178] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1743.151261] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1743.152336] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1743.153370] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1743.154382] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1743.155415] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1743.156481] RIP: 0010:transport_generic_free_cmd+0x4c/0x160 [target_core_mod]
[ 1743.157533] Code: 18 c7 44 24 10 00 00 00 00 65 48 8b 04 25 28 00 00 00 48 89 44 24 60 31 c0 e8 c0 a6 27 ed 85 ed c6 44 24 06 00 c6 44 24 07 00 <0f> 85 9b 00 00 00 f6 43 29 01 74 0a f6 43 06 02 0f 85 cd 00 00 00
[ 1743.159691] RSP: 0018:ffffa6370223fd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1743.160758] RAX: ffffa6370223fdb8 RBX: ffff895065a14da0 RCX: 0000000000000000
Message from syslogd@mehrangarh at Feb 10 15:48:34 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1743.161786] RDX: 0000000000000000 RSI: 000000000006a000 RDI: 0000000000000000
[ 1743.162828] RBP: 0000000000000001 R08: 0000000000000522 R09: 0000000000008000
[ 1743.163850] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1743.164872] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1743.165907] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1743.166981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1743.168011] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1743.169043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1743.170049] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1743.171064] Call Trace:
[ 1743.172063] iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1743.173091] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1743.174100] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1743.175114] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1743.176140] ? remove_wait_queue+0x60/0x60
[ 1743.177163] kthread+0x103/0x140
[ 1743.178160] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1743.179181] ? kthread_bind+0x10/0x10
[ 1743.180193] ret_from_fork+0x3a/0x50
[ 1771.134889] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1771.136058] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1771.136076] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1771.148984] irq event stamp: 0
[ 1771.150057] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1771.151148] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1771.152217] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1771.153234] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1771.154264] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1771.155270] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1771.156306] RIP: 0010:lock_release+0x142/0x2a0
[ 1771.157338] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1771.159436] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1771.160496] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1771.161546] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1771.162576] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1771.163586] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1771.164595] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1771.165598] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1771.166640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1771.167670] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1771.168711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1771.169738] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1771.170755] Call Trace:
[ 1771.171761] _raw_spin_unlock_bh+0x17/0x30
[ 1771.172774] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1771.173798] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1771.174813] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1771.175816] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1771.176849] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1771.177870] ? remove_wait_queue+0x60/0x60
[ 1771.178939] kthread+0x103/0x140
[ 1771.179937] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1771.180966] ? kthread_bind+0x10/0x10
[ 1771.181987] ret_from_fork+0x3a/0x50
[ 1799.134881] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1799.136073] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1799.136093] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1799.148664] irq event stamp: 0
[ 1799.149676] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1799.150691] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1799.151719] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1799.152734] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1799.153752] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1799.154788] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1799.155822] RIP: 0010:lock_release+0x142/0x2a0
[ 1799.156840] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1799.158973] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1799.159992] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1799.161008] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1799.162035] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1799.163046] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1799.164060] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1799.165069] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1799.166093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1799.167122] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1799.168153] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1799.169181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1799.170204] Call Trace:
[ 1799.171227] _raw_spin_unlock_bh+0x17/0x30
[ 1799.172231] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1799.173263] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1799.174295] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1799.175312] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1799.176337] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1799.177370] ? remove_wait_queue+0x60/0x60
[ 1799.178392] kthread+0x103/0x140
[ 1799.179393] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1799.180424] ? kthread_bind+0x10/0x10
[ 1799.181429] ret_from_fork+0x3a/0x50
[ 1827.134874] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1827.136077] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1827.136096] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1827.148658] irq event stamp: 0
[ 1827.149696] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1827.150742] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1827.151817] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1827.152850] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1827.153872] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1827.154938] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1827.155977] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 1827.157006] Code: fa 66 0f 1f 44 00 00 65 01 35 0c 10 f7 51 65 8b 05 05 10 f7 51 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 cd de 06 00 48 89 df 57 9d <0f> 1f 44 00 00 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 1827.159094] RSP: 0018:ffffa6370223fd80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1827.160134] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 1827.161169] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: 0000000000000246
[ 1827.162194] RBP: ffff895065a14da0 R08: 0000000000000000 R09: 0000000000000000
[ 1827.163214] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1827.164215] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1827.165232] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1827.166261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1827.167266] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1827.168291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1827.169310] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1827.170307] Call Trace:
[ 1827.171323] _raw_spin_lock_bh+0x1c/0x50
[ 1827.172336] __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1827.173360] iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1827.174380] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1827.175387] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1827.176419] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1827.177442] ? remove_wait_queue+0x60/0x60
[ 1827.178458] kthread+0x103/0x140
[ 1827.179467] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1827.180470] ? kthread_bind+0x10/0x10
[ 1827.181485] ret_from_fork+0x3a/0x50
[ 1855.134866] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1855.136065] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1855.136092] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1855.148497] irq event stamp: 0
[ 1855.149492] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 1855.150516] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1855.151544] softirqs last enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1855.152560] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1855.153568] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1855.154592] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1855.155593] RIP: 0010:lock_release+0x2c/0x2a0
[ 1855.156615] Code: 65 48 8b 14 25 40 8e 01 00 41 56 41 55 41 54 55 53 48 83 ec 10 44 8b 82 bc 0c 00 00 65 48 8b 04 25 28 00 00 00 48 89 44 24 08 <31> c0 45 85 c0 0f 85 10 01 00 00 48 89 fd 49 89 f4 9c 58 0f 1f 44
[ 1855.158663] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 1855.159659] RAX: fe6fc5eb227b1400 RBX: ffff895065a14ba8 RCX: 0000000000000000
[ 1855.160671] RDX: ffff8951948fa0c0 RSI: ffffffffc0ee80b9 RDI: ffff895065a14bc0
[ 1855.161686] RBP: ffff895065a14da0 R08: 0000000000000000 R09: 0000000000000000
[ 1855.162706] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1855.163730] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1855.164734] FS: 0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1855.165771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1855.166794] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1855.167799] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1855.168823] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Message from syslogd@mehrangarh at Feb 10 15:50:26 ...
kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1855.169813] Call Trace:
[ 1855.170833] _raw_spin_unlock_bh+0x17/0x30
[ 1855.171910] __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1855.172969] iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1855.174002] iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1855.175024] iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1855.176068] iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1855.177104] ? remove_wait_queue+0x60/0x60
[ 1855.178121] kthread+0x103/0x140
[ 1855.179144] ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1855.180180] ? kthread_bind+0x10/0x10
[ 1855.181207] ret_from_fork+0x3a/0x50
[ 1871.832621] ------------[ cut here ]------------
[ 1871.833683] NETDEV WATCHDOG: eno1 (igb): transmit queue 1 timed out
[ 1871.834783] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:443 dev_watchdog+0x2d9/0x2e0
[ 1871.835974] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1871.836017] ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1871.848417] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Tainted: G W L 5.6.0-rc1+ #18
[ 1871.849484] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1871.850551] RIP: 0010:dev_watchdog+0x2d9/0x2e0
[ 1871.851653] Code: da 90 ff eb ca 4c 89 f7 c6 05 b8 33 e9 00 01 e8 9d c9 fa ff 89 d9 48 89 c2 4c 89 f6 48 c7 c7 90 5f 1b af 31 c0 e8 47 bd 89 ff <0f> 0b e9 09 ff ff ff 0f 1f 44 00 00 41 57 ba 18 00 00 00 41 56 41
[ 1871.853858] RSP: 0018:ffffa6370003ce48 EFLAGS: 00010286
[ 1871.854981] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[ 1871.856101] RDX: ffff8951da8268c0 RSI: ffff8951da41a0f8 RDI: ffff8951da41a0f8
[ 1871.857219] RBP: ffff8951c57804b8 R08: 0000000000000000 R09: 0000000000000001
[ 1871.858314] R10: 0000000000000000 R11: ffff89506369f4f0 R12: 0000000000000008
[ 1871.859393] R13: ffff8951c57803e0 R14: ffff8951c5780000 R15: ffff8951c6b9fb00
[ 1871.860463] FS: 0000000000000000(0000) GS:ffff8951da400000(0000) knlGS:0000000000000000
[ 1871.861537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1871.862587] CR2: 0000558a0420e358 CR3: 0000000008a10006 CR4: 00000000003606e0
[ 1871.863659] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1871.864734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1871.865796] Call Trace:
[ 1871.866848] <IRQ>
[ 1871.867905] ? dev_deactivate_queue.constprop.56+0xd0/0xd0
[ 1871.869007] ? dev_deactivate_queue.constprop.56+0xd0/0xd0
[ 1871.870065] call_timer_fn+0xa7/0x2d0
[ 1871.871126] ? call_timer_fn+0x5/0x2d0
[ 1871.872190] ? dev_deactivate_queue.constprop.56+0xd0/0xd0
[ 1871.873243] run_timer_softirq+0x235/0x570
[ 1871.874288] ? clockevents_program_event+0x3f/0xf0
[ 1871.875319] __do_softirq+0xd6/0x419
[ 1871.876339] irq_exit+0xd9/0xf0
[ 1871.877361] smp_apic_timer_interrupt+0xac/0x240
[ 1871.878359] apic_timer_interrupt+0xf/0x20
[ 1871.879378] </IRQ>
[ 1871.880388] RIP: 0010:cpuidle_enter_state+0xf3/0x3d0
[ 1871.881381] Code: 44 8b 63 04 49 89 c6 0f 1f 44 00 00 31 ff e8 84 fe 9a ff 80 7c 24 0b 00 0f 85 de 01 00 00 e8 34 26 aa ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 56 02 00 00 49 63 d5 4c 2b 34 24 48 8d 04 52 48 8d
[ 1871.883444] RSP: 0018:ffffa637000d3e70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1871.884501] RAX: 0000000080000001 RBX: ffffc636fa402500 RCX: 000000000000001f
[ 1871.885536] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffffffffae7347dc
[ 1871.886551] RBP: ffffffffaf5c3c80 R08: 0000000000000002 R09: fffffff1d58cfed6
[ 1871.887567] R10: 0000000000000000 R11: 071c71c71c71c71c R12: 0000000000000001
[ 1871.888533] R13: 0000000000000004 R14: 000001b3d1eaf43f R15: 0000000000000001
[ 1871.889516] ? cpuidle_enter_state+0xec/0x3d0
[ 1871.890476] cpuidle_enter+0x29/0x40
[ 1871.891417] do_idle+0x1fb/0x2b0
[ 1871.892319] cpu_startup_entry+0x19/0x20
[ 1871.893225] start_secondary+0x171/0x1c0
[ 1871.894116] secondary_startup_64+0xa4/0xb0
[ 1871.894995] irq event stamp: 606396
[ 1871.895786] hardirqs last enabled at (606395): [<ffffffffae14025b>] rcu_idle_exit+0x2b/0x50
[ 1871.896548] hardirqs last disabled at (606396): [<ffffffffae0e477f>] do_idle+0x7f/0x2b0
[ 1871.897351] softirqs last enabled at (606392): [<ffffffffae0a86be>] irq_enter+0x5e/0x70
[ 1871.898169] softirqs last disabled at (606391): [<ffffffffae0a86a3>] irq_enter+0x43/0x70
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: IOZONE fails intermittently while toggling the interface
2020-02-07 14:00 IOZONE fails intermittently while toggling the interface Dakshaja Uppalapati
` (2 preceding siblings ...)
2020-02-10 14:31 ` Dakshaja Uppalapati
@ 2020-02-11 3:47 ` Bart Van Assche
2020-02-12 5:43 ` Dakshaja Uppalapati
4 siblings, 0 replies; 6+ messages in thread
From: Bart Van Assche @ 2020-02-11 3:47 UTC (permalink / raw)
To: target-devel
[-- Attachment #1: Type: text/plain, Size: 496 bytes --]
On 2020-02-10 06:19, Dakshaja Uppalapati wrote:
> I had run the test multiple times with the given patch, attached traces are seen.
Thanks for having tested the patch I had shared. Can you test the
attached patch on top of v5.6-rc1? It reverts the following two patches:
* e9d3009cb936 ("scsi: target: iscsi: Wait for all commands to finish
before freeing a session") # v5.5-rc1.
* 04060db41178 ("scsi: RDMA/isert: Fix a recently introduced regression
related to logout") # v5.5
Thanks,
Bart.
[-- Attachment #2: revert-recent-iscsi-and-iser-patches.patch --]
[-- Type: text/x-patch, Size: 3252 bytes --]
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index b273e421e910..a1a035270cab 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -2575,6 +2575,17 @@ isert_wait4logout(struct isert_conn *isert_conn)
}
}
+static void
+isert_wait4cmds(struct iscsi_conn *conn)
+{
+ isert_info("iscsi_conn %p\n", conn);
+
+ if (conn->sess) {
+ target_sess_cmd_list_set_waiting(conn->sess->se_sess);
+ target_wait_for_sess_cmds(conn->sess->se_sess);
+ }
+}
+
/**
* isert_put_unsol_pending_cmds() - Drop commands waiting for
* unsolicitate dataout
@@ -2622,6 +2633,7 @@ static void isert_wait_conn(struct iscsi_conn *conn)
ib_drain_qp(isert_conn->qp);
isert_put_unsol_pending_cmds(conn);
+ isert_wait4cmds(conn);
isert_wait4logout(isert_conn);
queue_work(isert_release_wq, &isert_conn->release_work);
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index b94ed4e30770..09e55ea0bf5d 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1165,9 +1165,7 @@ int iscsit_setup_scsi_cmd(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
hdr->cmdsn, be32_to_cpu(hdr->data_length), payload_length,
conn->cid);
- if (target_get_sess_cmd(&cmd->se_cmd, true) < 0)
- return iscsit_add_reject_cmd(cmd,
- ISCSI_REASON_WAITING_FOR_LOGOUT, buf);
+ target_get_sess_cmd(&cmd->se_cmd, true);
cmd->sense_reason = transport_lookup_cmd_lun(&cmd->se_cmd,
scsilun_to_int(&hdr->lun));
@@ -2004,9 +2002,7 @@ iscsit_handle_task_mgt_cmd(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
conn->sess->se_sess, 0, DMA_NONE,
TCM_SIMPLE_TAG, cmd->sense_buffer + 2);
- if (target_get_sess_cmd(&cmd->se_cmd, true) < 0)
- return iscsit_add_reject_cmd(cmd,
- ISCSI_REASON_WAITING_FOR_LOGOUT, buf);
+ target_get_sess_cmd(&cmd->se_cmd, true);
/*
* TASK_REASSIGN for ERL=2 / connection stays inside of
@@ -4149,6 +4145,9 @@ int iscsit_close_connection(
iscsit_stop_nopin_response_timer(conn);
iscsit_stop_nopin_timer(conn);
+ if (conn->conn_transport->iscsit_wait_conn)
+ conn->conn_transport->iscsit_wait_conn(conn);
+
/*
* During Connection recovery drop unacknowledged out of order
* commands for this connection, and prepare the other commands
@@ -4231,11 +4230,6 @@ int iscsit_close_connection(
* must wait until they have completed.
*/
iscsit_check_conn_usage_count(conn);
- target_sess_cmd_list_set_waiting(sess->se_sess);
- target_wait_for_sess_cmds(sess->se_sess);
-
- if (conn->conn_transport->iscsit_wait_conn)
- conn->conn_transport->iscsit_wait_conn(conn);
ahash_request_free(conn->conn_tx_hash);
if (conn->conn_rx_hash) {
diff --git a/include/scsi/iscsi_proto.h b/include/scsi/iscsi_proto.h
index 533f56733ba8..b71b5c4f418c 100644
--- a/include/scsi/iscsi_proto.h
+++ b/include/scsi/iscsi_proto.h
@@ -627,7 +627,6 @@ struct iscsi_reject {
#define ISCSI_REASON_BOOKMARK_INVALID 9
#define ISCSI_REASON_BOOKMARK_NO_RESOURCES 10
#define ISCSI_REASON_NEGOTIATION_RESET 11
-#define ISCSI_REASON_WAITING_FOR_LOGOUT 12
/* Max. number of Key=Value pairs in a text message */
#define MAX_KEY_VALUE_PAIRS 8192
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: IOZONE fails intermittently while toggling the interface
2020-02-07 14:00 IOZONE fails intermittently while toggling the interface Dakshaja Uppalapati
` (3 preceding siblings ...)
2020-02-11 3:47 ` Bart Van Assche
@ 2020-02-12 5:43 ` Dakshaja Uppalapati
4 siblings, 0 replies; 6+ messages in thread
From: Dakshaja Uppalapati @ 2020-02-12 5:43 UTC (permalink / raw)
To: target-devel
On Monday, February 02/10/20, 2020 at 19:47:41 -0800, Bart Van Assche wrote:
> On 2020-02-10 06:19, Dakshaja Uppalapati wrote:
> > I had run the test multiple times with the given patch, attached traces are seen.
>
> Thanks for having tested the patch I had shared. Can you test the
> attached patch on top of v5.6-rc1? It reverts the following two patches:
> * e9d3009cb936 ("scsi: target: iscsi: Wait for all commands to finish
> before freeing a session") # v5.5-rc1.
> * 04060db41178 ("scsi: RDMA/isert: Fix a recently introduced regression
> related to logout") # v5.5
Hi Bart,
With the given patch on 5.6-rc1, the test runs fine overnight.
Thanks,
Dakshaja
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-02-12 5:43 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-07 14:00 IOZONE fails intermittently while toggling the interface Dakshaja Uppalapati
2020-02-07 16:29 ` Bart Van Assche
2020-02-09 17:29 ` Bart Van Assche
2020-02-10 14:31 ` Dakshaja Uppalapati
2020-02-11 3:47 ` Bart Van Assche
2020-02-12 5:43 ` Dakshaja Uppalapati
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.