Re: IOZONE fails intermittently while toggling the interface

[Dakshaja Uppalapati <dakshaja@chelsio.com>]

[-- Attachment #1: Type: text/plain, Size: 1227 bytes --]

On Sunday, February 02/09/20, 2020 at 09:29:36 -0800, Bart Van Assche wrote:
> On 2020-02-07 06:00, Dakshaja Uppalapati wrote:
> > While testing iser with kernel 5.5, I see attached soft lockups.
> > Logged in and ran iozone on mounted disks. While the traffic is running, toggled the initiators interface down to up for 13 secs.
> > I see the attached hung task warnings in the target dmesg. 
> > # iozone -a -I -+d -g 32m
> > 
> > The same test with kernel-5.4 runs fine. I have suspected the following three commits which are all correlated, so removed these commits from 5.5 kernel, rebuilt the kernel and ran the same test then no issue is seen. Iozone is running successfully.
> > 
> > 80647a89eaf3f2: scsi: target: core: Release SPC-2 reservations when closing a session
> > e9d3009cb936b: scsi: target: iscsi: Wait for all commands to finish before freeing a session
> > 04060db41178c: scsi: RDMA/isert: Fix a recently introduced regression related to logout
> > 
> > I am trying to root cause the issue. Please do suggest me if you want me to  check some thing specific.
> 
> Does the attached patch help?

Hi Bart,

I had run the test multiple times with the given patch, attached traces are seen.


Thanks,
Dakshaja

[-- Attachment #2: dmesg_trace2.txt --]
[-- Type: text/plain, Size: 57685 bytes --]

[  532.046617] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[  532.047027] cxgb4 0000:03:00.4 ens1f4: link down
[  545.865331] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[  547.498413] ------------[ cut here ]------------
[  547.499049] refcount_t: underflow; use-after-free.
[  547.499757] WARNING: CPU: 5 PID: 801 at lib/refcount.c:28 refcount_warn_saturate+0x9b/0x100
[  547.500464] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  547.500533]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  547.512682] CPU: 5 PID: 801 Comm: kworker/5:1H Kdump: loaded Not tainted 5.6.0-rc1+ #18
[  547.513954] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  547.515337] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[  547.516662] RIP: 0010:refcount_warn_saturate+0x9b/0x100
[  547.517941] Code: e9 1e 01 01 e8 d6 73 bf ff 0f 0b c3 80 3d 98 e9 1e 01 00 75 a9 48 c7 c7 b0 f9 34 97 31 c0 c6 05 86 e9 1e 01 01 e8 b5 73 bf ff <0f> 0b c3 80 3d 7a e9 1e 01 00 75 88 48 c7 c7 58 f9 34 97 31 c0 c6
[  547.520824] RSP: 0018:ffffa2d14065fd50 EFLAGS: 00010282
[  547.522500] RAX: 0000000000000000 RBX: ffff8d8610a22c90 RCX: 0000000000000000
[  547.523976] RDX: ffff8d87dc0268c0 RSI: ffff8d87dbc1a0f8 RDI: ffff8d87dbc1a0f8
[  547.525567] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001
[  547.526989] R10: 0000000000000000 R11: ffff8d8583af86f0 R12: 0000000000000002
[  547.528551] R13: 0000000000000000 R14: 0000000000000010 R15: ffff8d87a488bc40
[  547.530003] FS:  0000000000000000(0000) GS:ffff8d87dbc00000(0000) knlGS:0000000000000000
[  547.531637] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  547.533166] CR2: 00007f52e7df0140 CR3: 00000002fd810002 CR4: 00000000003606e0
[  547.534786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  547.536466] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  547.538018] Call Trace:
[  547.539726]  transport_generic_free_cmd+0x155/0x160 [target_core_mod]
[  547.541466]  __ib_process_cq+0xc0/0x280 [ib_core]
[  547.543121]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  547.544848]  process_one_work+0x202/0x580
[  547.546620]  ? process_one_work+0x17c/0x580
[  547.548361]  worker_thread+0x4c/0x3f0
[  547.550065]  kthread+0x103/0x140
[  547.551877]  ? process_one_work+0x580/0x580
[  547.553700]  ? kthread_bind+0x10/0x10
[  547.555523]  ret_from_fork+0x3a/0x50
[  547.557310] irq event stamp: 730
[  547.559101] hardirqs last  enabled at (729): [<ffffffff96b858b4>] _raw_spin_unlock_irq+0x24/0x40
[  547.560943] hardirqs last disabled at (730): [<ffffffff96b7e44a>] __schedule+0xda/0x990
[  547.562821] softirqs last  enabled at (726): [<ffffffff969a3a24>] flush_backlog+0x134/0x150
[  547.564703] softirqs last disabled at (722): [<ffffffff969a38f5>] flush_backlog+0x5/0x150
[  547.566545] ---[ end trace e964c55ac82fd95e ]---
[  563.114653] iSCSI Login timeout on Network Portal 10.0.2.2:3260
[  572.936188] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  572.936293] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[  572.937403] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  572.937467]  ipmi_msghandler
[  572.938545] Modules linked in:
[  572.947394]  snd_timer
[  572.948499]  target_core_user
[  572.949546]  wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  572.950618]  uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm ipmi_msghandler snd_timer wmi snd soundcore
[  572.950643]  lpc_ich
[  572.951651] irq event stamp: 0
[  572.951655] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  572.954755]  i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  572.963700] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  572.963702] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  572.964863] irq event stamp: 0
[  572.964865] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  572.966050] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  572.966053] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G        W         5.6.0-rc1+ #18
[  572.967213] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  572.967216] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  572.970899] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  572.970903] RIP: 0010:lock_acquire+0x3a/0x1c0
[  572.972170] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  572.972172] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G        W         5.6.0-rc1+ #18
[  572.973451] Code: 40 8e 01 00 55 53 48 83 ec 30 8b 82 bc 0c 00 00 48 89 7c 24 20 85 c0 0f 85 ad 00 00 00 89 f5 41 89 cd 45 89 c6 4d 89 cf 9c 58 <0f> 1f 44 00 00 48 89 44 24 28 fa 66 0f 1f 44 00 00 c7 82 bc 0c 00
[  572.974678] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  572.974682] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[  572.975916] RSP: 0018:ffffa2d1430cfd40 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  572.977228] Code: 74 57 65 01 1d 32 08 d7 69 65 8b 05 2b 08 d7 69 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 d7 69 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[  572.978557] RAX: 0000000000000246 RBX: ffff8d8610a23aa0 RCX: 0000000000000000
[  572.978558] RDX: ffff8d878eb1c180 RSI: 0000000000000000 RDI: ffff8d8610a23ab8
[  572.979827] RSP: 0018:ffffa2d14308fdd0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  572.981123] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[  572.981124] R10: 0000000000000000 R11: ffffa2d1430cfdb0 R12: 0000000000000000
[  572.982452] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[  572.982453] RDX: ffff8d8786724180 RSI: 0000000000000201 RDI: ffffffff962a85f0
[  572.983742] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[  572.983743] FS:  0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[  572.985017] RBP: ffffffffc0db70fe R08: 0000000000000000 R09: 0000000000000000
[  572.985018] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[  572.986355] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Message from syslogd@mehrangarh at Feb 10 15:06:42 ...
 kernel:watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  572.986356] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[  572.989015] R13: ffff8d808737f800 R14: ffff8d808737fde0 R15: ffff8d86111af160
[  572.989016] FS:  0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[  572.990426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  572.991772] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  572.991773] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[  572.993133] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  572.993134] Call Trace:
[  572.995871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  572.997343]  ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[  572.998648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  572.998649] Call Trace:
[  572.999941]  _raw_spin_lock_bh+0x3d/0x50
[  573.001326]  __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[  573.002585]  ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[  573.003794]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  573.005047]  iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[  573.006314]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  573.007570]  __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[  573.008807]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  573.010048]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[  573.011296]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]

Message from syslogd@mehrangarh at Feb 10 15:06:42 ...
 kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[  573.012519]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  573.013714]  kthread+0x103/0x140
[  573.014900]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  573.016085]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  573.017215]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  573.018340]  ? kthread_bind+0x10/0x10
[  573.019435]  kthread+0x103/0x140
[  573.020532]  ret_from_fork+0x3a/0x50
[  573.021599]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  573.045424]  ? kthread_bind+0x10/0x10
[  573.046408]  ret_from_fork+0x3a/0x50
[  600.936308] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  600.936414] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[  600.937243] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  600.937275]  ipmi_msghandler
[  600.938049] Modules linked in:
[  600.944983]  snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  600.945964]  target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm ipmi_msghandler snd_timer
[  600.945985]  wmi
[  600.946934] irq event stamp: 0
[  600.946937] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  600.949897]  snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  600.958443] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  600.958445] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  600.959594] irq event stamp: 0
[  600.959596] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  600.960731] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  600.960733] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  600.961867] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  600.961869] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  600.965365] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  600.965369] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[  600.966604] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  600.966606] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  600.967804] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 55 ef 78 ff 48 89 ef e8 9d 3b 79 ff f6 c7 02 74 19 e8 b3 11 85 ff 48 89 df 57 9d <0f> 1f 44 00 00 65 ff 0d 9a 31 49 69 5b 5d c3 48 89 df 57 9d 0f 1f
[  600.968983] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  600.968986] RIP: 0010:lock_release+0xf/0x2a0
[  600.970199] RSP: 0018:ffffa2d1430cfd90 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[  600.971426] Code: c6 d0 23 30 97 48 c7 c7 a2 65 2e 97 31 c0 e8 08 b0 f8 ff 0f 0b e9 e8 fe ff ff 90 41 57 65 48 8b 14 25 40 8e 01 00 41 56 41 55 <41> 54 55 53 48 83 ec 10 44 8b 82 bc 0c 00 00 65 48 8b 04 25 28 00
[  600.972637] RAX: 0000000000000001 RBX: 0000000000000296 RCX: 0000000000000000
[  600.972638] RDX: 0000000000000006 RSI: ffffffffc0d53e4a RDI: 0000000000000296

Message from syslogd@mehrangarh at Feb 10 15:07:10 ...
 kernel:watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  600.973824] RSP: 0018:ffffa2d14308fdb8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  600.975009] RBP: ffff8d8610a23cf8 R08: 0000000000000000 R09: 0000000000000001
[  600.975010] R10: 0000000000000000 R11: ffffa2d1430cfdb0 R12: 0000000000000001
[  600.976228] RAX: 0000000000000000 RBX: ffff8d808737fe50 RCX: 0000000000000000
[  600.976230] RDX: ffff8d8786724180 RSI: ffffffffc0db70f1 RDI: ffff8d808737fe68
[  600.977458] R13: 0000000000000001 R14: ffff8d8610a23be0 R15: ffff8d8610a238e0
[  600.977460] FS:  0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[  600.978662] RBP: ffff8d86111af470 R08: 0000000000000001 R09: 0000000000000000
[  600.978663] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[  600.979845] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  600.979846] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[  600.982292] R13: ffff8d808737f800 R14: ffff8d808737fe50 R15: ffff8d86111af160
[  600.982294] FS:  0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[  600.983568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  600.984810] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  600.984811] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[  600.986052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Message from syslogd@mehrangarh at Feb 10 15:07:10 ...
 kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[  600.986053] Call Trace:
[  600.988635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  600.989941]  transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[  600.991230] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  600.991231] Call Trace:
[  600.992544]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[  600.993790]  _raw_spin_unlock_bh+0x17/0x30
[  600.995013]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  600.996240]  __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[  600.997468]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  600.998638]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  600.999801]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  601.000947]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  601.002088]  kthread+0x103/0x140
[  601.003223]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  601.004345]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  601.005483]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  601.006603]  ? kthread_bind+0x10/0x10
[  601.006606]  ret_from_fork+0x3a/0x50
[  601.007701]  kthread+0x103/0x140
[  601.030430]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  601.031328]  ? kthread_bind+0x10/0x10
[  601.032240]  ret_from_fork+0x3a/0x50
[  605.130023] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[  605.131611] cxgb4 0000:03:00.4 ens1f4: link down
[  612.501078] rcu: INFO: rcu_sched self-detected stall on CPU
[  612.501922] rcu:     13-....: (64820 ticks this GP) idle=bd6/1/0x4000000000000002 softirq=27999/27999 fqs=15343
[  612.503598]  (t=65002 jiffies g=69481 q=31952)
[  612.504590] NMI backtrace for cpu 13
[  612.505775] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  612.507577] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  612.509359] Call Trace:
[  612.511146]  <IRQ>
[  612.512865]  dump_stack+0x82/0xb3
[  612.514613]  nmi_cpu_backtrace+0x90/0xa0
[  612.516343]  ? lapic_can_unplug_cpu+0xa0/0xa0
[  612.518050]  nmi_trigger_cpumask_backtrace+0x73/0x130
[  612.519773]  rcu_dump_cpu_stacks+0x97/0xc5
[  612.521482]  rcu_sched_clock_irq+0x6ec/0x980
[  612.523172]  ? tick_sched_do_timer+0x70/0x70
[  612.524829]  update_process_times+0x24/0x50
[  612.526495]  tick_sched_handle+0x25/0x60
[  612.528128]  tick_sched_timer+0x37/0x70
[  612.529720]  __hrtimer_run_queues+0xe8/0x450
[  612.531315]  hrtimer_interrupt+0x122/0x270
[  612.532882]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[  612.534466]  smp_apic_timer_interrupt+0x86/0x240
[  612.536019]  apic_timer_interrupt+0xf/0x20
[  612.537576]  </IRQ>
[  612.539071] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[  612.540595] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c cf 69 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c cf 69 89 43 08 65 48 8b 04 25
[  612.543756] RSP: 0018:ffffa2d14308fdd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  612.545368] RAX: 0000000000000000 RBX: ffff8d808737fde0 RCX: 0000000000000000
[  612.547012] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8d808737fde0
[  612.548630] RBP: ffff8d86111af470 R08: 0000000000000001 R09: 0000000000000000
[  612.550237] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[  612.551819] R13: ffff8d808737f800 R14: ffff8d808737fde0 R15: ffff8d86111af160
[  612.553458]  ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[  612.555043]  __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[  612.556645]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  612.558229]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  612.559802]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  612.561401]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  612.562982]  kthread+0x103/0x140
[  612.564547]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  612.566140]  ? kthread_bind+0x10/0x10
[  612.567724]  ret_from_fork+0x3a/0x50
[  618.978720] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[  628.936291] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  628.937158] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  628.937191]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  628.947559] irq event stamp: 0
[  628.948639] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  628.949684] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80

Message from syslogd@mehrangarh at Feb 10 15:07:38 ...
 kernel:watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  628.950729] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  628.951764] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  628.952774] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  628.953800] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  628.954831] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[  628.955845] Code: 74 57 65 01 1d 32 08 d7 69 65 8b 05 2b 08 d7 69 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 d7 69 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[  628.957947] RSP: 0018:ffffa2d1430cfdd0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  628.959010] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[  628.960117] RDX: ffff8d878eb1c180 RSI: 0000000000000201 RDI: ffffffff962a85f0
[  628.961208] RBP: ffffffffc0db70f1 R08: 0000000000000000 R09: 0000000000000000
[  628.962312] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[  628.963440] R13: ffff8d8562ea2800 R14: ffff8d8562ea2e50 R15: ffff8d8610a238e0
[  628.964550] FS:  0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[  628.965630] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  628.966710] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[  628.967789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  628.968863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  628.969907] Call Trace:
[  628.970964]  __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[  628.972026]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  628.973100]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  628.974135]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  628.975201]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  628.976260]  kthread+0x103/0x140
[  628.977319]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  628.978418]  ? kthread_bind+0x10/0x10
[  628.979486]  ret_from_fork+0x3a/0x50
[  636.936393] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[  636.937563] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  636.937605]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]

Message from syslogd@mehrangarh at Feb 10 15:07:46 ...
 kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[  636.950692] irq event stamp: 0
[  636.951792] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  636.952889] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  636.953960] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  636.955021] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  636.956046] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  636.957104] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  636.958169] RIP: 0010:cpumask_next+0x0/0x20
[  636.959224] Code: 45 ff ff ff c6 00 00 eb be 48 83 c0 01 48 89 02 8d 41 ff 48 01 f8 80 38 22 75 9d c6 00 00 eb 98 cc cc cc cc cc cc cc cc cc cc <48> 89 f0 8b 35 bb 93 d6 00 83 c7 01 48 63 d7 48 89 c7 e8 39 66 b4
[  636.961414] RSP: 0018:ffffa2d14308fd20 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[  636.962540] RAX: 0000000000000003 RBX: ffffffffc0d8d660 RCX: ffff8d87db42d000
[  636.963646] RDX: ffff8d87db000000 RSI: ffffffff978c6920 RDI: 0000000000000003
[  636.964711] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000fff8
[  636.965780] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 000000000042d000
[  636.966837] R13: 0000000000000000 R14: ffff8d86111af460 R15: ffff8d86111af160
[  636.967885] FS:  0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[  636.968940] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  636.969990] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[  636.971038] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  636.972065] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  636.973064] Call Trace:
[  636.974057]  __is_kernel_percpu_address+0x39/0xa0
[  636.975056]  static_obj+0x25/0x50
[  636.976050]  lockdep_init_map+0x79/0x1f0
[  636.977041]  __init_waitqueue_head+0x36/0x50
[  636.978074]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[  636.979096]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[  636.980072]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  636.981071]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  636.982070]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  636.983063]  kthread+0x103/0x140
[  636.984045]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  636.985056]  ? kthread_bind+0x10/0x10
[  636.986050]  ret_from_fork+0x3a/0x50
[  656.936314] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  656.937474] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  656.937513]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  656.949583] irq event stamp: 0
[  656.950654] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  656.951731] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  656.952818] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  656.953887] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  656.954916] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  656.955949] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  656.956981] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[  656.957988] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c cf 69 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c cf 69 89 43 08 65 48 8b 04 25
[  656.959999] RSP: 0018:ffffa2d1430cfdc0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  656.961029] RAX: 0000000000000000 RBX: ffff8d8610a239f8 RCX: 0000000000000000
[  656.962055] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8d8610a239f8
[  656.963070] RBP: ffff8d8610a239f8 R08: 0000000000000001 R09: 0000000000000000
[  656.964089] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[  656.965167] R13: ffff8d8562ea2800 R14: ffff8d8610a23be0 R15: ffff8d8610a238e0
[  656.966184] FS:  0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[  656.967207] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  656.968231] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[  656.969271] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  656.970314] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  656.971391] Call Trace:
[  656.972500]  iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[  656.973582]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[  656.974664]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  656.975741]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  656.976817]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  656.977892]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  656.978940]  kthread+0x103/0x140
[  656.979992]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  656.981046]  ? kthread_bind+0x10/0x10
[  656.982083]  ret_from_fork+0x3a/0x50
[  664.936435] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[  664.937608] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  664.937646]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  664.950547] irq event stamp: 0
[  664.951626] hardirqs last  enabled at (0): [<0000000000000000>] 0x0

Message from syslogd@mehrangarh at Feb 10 15:08:14 ...
 kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[  664.952702] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  664.953783] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  664.954846] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  664.955900] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  664.956970] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  664.958041] RIP: 0010:__raw_spin_lock_init+0xa/0x50
[  664.959103] Code: c7 c7 08 c0 36 97 31 c0 e8 e7 f7 00 00 5b 48 c7 c7 b8 b4 39 97 31 c0 e9 d8 f7 00 00 5b c3 cc cc 0f 1f 44 00 00 41 54 49 89 d4 <55> 48 89 f5 be 38 00 00 00 53 48 89 fb e8 d4 82 ff ff 48 8d 7b 18
[  664.961224] RSP: 0018:ffffa2d14308fd78 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  664.962309] RAX: 0000000000000000 RBX: ffffa2d14308fdc0 RCX: 0000000000000000
[  664.963388] RDX: ffffffff9831f530 RSI: ffffffff97300412 RDI: ffffa2d14308fdc0
[  664.964490] RBP: ffffffffc0d67e09 R08: 0000000000000000 R09: 0000000000000000
[  664.965580] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: ffffffff9831f530
[  664.966667] R13: 0000000000000001 R14: ffff8d86111af460 R15: ffff8d86111af160
[  664.967745] FS:  0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[  664.968810] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  664.969884] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[  664.970957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  664.972011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  664.973058] Call Trace:
[  664.974097]  __init_waitqueue_head+0x25/0x50
[  664.975143]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[  664.976168]  ? __local_bh_enable_ip+0x80/0xf0
[  664.977188]  ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[  664.978236]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[  664.979289]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  664.980348]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  664.981479]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  664.982563]  kthread+0x103/0x140
[  664.983632]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  664.984703]  ? kthread_bind+0x10/0x10
[  664.985762]  ret_from_fork+0x3a/0x50
[  673.196146] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 6-... 13-... } 68051 jiffies s: 409 root: 0x2040/.
[  673.199958] rcu: blocking rcu_node structures:
[  673.201895] Task dump for CPU 6:
[  673.203851] iscsi_trx       R  running task        0 12756      2 0x8000408c
[  673.205900] Call Trace:
[  673.207825]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  673.209691]  ? retint_kernel+0x10/0x10
[  673.211611]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  673.213445]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  673.215228]  ? __is_kernel_percpu_address+0x39/0xa0
[  673.216826]  ? __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[  673.218524]  ? __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[  673.220046]  ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[  673.221765]  ? iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[  673.223383]  ? iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  673.224960]  ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  673.226588]  ? iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  673.228207]  ? kthread+0x103/0x140
[  673.229733]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  673.231326]  ? kthread_bind+0x10/0x10
[  673.232884]  ? ret_from_fork+0x3a/0x50
[  673.234527] Task dump for CPU 13:
[  673.236003] iscsi_trx       R  running task        0 12748      2 0x8000408c
[  673.237607] Call Trace:
[  673.239116]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  673.240644]  ? run_timer_softirq+0x52d/0x570
[  673.242090]  ? run_timer_softirq+0x52d/0x570
[  673.243596]  ? retint_kernel+0x10/0x10
[  673.245020]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  673.246562]  ? iscsit_free_all_datain_reqs+0x24/0xa0 [iscsi_target_mod]
[  673.247962]  ? __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[  673.249398]  ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[  673.250740]  ? iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  673.252186]  ? iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  673.253560]  ? iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  673.255025]  ? iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  673.256578]  ? kthread+0x103/0x140
[  673.257991]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  673.259536]  ? kthread_bind+0x10/0x10
[  673.260961]  ? ret_from_fork+0x3a/0x50
[  684.936368] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  684.937180] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  684.937210]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  684.946894] irq event stamp: 0
[  684.947821] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  684.948772] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  684.949743] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  684.950709] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  684.951673] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  684.952663] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  684.953657] RIP: 0010:lock_acquire+0xdf/0x1c0
[  684.954629] Code: 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d 0f 1f 44 00 00 48 83 c4 30 5b <5d> 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 38 d4 cf 69 89 c0 48 0f a3
[  684.956615] RSP: 0018:ffffa2d1430cfd88 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[  684.957650] RAX: ffff8d878eb1c180 RBX: ffff8d8610a239f8 RCX: 0000000000000000
[  684.958685] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[  684.959714] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[  684.960732] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  684.961746] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[  684.962756] FS:  0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[  684.963776] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  684.964769] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[  684.965769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  684.966790] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  684.967829] Call Trace:
[  684.968828]  _raw_spin_lock_bh+0x3d/0x50
[  684.969833]  ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[  684.970863]  iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[  684.971890]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[  684.972977]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  684.974006]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  684.975041]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  684.976070]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  684.977092]  kthread+0x103/0x140
[  684.978104]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  684.979120]  ? kthread_bind+0x10/0x10
[  684.980119]  ret_from_fork+0x3a/0x50
[  692.936490] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[  692.937632] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  692.937671]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]

Message from syslogd@mehrangarh at Feb 10 15:08:42 ...
 kernel:watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [iscsi_trx:12748]
[  692.949728] irq event stamp: 0
[  692.950765] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  692.951807] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  692.952857] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  692.953898] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  692.954934] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  692.955975] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  692.957004] RIP: 0010:lock_release+0x0/0x2a0
[  692.958019] Code: 83 3d b3 84 5b 01 00 0f 85 04 ff ff ff 48 c7 c6 d0 23 30 97 48 c7 c7 a2 65 2e 97 31 c0 e8 08 b0 f8 ff 0f 0b e9 e8 fe ff ff 90 <41> 57 65 48 8b 14 25 40 8e 01 00 41 56 41 55 41 54 55 53 48 83 ec
[  692.960047] RSP: 0018:ffffa2d14308fdd0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  692.961073] RAX: 0000000000000000 RBX: ffff8d808737fe50 RCX: 0000000000000000
[  692.962097] RDX: 0000000000000001 RSI: ffffffffc0db70f1 RDI: ffff8d808737fe68
[  692.963115] RBP: ffff8d86111af470 R08: 0000000000000001 R09: 0000000000000000
[  692.964139] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: 0000000000000001
[  692.965165] R13: ffff8d808737f800 R14: ffff8d808737fe50 R15: ffff8d86111af160
[  692.966206] FS:  0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[  692.967258] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  692.968301] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[  692.969351] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  692.970413] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  692.971509] Call Trace:
[  692.972586]  _raw_spin_unlock_bh+0x17/0x30
[  692.973666]  __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[  692.974745]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  692.975812]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  692.976886]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  692.977961]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  692.979021]  kthread+0x103/0x140
[  692.980049]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  692.981103]  ? kthread_bind+0x10/0x10
[  692.982141]  ret_from_fork+0x3a/0x50
[  712.936369] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [iscsi_trx:12756]
[  712.937563] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  712.937602]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  712.950360] irq event stamp: 0
[  712.951416] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  712.952502] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  712.953594] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  712.954669] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  712.955735] CPU: 6 PID: 12756 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  712.956820] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  712.957901] RIP: 0010:_raw_spin_unlock_irqrestore+0x3e/0x52
[  712.958967] Code: 24 10 e8 55 ef 78 ff 48 89 ef e8 9d 3b 79 ff f6 c7 02 74 19 e8 b3 11 85 ff 48 89 df 57 9d 0f 1f 44 00 00 65 ff 0d 9a 31 49 69 <5b> 5d c3 48 89 df 57 9d 0f 1f 44 00 00 e8 b0 10 85 ff eb e5 cc cc
[  712.961098] RSP: 0018:ffffa2d1430cfd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  712.962194] RAX: 0000000000000001 RBX: 0000000000000296 RCX: 0000000000000000
[  712.963247] RDX: 0000000000000006 RSI: ffffffffc0d53e4a RDI: 0000000000000296
[  712.964289] RBP: ffff8d8610a23cf8 R08: 0000000000000000 R09: 0000000000000001
[  712.965329] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[  712.966364] R13: 0000000000000001 R14: ffff8d8610a23be0 R15: ffff8d8610a238e0
[  712.967403] FS:  0000000000000000(0000) GS:ffff8d87dc200000(0000) knlGS:0000000000000000
[  712.968519] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  712.969577] CR2: 00007f52e7df0140 CR3: 00000002fd810001 CR4: 00000000003606e0
[  712.970632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  712.971700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  712.972754] Call Trace:
[  712.973786]  transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[  712.974854]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[  712.975906]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  712.976946]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  712.977989]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  712.979022]  kthread+0x103/0x140
[  712.980064]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  712.981111]  ? kthread_bind+0x10/0x10
[  712.982154]  ret_from_fork+0x3a/0x50
[  720.936424] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [iscsi_trx:12748]
[  720.937584] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr intel_rapl_common sb_edac ib_umad joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_generic rdma_cm iw_cm ledtrig_audio snd_hda_intel irqbypass snd_intel_dspcfg ib_cm snd_hda_codec ib_core crct10dif_pclmul iTCO_wdt iTCO_vendor_support ipmi_ssif crc32_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep crypto_simd snd_seq snd_seq_device cryptd ipmi_si glue_helper sg pcspkr ipmi_devintf snd_pcm
[  720.937622]  ipmi_msghandler snd_timer wmi snd soundcore lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci libahci crc32c_intel libata dca i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  720.950527] irq event stamp: 0
[  720.951618] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  720.952701] hardirqs last disabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  720.953765] softirqs last  enabled at (0): [<ffffffff9629d07f>] copy_process+0x76f/0x1f80
[  720.954818] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  720.955860] CPU: 13 PID: 12748 Comm: iscsi_trx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  720.956914] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  720.957958] RIP: 0010:lock_release+0x142/0x2a0
[  720.959010] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[  720.961159] RSP: 0018:ffffa2d14308fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  720.962243] RAX: ffff8d8786724180 RBX: ffff8d8786724180 RCX: 0000000000000000
[  720.963316] RDX: 0000000000000000 RSI: ffffffffc0db70f1 RDI: 0000000000000246
[  720.964373] RBP: ffff8d808737fe68 R08: 0000000000000000 R09: 0000000000000000
[  720.965462] R10: 0000000000000000 R11: ffffa2d14308fdb0 R12: ffffffffc0db70f1
[  720.966542] R13: 0000000000000246 R14: ffff8d808737fe50 R15: ffff8d86111af160
[  720.967600] FS:  0000000000000000(0000) GS:ffff8d87dec00000(0000) knlGS:0000000000000000
[  720.968680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  720.969754] CR2: 00007f4df5029248 CR3: 00000002fd810003 CR4: 00000000003606e0
[  720.970831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  720.971899] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  720.972954] Call Trace:
[  720.973993]  _raw_spin_unlock_bh+0x17/0x30
[  720.975037]  __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[  720.976089]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  720.977128]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  720.978149]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  720.979194]  iscsi_target_rx_thread+0xdf/0xf0 [iscsi_target_mod]
[  720.980237]  kthread+0x103/0x140
[  720.981271]  ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
[  720.982317]  ? kthread_bind+0x10/0x10
[  720.983414]  ret_from_fork+0x3a/0x50

[-- Attachment #3: dmesg_trace1.txt --]
[-- Type: text/plain, Size: 182833 bytes --]

[  271.548371] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.573691] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.592535] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.593316] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.594096] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.613094] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.631967] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.650686] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.669852] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[  271.688996] iSCSI/iqn.1994-05.com.redhat:1a48c21a4dd8: Unsupported SCSI Opcode 0xa3, sen
[root@mehrangarh ~]# dmesg -w
[  453.414110] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[  453.414689] cxgb4 0000:03:00.4 ens1f4: link down
[  467.162880] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[  468.807914] ------------[ cut here ]------------
[  468.808496] refcount_t: underflow; use-after-free.
[  468.809181] WARNING: CPU: 12 PID: 753 at lib/refcount.c:28 refcount_warn_saturate+0x9b/0
[  468.809910] Modules linked in: target_core_user uio target_core_pscsi target_core_file t_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_pxt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_ed_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_culni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[  468.809974]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi ables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea salgo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uv
[  468.817870] CPU: 12 PID: 753 Comm: kworker/12:1H Kdump: loaded Not tainted 5.6.0-rc1+ #1
[  468.818615] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  468.819379] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[  468.820144] RIP: 0010:refcount_warn_saturate+0x9b/0x100
[  468.820880] Code: e9 1e 01 01 e8 d6 73 bf ff 0f 0b c3 80 3d 98 e9 1e 01 00 75 a9 48 c7 c c7 c7 58 f9 14 af 31 c0 c6
[  468.822426] RSP: 0018:ffffa637004bbd50 EFLAGS: 00010282
[  468.823236] RAX: 0000000000000000 RBX: ffff895068723440 RCX: 0000000000000000
[  468.824060] RDX: ffff8951dea268c0 RSI: ffff8951de61a0f8 RDI: ffff8951de61a0f8
[  468.824827] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001
[  468.825621] R10: 0000000000000000 R11: ffff8951c491fef0 R12: 000000000000000f
[  468.826460] R13: 0000000000000080 R14: 0000000000000010 R15: ffff8951b6a07680
[  468.827300] FS:  0000000000000000(0000) GS:ffff8951de600000(0000) knlGS:0000000000000000
[  468.828166] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  468.829029] CR2: 0000555d41387060 CR3: 0000000008a10003 CR4: 00000000003606e0
[  468.829870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  468.830768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  468.831636] Call Trace:
[  468.832556]  transport_generic_free_cmd+0x155/0x160 [target_core_mod]
[  468.833477]  __ib_process_cq+0xc0/0x280 [ib_core]
[  468.834374]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  468.835274]  process_one_work+0x202/0x580
[  468.836194]  ? process_one_work+0x17c/0x580
[  468.837106]  worker_thread+0x4c/0x3f0
[  468.838020]  kthread+0x103/0x140
[  468.838901]  ? process_one_work+0x580/0x580
[  468.839835]  ? kthread_bind+0x10/0x10
[  468.840781]  ret_from_fork+0x3a/0x50
[  468.841717] irq event stamp: 2036
[  468.842642] hardirqs last  enabled at (2035): [<ffffffffae9858b4>] _raw_spin_unlock_irq+
[  468.843585] hardirqs last disabled at (2036): [<ffffffffae97e44a>] __schedule+0xda/0x990
[  468.844556] softirqs last  enabled at (2032): [<ffffffffae7a3a24>] flush_backlog+0x134/0
[  468.845513] softirqs last disabled at (2028): [<ffffffffae7a38f5>] flush_backlog+0x5/0x1
[  468.846456] ---[ end trace b89f796ccb2b161b ]---
[  779.866619] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[  779.867557] cxgb4 0000:03:00.4 ens1f4: link down
[  793.645349] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[  826.926701] cxgb4 0000:03:00.4: Port 0 link down, reason: Link Down
[  826.927629] cxgb4 0000:03:00.4 ens1f4: link down
[  840.735421] cxgb4 0000:03:00.4 ens1f4: link up, 100Gbps, full-duplex, Tx/Rx PAUSE
[  857.562340] iSCSI Login timeout on Network Portal 10.0.2.2:3260
[  867.136473] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[  867.137485] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[  867.137547]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  867.148103] irq event stamp: 0
[  867.149010] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  867.149971] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  867.150932] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  867.151865] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  867.152792] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W         5.6.0-rc1+ #18
[  867.153732] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  867.154656] RIP: 0010:do_raw_spin_lock+0x5c/0xa0
[  867.155586] Code: 47 31 c0 ba 01 00 00 00 f0 0f b1 13 0f 94 c2 84 d2 74 27 65 8b 05 0c 8c ef 51 89 43 08 65 48 8b 04 25 40 8e 01 00 48 89 43 10 <5b> c3 48 c7 c6 c8 41 10 af e8 07 03 00 00 eb a7 89 c6 48 89 df e8
[  867.157466] RSP: 0018:ffffa6370223fd98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[  867.158429] RAX: ffff8951948fa0c0 RBX: ffff89506770ce50 RCX: 0000000000000000
[  867.159394] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff89506770ce50
[  867.160359] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[  867.161327] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[  867.162300] R13: ffff89506770c800 R14: ffff89506770ce50 R15: ffff895065a14a90
[  867.163257] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[  867.164247] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  867.165247] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[  867.166259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  867.167273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  867.168285] Call Trace:
[  867.169305]  __iscsit_free_cmd+0x121/0x2c0 [iscsi_target_mod]
[  867.170322]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  867.171337]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  867.172360]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  867.173373]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[  867.174395]  ? remove_wait_queue+0x60/0x60
[  867.175408]  kthread+0x103/0x140
[  867.176400]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[  867.177420]  ? kthread_bind+0x10/0x10
[  867.178437]  ret_from_fork+0x3a/0x50

Message from syslogd@mehrangarh at Feb 10 15:33:58 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[  895.136221] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[  895.137439] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[  895.137460]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  895.150359] irq event stamp: 0
[  895.151458] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  895.152546] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  895.153623] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  895.154660] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  895.155686] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  895.156739] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  895.157795] RIP: 0010:__is_kernel_percpu_address+0x4e/0xa0
[  895.158841] Code: 00 00 00 53 48 89 fb 48 c7 c6 20 69 6c af 89 c7 e8 a7 34 6e 00 3b 05 65 c8 44 01 73 4c 89 c2 48 89 e9 48 03 0c d5 80 59 1c af <48> 39 d9 48 89 ca 77 d5 4a 8d 0c 21 48 39 cb 73 cc 4d 85 ed 74 32
[  895.160972] RSP: 0018:ffffa6370223fce8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[  895.162023] RAX: 000000000000000d RBX: ffffffffc0ebe660 RCX: ffff8951dec00000
[  895.163048] RDX: 000000000000000d RSI: 0000000000000000 RDI: ffffffffaf6c6920
[  895.164088] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000e000
[  895.165089] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[  895.166101] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[  895.167116] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[  895.168141] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  895.169158] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[  895.170152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  895.171164] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  895.172139] Call Trace:
[  895.173132]  static_obj+0x25/0x50
[  895.174103]  lockdep_init_map+0x79/0x1f0
[  895.175092]  __init_waitqueue_head+0x36/0x50
[  895.176058]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[  895.177054]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[  895.178046]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  895.179019]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  895.180019]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]

Message from syslogd@mehrangarh at Feb 10 15:34:26 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[  895.181012]  ? remove_wait_queue+0x60/0x60
[  895.182005]  kthread+0x103/0x140
[  895.182994]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[  895.183987]  ? kthread_bind+0x10/0x10
[  895.184979]  ret_from_fork+0x3a/0x50
[  907.379831] rcu: INFO: rcu_sched self-detected stall on CPU
[  907.380817] rcu:     11-....: (64908 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=16230
[  907.382776]  (t=65000 jiffies g=68413 q=33708)
[  907.383730] NMI backtrace for cpu 11
[  907.385490] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  907.387402] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  907.389266] Call Trace:
[  907.391022]  <IRQ>
[  907.392690]  dump_stack+0x82/0xb3
[  907.394377]  nmi_cpu_backtrace+0x90/0xa0
[  907.395963]  ? lapic_can_unplug_cpu+0xa0/0xa0
[  907.397530]  nmi_trigger_cpumask_backtrace+0x73/0x130
[  907.399058]  rcu_dump_cpu_stacks+0x97/0xc5
[  907.400560]  rcu_sched_clock_irq+0x6ec/0x980
[  907.402046]  ? tick_sched_do_timer+0x70/0x70
[  907.403521]  update_process_times+0x24/0x50
[  907.404996]  tick_sched_handle+0x25/0x60
[  907.406464]  tick_sched_timer+0x37/0x70
[  907.407866]  __hrtimer_run_queues+0xe8/0x450
[  907.409292]  hrtimer_interrupt+0x122/0x270
[  907.410635]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[  907.412051]  smp_apic_timer_interrupt+0x86/0x240
[  907.413394]  apic_timer_interrupt+0xf/0x20
[  907.414659]  </IRQ>
[  907.415923] RIP: 0010:lock_release+0x142/0x2a0
[  907.417209] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[  907.419901] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  907.421333] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[  907.422706] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[  907.424108] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[  907.425499] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[  907.426863] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[  907.428317]  ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[  907.429691]  ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[  907.431097]  _raw_spin_unlock_bh+0x17/0x30
[  907.432502]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[  907.433888]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[  907.435322]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  907.436689]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  907.438148]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[  907.439511]  ? remove_wait_queue+0x60/0x60
[  907.440901]  kthread+0x103/0x140
[  907.442332]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[  907.443714]  ? kthread_bind+0x10/0x10
[  907.445122]  ret_from_fork+0x3a/0x50
[  935.135942] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[  935.136806] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[  935.136838]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  935.146955] irq event stamp: 0
[  935.147949] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  935.148945] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  935.150016] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  935.150950] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  935.151901] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  935.152847] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  935.153834] RIP: 0010:lock_acquire+0xd5/0x1c0

Message from syslogd@mehrangarh at Feb 10 15:35:06 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[  935.154807] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[  935.156848] RSP: 0018:ffffa6370223fd00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  935.157878] RAX: ffff8951948fa0c0 RBX: 0000000000000246 RCX: 0000000000000000
[  935.158913] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[  935.160016] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[  935.161037] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[  935.162054] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[  935.163040] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[  935.164025] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  935.165025] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[  935.166016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  935.167019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  935.168018] Call Trace:
[  935.169010]  ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[  935.170006]  _raw_spin_lock_bh+0x3d/0x50
[  935.171016]  ? iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[  935.172037]  iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[  935.173058]  __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[  935.174079]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[  935.175097]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  935.176115]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  935.177118]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[  935.178139]  ? remove_wait_queue+0x60/0x60
[  935.179143]  kthread+0x103/0x140
[  935.180152]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[  935.181156]  ? kthread_bind+0x10/0x10
[  935.182167]  ret_from_fork+0x3a/0x50
[  963.135798] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[  963.136936] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[  963.136962]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  963.148871] irq event stamp: 0
[  963.149921] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  963.150979] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  963.152059] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  963.153059] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  963.154028] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  963.155023] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  963.156018] RIP: 0010:iscsit_free_r2ts_from_list+0x10/0x70 [iscsi_target_mod]
[  963.157006] Code: d8 01 00 00 be ff ff ff ff e8 0c 8e 22 ed 85 c0 75 ab 0f 0b eb a7 0f 1f 40 00 0f 1f 44 00 00 41 55 4c 8d af c0 01 00 00 41 54 <55> 48 89 fd 4c 89 ef 4c 8d a5 08 02 00 00 53 e8 0c de a9 ed 48 8b
[  963.158989] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  963.160009] RAX: 0000000000000000 RBX: ffff895065a14a90 RCX: 0000000000000000
[  963.161022] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffff895065a14a90
[  963.162043] RBP: ffff895065a14da0 R08: 0000000000000000 R09: 0000000000000000
[  963.163084] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[  963.164123] R13: ffff895065a14c50 R14: ffff895065a14d90 R15: ffff895065a14a90
[  963.165141] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[  963.166185] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  963.167227] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[  963.168265] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  963.169298] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  963.170340] Call Trace:
[  963.171379]  __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[  963.172422]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[  963.173444]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  963.174487]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  963.175520]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[  963.176577]  ? remove_wait_queue+0x60/0x60
[  963.177612]  kthread+0x103/0x140
[  963.178629]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[  963.179658]  ? kthread_bind+0x10/0x10
[  963.180687]  ret_from_fork+0x3a/0x50
[  984.025504] INFO: task iscsi_np:7853 blocked for more than 122 seconds.
[  984.027401]       Tainted: G        W    L    5.6.0-rc1+ #18
[  984.029326] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  984.031370] iscsi_np        D    0  7853      2 0x80004084
[  984.033356] Call Trace:
[  984.035312]  ? __schedule+0x38c/0x990
[  984.037350]  schedule+0x5f/0xd0
[  984.039211]  schedule_timeout+0x1e4/0x310
[  984.041057]  ? wait_for_completion+0x49/0x190
[  984.042900]  wait_for_completion+0x12e/0x190
[  984.044712]  ? wake_up_q+0xa0/0xa0
[  984.046268]  iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[  984.047983]  iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[  984.049674]  iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[  984.051169]  iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[  984.052806]  kthread+0x103/0x140
[  984.054412]  ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[  984.055970]  ? kthread_bind+0x10/0x10
[  984.057541]  ret_from_fork+0x3a/0x50
[  984.059021] INFO: lockdep is turned off.
[  991.135707] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[  991.136638] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[  991.136668]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[  991.147124] irq event stamp: 0
[  991.148106] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[  991.149103] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[  991.150110] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80

Message from syslogd@mehrangarh at Feb 10 15:36:02 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[  991.151077] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  991.152037] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[  991.153013] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[  991.153994] RIP: 0010:__is_kernel_percpu_address+0x41/0xa0
[  991.154951] Code: 00 49 81 ec 00 00 00 00 55 48 c7 c5 00 00 00 00 53 48 89 fb 48 c7 c6 20 69 6c af 89 c7 e8 a7 34 6e 00 3b 05 65 c8 44 01 73 4c <89> c2 48 89 e9 48 03 0c d5 80 59 1c af 48 39 d9 48 89 ca 77 d5 4a
[  991.156945] RSP: 0018:ffffa6370223fce8 EFLAGS: 00000283 ORIG_RAX: ffffffffffffff13
[  991.157964] RAX: 000000000000000e RBX: ffffffffc0ebe660 RCX: 0000000000000000
[  991.158965] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[  991.159964] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000c000
[  991.160981] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[  991.161973] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[  991.162958] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[  991.163957] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  991.164951] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[  991.165953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  991.166959] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  991.167945] Call Trace:
[  991.168948]  static_obj+0x25/0x50
[  991.169950]  lockdep_init_map+0x79/0x1f0
[  991.170953]  __init_waitqueue_head+0x36/0x50
[  991.171967]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[  991.172982]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[  991.173971]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[  991.174964]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[  991.175979]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[  991.176990]  ? remove_wait_queue+0x60/0x60
[  991.177992]  kthread+0x103/0x140
[  991.178969]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[  991.179960]  ? kthread_bind+0x10/0x10
[  991.180967]  ret_from_fork+0x3a/0x50
[ 1019.135673] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1019.136829] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1019.136854]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1019.148830] irq event stamp: 0
[ 1019.149870] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1019.150923] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1019.151990] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1019.153005] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1019.154006] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1019.155002] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1019.155993] RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x52
[ 1019.156982] Code: fd 48 8d 7f 18 53 48 89 f3 48 8b 74 24 10 e8 55 ef 78 ff 48 89 ef e8 9d 3b 79 ff f6 c7 02 74 19 e8 b3 11 85 ff 48 89 df 57 9d <0f> 1f 44 00 00 65 ff 0d 9a 31 69 51 5b 5d c3 48 89 df 57 9d 0f 1f
[ 1019.158939] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[ 1019.159927] RAX: 0000000000000001 RBX: 0000000000000296 RCX: 0000000000000000
[ 1019.160919] RDX: 000000000000000b RSI: ffffffffc0e84e4a RDI: 0000000000000296
[ 1019.161931] RBP: ffff895065a14ea8 R08: 0000000000000000 R09: 0000000000000001
[ 1019.162946] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1019.163963] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1019.164974] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1019.165988] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1019.166997] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1019.168014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1019.169030] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1019.170041] Call Trace:
[ 1019.171064]  transport_generic_free_cmd+0x12a/0x160 [target_core_mod]
[ 1019.172104]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1019.173139]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1019.174177]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1019.175201]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1019.176238]  ? remove_wait_queue+0x60/0x60
[ 1019.177268]  kthread+0x103/0x140
[ 1019.178288]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1019.179298]  ? kthread_bind+0x10/0x10
[ 1019.180322]  ret_from_fork+0x3a/0x50
[ 1047.135629] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1047.136825] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1047.136848]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1047.149702] irq event stamp: 0
[ 1047.150766] hardirqs last  enabled at (0): [<0000000000000000>] 0x0

Message from syslogd@mehrangarh at Feb 10 15:36:58 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1047.151853] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1047.152927] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1047.153951] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1047.154970] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1047.156013] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1047.157062] RIP: 0010:lock_release+0x142/0x2a0
[ 1047.158097] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1047.160204] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1047.161261] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1047.162302] RDX: 0000000000000000 RSI: ffffffffc0ee80fe RDI: 0000000000000246
[ 1047.163336] RBP: ffff89506770cdf8 R08: 0000000000000000 R09: 0000000000000000
[ 1047.164334] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80fe
[ 1047.165339] R13: 0000000000000246 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1047.166356] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1047.167385] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1047.168423] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1047.169442] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1047.170474] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1047.171469] Call Trace:
[ 1047.172487]  _raw_spin_unlock_bh+0x17/0x30
[ 1047.173521]  __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1047.174554]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1047.175576]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1047.176606]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1047.177692]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1047.178727]  ? remove_wait_queue+0x60/0x60
[ 1047.179776]  kthread+0x103/0x140
[ 1047.180795]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1047.181805]  ? kthread_bind+0x10/0x10
[ 1047.182825]  ret_from_fork+0x3a/0x50
[ 1075.135523] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1075.136720] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1075.136744]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1075.149337] irq event stamp: 0

Message from syslogd@mehrangarh at Feb 10 15:37:26 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1075.150374] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1075.151422] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1075.152498] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1075.153528] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1075.154602] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1075.155645] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1075.156685] RIP: 0010:_raw_spin_unlock_bh+0x5/0x30
[ 1075.157713] Code: 48 89 fb 48 8d 7f 18 e8 99 f6 78 ff 48 89 df e8 e1 42 79 ff 65 ff 0d f2 38 69 51 5b c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <53> 48 8b 74 24 08 48 89 fb 48 8d 7f 18 e8 69 f6 78 ff 48 89 df e8
[ 1075.159780] RSP: 0018:ffffa6370223fda0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1075.160800] RAX: ffff895065a14c98 RBX: ffff895065a14a90 RCX: 0000000000000000
[ 1075.161832] RDX: ffff895065a14c98 RSI: 0000000000000000 RDI: ffff895065a14c50
[ 1075.162858] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[ 1075.163879] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1075.164891] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1075.165914] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1075.166931] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1075.167961] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1075.168992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1075.170016] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1075.171037] Call Trace:
[ 1075.172049]  __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1075.173098]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1075.174112]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1075.175131]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1075.176198]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1075.177219]  ? remove_wait_queue+0x60/0x60
[ 1075.178239]  kthread+0x103/0x140
[ 1075.179251]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1075.180288]  ? kthread_bind+0x10/0x10
[ 1075.181310]  ret_from_fork+0x3a/0x50
[ 1102.382142] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1102.383188] rcu:     11-....: (259573 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=64897
[ 1102.384357]  (t=260003 jiffies g=68413 q=44407)
[ 1102.385505] NMI backtrace for cpu 11
[ 1102.386852] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1102.388898] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1102.390929] Call Trace:
[ 1102.392907]  <IRQ>
[ 1102.394761]  dump_stack+0x82/0xb3
[ 1102.396614]  nmi_cpu_backtrace+0x90/0xa0
[ 1102.398356]  ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1102.400051]  nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1102.401788]  rcu_dump_cpu_stacks+0x97/0xc5
[ 1102.403461]  rcu_sched_clock_irq+0x6ec/0x980
[ 1102.405023]  ? tick_sched_do_timer+0x70/0x70
[ 1102.406624]  update_process_times+0x24/0x50
[ 1102.408100]  tick_sched_handle+0x25/0x60
[ 1102.409665]  tick_sched_timer+0x37/0x70
[ 1102.411091]  __hrtimer_run_queues+0xe8/0x450
[ 1102.412659]  hrtimer_interrupt+0x122/0x270
[ 1102.414115]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1102.415636]  smp_apic_timer_interrupt+0x86/0x240
[ 1102.417044]  apic_timer_interrupt+0xf/0x20
[ 1102.418537]  </IRQ>
[ 1102.419888] RIP: 0010:__is_kernel_percpu_address+0x91/0xa0
[ 1102.421233] Code: 49 89 5d 00 48 63 05 5e 14 ea 01 48 03 1c c5 80 59 1c af b8 01 00 00 00 48 01 dd 5b 49 89 6d 00 5d 41 5c 41 5d c3 31 c0 5b 5d <41> 5c 41 5d c3 b8 01 00 00 00 eb f2 0f 1f 00 0f 1f 44 00 00 31 f6
[ 1102.424069] RSP: 0018:ffffa6370223fcf8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1102.425575] RAX: 0000000000000000 RBX: ffffffffc0ebe660 RCX: 0000000000000010
[ 1102.426993] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1102.428511] RBP: ffffffffc0ebe660 R08: 0000000000000000 R09: 0000000000008000
[ 1102.429927] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1102.431383] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1102.432805]  static_obj+0x25/0x50
[ 1102.434186]  lockdep_init_map+0x79/0x1f0
[ 1102.435694]  __init_waitqueue_head+0x36/0x50
[ 1102.437089]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1102.438589]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1102.439977]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1102.441437]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1102.442882]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1102.444296]  ? remove_wait_queue+0x60/0x60
[ 1102.445738]  kthread+0x103/0x140
[ 1102.447100]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1102.448572]  ? kthread_bind+0x10/0x10
[ 1102.449947]  ret_from_fork+0x3a/0x50
[ 1106.905189] INFO: task iscsi_np:7853 blocked for more than 245 seconds.
[ 1106.906566]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1106.907916] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1106.909399] iscsi_np        D    0  7853      2 0x80004084
[ 1106.910762] Call Trace:
[ 1106.912196]  ? __schedule+0x38c/0x990
[ 1106.913533]  schedule+0x5f/0xd0
[ 1106.914900]  schedule_timeout+0x1e4/0x310
[ 1106.916261]  ? wait_for_completion+0x49/0x190
[ 1106.917555]  wait_for_completion+0x12e/0x190
[ 1106.918923]  ? wake_up_q+0xa0/0xa0
[ 1106.920314]  iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1106.921641]  iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1106.923045]  iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1106.924553]  iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1106.926007]  kthread+0x103/0x140
[ 1106.927358]  ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1106.928857]  ? kthread_bind+0x10/0x10
[ 1106.930320]  ret_from_fork+0x3a/0x50
[ 1106.931678] INFO: task sosreport:8582 blocked for more than 122 seconds.
[ 1106.933131]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1106.934651] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1106.936132] sosreport       D    0  8582   8581 0x00000080
[ 1106.937680] Call Trace:
[ 1106.939173]  ? __schedule+0x38c/0x990
[ 1106.940620]  ? avc_has_perm_noaudit+0x37/0x250
[ 1106.942108]  schedule+0x5f/0xd0
[ 1106.943618]  schedule_timeout+0x1e4/0x310
[ 1106.945022]  ? wait_for_completion+0x49/0x190
[ 1106.946519]  wait_for_completion+0x12e/0x190
[ 1106.947908]  ? wake_up_q+0xa0/0xa0
[ 1106.949165]  __wait_rcu_gp+0xfd/0x140
[ 1106.950607]  synchronize_rcu.part.75+0x3e/0x70
[ 1106.951963]  ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1106.953335]  ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1106.954621]  rcu_sync_enter+0x54/0xf0
[ 1106.955974]  percpu_down_write+0x15/0xe0
[ 1106.957342]  ? seq_read+0x327/0x3e0
[ 1106.958591]  locks_start+0x28/0x50
[ 1106.959913]  seq_read+0x1da/0x3e0
[ 1106.961229]  proc_reg_read+0x3c/0x70
[ 1106.962444]  vfs_read+0xac/0x160
[ 1106.963763]  ksys_read+0x5c/0xd0
[ 1106.965020]  do_syscall_64+0x60/0x270
[ 1106.966272]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1106.967441] RIP: 0033:0x7f37d1ec3f70
[ 1106.968726] Code: Bad RIP value.
[ 1106.969974] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1106.971227] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1106.972487] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1106.973762] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1106.975005] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1106.976185] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1106.977384] INFO: lockdep is turned off.
[ 1127.135327] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1127.136074] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1127.136106]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1127.145283] irq event stamp: 0

Message from syslogd@mehrangarh at Feb 10 15:38:18 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1127.146134] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1127.146989] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1127.147875] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1127.148752] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1127.149629] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1127.150506] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1127.151389] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 1127.152213] Code: fa 66 0f 1f 44 00 00 65 01 35 0c 10 f7 51 65 8b 05 05 10 f7 51 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 cd de 06 00 48 89 df 57 9d <0f> 1f 44 00 00 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 1127.154020] RSP: 0018:ffffa6370223fd58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1127.154972] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 1127.155941] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: 0000000000000246
[ 1127.156908] RBP: ffff895065a14a90 R08: 0000000000000000 R09: 0000000000000000
[ 1127.157873] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffff895065a14c98
[ 1127.158826] R13: ffff895065a14c50 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1127.159797] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1127.160780] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1127.161767] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1127.162766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1127.163768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1127.164768] Call Trace:
[ 1127.165767]  _raw_spin_lock_bh+0x1c/0x50
[ 1127.166776]  iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 1127.167799]  __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1127.168823]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1127.169848]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1127.170884]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1127.171918]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1127.172944]  ? remove_wait_queue+0x60/0x60
[ 1127.173965]  kthread+0x103/0x140
[ 1127.174983]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1127.175993]  ? kthread_bind+0x10/0x10
[ 1127.177019]  ret_from_fork+0x3a/0x50
[ 1155.135221] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1155.136412] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1155.136435]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1155.148788] irq event stamp: 0

Message from syslogd@mehrangarh at Feb 10 15:38:46 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1155.149801] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1155.150826] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1155.151842] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1155.152873] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1155.153897] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1155.154928] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1155.155943] RIP: 0010:lock_release+0x142/0x2a0
[ 1155.156947] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1155.159007] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1155.160033] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1155.161053] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1155.162071] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1155.163102] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1155.164105] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1155.165108] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1155.166114] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1155.167133] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1155.168157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1155.169177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1155.170201] Call Trace:
[ 1155.171287]  _raw_spin_unlock_bh+0x17/0x30
[ 1155.172319]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1155.173352]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1155.174379]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1155.175412]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1155.176452]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1155.177474]  ? remove_wait_queue+0x60/0x60
[ 1155.178479]  kthread+0x103/0x140
[ 1155.179477]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1155.180486]  ? kthread_bind+0x10/0x10
[ 1155.181507]  ret_from_fork+0x3a/0x50
[ 1183.135153] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1183.136353] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1183.136377]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1183.148928] irq event stamp: 0
[ 1183.149969] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1183.151016] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80

Message from syslogd@mehrangarh at Feb 10 15:39:14 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1183.152046] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1183.153057] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1183.154077] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1183.155089] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1183.156124] RIP: 0010:_find_next_bit.constprop.1+0x66/0x70
[ 1183.157196] Code: 8b 0c d7 74 04 4e 23 0c d6 4d 31 c1 75 0d 48 83 c1 40 48 39 ca 77 de 48 89 d0 c3 f3 49 0f bc c1 48 01 c8 48 39 d0 48 0f 47 c2 <c3> 66 0f 1f 84 00 00 00 00 00 45 31 c0 eb 8b 90 66 2e 0f 1f 84 00
[ 1183.159300] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000283 ORIG_RAX: ffffffffffffff13
[ 1183.160336] RAX: 0000000000000004 RBX: ffffffffc0ebe660 RCX: 0000000000000000
[ 1183.161371] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1183.162399] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000fff0
[ 1183.163426] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1183.164448] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1183.165465] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1183.166486] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1183.167503] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1183.168526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1183.169524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1183.170536] Call Trace:
[ 1183.171550]  cpumask_next+0x17/0x20
[ 1183.172553]  __is_kernel_percpu_address+0x39/0xa0
[ 1183.173564]  static_obj+0x25/0x50
[ 1183.174565]  lockdep_init_map+0x79/0x1f0
[ 1183.175565]  __init_waitqueue_head+0x36/0x50
[ 1183.176552]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1183.177564]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1183.178566]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1183.179569]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1183.180577]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1183.181583]  ? remove_wait_queue+0x60/0x60
[ 1183.182583]  kthread+0x103/0x140
[ 1183.183581]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1183.184581]  ? kthread_bind+0x10/0x10
[ 1183.185563]  ret_from_fork+0x3a/0x50
[ 1211.135147] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1211.136267] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1211.136286]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1211.147682] irq event stamp: 0

Message from syslogd@mehrangarh at Feb 10 15:39:42 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1211.148694] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1211.149722] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1211.150753] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1211.151746] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1211.152708] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1211.153690] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1211.154652] RIP: 0010:transport_generic_free_cmd+0x58/0x160 [target_core_mod]
[ 1211.155601] Code: 04 25 28 00 00 00 48 89 44 24 60 31 c0 e8 c0 a6 27 ed 85 ed c6 44 24 06 00 c6 44 24 07 00 0f 85 9b 00 00 00 f6 43 29 01 74 0a <f6> 43 06 02 0f 85 cd 00 00 00 80 7c 24 06 00 74 0c 48 8d 44 24 10
[ 1211.157568] RSP: 0018:ffffa6370223fd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1211.158575] RAX: 0000000000000001 RBX: ffff895065a14da0 RCX: 0000000000000000
[ 1211.159570] RDX: 000000000000000b RSI: ffffffffc0e84e4a RDI: 0000000000000296
[ 1211.160581] RBP: ffff895065a14ea8 R08: 0000000000000000 R09: 0000000000000001
[ 1211.161589] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1211.162574] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1211.163572] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1211.164579] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1211.165590] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1211.166604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1211.167620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1211.168629] Call Trace:
[ 1211.169644]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1211.170667]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1211.171702]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1211.172738]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1211.173765]  ? remove_wait_queue+0x60/0x60
[ 1211.174767]  kthread+0x103/0x140
[ 1211.175781]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1211.176800]  ? kthread_bind+0x10/0x10
[ 1211.177791]  ret_from_fork+0x3a/0x50
[ 1229.784929] INFO: task iscsi_np:7853 blocked for more than 368 seconds.
[ 1229.786796]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1229.788694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1229.790784] iscsi_np        D    0  7853      2 0x80004084
[ 1229.792835] Call Trace:
[ 1229.794782]  ? __schedule+0x38c/0x990
[ 1229.796717]  schedule+0x5f/0xd0
[ 1229.798633]  schedule_timeout+0x1e4/0x310
[ 1229.800384]  ? wait_for_completion+0x49/0x190
[ 1229.802226]  wait_for_completion+0x12e/0x190
[ 1229.803981]  ? wake_up_q+0xa0/0xa0
[ 1229.805554]  iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1229.807369]  iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1229.808991]  iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1229.810578]  iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1229.812228]  kthread+0x103/0x140
[ 1229.813771]  ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1229.815344]  ? kthread_bind+0x10/0x10
[ 1229.816853]  ret_from_fork+0x3a/0x50
[ 1229.818407] INFO: task sosreport:8582 blocked for more than 245 seconds.
[ 1229.819984]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1229.821556] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1229.823138] sosreport       D    0  8582   8581 0x00000080
[ 1229.824567] Call Trace:
[ 1229.826086]  ? __schedule+0x38c/0x990
[ 1229.827527]  ? avc_has_perm_noaudit+0x37/0x250
[ 1229.828984]  schedule+0x5f/0xd0
[ 1229.830284]  schedule_timeout+0x1e4/0x310
[ 1229.831712]  ? wait_for_completion+0x49/0x190
[ 1229.833162]  wait_for_completion+0x12e/0x190
[ 1229.834661]  ? wake_up_q+0xa0/0xa0
[ 1229.836146]  __wait_rcu_gp+0xfd/0x140
[ 1229.837500]  synchronize_rcu.part.75+0x3e/0x70
[ 1229.838864]  ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1229.840304]  ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1229.841524]  rcu_sync_enter+0x54/0xf0
[ 1229.842840]  percpu_down_write+0x15/0xe0
[ 1229.844237]  ? seq_read+0x327/0x3e0
[ 1229.845609]  locks_start+0x28/0x50
[ 1229.846936]  seq_read+0x1da/0x3e0
[ 1229.848216]  proc_reg_read+0x3c/0x70
[ 1229.849636]  vfs_read+0xac/0x160
[ 1229.850834]  ksys_read+0x5c/0xd0
[ 1229.852130]  do_syscall_64+0x60/0x270
[ 1229.853313]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1229.854604] RIP: 0033:0x7f37d1ec3f70
[ 1229.855872] Code: Bad RIP value.
[ 1229.857136] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1229.858305] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1229.859573] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1229.860752] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1229.861984] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1229.863156] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1229.864482] INFO: lockdep is turned off.
[ 1239.135142] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1239.135898] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1239.135929]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1239.145136] irq event stamp: 0
[ 1239.146022] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1239.146901] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1239.147748] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1239.148606] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1239.149481] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18

Message from syslogd@mehrangarh at Feb 10 15:40:10 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1239.150373] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1239.151275] RIP: 0010:_find_next_bit.constprop.1+0x19/0x70
[ 1239.152121] Code: c0 48 01 d0 c3 48 89 f0 f3 c3 0f 1f 80 00 00 00 00 48 39 d1 73 4e 48 89 c8 48 c1 e8 06 48 85 f6 4c 8b 0c c7 74 04 4c 23 0c c6 <48> c7 c0 ff ff ff ff 4d 31 c1 48 d3 e0 48 83 e1 c0 49 21 c1 74 1b
[ 1239.153959] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1239.154889] RAX: 0000000000000000 RBX: ffffffffc0ebe660 RCX: 000000000000000e
[ 1239.155824] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1239.156784] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000ffff
[ 1239.157754] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1239.158722] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1239.159691] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1239.160651] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1239.161628] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1239.162621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1239.163616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1239.164611] Call Trace:
[ 1239.165587]  cpumask_next+0x17/0x20
[ 1239.166584]  __is_kernel_percpu_address+0x39/0xa0
[ 1239.167589]  static_obj+0x25/0x50
[ 1239.168587]  lockdep_init_map+0x79/0x1f0
[ 1239.169565]  __init_waitqueue_head+0x36/0x50
[ 1239.170557]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1239.171570]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1239.172570]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1239.173557]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1239.174569]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1239.175561]  ? remove_wait_queue+0x60/0x60
[ 1239.176567]  kthread+0x103/0x140
[ 1239.177566]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1239.178567]  ? kthread_bind+0x10/0x10
[ 1239.179548]  ret_from_fork+0x3a/0x50
[ 1267.135137] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1267.136257] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1267.136276]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1267.147700] irq event stamp: 0
[ 1267.148682] hardirqs last  enabled at (0): [<0000000000000000>] 0x0

Message from syslogd@mehrangarh at Feb 10 15:40:38 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1267.149712] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1267.150726] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1267.151720] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1267.152703] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1267.153683] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1267.154654] RIP: 0010:lock_acquire+0x20/0x1c0
[ 1267.155612] Code: 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54 41 89 d4 65 48 8b 14 25 40 8e 01 00 55 53 48 83 ec 30 8b 82 bc 0c 00 00 <48> 89 7c 24 20 85 c0 0f 85 ad 00 00 00 89 f5 41 89 cd 45 89 c6 4d
[ 1267.157579] RSP: 0018:ffffa6370223fd28 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 1267.158586] RAX: 0000000000000000 RBX: ffff89506770cde0 RCX: 0000000000000000
[ 1267.159598] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: ffff89506770cdf8
[ 1267.160599] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[ 1267.161611] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1267.162618] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1267.163622] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1267.164638] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1267.165655] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1267.166660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1267.167685] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1267.168706] Call Trace:
[ 1267.169725]  _raw_spin_lock_bh+0x3d/0x50
[ 1267.170749]  ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1267.171783]  __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1267.172799]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1267.173830]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1267.174852]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1267.175903]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1267.176900]  ? remove_wait_queue+0x60/0x60
[ 1267.177889]  kthread+0x103/0x140
[ 1267.178880]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1267.179904]  ? kthread_bind+0x10/0x10
[ 1267.180918]  ret_from_fork+0x3a/0x50
[ 1295.135131] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1295.136324] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1295.136343]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1295.148740] irq event stamp: 0
[ 1295.149740] hardirqs last  enabled at (0): [<0000000000000000>] 0x0

Message from syslogd@mehrangarh at Feb 10 15:41:06 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1295.150752] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1295.151771] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1295.152784] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1295.153814] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1295.154834] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1295.155858] RIP: 0010:lock_release+0x142/0x2a0
[ 1295.156882] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1295.158947] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1295.159947] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1295.160945] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1295.161939] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1295.162962] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1295.163987] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1295.165006] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1295.166031] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1295.167037] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1295.168055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1295.169075] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1295.170096] Call Trace:
[ 1295.171138]  _raw_spin_unlock_bh+0x17/0x30
[ 1295.172209]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1295.173281]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1295.174291]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1295.175306]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1295.176342]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1295.177371]  ? remove_wait_queue+0x60/0x60
[ 1295.178376]  kthread+0x103/0x140
[ 1295.179389]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1295.180413]  ? kthread_bind+0x10/0x10
[ 1295.181506]  ret_from_fork+0x3a/0x50
[ 1297.384853] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1297.385888] rcu:     11-....: (454193 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=113554
[ 1297.387143]  (t=455006 jiffies g=68413 q=53587)
[ 1297.388237] NMI backtrace for cpu 11
[ 1297.389469] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1297.391434] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1297.393367] Call Trace:
[ 1297.395270]  <IRQ>
[ 1297.397059]  dump_stack+0x82/0xb3
[ 1297.398791]  nmi_cpu_backtrace+0x90/0xa0
[ 1297.400554]  ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1297.402254]  nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1297.403818]  rcu_dump_cpu_stacks+0x97/0xc5
[ 1297.405460]  rcu_sched_clock_irq+0x6ec/0x980
[ 1297.406994]  ? tick_sched_do_timer+0x70/0x70
[ 1297.408524]  update_process_times+0x24/0x50
[ 1297.410024]  tick_sched_handle+0x25/0x60
[ 1297.411530]  tick_sched_timer+0x37/0x70
[ 1297.413023]  __hrtimer_run_queues+0xe8/0x450
[ 1297.414514]  hrtimer_interrupt+0x122/0x270
[ 1297.415973]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1297.417445]  smp_apic_timer_interrupt+0x86/0x240
[ 1297.418840]  apic_timer_interrupt+0xf/0x20
[ 1297.420274]  </IRQ>
[ 1297.421576] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1297.422893] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1297.425691] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1297.427145] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1297.428596] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1297.430067] RBP: ffffffffc0ee80fe R08: 0000000000000000 R09: 0000000000000000
[ 1297.431489] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1297.432884] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1297.434335]  ? __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1297.435723]  ? __local_bh_enable_ip+0x80/0xf0
[ 1297.437226]  __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1297.438628]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1297.440064]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1297.441526]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1297.442964]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1297.444449]  ? remove_wait_queue+0x60/0x60
[ 1297.445855]  kthread+0x103/0x140
[ 1297.447323]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1297.448722]  ? kthread_bind+0x10/0x10
[ 1297.450222]  ret_from_fork+0x3a/0x50
[ 1323.135126] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1323.135996] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1323.136022]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1323.146274] irq event stamp: 0
[ 1323.147279] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1323.148301] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1323.149331] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1323.150313] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1323.151292] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1323.152291] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1323.153301] RIP: 0010:__module_address+0x5c/0xf0
[ 1323.154297] Code: 83 e2 01 48 8d 42 04 48 8b 04 c5 c8 9a 40 af 48 85 c0 74 2f 48 8d 3c 12 48 01 d7 48 c1 e7 03 48 f7 df 48 8d 14 38 48 8b 4a e0 <48> 39 cb 72 61 8b 72 e8 48 01 f1 48 39 cb 72 0b 48 8b 40 08 48 85
[ 1323.156379] RSP: 0018:ffffa6370223fcf8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 1323.157441] RAX: ffffffffc0a5ade0 RBX: ffffffffc0ebe660 RCX: ffffffffc0a51000
[ 1323.158510] RDX: ffffffffc0a5ade0 RSI: 000000000001a000 RDI: 0000000000000000
[ 1323.159580] RBP: ffffffffc0ebe660 R08: 0000000000000522 R09: 0000000000008000
[ 1323.160646] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1323.161695] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1323.162726] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1323.163756] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1323.164762] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1323.165793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1323.166804] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1323.167824] Call Trace:
[ 1323.168829]  is_module_address+0x11/0x20
[ 1323.169826]  static_obj+0x31/0x50
[ 1323.170813]  lockdep_init_map+0x79/0x1f0
[ 1323.171798]  __init_waitqueue_head+0x36/0x50
[ 1323.172788]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1323.173782]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1323.174786]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1323.175773]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1323.176781]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1323.177788]  ? remove_wait_queue+0x60/0x60
[ 1323.178771]  kthread+0x103/0x140
[ 1323.179751]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1323.180742]  ? kthread_bind+0x10/0x10
[ 1323.181735]  ret_from_fork+0x3a/0x50
[ 1351.135121] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1351.136243] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1351.136262]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1351.147911] irq event stamp: 0
[ 1351.148885] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1351.149908] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1351.150941] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1351.151930] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1351.152877] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1351.153871] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1351.154832] RIP: 0010:_find_next_bit.constprop.1+0x19/0x70
[ 1351.155793] Code: c0 48 01 d0 c3 48 89 f0 f3 c3 0f 1f 80 00 00 00 00 48 39 d1 73 4e 48 89 c8 48 c1 e8 06 48 85 f6 4c 8b 0c c7 74 04 4c 23 0c c6 <48> c7 c0 ff ff ff ff 4d 31 c1 48 d3 e0 48 83 e1 c0 49 21 c1 74 1b
[ 1351.157773] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1351.158780] RAX: 0000000000000000 RBX: ffffffffc0ebe660 RCX: 0000000000000008
[ 1351.159778] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1351.160794] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000ffff
[ 1351.161805] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1351.162812] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1351.163800] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1351.164803] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1351.165794] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1351.166810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1351.167808] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1351.168804] Call Trace:
[ 1351.169799]  cpumask_next+0x17/0x20
[ 1351.170787]  __is_kernel_percpu_address+0x39/0xa0
[ 1351.171781]  static_obj+0x25/0x50
[ 1351.172777]  lockdep_init_map+0x79/0x1f0
[ 1351.173759]  __init_waitqueue_head+0x36/0x50
[ 1351.174764]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1351.175772]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1351.176756]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1351.177760]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1351.178772]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1351.179761]  ? remove_wait_queue+0x60/0x60
[ 1351.180761]  kthread+0x103/0x140
[ 1351.181762]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1351.182764]  ? kthread_bind+0x10/0x10
[ 1351.183729]  ret_from_fork+0x3a/0x50
[ 1352.664902] INFO: task iscsi_np:7853 blocked for more than 491 seconds.
[ 1352.666638]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1352.668358] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1352.670191] iscsi_np        D    0  7853      2 0x80004084
[ 1352.672011] Call Trace:
[ 1352.673617]  ? __schedule+0x38c/0x990
[ 1352.675307]  schedule+0x5f/0xd0
[ 1352.676907]  schedule_timeout+0x1e4/0x310
[ 1352.678374]  ? wait_for_completion+0x49/0x190
[ 1352.679886]  wait_for_completion+0x12e/0x190
[ 1352.681345]  ? wake_up_q+0xa0/0xa0
[ 1352.682810]  iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1352.684332]  iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1352.685836]  iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1352.687433]  iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1352.688927]  kthread+0x103/0x140
[ 1352.690354]  ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1352.691812]  ? kthread_bind+0x10/0x10
[ 1352.693301]  ret_from_fork+0x3a/0x50
[ 1352.694722] INFO: task sosreport:8582 blocked for more than 368 seconds.
[ 1352.696182]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1352.697561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1352.698967] sosreport       D    0  8582   8581 0x00000080
[ 1352.700316] Call Trace:
[ 1352.701723]  ? __schedule+0x38c/0x990
[ 1352.703194]  ? avc_has_perm_noaudit+0x37/0x250
[ 1352.704651]  schedule+0x5f/0xd0
[ 1352.706020]  schedule_timeout+0x1e4/0x310
[ 1352.707434]  ? wait_for_completion+0x49/0x190
[ 1352.708787]  wait_for_completion+0x12e/0x190
[ 1352.710234]  ? wake_up_q+0xa0/0xa0
[ 1352.711614]  __wait_rcu_gp+0xfd/0x140
[ 1352.712952]  synchronize_rcu.part.75+0x3e/0x70
[ 1352.714257]  ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1352.715603]  ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1352.716954]  rcu_sync_enter+0x54/0xf0
[ 1352.718259]  percpu_down_write+0x15/0xe0
[ 1352.719660]  ? seq_read+0x327/0x3e0
[ 1352.721039]  locks_start+0x28/0x50
[ 1352.722319]  seq_read+0x1da/0x3e0
[ 1352.723677]  proc_reg_read+0x3c/0x70
[ 1352.724939]  vfs_read+0xac/0x160
[ 1352.726243]  ksys_read+0x5c/0xd0
[ 1352.727516]  do_syscall_64+0x60/0x270
[ 1352.728778]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1352.730102] RIP: 0033:0x7f37d1ec3f70
[ 1352.731311] Code: Bad RIP value.
[ 1352.732578] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1352.733875] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1352.735194] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1352.736503] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1352.737734] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1352.739001] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1352.740269] INFO: lockdep is turned off.
[ 1379.135114] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1379.135862] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1379.135889]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1379.145062] irq event stamp: 0
[ 1379.145955] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1379.146806] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1379.147676] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1379.148552] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1379.149429] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1379.150324] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1379.151225] RIP: 0010:lock_release+0x14c/0x2a0
[ 1379.152060] Code: 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d 0f 1f 44 00 00 48 8b 44 24 08 <65> 48 33 04 25 28 00 00 00 0f 85 e9 00 00 00 48 83 c4 10 5b 5d 41
[ 1379.153879] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1379.154802] RAX: fe6fc5eb227b1400 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1379.155758] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246

Message from syslogd@mehrangarh at Feb 10 15:42:30 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1379.156725] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1379.157700] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1379.158679] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1379.159640] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1379.160628] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1379.161602] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1379.162604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1379.163611] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1379.164599] Call Trace:
[ 1379.165605]  _raw_spin_unlock_bh+0x17/0x30
[ 1379.166604]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1379.167630]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1379.168657]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1379.169688]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1379.170713]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1379.171730]  ? remove_wait_queue+0x60/0x60
[ 1379.172753]  kthread+0x103/0x140
[ 1379.173756]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1379.174767]  ? kthread_bind+0x10/0x10
[ 1379.175787]  ret_from_fork+0x3a/0x50
[ 1407.135105] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1407.136304] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1407.136323]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1407.148898] irq event stamp: 0
[ 1407.149911] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1407.150934] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1407.151988] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1407.152994] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1407.154011] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1407.155025] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1407.156071] RIP: 0010:target_put_sess_cmd+0x19/0x50 [target_core_mod]
[ 1407.157115] Code: ed e9 aa fe ff ff 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 97 40 01 00 00 b8 ff ff ff ff f0 0f c1 87 40 01 00 00 <83> f8 01 74 07 85 c0 7e 11 31 c0 c3 48 89 d7 e8 23 fb ff ff b8 01

Message from syslogd@mehrangarh at Feb 10 15:42:58 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1407.159219] RSP: 0018:ffffa6370223fe08 EFLAGS: 00000287 ORIG_RAX: ffffffffffffff13
[ 1407.160256] RAX: 00000000c0000000 RBX: ffff89506770c800 RCX: 0000000000000000
[ 1407.161288] RDX: ffff895065a14ee0 RSI: 0000000000000201 RDI: ffff895065a14da0
[ 1407.162318] RBP: ffffa6370223fe38 R08: 0000000000000000 R09: 0000000000000000
[ 1407.163333] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffff89506770e000
[ 1407.164367] R13: ffff895065a14a90 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1407.165379] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1407.166420] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1407.167451] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1407.168488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1407.169521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1407.170550] Call Trace:
[ 1407.171583]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1407.172627]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1407.173661]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1407.174709]  ? remove_wait_queue+0x60/0x60
[ 1407.175751]  kthread+0x103/0x140
[ 1407.176772]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1407.177814]  ? kthread_bind+0x10/0x10
[ 1407.178858]  ret_from_fork+0x3a/0x50
[ 1435.135096] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1435.136305] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1435.136325]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1435.149704] irq event stamp: 0
[ 1435.150793] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1435.151897] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1435.152998] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1435.154042] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1435.155043] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1435.156084] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1435.157161] RIP: 0010:find_next_bit+0x9/0x10
[ 1435.158212] Code: ee 81 e6 ff 00 00 00 49 89 34 24 41 5c c3 48 89 d8 5b 5d 41 5c c3 90 66 2e 0f 1f 84 00 00 00 00 00 48 89 d1 45 31 c0 48 89 f2 <31> f6 e9 f0 fe ff ff 48 8b 0a 48 89 0e 48 89 c8 f0 48 0f b1 3a 48
[ 1435.160373] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1435.161441] RAX: ffffffffaf6c6920 RBX: ffffffffc0ebe660 RCX: 000000000000000e

Message from syslogd@mehrangarh at Feb 10 15:43:26 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1435.162524] RDX: 0000000000000010 RSI: 0000000000000010 RDI: ffffffffaf6c6920
[ 1435.163595] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000e000
[ 1435.164646] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1435.165684] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1435.166699] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1435.167717] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1435.168751] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1435.169790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1435.170820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1435.171852] Call Trace:
[ 1435.172814]  cpumask_next+0x17/0x20
[ 1435.173810]  __is_kernel_percpu_address+0x39/0xa0
[ 1435.174813]  static_obj+0x25/0x50
[ 1435.175791]  lockdep_init_map+0x79/0x1f0
[ 1435.176786]  __init_waitqueue_head+0x36/0x50
[ 1435.177782]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1435.178793]  ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 1435.179786]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1435.180776]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1435.181787]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1435.182797]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1435.183793]  ? remove_wait_queue+0x60/0x60
[ 1435.184790]  kthread+0x103/0x140
[ 1435.185776]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1435.186766]  ? kthread_bind+0x10/0x10
[ 1435.187717]  ret_from_fork+0x3a/0x50
[ 1463.135086] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1463.136192] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1463.136214]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1463.147526] irq event stamp: 0
[ 1463.148550] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1463.149576] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1463.150602] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1463.151594] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1463.152568] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1463.153519] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1463.154488] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 1463.155428] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c ef 51 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c ef 51 89 43 08 65 48 8b 04 25
[ 1463.157402] RSP: 0018:ffffa6370223fd80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1463.158414] RAX: 0000000000000000 RBX: ffff895065a14ba8 RCX: 0000000000000000

Message from syslogd@mehrangarh at Feb 10 15:43:54 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1463.159431] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff895065a14ba8
[ 1463.160445] RBP: ffff895065a14ba8 R08: 0000000000000001 R09: 0000000000000000
[ 1463.161454] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1463.162459] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1463.163467] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1463.164475] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1463.165492] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1463.166521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1463.167551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1463.168577] Call Trace:
[ 1463.169610]  iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1463.170653]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1463.171681]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1463.172722]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1463.173768]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1463.174839]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1463.175880]  ? remove_wait_queue+0x60/0x60
[ 1463.176912]  kthread+0x103/0x140
[ 1463.177936]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1463.178971]  ? kthread_bind+0x10/0x10
[ 1463.179974]  ret_from_fork+0x3a/0x50
[ 1475.544863] INFO: task iscsi_np:7853 blocked for more than 614 seconds.
[ 1475.546777]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1475.548695] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1475.550755] iscsi_np        D    0  7853      2 0x80004084
[ 1475.552779] Call Trace:
[ 1475.554699]  ? __schedule+0x38c/0x990
[ 1475.556663]  schedule+0x5f/0xd0
[ 1475.558522]  schedule_timeout+0x1e4/0x310
[ 1475.560343]  ? wait_for_completion+0x49/0x190
[ 1475.562164]  wait_for_completion+0x12e/0x190
[ 1475.563903]  ? wake_up_q+0xa0/0xa0
[ 1475.565483]  iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1475.567206]  iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1475.568704]  iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1475.570395]  iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1475.571950]  kthread+0x103/0x140
[ 1475.573433]  ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1475.575017]  ? kthread_bind+0x10/0x10
[ 1475.576548]  ret_from_fork+0x3a/0x50
[ 1475.578194] INFO: task sosreport:8582 blocked for more than 491 seconds.
[ 1475.579670]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1475.581319] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1475.582873] sosreport       D    0  8582   8581 0x00000080
[ 1475.584374] Call Trace:
[ 1475.585905]  ? __schedule+0x38c/0x990
[ 1475.587352]  ? avc_has_perm_noaudit+0x37/0x250
[ 1475.588790]  schedule+0x5f/0xd0
[ 1475.590313]  schedule_timeout+0x1e4/0x310
[ 1475.591707]  ? wait_for_completion+0x49/0x190
[ 1475.593253]  wait_for_completion+0x12e/0x190
[ 1475.594683]  ? wake_up_q+0xa0/0xa0
[ 1475.596185]  __wait_rcu_gp+0xfd/0x140
[ 1475.597602]  synchronize_rcu.part.75+0x3e/0x70
[ 1475.599036]  ? rcu_cpu_kthread+0x3d0/0x3d0
[ 1475.600361]  ? __bpf_trace_rcu_utilization+0x10/0x10
[ 1475.601740]  rcu_sync_enter+0x54/0xf0
[ 1475.603213]  percpu_down_write+0x15/0xe0
[ 1475.604583]  ? seq_read+0x327/0x3e0
[ 1475.605968]  locks_start+0x28/0x50
[ 1475.607319]  seq_read+0x1da/0x3e0
[ 1475.608640]  proc_reg_read+0x3c/0x70
[ 1475.609957]  vfs_read+0xac/0x160
[ 1475.611240]  ksys_read+0x5c/0xd0
[ 1475.612532]  do_syscall_64+0x60/0x270
[ 1475.613786]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 1475.615058] RIP: 0033:0x7f37d1ec3f70
[ 1475.616266] Code: Bad RIP value.
[ 1475.617525] RSP: 002b:00007fff8bdc0da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1475.618770] RAX: ffffffffffffffda RBX: 0000000000e3b460 RCX: 00007f37d1ec3f70
[ 1475.620177] RDX: 0000000000004000 RSI: 0000000000e3e174 RDI: 000000000000000b
[ 1475.621477] RBP: 00007f37d21983a0 R08: ffffffffffffffff R09: 0000000000000000
[ 1475.622727] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000004000
[ 1475.624003] R13: 0000000000000d70 R14: 00007f37d2197858 R15: 0000000000e3e174
[ 1475.625272] INFO: lockdep is turned off.
[ 1491.135077] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1491.135822] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1491.135850]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1491.145026] irq event stamp: 0
[ 1491.145926] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1491.146788] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1491.147661] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1491.148519] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1491.149393] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1491.150287] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1491.151171] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1491.152002] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1491.153806] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1491.154733] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1491.155683] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0

Message from syslogd@mehrangarh at Feb 10 15:44:22 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1491.156634] RBP: ffffffffc0ee80b9 R08: 0000000000000000 R09: 0000000000000000
[ 1491.157591] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1491.158567] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1491.159545] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1491.160520] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1491.161514] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1491.162522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1491.163515] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1491.164507] Call Trace:
[ 1491.165518]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1491.166532]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1491.167547]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1491.168569]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1491.169613]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1491.170642]  ? remove_wait_queue+0x60/0x60
[ 1491.171661]  kthread+0x103/0x140
[ 1491.172671]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1491.173726]  ? kthread_bind+0x10/0x10
[ 1491.174756]  ret_from_fork+0x3a/0x50
[ 1492.387798] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1492.388846] rcu:     11-....: (648817 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=162211
[ 1492.390039]  (t=650009 jiffies g=68413 q=59730)
[ 1492.391114] NMI backtrace for cpu 11
[ 1492.392368] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1492.394391] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1492.396382] Call Trace:
[ 1492.398317]  <IRQ>
[ 1492.400218]  dump_stack+0x82/0xb3
[ 1492.402032]  nmi_cpu_backtrace+0x90/0xa0
[ 1492.403779]  ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1492.405517]  nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1492.407221]  rcu_dump_cpu_stacks+0x97/0xc5
[ 1492.408836]  rcu_sched_clock_irq+0x6ec/0x980
[ 1492.410470]  ? tick_sched_do_timer+0x70/0x70
[ 1492.412031]  update_process_times+0x24/0x50
[ 1492.413525]  tick_sched_handle+0x25/0x60
[ 1492.415050]  tick_sched_timer+0x37/0x70
[ 1492.416551]  __hrtimer_run_queues+0xe8/0x450
[ 1492.418090]  hrtimer_interrupt+0x122/0x270
[ 1492.419592]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1492.421126]  smp_apic_timer_interrupt+0x86/0x240
[ 1492.422552]  apic_timer_interrupt+0xf/0x20
[ 1492.424002]  </IRQ>
[ 1492.425395] RIP: 0010:lock_acquire+0xde/0x1c0
[ 1492.426726] Code: 41 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d 0f 1f 44 00 00 48 83 c4 30 <5b> 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 38 d4 ef 51 89 c0 48 0f
[ 1492.429577] RSP: 0018:ffffa6370223fd58 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[ 1492.431014] RAX: ffff8951948fa0c0 RBX: 0000000000000246 RCX: 0000000000000000
[ 1492.432456] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 1492.433890] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 1492.435334] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1492.436699] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 1492.438142]  _raw_spin_lock_bh+0x3d/0x50
[ 1492.439459]  ? __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1492.440830]  __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1492.442253]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1492.443622]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1492.445051]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1492.446440]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1492.447835]  ? remove_wait_queue+0x60/0x60
[ 1492.449262]  kthread+0x103/0x140
[ 1492.450604]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1492.452012]  ? kthread_bind+0x10/0x10
[ 1492.453400]  ret_from_fork+0x3a/0x50
[ 1519.135050] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1519.135886] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1519.135914]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1519.145816] irq event stamp: 0
[ 1519.146738] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1519.147678] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1519.148643] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1519.149603] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1519.150562] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1519.151562] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1519.152550] RIP: 0010:do_raw_spin_lock+0x3e/0xa0
[ 1519.153536] Code: 8b 43 10 65 48 8b 14 25 40 8e 01 00 48 39 c2 74 66 8b 53 08 65 8b 05 29 8c ef 51 39 c2 74 47 31 c0 ba 01 00 00 00 f0 0f b1 13 <0f> 94 c2 84 d2 74 27 65 8b 05 0c 8c ef 51 89 43 08 65 48 8b 04 25
[ 1519.155576] RSP: 0018:ffffa6370223fd98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1519.156620] RAX: 0000000000000000 RBX: ffff89506770cde0 RCX: 0000000000000000
[ 1519.157667] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff89506770cde0
[ 1519.158720] RBP: ffff895065a14da0 R08: 0000000000000001 R09: 0000000000000000
[ 1519.159777] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001

Message from syslogd@mehrangarh at Feb 10 15:44:50 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1519.160829] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1519.161844] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1519.162862] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1519.163871] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1519.164890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1519.165911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1519.166909] Call Trace:
[ 1519.167911]  __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1519.168944]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1519.169974]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1519.171007]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1519.172087]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1519.173128]  ? remove_wait_queue+0x60/0x60
[ 1519.174206]  kthread+0x103/0x140
[ 1519.175227]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1519.176241]  ? kthread_bind+0x10/0x10
[ 1519.177273]  ret_from_fork+0x3a/0x50
[ 1547.135018] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1547.136213] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1547.136235]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1547.149143] irq event stamp: 0
[ 1547.150238] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1547.151327] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1547.152406] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1547.153419] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1547.154446] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1547.155479] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1547.156534] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1547.157575] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1547.159702] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1547.160772] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1547.161824] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1547.162859] RBP: ffffffffc0ee80b9 R08: 0000000000000000 R09: 0000000000000000
[ 1547.163890] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1547.164916] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1547.165950] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1547.166996] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1547.168069] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1547.169112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1547.170149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1547.171175] Call Trace:
[ 1547.172203]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1547.173246]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1547.174273]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1547.175316]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1547.176365]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1547.177411]  ? remove_wait_queue+0x60/0x60
[ 1547.178429]  kthread+0x103/0x140
[ 1547.179474]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1547.180508]  ? kthread_bind+0x10/0x10
[ 1547.181544]  ret_from_fork+0x3a/0x50
[ 1575.134987] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1575.136187] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1575.136206]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1575.149191] irq event stamp: 0
[ 1575.150277] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1575.151354] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1575.152408] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1575.153441] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1575.154481] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1575.155531] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1575.156597] RIP: 0010:transport_generic_free_cmd+0x58/0x160 [target_core_mod]
[ 1575.157641] Code: 04 25 28 00 00 00 48 89 44 24 60 31 c0 e8 c0 a6 27 ed 85 ed c6 44 24 06 00 c6 44 24 07 00 0f 85 9b 00 00 00 f6 43 29 01 74 0a <f6> 43 06 02 0f 85 cd 00 00 00 80 7c 24 06 00 74 0c 48 8d 44 24 10
[ 1575.159771] RSP: 0018:ffffa6370223fd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1575.160810] RAX: 0000000000000001 RBX: ffff895065a14da0 RCX: 0000000000000000

Message from syslogd@mehrangarh at Feb 10 15:45:46 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1575.161860] RDX: 000000000000000b RSI: ffffffffc0e84e4a RDI: 0000000000000296
[ 1575.162889] RBP: ffff895065a14ea8 R08: 0000000000000000 R09: 0000000000000001
[ 1575.163918] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1575.164943] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1575.165964] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1575.167050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1575.168084] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1575.169119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1575.170145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1575.171181] Call Trace:
[ 1575.172200]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1575.173220]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1575.174252]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1575.175283]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1575.176316]  ? remove_wait_queue+0x60/0x60
[ 1575.177344]  kthread+0x103/0x140
[ 1575.178363]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1575.179386]  ? kthread_bind+0x10/0x10
[ 1575.180384]  ret_from_fork+0x3a/0x50
[ 1598.424741] INFO: task iscsi_np:7853 blocked for more than 737 seconds.
[ 1598.426598]       Tainted: G        W    L    5.6.0-rc1+ #18
[ 1598.428502] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1598.430588] iscsi_np        D    0  7853      2 0x80004084
[ 1598.432649] Call Trace:
[ 1598.434613]  ? __schedule+0x38c/0x990
[ 1598.436602]  schedule+0x5f/0xd0
[ 1598.438494]  schedule_timeout+0x1e4/0x310
[ 1598.440363]  ? wait_for_completion+0x49/0x190
[ 1598.442176]  wait_for_completion+0x12e/0x190
[ 1598.443979]  ? wake_up_q+0xa0/0xa0
[ 1598.445693]  iscsi_check_for_session_reinstatement+0x1dd/0x270 [iscsi_target_mod]
[ 1598.447393]  iscsi_target_do_login+0xe8/0x5d0 [iscsi_target_mod]
[ 1598.448970]  iscsi_target_start_negotiation+0x51/0xa0 [iscsi_target_mod]
[ 1598.450677]  iscsi_target_login_thread+0x93d/0x10f0 [iscsi_target_mod]
[ 1598.452286]  kthread+0x103/0x140
[ 1598.453651]  ? iscsi_target_login_sess_out+0x200/0x200 [iscsi_target_mod]
[ 1598.455287]  ? kthread_bind+0x10/0x10
[ 1598.456822]  ret_from_fork+0x3a/0x50
[ 1598.458278] INFO: lockdep is turned off.
[ 1603.134955] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1603.135870] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1603.135896]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1603.146278] irq event stamp: 0
[ 1603.147263] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1603.148229] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1603.149202] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1603.150170] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1603.151130] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1603.152105] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1603.153085] RIP: 0010:_find_next_bit.constprop.1+0x5f/0x70
[ 1603.154059] Code: c1 ea 06 48 85 f6 4e 8b 0c d7 74 04 4e 23 0c d6 4d 31 c1 75 0d 48 83 c1 40 48 39 ca 77 de 48 89 d0 c3 f3 49 0f bc c1 48 01 c8 <48> 39 d0 48 0f 47 c2 c3 66 0f 1f 84 00 00 00 00 00 45 31 c0 eb 8b
[ 1603.156051] RSP: 0018:ffffa6370223fcd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1603.157070] RAX: 000000000000000d RBX: ffffffffc0ebe660 RCX: 0000000000000000
[ 1603.158071] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffffaf6c6920
[ 1603.159069] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000e000
[ 1603.160067] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 000000000042d000
[ 1603.161076] R13: 0000000000000000 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1603.162062] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1603.163063] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1603.164037] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1603.165023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1603.166008] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1603.166976] Call Trace:
[ 1603.167939]  cpumask_next+0x17/0x20
[ 1603.168970]  __is_kernel_percpu_address+0x39/0xa0
[ 1603.169938]  static_obj+0x25/0x50
[ 1603.170970]  lockdep_init_map+0x79/0x1f0
[ 1603.171926]  __init_waitqueue_head+0x36/0x50
[ 1603.172927]  transport_generic_free_cmd+0x40/0x160 [target_core_mod]
[ 1603.173889]  ? __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 1603.174876]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1603.175863]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1603.176845]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1603.177809]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1603.178796]  ? remove_wait_queue+0x60/0x60
[ 1603.179787]  kthread+0x103/0x140
[ 1603.180764]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1603.181730]  ? kthread_bind+0x10/0x10
[ 1603.182670]  ret_from_fork+0x3a/0x50
[ 1631.134928] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1631.136029] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1631.136047]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1631.147287] irq event stamp: 0
[ 1631.148312] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1631.149344] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1631.150371] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1631.151360] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1631.152330] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1631.153279] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1631.154242] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 1631.155179] Code: fa 66 0f 1f 44 00 00 65 01 35 0c 10 f7 51 65 8b 05 05 10 f7 51 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 cd de 06 00 48 89 df 57 9d <0f> 1f 44 00 00 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 1631.157153] RSP: 0018:ffffa6370223fd58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1631.158164] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 1631.159162] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: 0000000000000246
[ 1631.160169] RBP: ffff895065a14a90 R08: 0000000000000000 R09: 0000000000000000
[ 1631.161151] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffff895065a14c98
[ 1631.162150] R13: ffff895065a14c50 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1631.163149] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1631.164165] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1631.165170] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1631.166190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1631.167211] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1631.168230] Call Trace:
[ 1631.169244]  _raw_spin_lock_bh+0x1c/0x50
[ 1631.170267]  ? __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1631.171294]  iscsit_free_r2ts_from_list+0x24/0x70 [iscsi_target_mod]
[ 1631.172323]  __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1631.173334]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1631.174362]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1631.175375]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1631.176408]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1631.177448]  ? remove_wait_queue+0x60/0x60
[ 1631.178479]  kthread+0x103/0x140
[ 1631.179479]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1631.180485]  ? kthread_bind+0x10/0x10
[ 1631.181509]  ret_from_fork+0x3a/0x50
[ 1659.134920] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1659.136098] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1659.136117]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1659.148275] irq event stamp: 0
[ 1659.149300] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1659.150358] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1659.151420] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1659.152435] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1659.153455] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1659.154477] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1659.155503] RIP: 0010:lock_release+0x142/0x2a0
[ 1659.156506] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1659.158531] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1659.159538] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1659.160569] RDX: 0000000000000000 RSI: ffffffffc0ee80f1 RDI: 0000000000000246
[ 1659.161604] RBP: ffff89506770ce68 R08: 0000000000000000 R09: 0000000000000000
[ 1659.162620] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80f1
[ 1659.163648] R13: 0000000000000246 R14: ffff89506770ce50 R15: ffff895065a14a90
[ 1659.164684] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1659.165687] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1659.166705] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1659.167733] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1659.168744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1659.169740] Call Trace:
[ 1659.170760]  _raw_spin_unlock_bh+0x17/0x30
[ 1659.171785]  __iscsit_free_cmd+0x281/0x2c0 [iscsi_target_mod]
[ 1659.172787]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1659.173785]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1659.174813]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1659.175849]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1659.176876]  ? remove_wait_queue+0x60/0x60
[ 1659.177893]  kthread+0x103/0x140
[ 1659.178935]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1659.180009]  ? kthread_bind+0x10/0x10
[ 1659.181034]  ret_from_fork+0x3a/0x50
[ 1687.134913] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1687.136110] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1687.136128]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1687.148735] irq event stamp: 0
[ 1687.149772] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1687.150799] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1687.151856] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1687.152845] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1687.153868] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1687.154882] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1687.155892] RIP: 0010:lock_acquire+0xd5/0x1c0
[ 1687.156981] Code: 08 49 c1 e9 09 49 83 f1 01 41 83 e1 01 e8 e3 e7 ff ff 65 48 8b 04 25 40 8e 01 00 48 89 df c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b
[ 1687.159072] RSP: 0018:ffffa6370223fd10 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1687.160111] RAX: ffff8951948fa0c0 RBX: 0000000000000246 RCX: 0000000000000000
[ 1687.161144] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000246
[ 1687.162169] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 1687.163187] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000000
[ 1687.164184] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[ 1687.165189] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1687.166201] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1687.167189] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1687.168180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1687.169184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1687.170165] Call Trace:
[ 1687.171166]  ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1687.172173]  _raw_spin_lock_bh+0x3d/0x50
[ 1687.173164]  ? iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1687.174183]  iscsit_stop_dataout_timer+0x19/0x80 [iscsi_target_mod]
[ 1687.175207]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1687.176229]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1687.177230]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1687.178250]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1687.179272]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1687.180294]  ? remove_wait_queue+0x60/0x60
[ 1687.181292]  kthread+0x103/0x140
[ 1687.182300]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1687.183318]  ? kthread_bind+0x10/0x10
[ 1687.184331]  ret_from_fork+0x3a/0x50
[ 1687.390634] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1687.391599] rcu:     11-....: (843434 ticks this GP) idle=c92/1/0x4000000000000002 softirq=23820/23820 fqs=210868
[ 1687.393510]  (t=845012 jiffies g=68413 q=64187)
[ 1687.394480] NMI backtrace for cpu 11
[ 1687.396158] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1687.398016] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1687.399758] Call Trace:
[ 1687.401478]  <IRQ>
[ 1687.403179]  dump_stack+0x82/0xb3
[ 1687.404799]  nmi_cpu_backtrace+0x90/0xa0
[ 1687.406393]  ? lapic_can_unplug_cpu+0xa0/0xa0
[ 1687.407982]  nmi_trigger_cpumask_backtrace+0x73/0x130
[ 1687.409462]  rcu_dump_cpu_stacks+0x97/0xc5
[ 1687.411032]  rcu_sched_clock_irq+0x6ec/0x980
[ 1687.412473]  ? tick_sched_do_timer+0x70/0x70
[ 1687.414044]  update_process_times+0x24/0x50
[ 1687.415506]  tick_sched_handle+0x25/0x60
[ 1687.417045]  tick_sched_timer+0x37/0x70
[ 1687.418449]  __hrtimer_run_queues+0xe8/0x450
[ 1687.419899]  hrtimer_interrupt+0x122/0x270
[ 1687.421276]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 1687.422601]  smp_apic_timer_interrupt+0x86/0x240
[ 1687.423965]  apic_timer_interrupt+0xf/0x20
[ 1687.425230]  </IRQ>
[ 1687.426499] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1687.427826] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1687.430671] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1687.432171] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1687.433581] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1687.435084] RBP: ffffffffc0ee80c1 R08: 0000000000000000 R09: 0000000000000000
[ 1687.436489] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1687.438011] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1687.439433]  ? __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1687.440904]  ? __local_bh_enable_ip+0x80/0xf0
[ 1687.442353]  __iscsit_free_cmd+0x251/0x2c0 [iscsi_target_mod]
[ 1687.443808]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1687.445254]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1687.446679]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1687.448160]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1687.449577]  ? remove_wait_queue+0x60/0x60
[ 1687.451080]  kthread+0x103/0x140
[ 1687.452459]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1687.453921]  ? kthread_bind+0x10/0x10
[ 1687.455316]  ret_from_fork+0x3a/0x50
[ 1715.134905] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [iscsi_ttx:8484]
[ 1715.135762] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1715.135786]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1715.145998] irq event stamp: 0
[ 1715.146998] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1715.148020] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1715.149053] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1715.150034] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1715.151013] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1715.152011] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1715.153021] RIP: 0010:__local_bh_enable_ip+0x87/0xf0
[ 1715.154030] Code: 74 57 65 01 1d 32 08 f7 51 65 8b 05 2b 08 f7 51 a9 00 ff 1f 00 74 55 65 ff 0d 1d 08 f7 51 e8 20 e8 12 00 fb 66 0f 1f 44 00 00 <5b> 5d c3 8b 80 88 0c 00 00 85 c0 75 a7 80 3d 88 ea 5e 01 00 75 9e
[ 1715.156116] RSP: 0018:ffffa6370223fd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1715.157181] RAX: 0000000000000000 RBX: 00000000fffffe00 RCX: 0000000000000000
[ 1715.158249] RDX: ffff8951948fa0c0 RSI: 0000000000000201 RDI: ffffffffae0a85f0
[ 1715.159323] RBP: ffffffffc0ee80fe R08: 0000000000000000 R09: 0000000000000000
[ 1715.160381] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1715.161440] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1715.162485] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1715.163531] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1715.164574] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1715.165621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1715.166656] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1715.167673] Call Trace:
[ 1715.168706]  __iscsit_free_cmd+0x28e/0x2c0 [iscsi_target_mod]
[ 1715.169719]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1715.170754]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1715.171769]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1715.172816]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1715.173861]  ? remove_wait_queue+0x60/0x60
[ 1715.174913]  kthread+0x103/0x140
[ 1715.175986]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1715.177064]  ? kthread_bind+0x10/0x10
[ 1715.178095]  ret_from_fork+0x3a/0x50
[ 1743.134897] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1743.136093] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1743.136113]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1743.149085] irq event stamp: 0
[ 1743.150178] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1743.151261] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1743.152336] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1743.153370] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1743.154382] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1743.155415] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1743.156481] RIP: 0010:transport_generic_free_cmd+0x4c/0x160 [target_core_mod]
[ 1743.157533] Code: 18 c7 44 24 10 00 00 00 00 65 48 8b 04 25 28 00 00 00 48 89 44 24 60 31 c0 e8 c0 a6 27 ed 85 ed c6 44 24 06 00 c6 44 24 07 00 <0f> 85 9b 00 00 00 f6 43 29 01 74 0a f6 43 06 02 0f 85 cd 00 00 00
[ 1743.159691] RSP: 0018:ffffa6370223fd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 1743.160758] RAX: ffffa6370223fdb8 RBX: ffff895065a14da0 RCX: 0000000000000000

Message from syslogd@mehrangarh at Feb 10 15:48:34 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1743.161786] RDX: 0000000000000000 RSI: 000000000006a000 RDI: 0000000000000000
[ 1743.162828] RBP: 0000000000000001 R08: 0000000000000522 R09: 0000000000008000
[ 1743.163850] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1743.164872] R13: 0000000000000001 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1743.165907] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1743.166981] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1743.168011] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1743.169043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1743.170049] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1743.171064] Call Trace:
[ 1743.172063]  iscsit_free_cmd+0x58/0xa0 [iscsi_target_mod]
[ 1743.173091]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1743.174100]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1743.175114]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1743.176140]  ? remove_wait_queue+0x60/0x60
[ 1743.177163]  kthread+0x103/0x140
[ 1743.178160]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1743.179181]  ? kthread_bind+0x10/0x10
[ 1743.180193]  ret_from_fork+0x3a/0x50
[ 1771.134889] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1771.136058] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1771.136076]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1771.148984] irq event stamp: 0
[ 1771.150057] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1771.151148] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1771.152217] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1771.153234] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1771.154264] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1771.155270] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1771.156306] RIP: 0010:lock_release+0x142/0x2a0
[ 1771.157338] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1771.159436] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1771.160496] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1771.161546] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1771.162576] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1771.163586] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1771.164595] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1771.165598] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1771.166640] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1771.167670] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1771.168711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1771.169738] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1771.170755] Call Trace:
[ 1771.171761]  _raw_spin_unlock_bh+0x17/0x30
[ 1771.172774]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1771.173798]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1771.174813]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1771.175816]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1771.176849]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1771.177870]  ? remove_wait_queue+0x60/0x60
[ 1771.178939]  kthread+0x103/0x140
[ 1771.179937]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1771.180966]  ? kthread_bind+0x10/0x10
[ 1771.181987]  ret_from_fork+0x3a/0x50
[ 1799.134881] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1799.136073] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1799.136093]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1799.148664] irq event stamp: 0
[ 1799.149676] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1799.150691] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1799.151719] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1799.152734] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1799.153752] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1799.154788] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1799.155822] RIP: 0010:lock_release+0x142/0x2a0
[ 1799.156840] Code: 11 44 2b 3c 24 44 39 bb b8 0c 00 00 0f 85 34 01 00 00 65 48 8b 04 25 40 8e 01 00 4c 89 ef c7 80 bc 0c 00 00 00 00 00 00 57 9d <0f> 1f 44 00 00 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 0f 85 e9
[ 1799.158973] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1799.159992] RAX: ffff8951948fa0c0 RBX: ffff8951948fa0c0 RCX: 0000000000000000
[ 1799.161008] RDX: 0000000000000000 RSI: ffffffffc0ee80b9 RDI: 0000000000000246
[ 1799.162035] RBP: ffff895065a14bc0 R08: 0000000000000000 R09: 0000000000000000
[ 1799.163046] R10: 0000000000000000 R11: ffffa6370223fd70 R12: ffffffffc0ee80b9
[ 1799.164060] R13: 0000000000000246 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1799.165069] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1799.166093] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1799.167122] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1799.168153] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1799.169181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1799.170204] Call Trace:
[ 1799.171227]  _raw_spin_unlock_bh+0x17/0x30
[ 1799.172231]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1799.173263]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1799.174295]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1799.175312]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1799.176337]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1799.177370]  ? remove_wait_queue+0x60/0x60
[ 1799.178392]  kthread+0x103/0x140
[ 1799.179393]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1799.180424]  ? kthread_bind+0x10/0x10
[ 1799.181429]  ret_from_fork+0x3a/0x50
[ 1827.134874] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1827.136077] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1827.136096]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1827.148658] irq event stamp: 0
[ 1827.149696] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1827.150742] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1827.151817] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1827.152850] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1827.153872] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1827.154938] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1827.155977] RIP: 0010:__local_bh_disable_ip+0x48/0x60
[ 1827.157006] Code: fa 66 0f 1f 44 00 00 65 01 35 0c 10 f7 51 65 8b 05 05 10 f7 51 48 31 c6 f7 c6 00 ff 00 00 75 05 e8 cd de 06 00 48 89 df 57 9d <0f> 1f 44 00 00 5b c3 0f 0b eb c1 0f 1f 00 66 2e 0f 1f 84 00 00 00
[ 1827.159094] RSP: 0018:ffffa6370223fd80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1827.160134] RAX: 0000000000000201 RBX: 0000000000000246 RCX: 0000000000000000
[ 1827.161169] RDX: ffff8951948fa0c0 RSI: 0000000000000000 RDI: 0000000000000246
[ 1827.162194] RBP: ffff895065a14da0 R08: 0000000000000000 R09: 0000000000000000
[ 1827.163214] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1827.164215] R13: ffff89506770c800 R14: ffff89506770cde0 R15: ffff895065a14a90
[ 1827.165232] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1827.166261] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1827.167266] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1827.168291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1827.169310] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1827.170307] Call Trace:
[ 1827.171323]  _raw_spin_lock_bh+0x1c/0x50
[ 1827.172336]  __iscsit_free_cmd+0xa9/0x2c0 [iscsi_target_mod]
[ 1827.173360]  iscsit_free_cmd+0x48/0xa0 [iscsi_target_mod]
[ 1827.174380]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1827.175387]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1827.176419]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1827.177442]  ? remove_wait_queue+0x60/0x60
[ 1827.178458]  kthread+0x103/0x140
[ 1827.179467]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1827.180470]  ? kthread_bind+0x10/0x10
[ 1827.181485]  ret_from_fork+0x3a/0x50
[ 1855.134866] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1855.136065] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1855.136092]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1855.148497] irq event stamp: 0
[ 1855.149492] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 1855.150516] hardirqs last disabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1855.151544] softirqs last  enabled at (0): [<ffffffffae09d07f>] copy_process+0x76f/0x1f80
[ 1855.152560] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 1855.153568] CPU: 11 PID: 8484 Comm: iscsi_ttx Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1855.154592] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1855.155593] RIP: 0010:lock_release+0x2c/0x2a0
[ 1855.156615] Code: 65 48 8b 14 25 40 8e 01 00 41 56 41 55 41 54 55 53 48 83 ec 10 44 8b 82 bc 0c 00 00 65 48 8b 04 25 28 00 00 00 48 89 44 24 08 <31> c0 45 85 c0 0f 85 10 01 00 00 48 89 fd 49 89 f4 9c 58 0f 1f 44
[ 1855.158663] RSP: 0018:ffffa6370223fd50 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 1855.159659] RAX: fe6fc5eb227b1400 RBX: ffff895065a14ba8 RCX: 0000000000000000
[ 1855.160671] RDX: ffff8951948fa0c0 RSI: ffffffffc0ee80b9 RDI: ffff895065a14bc0
[ 1855.161686] RBP: ffff895065a14da0 R08: 0000000000000000 R09: 0000000000000000
[ 1855.162706] R10: 0000000000000000 R11: ffffa6370223fd70 R12: 0000000000000001
[ 1855.163730] R13: ffff89506770c800 R14: ffff895065a14d90 R15: ffff895065a14a90
[ 1855.164734] FS:  0000000000000000(0000) GS:ffff8951de000000(0000) knlGS:0000000000000000
[ 1855.165771] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1855.166794] CR2: 00007faeb911c248 CR3: 0000000008a10002 CR4: 00000000003606e0
[ 1855.167799] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1855.168823] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Message from syslogd@mehrangarh at Feb 10 15:50:26 ...
 kernel:watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [iscsi_ttx:8484]
[ 1855.169813] Call Trace:
[ 1855.170833]  _raw_spin_unlock_bh+0x17/0x30
[ 1855.171910]  __iscsit_free_cmd+0x249/0x2c0 [iscsi_target_mod]
[ 1855.172969]  iscsit_free_cmd+0x76/0xa0 [iscsi_target_mod]
[ 1855.174002]  iscsit_close_connection+0x570/0x920 [iscsi_target_mod]
[ 1855.175024]  iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
[ 1855.176068]  iscsi_target_tx_thread+0x173/0x210 [iscsi_target_mod]
[ 1855.177104]  ? remove_wait_queue+0x60/0x60
[ 1855.178121]  kthread+0x103/0x140
[ 1855.179144]  ? iscsit_thread_get_cpumask+0xa0/0xa0 [iscsi_target_mod]
[ 1855.180180]  ? kthread_bind+0x10/0x10
[ 1855.181207]  ret_from_fork+0x3a/0x50
[ 1871.832621] ------------[ cut here ]------------
[ 1871.833683] NETDEV WATCHDOG: eno1 (igb): transmit queue 1 timed out
[ 1871.834783] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:443 dev_watchdog+0x2d9/0x2e0
[ 1871.835974] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock mlx5_ib mlx5_core mlxfw pci_hyperv_intf nvmet_tcp nvmet_rdma nvmet rdma_ucm iw_cxgb4 ib_uverbs cxgb4 libcxgb netconsole macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rfkill ib_isert iscsi_target_mod target_core_mod sunrpc intel_rapl_msr ib_umad intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec rdma_cm iw_cm ib_cm ib_core snd_hda_core irqbypass snd_hwdep crct10dif_pclmul snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device iTCO_wdt aesni_intel iTCO_vendor_support
[ 1871.836017]  ipmi_ssif crypto_simd snd_pcm snd_timer cryptd glue_helper ipmi_si snd wmi pcspkr ipmi_devintf sg soundcore ipmi_msghandler lpc_ich i2c_i801 dm_multipath sch_fq_codel ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt t10_pi fb_sys_fops drm igb crc32c_intel ahci libahci dca libata i2c_algo_bit i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_uverbs]
[ 1871.848417] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Tainted: G        W    L    5.6.0-rc1+ #18
[ 1871.849484] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
[ 1871.850551] RIP: 0010:dev_watchdog+0x2d9/0x2e0
[ 1871.851653] Code: da 90 ff eb ca 4c 89 f7 c6 05 b8 33 e9 00 01 e8 9d c9 fa ff 89 d9 48 89 c2 4c 89 f6 48 c7 c7 90 5f 1b af 31 c0 e8 47 bd 89 ff <0f> 0b e9 09 ff ff ff 0f 1f 44 00 00 41 57 ba 18 00 00 00 41 56 41
[ 1871.853858] RSP: 0018:ffffa6370003ce48 EFLAGS: 00010286
[ 1871.854981] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[ 1871.856101] RDX: ffff8951da8268c0 RSI: ffff8951da41a0f8 RDI: ffff8951da41a0f8
[ 1871.857219] RBP: ffff8951c57804b8 R08: 0000000000000000 R09: 0000000000000001
[ 1871.858314] R10: 0000000000000000 R11: ffff89506369f4f0 R12: 0000000000000008
[ 1871.859393] R13: ffff8951c57803e0 R14: ffff8951c5780000 R15: ffff8951c6b9fb00
[ 1871.860463] FS:  0000000000000000(0000) GS:ffff8951da400000(0000) knlGS:0000000000000000
[ 1871.861537] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1871.862587] CR2: 0000558a0420e358 CR3: 0000000008a10006 CR4: 00000000003606e0
[ 1871.863659] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1871.864734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1871.865796] Call Trace:
[ 1871.866848]  <IRQ>
[ 1871.867905]  ? dev_deactivate_queue.constprop.56+0xd0/0xd0
[ 1871.869007]  ? dev_deactivate_queue.constprop.56+0xd0/0xd0
[ 1871.870065]  call_timer_fn+0xa7/0x2d0
[ 1871.871126]  ? call_timer_fn+0x5/0x2d0
[ 1871.872190]  ? dev_deactivate_queue.constprop.56+0xd0/0xd0
[ 1871.873243]  run_timer_softirq+0x235/0x570
[ 1871.874288]  ? clockevents_program_event+0x3f/0xf0
[ 1871.875319]  __do_softirq+0xd6/0x419
[ 1871.876339]  irq_exit+0xd9/0xf0
[ 1871.877361]  smp_apic_timer_interrupt+0xac/0x240
[ 1871.878359]  apic_timer_interrupt+0xf/0x20
[ 1871.879378]  </IRQ>
[ 1871.880388] RIP: 0010:cpuidle_enter_state+0xf3/0x3d0
[ 1871.881381] Code: 44 8b 63 04 49 89 c6 0f 1f 44 00 00 31 ff e8 84 fe 9a ff 80 7c 24 0b 00 0f 85 de 01 00 00 e8 34 26 aa ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 56 02 00 00 49 63 d5 4c 2b 34 24 48 8d 04 52 48 8d
[ 1871.883444] RSP: 0018:ffffa637000d3e70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1871.884501] RAX: 0000000080000001 RBX: ffffc636fa402500 RCX: 000000000000001f
[ 1871.885536] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffffffffae7347dc
[ 1871.886551] RBP: ffffffffaf5c3c80 R08: 0000000000000002 R09: fffffff1d58cfed6
[ 1871.887567] R10: 0000000000000000 R11: 071c71c71c71c71c R12: 0000000000000001
[ 1871.888533] R13: 0000000000000004 R14: 000001b3d1eaf43f R15: 0000000000000001
[ 1871.889516]  ? cpuidle_enter_state+0xec/0x3d0
[ 1871.890476]  cpuidle_enter+0x29/0x40
[ 1871.891417]  do_idle+0x1fb/0x2b0
[ 1871.892319]  cpu_startup_entry+0x19/0x20
[ 1871.893225]  start_secondary+0x171/0x1c0
[ 1871.894116]  secondary_startup_64+0xa4/0xb0
[ 1871.894995] irq event stamp: 606396
[ 1871.895786] hardirqs last  enabled at (606395): [<ffffffffae14025b>] rcu_idle_exit+0x2b/0x50
[ 1871.896548] hardirqs last disabled at (606396): [<ffffffffae0e477f>] do_idle+0x7f/0x2b0
[ 1871.897351] softirqs last  enabled at (606392): [<ffffffffae0a86be>] irq_enter+0x5e/0x70
[ 1871.898169] softirqs last disabled at (606391): [<ffffffffae0a86a3>] irq_enter+0x43/0x70