Re: RCU stall

Re: RCU stall

[Paul E. McKenney]

On Tue, Mar 22, 2016 at 01:20:02PM -0700, Bart Van Assche wrote:
> Hello Paul,
> 
> Maybe you remember that we met briefly during the LinuxCon VIP
> reception in the Chihuly Garden in Seattle (August 2015)?
> 
> Yesterday I tried to verify a change in the ib_srpt driver on my
> setup by letting the ib_srp and ib_srpt drivers communicate with
> each other on the same system using the loopback functionality
> provided by Mellanox ConnectX-3 HCA's. Instead of running into any
> SRP related issues I hit something that looks like an RCU stall. I
> can reproduce this stall with the v4.4.6 and v4.5-rc7 kernels (I
> have not yet tried any other kernels). Can you have a look at the
> call traces in the attached netconsole output and check whether or
> not this is related to the RCU implementation in the Linux kernel?
> The test I ran is as follows:
> * Load and configure the ib_srpt target driver such that it exports
> a RAM disk (brd) as a SCSI device.
> * Load and configure the ib_srp initiator driver (which is a SCSI
> initiator driver).
> * Start the srp_daemon. This daemon makes the ib_srp initiator
> driver log in to the ib_srpt target driver.
> * Activate multipath on top of the SCSI devices created by ib_srp.
> * Make fio send a workload with a high queue depth to the ib_srp
> initiator driver.
> * Restart the SCSI target stack (ib_srpt + the kernel modules used
> by that driver).
> 
> I can post this information on a public mailing list if you would
> prefer this.

No need, I added LKML on CC.  Depending on what you find, you might
want to add more people or more lists.

You are getting a soft lockup as well as an RCU CPU stall warning, so
it looks like something is taking a very long time in blk_done_softirq().

You have multiple occurrences at different times, so it looks to be
a long time as opposed to an infinite time.  Are you perhaps doing
something that would make a huge amount of work for blk_done_softirq()?

See Documentation/RCU/stallwarn.txt in the kernel source tree for more
info on how to debug this sort of thing.

							Thanx, Paul

> Thanks,
> 
> Bart.

> [10416.888346] device-mapper: multipath: Failing path 8:112.
> [10416.894423] device-mapper: multipath: Failing path 8:96.
> [10416.906746] device-mapper: multipath: Failing path 8:240.
> [10416.907224] device-mapper: multipath: Failing path 66:80.
> [10416.915126] device-mapper: multipath: Failing path 65:192.
> [10416.915479] device-mapper: multipath: Failing path 66:160.
> [10416.990626] device-mapper: multipath: Failing path 67:144.
> [10417.961194] device-mapper: multipath: Failing path 66:48.
> [10417.968416] device-mapper: multipath: Failing path 67:128.
> [10417.976784] device-mapper: multipath: Failing path 67:64.
> [10418.996938] device-mapper: multipath: Failing path 8:128.
> [10418.997469] device-mapper: multipath: Failing path 65:80.
> [10419.004872] device-mapper: multipath: Failing path 65:128.
> [10419.006014] device-mapper: multipath: Failing path 8:176.
> [10419.007623] device-mapper: multipath: Failing path 67:96.
> [10419.011766] device-mapper: multipath: Failing path 65:64.
> [10420.054544] scsi_io_completion: 22 callbacks suppressed
> [10420.054632] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.054704] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10420.054771] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.054847] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10420.054917] blk_update_request: 22 callbacks suppressed
> [10420.054989] blk_update_request: I/O error, dev sdj, sector 1952320
> [10420.055114] device-mapper: multipath: Failing path 8:144.
> [10420.060874] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.060927] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10420.060965] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.061010] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00
> [10420.061051] blk_update_request: I/O error, dev sdz, sector 704
> [10420.061118] device-mapper: multipath: Failing path 65:144.
> [10420.067122] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.067173] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10420.067224] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.067289] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10420.067351] blk_update_request: I/O error, dev sdk, sector 1952320
> [10420.067445] device-mapper: multipath: Failing path 8:160.
> [10420.067825] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.067894] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] 
> [10420.067954] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.068017] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10420.068075] blk_update_request: I/O error, dev sdaa, sector 1952320
> [10420.068165] device-mapper: multipath: Failing path 65:160.
> [10420.073375] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.073456] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10420.073520] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.073582] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 01 10 00 00 60 00
> [10420.073641] blk_update_request: I/O error, dev sdad, sector 272
> [10420.073738] device-mapper: multipath: Failing path 65:208.
> [10420.075521] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.075642] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10420.075748] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.075855] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10420.075955] blk_update_request: I/O error, dev sdo, sector 608
> [10420.076204] device-mapper: multipath: Failing path 8:224.
> [10420.081491] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.081597] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10420.081685] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.081778] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 60 00
> [10420.081850] blk_update_request: I/O error, dev sdai, sector 1952368
> [10420.081939] device-mapper: multipath: Failing path 66:32.
> [10420.157841] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.157922] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10420.157982] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.158023] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10420.158059] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10420.158124] device-mapper: multipath: Failing path 67:224.
> [10420.184232] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.184336] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10420.184417] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.184505] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10420.184588] blk_update_request: I/O error, dev sdas, sector 1952320
> [10420.184713] device-mapper: multipath: Failing path 66:192.
> [10420.190423] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10420.190528] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10420.190711] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10420.190805] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00
> [10420.190921] blk_update_request: I/O error, dev sdar, sector 704
> [10420.191092] device-mapper: multipath: Failing path 66:176.
> [10421.152191] device-mapper: multipath: Failing path 65:176.
> [10421.152905] device-mapper: multipath: Failing path 66:208.
> [10421.158332] device-mapper: multipath: Failing path 66:64.
> [10421.169109] device-mapper: multipath: Failing path 68:16.
> [10421.169450] device-mapper: multipath: Failing path 67:192.
> [10421.178113] device-mapper: multipath: Failing path 68:0.
> [10422.226466] device-mapper: multipath: Failing path 8:112.
> [10422.233709] device-mapper: multipath: Failing path 8:96.
> [10422.247912] device-mapper: multipath: Failing path 8:240.
> [10422.248844] device-mapper: multipath: Failing path 66:80.
> [10422.255501] device-mapper: multipath: Failing path 65:192.
> [10422.261653] device-mapper: multipath: Failing path 66:160.
> [10422.324643] device-mapper: multipath: Failing path 67:144.
> [10423.297527] device-mapper: multipath: Failing path 66:48.
> [10423.307128] device-mapper: multipath: Failing path 67:128.
> [10423.314250] device-mapper: multipath: Failing path 67:64.
> [10424.336170] device-mapper: multipath: Failing path 8:128.
> [10424.336732] device-mapper: multipath: Failing path 65:80.
> [10424.345244] device-mapper: multipath: Failing path 65:128.
> [10424.345717] device-mapper: multipath: Failing path 65:64.
> [10424.346791] device-mapper: multipath: Failing path 8:176.
> [10424.354214] device-mapper: multipath: Failing path 67:96.
> [10425.390756] scsi_io_completion: 22 callbacks suppressed
> [10425.390886] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.390997] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10425.391099] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.391214] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10425.391321] blk_update_request: 22 callbacks suppressed
> [10425.391415] blk_update_request: I/O error, dev sdj, sector 1952320
> [10425.391564] device-mapper: multipath: Failing path 8:144.
> [10425.398476] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.398564] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10425.398632] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.398703] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00
> [10425.398767] blk_update_request: I/O error, dev sdz, sector 704
> [10425.398866] device-mapper: multipath: Failing path 65:144.
> [10425.400705] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.400779] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10425.400845] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.400911] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10425.400969] blk_update_request: I/O error, dev sdk, sector 1952320
> [10425.401137] device-mapper: multipath: Failing path 8:160.
> [10425.401616] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.401684] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] 
> [10425.401743] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.401807] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10425.401866] blk_update_request: I/O error, dev sdaa, sector 1952320
> [10425.401967] device-mapper: multipath: Failing path 65:160.
> [10425.404794] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.404879] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10425.404952] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.405029] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 00 78 00 00 08 00
> [10425.405103] blk_update_request: I/O error, dev sdai, sector 120
> [10425.405273] device-mapper: multipath: Failing path 66:32.
> [10425.409526] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.409609] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10425.409674] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.409741] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00
> [10425.409806] blk_update_request: I/O error, dev sdo, sector 704
> [10425.409971] device-mapper: multipath: Failing path 8:224.
> [10425.411359] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.411471] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10425.411547] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.411614] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 e8 00
> [10425.411680] blk_update_request: I/O error, dev sdas, sector 768
> [10425.411770] device-mapper: multipath: Failing path 66:192.
> [10425.417081] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.417156] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10425.417217] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.417283] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00
> [10425.417351] blk_update_request: I/O error, dev sdad, sector 1952688
> [10425.417512] device-mapper: multipath: Failing path 65:208.
> [10425.483947] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.484013] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10425.484076] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.484127] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10425.484177] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10425.484245] device-mapper: multipath: Failing path 67:224.
> [10425.497810] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10425.497866] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10425.497910] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10425.497957] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00
> [10425.497999] blk_update_request: I/O error, dev sdar, sector 1952280
> [10425.498065] device-mapper: multipath: Failing path 66:176.
> [10426.456675] device-mapper: multipath: Failing path 65:176.
> [10426.457202] device-mapper: multipath: Failing path 66:208.
> [10426.462722] device-mapper: multipath: Failing path 66:64.
> [10426.472875] device-mapper: multipath: Failing path 68:16.
> [10426.473257] device-mapper: multipath: Failing path 67:192.
> [10426.477439] device-mapper: multipath: Failing path 68:0.
> [10427.509583] device-mapper: multipath: Failing path 8:112.
> [10427.516675] device-mapper: multipath: Failing path 8:96.
> [10427.529858] device-mapper: multipath: Failing path 8:240.
> [10427.530437] device-mapper: multipath: Failing path 66:80.
> [10427.537627] device-mapper: multipath: Failing path 65:192.
> [10427.544651] device-mapper: multipath: Failing path 66:160.
> [10427.616573] device-mapper: multipath: Failing path 67:144.
> [10428.580910] device-mapper: multipath: Failing path 66:48.
> [10428.589846] device-mapper: multipath: Failing path 67:128.
> [10428.601369] device-mapper: multipath: Failing path 67:64.
> [10429.602027] device-mapper: multipath: Failing path 8:128.
> [10429.607145] device-mapper: multipath: Failing path 65:80.
> [10429.609131] device-mapper: multipath: Failing path 65:128.
> [10429.619473] device-mapper: multipath: Failing path 8:176.
> [10429.712106] device-mapper: multipath: Failing path 67:96.
> [10430.675345] scsi_io_completion: 21 callbacks suppressed
> [10430.675424] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.675493] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10430.675563] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.675645] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10430.675711] blk_update_request: 21 callbacks suppressed
> [10430.675777] blk_update_request: I/O error, dev sdj, sector 1952320
> [10430.675880] device-mapper: multipath: Failing path 8:144.
> [10430.677763] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.677884] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10430.677980] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.678092] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00
> [10430.678189] blk_update_request: I/O error, dev sdz, sector 2048
> [10430.678408] device-mapper: multipath: Failing path 65:144.
> [10430.680621] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.680774] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10430.680903] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.681010] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10430.681132] blk_update_request: I/O error, dev sdk, sector 1952320
> [10430.681357] device-mapper: multipath: Failing path 8:160.
> [10430.686148] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.686267] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] 
> [10430.686364] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.686468] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00
> [10430.686557] blk_update_request: I/O error, dev sdaa, sector 32
> [10430.687034] device-mapper: multipath: Failing path 65:160.
> [10430.687938] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.688015] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10430.688077] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.688137] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10430.688193] blk_update_request: I/O error, dev sdu, sector 608
> [10430.688319] device-mapper: multipath: Failing path 65:64.
> [10430.690959] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.691031] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10430.691091] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.691151] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 b8 00
> [10430.691211] blk_update_request: I/O error, dev sdo, sector 768
> [10430.691397] device-mapper: multipath: Failing path 8:224.
> [10430.694975] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.695052] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10430.695115] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.695182] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 00 00 00 18 00
> [10430.695240] blk_update_request: I/O error, dev sdad, sector 1952256
> [10430.695480] device-mapper: multipath: Failing path 65:208.
> [10430.776287] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.776347] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10430.776388] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.776430] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cd b8 00 00 08 00
> [10430.776471] blk_update_request: I/O error, dev sdai, sector 1953208
> [10430.776537] device-mapper: multipath: Failing path 66:32.
> [10430.779410] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.779467] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10430.779511] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.779562] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00
> [10430.779606] blk_update_request: I/O error, dev sdas, sector 1952280
> [10430.779712] device-mapper: multipath: Failing path 66:192.
> [10430.782147] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10430.782219] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10430.782279] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10430.782342] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10430.782402] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10430.782491] device-mapper: multipath: Failing path 67:224.
> [10430.784545] device-mapper: multipath: Failing path 66:176.
> [10431.767557] device-mapper: multipath: Failing path 65:176.
> [10431.767921] device-mapper: multipath: Failing path 66:208.
> [10431.773168] device-mapper: multipath: Failing path 66:64.
> [10431.793497] device-mapper: multipath: Failing path 68:16.
> [10431.794007] device-mapper: multipath: Failing path 67:192.
> [10431.798234] device-mapper: multipath: Failing path 68:0.
> [10432.827559] device-mapper: multipath: Failing path 8:112.
> [10432.834048] device-mapper: multipath: Failing path 8:96.
> [10432.847571] device-mapper: multipath: Failing path 8:240.
> [10432.848142] device-mapper: multipath: Failing path 66:80.
> [10432.854439] device-mapper: multipath: Failing path 65:192.
> [10432.859782] device-mapper: multipath: Failing path 66:160.
> [10432.925961] device-mapper: multipath: Failing path 67:144.
> [10433.916020] device-mapper: multipath: Failing path 66:48.
> [10433.923574] device-mapper: multipath: Failing path 67:128.
> [10433.931521] device-mapper: multipath: Failing path 67:64.
> [10434.944355] device-mapper: multipath: Failing path 8:128.
> [10434.944780] device-mapper: multipath: Failing path 65:80.
> [10434.946445] device-mapper: multipath: Failing path 67:96.
> [10434.953279] device-mapper: multipath: Failing path 65:128.
> [10434.961368] device-mapper: multipath: Failing path 8:176.
> [10435.997587] scsi_io_completion: 22 callbacks suppressed
> [10435.997666] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10435.997735] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10435.997800] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10435.997868] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10435.997932] blk_update_request: 22 callbacks suppressed
> [10435.997992] blk_update_request: I/O error, dev sdj, sector 1952320
> [10435.998093] device-mapper: multipath: Failing path 8:144.
> [10435.998848] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10435.998940] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10435.999015] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10435.999136] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10435.999212] blk_update_request: I/O error, dev sdz, sector 1952320
> [10435.999358] device-mapper: multipath: Failing path 65:144.
> [10436.003810] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.003924] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10436.004034] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.004182] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10436.004300] blk_update_request: I/O error, dev sdk, sector 1952320
> [10436.004486] device-mapper: multipath: Failing path 8:160.
> [10436.006466] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.006572] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] 
> [10436.006677] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.006835] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10436.006943] blk_update_request: I/O error, dev sdaa, sector 1952320
> [10436.007132] device-mapper: multipath: Failing path 65:160.
> [10436.009417] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.009500] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10436.009563] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.009626] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 b8 00 00 48 00
> [10436.009686] blk_update_request: I/O error, dev sdu, sector 696
> [10436.009832] device-mapper: multipath: Failing path 65:64.
> [10436.013058] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.013131] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10436.013213] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.013311] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00
> [10436.013405] blk_update_request: I/O error, dev sdad, sector 32
> [10436.013575] device-mapper: multipath: Failing path 65:208.
> [10436.016633] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.016704] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10436.016760] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.016820] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00
> [10436.016876] blk_update_request: I/O error, dev sdo, sector 704
> [10436.017086] device-mapper: multipath: Failing path 8:224.
> [10436.022985] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.023062] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10436.023132] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.023201] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 60 00
> [10436.023264] blk_update_request: I/O error, dev sdas, sector 1952368
> [10436.023405] device-mapper: multipath: Failing path 66:192.
> [10436.024977] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.025082] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10436.025178] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.025271] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cb 20 00 00 30 00
> [10436.025328] blk_update_request: I/O error, dev sdai, sector 1952544
> [10436.025472] device-mapper: multipath: Failing path 66:32.
> [10436.119145] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10436.119237] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10436.119305] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10436.119380] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10436.119486] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10436.119638] device-mapper: multipath: Failing path 67:224.
> [10436.133091] device-mapper: multipath: Failing path 66:176.
> [10437.082788] device-mapper: multipath: Failing path 65:176.
> [10437.083248] device-mapper: multipath: Failing path 66:208.
> [10437.083653] device-mapper: multipath: Failing path 68:16.
> [10437.083928] device-mapper: multipath: Failing path 67:192.
> [10437.087703] device-mapper: multipath: Failing path 68:0.
> [10437.088448] device-mapper: multipath: Failing path 66:64.
> [10438.129087] device-mapper: multipath: Failing path 8:112.
> [10438.138623] device-mapper: multipath: Failing path 8:96.
> [10438.151534] device-mapper: multipath: Failing path 8:240.
> [10438.152048] device-mapper: multipath: Failing path 66:80.
> [10438.158935] device-mapper: multipath: Failing path 65:192.
> [10438.164948] device-mapper: multipath: Failing path 66:160.
> [10438.232625] device-mapper: multipath: Failing path 67:144.
> [10439.204539] device-mapper: multipath: Failing path 66:48.
> [10439.212592] device-mapper: multipath: Failing path 67:128.
> [10439.220192] device-mapper: multipath: Failing path 67:64.
> [10440.234002] device-mapper: multipath: Failing path 8:128.
> [10440.234585] device-mapper: multipath: Failing path 65:80.
> [10440.237500] device-mapper: multipath: Failing path 67:96.
> [10440.241884] device-mapper: multipath: Failing path 8:176.
> [10440.241930] device-mapper: multipath: Failing path 65:128.
> [10441.293227] scsi_io_completion: 22 callbacks suppressed
> [10441.293301] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.293360] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10441.293420] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.293673] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10441.293733] blk_update_request: 22 callbacks suppressed
> [10441.293785] blk_update_request: I/O error, dev sdj, sector 1952320
> [10441.293916] device-mapper: multipath: Failing path 8:144.
> [10441.294596] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.294703] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10441.294799] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.294939] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10441.295032] blk_update_request: I/O error, dev sdz, sector 1952320
> [10441.295203] device-mapper: multipath: Failing path 65:144.
> [10441.297186] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.297273] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10441.297341] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.297412] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10441.297518] blk_update_request: I/O error, dev sdk, sector 1952320
> [10441.297744] device-mapper: multipath: Failing path 8:160.
> [10441.300255] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.300340] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] 
> [10441.300419] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.300535] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 00 00 38 00 00 08 00
> [10441.300626] blk_update_request: I/O error, dev sdaa, sector 56
> [10441.300861] device-mapper: multipath: Failing path 65:160.
> [10441.302907] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.302972] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10441.303045] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.303132] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 70 00 00 60 00
> [10441.303215] blk_update_request: I/O error, dev sdu, sector 624
> [10441.303399] device-mapper: multipath: Failing path 65:64.
> [10441.306683] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.306792] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10441.306862] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.306937] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10441.307000] blk_update_request: I/O error, dev sdo, sector 608
> [10441.307201] device-mapper: multipath: Failing path 8:224.
> [10441.307621] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.307685] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10441.307739] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.307799] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10441.307855] blk_update_request: I/O error, dev sdad, sector 608
> [10441.307939] device-mapper: multipath: Failing path 65:208.
> [10441.317400] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.317503] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10441.317591] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.317649] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cb 20 00 00 30 00
> [10441.317704] blk_update_request: I/O error, dev sdai, sector 1952544
> [10441.317802] device-mapper: multipath: Failing path 66:32.
> [10441.387950] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.388012] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10441.388060] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.388108] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10441.388156] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10441.388227] device-mapper: multipath: Failing path 67:224.
> [10441.396210] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10441.396260] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10441.396308] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10441.396377] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10441.396436] blk_update_request: I/O error, dev sdas, sector 1952320
> [10441.396525] device-mapper: multipath: Failing path 66:192.
> [10441.404557] device-mapper: multipath: Failing path 66:176.
> [10442.380325] device-mapper: multipath: Failing path 65:176.
> [10442.380917] device-mapper: multipath: Failing path 66:208.
> [10442.385656] device-mapper: multipath: Failing path 66:64.
> [10442.398393] device-mapper: multipath: Failing path 68:16.
> [10442.398714] device-mapper: multipath: Failing path 67:192.
> [10442.403546] device-mapper: multipath: Failing path 68:0.
> [10443.438787] device-mapper: multipath: Failing path 8:112.
> [10443.445929] device-mapper: multipath: Failing path 8:96.
> [10443.452757] device-mapper: multipath: Failing path 67:144.
> [10443.456211] device-mapper: multipath: Failing path 8:240.
> [10443.457007] device-mapper: multipath: Failing path 66:80.
> [10443.465293] device-mapper: multipath: Failing path 65:192.
> [10443.470696] device-mapper: multipath: Failing path 66:160.
> [10444.503706] device-mapper: multipath: Failing path 66:48.
> [10444.510208] device-mapper: multipath: Failing path 67:128.
> [10444.517049] device-mapper: multipath: Failing path 67:64.
> [10445.536571] device-mapper: multipath: Failing path 8:128.
> [10445.537057] device-mapper: multipath: Failing path 65:80.
> [10445.544179] device-mapper: multipath: Failing path 65:128.
> [10445.546885] device-mapper: multipath: Failing path 8:176.
> [10445.551266] device-mapper: multipath: Failing path 67:96.
> [10446.585026] scsi_io_completion: 22 callbacks suppressed
> [10446.585106] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.585175] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10446.585238] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.585305] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10446.585368] blk_update_request: 22 callbacks suppressed
> [10446.585427] blk_update_request: I/O error, dev sdj, sector 1952320
> [10446.585532] device-mapper: multipath: Failing path 8:144.
> [10446.587225] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.587328] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10446.587409] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.587544] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00
> [10446.587626] blk_update_request: I/O error, dev sdz, sector 24
> [10446.588770] device-mapper: multipath: Failing path 65:144.
> [10446.596518] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.596664] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10446.596783] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.596934] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10446.597056] blk_update_request: I/O error, dev sdu, sector 608
> [10446.597236] device-mapper: multipath: Failing path 65:64.
> [10446.612060] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.612156] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10446.612221] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.612292] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10446.612357] blk_update_request: I/O error, dev sdk, sector 1952320
> [10446.612575] device-mapper: multipath: Failing path 8:160.
> [10446.613185] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.613324] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10446.613442] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.613576] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10446.613694] blk_update_request: I/O error, dev sdad, sector 1952320
> [10446.613897] device-mapper: multipath: Failing path 65:208.
> [10446.618945] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.619076] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10446.619186] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.619329] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00
> [10446.619458] blk_update_request: I/O error, dev sdai, sector 32
> [10446.619717] device-mapper: multipath: Failing path 66:32.
> [10446.623887] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.624020] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10446.624136] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.624269] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10446.624374] blk_update_request: I/O error, dev sdo, sector 608
> [10446.624660] device-mapper: multipath: Failing path 8:224.
> [10446.630687] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.630831] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10446.630956] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.631081] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 90 00
> [10446.631193] blk_update_request: I/O error, dev sdas, sector 1952368
> [10446.631370] device-mapper: multipath: Failing path 66:192.
> [10446.701475] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.701555] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10446.701616] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.701679] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10446.701738] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10446.701834] device-mapper: multipath: Failing path 67:224.
> [10446.722225] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10446.722330] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10446.722422] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10446.722506] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10446.722604] blk_update_request: I/O error, dev sdar, sector 1952320
> [10446.722745] device-mapper: multipath: Failing path 66:176.
> [10447.703907] device-mapper: multipath: Failing path 65:160.
> [10447.704418] device-mapper: multipath: Failing path 66:64.
> [10447.711405] device-mapper: multipath: Failing path 65:176.
> [10447.717415] device-mapper: multipath: Failing path 66:208.
> [10447.736685] device-mapper: multipath: Failing path 68:16.
> [10447.737169] device-mapper: multipath: Failing path 67:192.
> [10447.742386] device-mapper: multipath: Failing path 68:0.
> [10448.746828] device-mapper: multipath: Failing path 8:112.
> [10448.752272] device-mapper: multipath: Failing path 8:96.
> [10448.777442] device-mapper: multipath: Failing path 8:240.
> [10448.778049] device-mapper: multipath: Failing path 66:80.
> [10448.783876] device-mapper: multipath: Failing path 65:192.
> [10448.784283] device-mapper: multipath: Failing path 66:160.
> [10448.846609] device-mapper: multipath: Failing path 67:144.
> [10449.839649] device-mapper: multipath: Failing path 66:48.
> [10449.846890] device-mapper: multipath: Failing path 67:128.
> [10449.855407] device-mapper: multipath: Failing path 67:64.
> [10450.875424] device-mapper: multipath: Failing path 8:128.
> [10450.875881] device-mapper: multipath: Failing path 65:80.
> [10450.876903] device-mapper: multipath: Failing path 67:96.
> [10450.883995] device-mapper: multipath: Failing path 65:128.
> [10450.886613] device-mapper: multipath: Failing path 8:176.
> [10451.926827] scsi_io_completion: 22 callbacks suppressed
> [10451.926909] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10451.926974] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10451.927034] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10451.927099] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10451.927160] blk_update_request: 22 callbacks suppressed
> [10451.927219] blk_update_request: I/O error, dev sdj, sector 1952320
> [10451.927337] device-mapper: multipath: Failing path 8:144.
> [10451.930152] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10451.930249] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10451.930332] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10451.930451] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 78 00 00 08 00
> [10451.930536] blk_update_request: I/O error, dev sdz, sector 120
> [10451.930684] device-mapper: multipath: Failing path 65:144.
> [10451.935021] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10451.935127] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10451.935186] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10451.935253] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10451.935309] blk_update_request: I/O error, dev sdu, sector 608
> [10451.935450] device-mapper: multipath: Failing path 65:64.
> [10451.937343] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10451.937431] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10451.937506] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10451.937625] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10451.937705] blk_update_request: I/O error, dev sdk, sector 1952320
> [10451.937857] device-mapper: multipath: Failing path 8:160.
> [10451.944256] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10451.944355] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10451.944434] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10451.944502] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 10 00 00 08 00
> [10451.944560] blk_update_request: I/O error, dev sdad, sector 16
> [10451.944650] device-mapper: multipath: Failing path 65:208.
> [10451.948096] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10451.948166] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10451.948221] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10451.948282] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 60 00 00 60 00
> [10451.948338] blk_update_request: I/O error, dev sdo, sector 864
> [10451.948431] device-mapper: multipath: Failing path 8:224.
> [10451.950866] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10451.950938] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10451.950992] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10451.951048] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d c9 80 00 00 80 00
> [10451.951102] blk_update_request: I/O error, dev sdai, sector 1952128
> [10451.951194] device-mapper: multipath: Failing path 66:32.
> [10452.032024] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10452.032084] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10452.032122] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10452.032163] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10452.032199] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10452.032299] device-mapper: multipath: Failing path 67:224.
> [10452.038388] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10452.038449] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10452.038500] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10452.038552] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10452.038596] blk_update_request: I/O error, dev sdas, sector 1952320
> [10452.038721] device-mapper: multipath: Failing path 66:192.
> [10452.045632] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10452.045688] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10452.045733] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10452.045779] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00
> [10452.045822] blk_update_request: I/O error, dev sdar, sector 704
> [10452.045927] device-mapper: multipath: Failing path 66:176.
> [10453.023588] device-mapper: multipath: Failing path 65:160.
> [10453.024299] device-mapper: multipath: Failing path 66:64.
> [10453.030250] device-mapper: multipath: Failing path 65:176.
> [10453.036030] device-mapper: multipath: Failing path 66:208.
> [10453.082442] device-mapper: multipath: Failing path 68:16.
> [10453.082943] device-mapper: multipath: Failing path 67:192.
> [10453.088366] device-mapper: multipath: Failing path 68:0.
> [10454.101354] device-mapper: multipath: Failing path 8:112.
> [10454.109728] device-mapper: multipath: Failing path 8:96.
> [10454.122180] device-mapper: multipath: Failing path 8:240.
> [10454.123095] device-mapper: multipath: Failing path 66:80.
> [10454.133511] device-mapper: multipath: Failing path 65:192.
> [10454.218507] device-mapper: multipath: Failing path 67:144.
> [10454.225052] device-mapper: multipath: Failing path 66:160.
> [10455.184119] device-mapper: multipath: Failing path 66:48.
> [10455.192576] device-mapper: multipath: Failing path 67:128.
> [10455.199838] device-mapper: multipath: Failing path 67:64.
> [10456.219579] device-mapper: multipath: Failing path 8:128.
> [10456.220649] device-mapper: multipath: Failing path 65:80.
> [10456.224363] device-mapper: multipath: Failing path 8:176.
> [10456.228154] device-mapper: multipath: Failing path 65:128.
> [10456.236087] device-mapper: multipath: Failing path 67:96.
> [10457.265228] scsi_io_completion: 22 callbacks suppressed
> [10457.265309] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.265376] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10457.265441] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.265510] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10457.265573] blk_update_request: 22 callbacks suppressed
> [10457.265631] blk_update_request: I/O error, dev sdj, sector 1952320
> [10457.265727] device-mapper: multipath: Failing path 8:144.
> [10457.266608] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.266722] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10457.266805] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.266929] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d c1 e8 00 00 08 00
> [10457.267015] blk_update_request: I/O error, dev sdz, sector 1950184
> [10457.267159] device-mapper: multipath: Failing path 65:144.
> [10457.273904] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.274003] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10457.274078] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.274161] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 88 00 00 60 00
> [10457.274236] blk_update_request: I/O error, dev sdu, sector 648
> [10457.274357] device-mapper: multipath: Failing path 65:64.
> [10457.275488] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.275583] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10457.275670] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.275796] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10457.275874] blk_update_request: I/O error, dev sdk, sector 1952320
> [10457.276184] device-mapper: multipath: Failing path 8:160.
> [10457.276668] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.276766] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10457.276841] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.276925] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10457.277005] blk_update_request: I/O error, dev sdad, sector 1952320
> [10457.277106] device-mapper: multipath: Failing path 65:208.
> [10457.285381] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.285455] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10457.285512] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.285573] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 f0 00 00 10 00
> [10457.285817] blk_update_request: I/O error, dev sdo, sector 752
> [10457.285944] device-mapper: multipath: Failing path 8:224.
> [10457.286392] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.286477] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10457.286552] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.286632] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 02 f0 00 00 10 00
> [10457.286707] blk_update_request: I/O error, dev sdai, sector 752
> [10457.286828] device-mapper: multipath: Failing path 66:32.
> [10457.294129] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.294220] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10457.294295] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.294364] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00
> [10457.294424] blk_update_request: I/O error, dev sdas, sector 1952688
> [10457.294523] device-mapper: multipath: Failing path 66:192.
> [10457.372572] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.372641] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10457.372688] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.372737] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10457.372783] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10457.372852] device-mapper: multipath: Failing path 67:224.
> [10457.379743] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10457.379833] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10457.379889] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10457.379955] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10457.380011] blk_update_request: I/O error, dev sdar, sector 1952320
> [10457.380163] device-mapper: multipath: Failing path 66:176.
> [10458.355184] device-mapper: multipath: Failing path 65:160.
> [10458.355600] device-mapper: multipath: Failing path 66:64.
> [10458.362158] device-mapper: multipath: Failing path 65:176.
> [10458.367380] device-mapper: multipath: Failing path 66:208.
> [10458.388284] device-mapper: multipath: Failing path 68:16.
> [10458.389179] device-mapper: multipath: Failing path 67:192.
> [10458.395063] device-mapper: multipath: Failing path 68:0.
> [10459.421645] device-mapper: multipath: Failing path 8:112.
> [10459.427728] device-mapper: multipath: Failing path 8:96.
> [10459.433866] device-mapper: multipath: Failing path 67:144.
> [10459.444039] device-mapper: multipath: Failing path 8:240.
> [10459.444657] device-mapper: multipath: Failing path 66:80.
> [10459.453974] device-mapper: multipath: Failing path 65:192.
> [10459.454422] device-mapper: multipath: Failing path 66:160.
> [10460.482933] device-mapper: multipath: Failing path 66:48.
> [10460.488513] device-mapper: multipath: Failing path 67:128.
> [10460.495239] device-mapper: multipath: Failing path 67:64.
> [10461.513588] device-mapper: multipath: Failing path 8:128.
> [10461.514034] device-mapper: multipath: Failing path 65:80.
> [10461.521182] device-mapper: multipath: Failing path 65:128.
> [10461.522550] device-mapper: multipath: Failing path 8:176.
> [10461.528881] device-mapper: multipath: Failing path 67:96.
> [10462.564110] scsi_io_completion: 22 callbacks suppressed
> [10462.564202] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.564288] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10462.564368] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.564466] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10462.564557] blk_update_request: 22 callbacks suppressed
> [10462.564640] blk_update_request: I/O error, dev sdj, sector 1952320
> [10462.564763] device-mapper: multipath: Failing path 8:144.
> [10462.566732] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.566834] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10462.566929] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.567055] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00
> [10462.567137] blk_update_request: I/O error, dev sdz, sector 24
> [10462.567301] device-mapper: multipath: Failing path 65:144.
> [10462.573104] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.573192] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10462.573263] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.573332] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10462.573390] blk_update_request: I/O error, dev sdu, sector 608
> [10462.573467] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.573472] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10462.573477] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.573481] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10462.573485] blk_update_request: I/O error, dev sdk, sector 1952320
> [10462.573547] device-mapper: multipath: Failing path 8:160.
> [10462.574255] device-mapper: multipath: Failing path 65:64.
> [10462.574825] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.574884] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10462.574938] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.574989] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00
> [10462.575044] blk_update_request: I/O error, dev sdad, sector 2048
> [10462.575121] device-mapper: multipath: Failing path 65:208.
> [10462.581259] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.581316] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10462.581365] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.581413] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 b0 00
> [10462.581469] blk_update_request: I/O error, dev sdo, sector 768
> [10462.581594] device-mapper: multipath: Failing path 8:224.
> [10462.583891] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.583954] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10462.584001] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.584053] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d c9 60 00 00 60 00
> [10462.584103] blk_update_request: I/O error, dev sdai, sector 1952096
> [10462.584219] device-mapper: multipath: Failing path 66:32.
> [10462.686906] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.686982] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10462.687040] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.687097] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10462.687151] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10462.687234] device-mapper: multipath: Failing path 67:224.
> [10462.692048] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.692112] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10462.692159] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.692210] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10462.692259] blk_update_request: I/O error, dev sdas, sector 1952320
> [10462.692330] device-mapper: multipath: Failing path 66:192.
> [10462.699023] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10462.699089] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10462.699139] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10462.699190] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 60 00
> [10462.699246] blk_update_request: I/O error, dev sdar, sector 768
> [10462.699413] device-mapper: multipath: Failing path 66:176.
> [10463.689681] device-mapper: multipath: Failing path 65:160.
> [10463.690254] device-mapper: multipath: Failing path 66:64.
> [10463.697821] device-mapper: multipath: Failing path 65:176.
> [10463.705424] device-mapper: multipath: Failing path 66:208.
> [10463.726975] device-mapper: multipath: Failing path 68:16.
> [10463.727373] device-mapper: multipath: Failing path 67:192.
> [10463.736754] device-mapper: multipath: Failing path 68:0.
> [10464.765376] device-mapper: multipath: Failing path 8:112.
> [10464.772388] device-mapper: multipath: Failing path 8:96.
> [10464.784249] device-mapper: multipath: Failing path 8:240.
> [10464.784747] device-mapper: multipath: Failing path 66:80.
> [10464.792111] device-mapper: multipath: Failing path 65:192.
> [10464.798740] device-mapper: multipath: Failing path 66:160.
> [10464.852449] device-mapper: multipath: Failing path 67:144.
> [10465.837027] device-mapper: multipath: Failing path 66:48.
> [10465.840983] device-mapper: multipath: Failing path 67:128.
> [10465.847920] device-mapper: multipath: Failing path 67:64.
> [10466.866019] device-mapper: multipath: Failing path 8:128.
> [10466.866759] device-mapper: multipath: Failing path 65:80.
> [10466.872622] device-mapper: multipath: Failing path 8:176.
> [10466.874797] device-mapper: multipath: Failing path 65:128.
> [10466.882759] device-mapper: multipath: Failing path 67:96.
> [10467.910924] scsi_io_completion: 22 callbacks suppressed
> [10467.910999] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10467.911058] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10467.911116] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10467.911178] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10467.911235] blk_update_request: 22 callbacks suppressed
> [10467.911287] blk_update_request: I/O error, dev sdj, sector 1952320
> [10467.911374] device-mapper: multipath: Failing path 8:144.
> [10467.912763] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10467.912851] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10467.912921] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10467.912996] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00
> [10467.913102] blk_update_request: I/O error, dev sdz, sector 2048
> [10467.913339] device-mapper: multipath: Failing path 65:144.
> [10467.919680] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10467.919777] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10467.919873] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10467.919942] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10467.920021] blk_update_request: I/O error, dev sdu, sector 608
> [10467.920234] device-mapper: multipath: Failing path 65:64.
> [10467.920949] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10467.921039] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10467.921110] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10467.921225] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10467.921310] blk_update_request: I/O error, dev sdk, sector 1952320
> [10467.921462] device-mapper: multipath: Failing path 8:160.
> [10467.925848] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10467.925922] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10467.925987] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10467.926058] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00
> [10467.926266] blk_update_request: I/O error, dev sdad, sector 32
> [10467.926765] device-mapper: multipath: Failing path 65:208.
> [10467.931080] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10467.931152] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10467.931207] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10467.931262] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 88 00
> [10467.931316] blk_update_request: I/O error, dev sdo, sector 768
> [10467.931409] device-mapper: multipath: Failing path 8:224.
> [10467.935526] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10467.935597] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10467.935657] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10467.935717] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d ca 00 00 00 18 00
> [10467.935772] blk_update_request: I/O error, dev sdai, sector 1952256
> [10467.935916] device-mapper: multipath: Failing path 66:32.
> [10468.010601] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10468.010674] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10468.010723] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10468.010775] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10468.010823] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10468.010900] device-mapper: multipath: Failing path 67:224.
> [10468.017011] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10468.017091] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10468.017156] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10468.017224] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 68 00 00 08 00
> [10468.017293] blk_update_request: I/O error, dev sdas, sector 1952360
> [10468.017447] device-mapper: multipath: Failing path 66:192.
> [10468.018534] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10468.018613] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10468.018679] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10468.018750] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00
> [10468.018815] blk_update_request: I/O error, dev sdar, sector 1952280
> [10468.018916] device-mapper: multipath: Failing path 66:176.
> [10469.006483] device-mapper: multipath: Failing path 65:160.
> [10469.006965] device-mapper: multipath: Failing path 66:64.
> [10469.012283] device-mapper: multipath: Failing path 65:176.
> [10469.016967] device-mapper: multipath: Failing path 66:208.
> [10469.047129] device-mapper: multipath: Failing path 68:16.
> [10469.047451] device-mapper: multipath: Failing path 67:192.
> [10469.052379] device-mapper: multipath: Failing path 68:0.
> [10470.072818] device-mapper: multipath: Failing path 8:112.
> [10470.079865] device-mapper: multipath: Failing path 8:96.
> [10470.101784] device-mapper: multipath: Failing path 8:240.
> [10470.102397] device-mapper: multipath: Failing path 66:80.
> [10470.111733] device-mapper: multipath: Failing path 65:192.
> [10470.112277] device-mapper: multipath: Failing path 66:160.
> [10470.169999] device-mapper: multipath: Failing path 67:144.
> [10471.157267] device-mapper: multipath: Failing path 66:48.
> [10471.166331] device-mapper: multipath: Failing path 67:128.
> [10471.172806] device-mapper: multipath: Failing path 67:64.
> [10472.191121] device-mapper: multipath: Failing path 8:128.
> [10472.191627] device-mapper: multipath: Failing path 65:80.
> [10472.192259] device-mapper: multipath: Failing path 8:176.
> [10472.198851] device-mapper: multipath: Failing path 65:128.
> [10472.204424] device-mapper: multipath: Failing path 67:96.
> [10473.252723] scsi_io_completion: 22 callbacks suppressed
> [10473.252798] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.252860] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10473.252918] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.252981] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10473.253045] blk_update_request: 22 callbacks suppressed
> [10473.253099] blk_update_request: I/O error, dev sdj, sector 1952320
> [10473.253236] device-mapper: multipath: Failing path 8:144.
> [10473.254825] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.254913] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10473.254982] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.255051] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d c1 e8 00 00 08 00
> [10473.255118] blk_update_request: I/O error, dev sdz, sector 1950184
> [10473.255219] device-mapper: multipath: Failing path 65:144.
> [10473.262826] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.262921] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10473.262984] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.263052] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00
> [10473.263123] blk_update_request: I/O error, dev sdu, sector 608
> [10473.263268] device-mapper: multipath: Failing path 65:64.
> [10473.264610] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.264682] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10473.264744] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.264807] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10473.264861] blk_update_request: I/O error, dev sdk, sector 1952320
> [10473.265012] device-mapper: multipath: Failing path 8:160.
> [10473.265457] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.265532] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10473.265589] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.265648] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10473.265704] blk_update_request: I/O error, dev sdad, sector 1952320
> [10473.265790] device-mapper: multipath: Failing path 65:208.
> [10473.272881] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.272984] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10473.273069] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.273166] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 60 00
> [10473.273255] blk_update_request: I/O error, dev sdai, sector 512
> [10473.273479] device-mapper: multipath: Failing path 66:32.
> [10473.273980] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.274079] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10473.274168] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.274272] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 60 00
> [10473.274362] blk_update_request: I/O error, dev sdo, sector 512
> [10473.274532] device-mapper: multipath: Failing path 8:224.
> [10473.286394] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.286470] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10473.286533] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.286597] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00
> [10473.286658] blk_update_request: I/O error, dev sdas, sector 1952688
> [10473.287437] device-mapper: multipath: Failing path 66:192.
> [10473.363750] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.363816] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10473.363861] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.363908] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10473.363957] blk_update_request: I/O error, dev sdbk, sector 1952320
> [10473.364066] device-mapper: multipath: Failing path 67:224.
> [10473.365091] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10473.365138] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10473.365175] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10473.365215] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 28 00 00 08 00
> [10473.365269] blk_update_request: I/O error, dev sdar, sector 1952296
> [10473.365347] device-mapper: multipath: Failing path 66:176.
> [10474.346552] device-mapper: multipath: Failing path 65:160.
> [10474.347094] device-mapper: multipath: Failing path 66:64.
> [10474.354623] device-mapper: multipath: Failing path 65:176.
> [10474.362549] device-mapper: multipath: Failing path 66:208.
> [10474.363551] device-mapper: multipath: Failing path 68:16.
> [10474.364351] device-mapper: multipath: Failing path 67:192.
> [10474.369349] device-mapper: multipath: Failing path 68:0.
> [10475.402882] device-mapper: multipath: Failing path 8:112.
> [10475.409928] device-mapper: multipath: Failing path 8:96.
> [10475.422390] device-mapper: multipath: Failing path 8:240.
> [10475.423089] device-mapper: multipath: Failing path 66:80.
> [10475.431476] device-mapper: multipath: Failing path 65:192.
> [10475.437756] device-mapper: multipath: Failing path 66:160.
> [10475.497531] device-mapper: multipath: Failing path 67:144.
> [10476.475289] device-mapper: multipath: Failing path 66:48.
> [10476.481673] device-mapper: multipath: Failing path 67:128.
> [10476.487874] device-mapper: multipath: Failing path 67:64.
> [10477.508134] device-mapper: multipath: Failing path 8:128.
> [10477.508759] device-mapper: multipath: Failing path 65:80.
> [10477.509813] device-mapper: multipath: Failing path 67:96.
> [10477.514796] device-mapper: multipath: Failing path 8:176.
> [10477.515948] device-mapper: multipath: Failing path 65:128.
> [10478.553204] scsi_io_completion: 22 callbacks suppressed
> [10478.553279] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.553345] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] 
> [10478.553413] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.553482] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10478.553554] blk_update_request: 22 callbacks suppressed
> [10478.553611] blk_update_request: I/O error, dev sdj, sector 1952320
> [10478.553731] device-mapper: multipath: Failing path 8:144.
> [10478.554502] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.554579] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] 
> [10478.554637] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.554700] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10478.554757] blk_update_request: I/O error, dev sdz, sector 1952320
> [10478.555100] device-mapper: multipath: Failing path 65:144.
> [10478.560256] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.560329] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10478.560385] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.560449] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10478.560503] blk_update_request: I/O error, dev sdk, sector 1952320
> [10478.560729] device-mapper: multipath: Failing path 8:160.
> [10478.561184] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.561303] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10478.561404] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.561504] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10478.561610] blk_update_request: I/O error, dev sdad, sector 1952320
> [10478.561831] device-mapper: multipath: Failing path 65:208.
> [10478.562889] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.562993] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] 
> [10478.563053] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.563116] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 68 00 00 60 00
> [10478.563174] blk_update_request: I/O error, dev sdu, sector 616
> [10478.563274] device-mapper: multipath: Failing path 65:64.
> [10478.563686] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.563771] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10478.563848] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.563931] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00
> [10478.564008] blk_update_request: I/O error, dev sdas, sector 24
> [10478.564132] device-mapper: multipath: Failing path 66:192.
> [10478.570509] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.570583] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] 
> [10478.570643] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.570705] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 01 00 00
> [10478.570762] blk_update_request: I/O error, dev sdar, sector 768
> [10478.570853] device-mapper: multipath: Failing path 66:176.
> [10478.571267] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.571333] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10478.571400] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.571463] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 01 00 00
> [10478.571522] blk_update_request: I/O error, dev sdai, sector 768
> [10478.571624] device-mapper: multipath: Failing path 66:32.
> [10478.577878] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.577956] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] 
> [10478.578016] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.578078] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 10 00 00 00 08 00
> [10478.578136] blk_update_request: I/O error, dev sdo, sector 4096
> [10478.578222] device-mapper: multipath: Failing path 8:224.
> [10478.646314] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10478.646387] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] 
> [10478.646439] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10478.646491] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d cc f0 00 00 08 00
> [10478.646541] blk_update_request: I/O error, dev sdbk, sector 1953008
> [10478.646616] device-mapper: multipath: Failing path 67:224.
> [10479.627635] device-mapper: multipath: Failing path 65:160.
> [10479.628207] device-mapper: multipath: Failing path 66:64.
> [10479.636021] device-mapper: multipath: Failing path 65:176.
> [10479.643095] device-mapper: multipath: Failing path 66:208.
> [10479.677320] device-mapper: multipath: Failing path 68:16.
> [10479.677851] device-mapper: multipath: Failing path 67:192.
> [10479.683638] device-mapper: multipath: Failing path 68:0.
> [10480.677063] device-mapper: multipath: Failing path 8:112.
> [10480.681958] device-mapper: multipath: Failing path 8:96.
> [10480.683898] device-mapper: multipath: Failing path 8:240.
> [10480.684200] device-mapper: multipath: Failing path 66:80.
> [10480.685455] device-mapper: multipath: Failing path 67:144.
> [10480.689543] device-mapper: multipath: Failing path 65:192.
> [10480.694580] device-mapper: multipath: Failing path 66:160.
> [10481.719022] device-mapper: multipath: Failing path 66:48.
> [10481.729178] device-mapper: multipath: Failing path 67:128.
> [10481.734260] device-mapper: multipath: Failing path 67:64.
> [10482.756839] device-mapper: multipath: Failing path 8:128.
> [10482.763827] device-mapper: multipath: Failing path 65:80.
> [10482.773300] device-mapper: multipath: Failing path 65:128.
> [10482.780948] device-mapper: multipath: Failing path 8:176.
> [10483.833855] scsi_io_completion: 21 callbacks suppressed
> [10483.833924] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10483.833996] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] 
> [10483.834059] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10483.834123] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10483.834192] blk_update_request: 21 callbacks suppressed
> [10483.834252] blk_update_request: I/O error, dev sdad, sector 1952320
> [10483.834482] device-mapper: multipath: Failing path 65:208.
> [10483.835438] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10483.835506] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] 
> [10483.835566] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10483.835631] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10483.835696] blk_update_request: I/O error, dev sdas, sector 1952320
> [10483.835964] device-mapper: multipath: Failing path 66:192.
> [10483.842954] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10483.843039] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] 
> [10483.843118] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10483.843199] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 60 00
> [10483.843271] blk_update_request: I/O error, dev sdai, sector 768
> [10483.843552] device-mapper: multipath: Failing path 66:32.
> [10484.844848] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10484.844932] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] 
> [10484.845002] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10484.845081] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00
> [10484.845152] blk_update_request: I/O error, dev sdk, sector 1952320
> [10484.845737] device-mapper: multipath: Failing path 8:160.
> [10509.568010] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
> [10509.568088] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10509.575101] irq event stamp: 15233868
> [10509.575158] hardirqs last  enabled at (15233867): [<ffffffff81578167>] _raw_spin_unlock_irq+0x27/0x40
> [10509.575265] hardirqs last disabled at (15233868): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10509.575370] softirqs last  enabled at (15233850): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10509.575836] softirqs last disabled at (15233743): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10509.575960] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10509.576023] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10509.576086] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10509.576149] RIP: 0010:[<ffffffff8157816f>]  [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40
> [10509.576263] RSP: 0018:ffff880400fcbcd8  EFLAGS: 00000246
> [10509.576321] RAX: ffff88040120af80 RBX: ffff8800694542c0 RCX: 0000000000000001
> [10509.576383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80
> [10509.576440] RBP: ffff880400fcbce0 R08: ffff8800694543c0 R09: 0000000000000000
> [10509.576498] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88040caf2290
> [10509.576557] R13: ffff880069453a00 R14: ffff8803b724ab80 R15: 0000000000000000
> [10509.576611] FS:  0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000
> [10509.576667] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10509.576720] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10509.576772] Stack:
> [10509.576821]  ffff880455f0dcd0 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80
> [10509.577057]  ffff88040caf25d8 ffff8804203cefa8 ffff880455f0dd50 0000000000000046
> [10509.577310]  ffff880455f0dd00 ffff880069453a00 ffff880069453a00 0000000000000282
> [10509.577552] Call Trace:
> [10509.577606]  [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600
> [10509.577666]  [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40
> [10509.577723]  [<ffffffff81281175>] __elv_add_request+0x75/0x1f0
> [10509.577781]  [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190
> [10509.577845]  [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod]
> [10509.577909]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10509.577969]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10509.578028]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10509.578085]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10509.578142]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10509.578199]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10509.578255]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10509.578310] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 
> [10537.567212] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
> [10537.567287] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10537.573758] irq event stamp: 30645910
> [10537.573810] hardirqs last  enabled at (30645909): [<ffffffff81578167>] _raw_spin_unlock_irq+0x27/0x40
> [10537.573918] hardirqs last disabled at (30645910): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10537.574020] softirqs last  enabled at (30645860): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10537.574122] softirqs last disabled at (30645755): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10537.574223] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10537.574279] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10537.574335] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10537.574390] RIP: 0010:[<ffffffff8157816f>]  [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40
> [10537.574492] RSP: 0018:ffff880400fcbcd8  EFLAGS: 00000246
> [10537.574543] RAX: ffff88040120af80 RBX: ffff880444673440 RCX: 0000000000000001
> [10537.574594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80
> [10537.574645] RBP: ffff880400fcbce0 R08: ffff880444673540 R09: 0000000000000000
> [10537.575221] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8804426a4520
> [10537.575274] R13: ffff880444672b80 R14: ffff88033fd9a358 R15: 0000000000000000
> [10537.575327] FS:  0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000
> [10537.575382] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10537.575435] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10537.575487] Stack:
> [10537.575536]  ffff88040161d388 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80
> [10537.575783]  ffff8804426a4868 ffff880415a49c20 ffff88040161d408 0000000000000046
> [10537.576029]  ffff88040161d3b8 ffff880444672b80 ffff880444672b80 0000000000000282
> [10537.576270] Call Trace:
> [10537.576324]  [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600
> [10537.576384]  [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40
> [10537.576441]  [<ffffffff81281175>] __elv_add_request+0x75/0x1f0
> [10537.576498]  [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190
> [10537.576561]  [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod]
> [10537.576624]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10537.576685]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10537.576742]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10537.576798]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10537.576855]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10537.576911]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10537.576967]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10537.577022] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 
> [10546.926940] INFO: rcu_sched self-detected stall on CPU
> [10546.927120] 	3-...: (6497 ticks this GP) idle=fb9/140000000000002/0 softirq=2044956/2045037 fqs=5414 
> [10546.927219] 	 (t=6500 jiffies g=219289 c=219288 q=7233211)
> [10546.927401] Task dump for CPU 3:
> [10546.927486] kdmwork-254:2   R  running task        0   358      2 0x00000008
> [10546.927736]  0000000000000002 ffff88046f263848 ffffffff8108195f 0000000000000003
> [10546.928161]  ffffffff81a40a40 ffff88046f263860 ffffffff81084742 0000000000000004
> [10546.928578]  ffff88046f263890 ffffffff810b5e79 ffff88046f275400 ffffffff81a40a40
> [10546.928989] Call Trace:
> [10546.929076]  <IRQ>  [<ffffffff8108195f>] sched_show_task+0xbf/0x150
> [10546.929253]  [<ffffffff81084742>] dump_cpu_task+0x32/0x40
> [10546.929346]  [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0
> [10546.929438]  [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730
> [10546.929529]  [<ffffffff8109e769>] ? cpuacct_account_field+0xd9/0x1b0
> [10546.929621]  [<ffffffff8109e690>] ? cpuacct_charge+0x1a0/0x1a0
> [10546.929715]  [<ffffffff810bc1c4>] update_process_times+0x34/0x60
> [10546.929807]  [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50
> [10546.929899]  [<ffffffff810cb148>] tick_sched_timer+0x38/0x70
> [10546.929991]  [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30
> [10546.930083]  [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0
> [10546.930176]  [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0
> [10546.930267]  [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10
> [10546.930360]  [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90
> [10546.930451]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10546.930544]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10546.930641]  [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50
> [10546.930732]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10546.930835]  [<ffffffff810b71c2>] ? call_rcu_sched+0x12/0x20
> [10546.930929]  [<ffffffff8117baf8>] ? put_object+0x28/0x40
> [10546.931023]  [<ffffffff810a2bf1>] ? debug_check_no_locks_freed+0xb1/0x140
> [10546.931117]  [<ffffffff811d0a12>] ? kiocb_free+0x32/0x40
> [10546.931209]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10546.931301]  [<ffffffff811d0a12>] kiocb_free+0x32/0x40
> [10546.931395]  [<ffffffff811d1b75>] aio_complete+0x1e5/0x3c0
> [10546.931488]  [<ffffffff811bef45>] dio_complete+0x75/0x1d0
> [10546.931582]  [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130
> [10546.931675]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10546.931769]  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
> [10546.931867]  [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod]
> [10546.931961]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10546.932052]  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
> [10546.932147]  [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0
> [10546.932238]  [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610
> [10546.932333]  [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120
> [10546.932423]  [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140
> [10546.932516]  [<ffffffff8128c742>] blk_done_softirq+0x72/0x90
> [10546.932607]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10546.932700]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10546.932792]  [<ffffffff810065a5>] do_IRQ+0x65/0x110
> [10546.932884]  [<ffffffff815793c9>] common_interrupt+0x89/0x89
> [10546.932973]  <EOI>  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10546.933164]  [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
> [10546.933265]  [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
> [10546.933365]  [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
> [10546.933464]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10546.933561]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10546.933653]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10546.933747]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10546.933841]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10546.933934]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10546.934033]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10573.566190] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
> [10573.566286] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10573.575752] irq event stamp: 50535115
> [10573.575821] hardirqs last  enabled at (50535114): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10573.575959] hardirqs last disabled at (50535115): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10573.576115] softirqs last  enabled at (50535044): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10573.576259] softirqs last disabled at (50535061): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10573.576410] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10573.576491] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10573.576571] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10573.576650] RIP: 0010:[<ffffffff815781b3>]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10573.576797] RSP: 0018:ffff88046f263b88  EFLAGS: 00000296
> [10573.576868] RAX: ffff88040120af80 RBX: 0000000000000296 RCX: 0000000000000000
> [10573.576938] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88040120af80
> [10573.577007] RBP: ffff88046f263b98 R08: ffff88046f263b08 R09: 0000000000000000
> [10573.577078] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880461584e00
> [10573.577162] R13: ffffea0009ec0400 R14: ffff880461584e00 R15: ffff880461488d80
> [10573.577240] FS:  0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000
> [10573.577323] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10573.577399] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10573.577475] Stack:
> [10573.577546]  ffff880461584e00 0000000000000001 ffff88046f263c48 ffffffff811640d6
> [10573.577881]  ffff88046f263bd8 ffff88040120af80 ffff880266ffaa80 ffff88027b011540
> [10573.578207]  ffff88027b011540 ffff000109ec0400 0000000000000296 ffff88046f263bf0
> [10573.578538] Call Trace:
> [10573.578608]  <IRQ> 
> [10573.578690]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10573.578860]  [<ffffffff810b71c2>] ? call_rcu_sched+0x12/0x20
> [10573.578947]  [<ffffffff8117baf8>] ? put_object+0x28/0x40
> [10573.579029]  [<ffffffff810a2bf1>] ? debug_check_no_locks_freed+0xb1/0x140
> [10573.579114]  [<ffffffff811bef54>] ? dio_complete+0x84/0x1d0
> [10573.579203]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10573.579289]  [<ffffffff811bef54>] dio_complete+0x84/0x1d0
> [10573.579372]  [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130
> [10573.579453]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10573.579534]  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
> [10573.579622]  [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod]
> [10573.579704]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10573.579784]  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
> [10573.579866]  [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0
> [10573.579946]  [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610
> [10573.580030]  [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120
> [10573.580118]  [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140
> [10573.580210]  [<ffffffff8128c742>] blk_done_softirq+0x72/0x90
> [10573.580296]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10573.580374]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10573.580453]  [<ffffffff810065a5>] do_IRQ+0x65/0x110
> [10573.580535]  [<ffffffff815793c9>] common_interrupt+0x89/0x89
> [10573.580614]  <EOI> 
> [10573.580695]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10573.580853]  [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
> [10573.580942]  [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
> [10573.581032]  [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
> [10573.581122]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10573.581205]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10573.581284]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10573.581367]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10573.581448]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10573.581530]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10573.581609]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10573.581691] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 
> [10601.565455] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
> [10601.565532] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10601.572008] irq event stamp: 65965438
> [10601.572063] hardirqs last  enabled at (65965437): [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d
> [10601.572175] hardirqs last disabled at (65965438): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10601.572985] softirqs last  enabled at (65965386): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10601.573093] softirqs last disabled at (65965185): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10601.573195] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10601.573250] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10601.573304] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10601.573357] RIP: 0010:[<ffffffff8157816f>]  [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40
> [10601.573457] RSP: 0018:ffff880400fcbcd8  EFLAGS: 00000246
> [10601.573507] RAX: ffff88040120af80 RBX: ffff8800694542c0 RCX: 0000000000000001
> [10601.573557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80
> [10601.573609] RBP: ffff880400fcbce0 R08: ffff8800694543c0 R09: 0000000000000000
> [10601.573670] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88040caf2290
> [10601.573724] R13: ffff880069453a00 R14: ffff88020d55f230 R15: 0000000000000000
> [10601.573779] FS:  0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000
> [10601.573835] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10601.573888] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10601.573941] Stack:
> [10601.573990]  ffff880455f0dcd0 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80
> [10601.574230]  ffff88040caf25d8 ffff8804203cefa8 ffff880455f0dd50 0000000000000046
> [10601.574464]  ffff880455f0dd00 ffff880069453a00 ffff880069453a00 0000000000000282
> [10601.574694] Call Trace:
> [10601.574746]  [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600
> [10601.574804]  [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40
> [10601.575744]  [<ffffffff81281175>] __elv_add_request+0x75/0x1f0
> [10601.575810]  [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190
> [10601.575876]  [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod]
> [10601.575941]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10601.576004]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10601.576063]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10601.576122]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10601.576181]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10601.576240]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10601.576299]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10601.576356] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 
> [10629.564604] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
> [10629.564704] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10629.574344] irq event stamp: 81405685
> [10629.574414] hardirqs last  enabled at (81405684): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10629.574557] hardirqs last disabled at (81405685): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10629.574713] softirqs last  enabled at (81405466): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10629.574858] softirqs last disabled at (81405547): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10629.574999] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10629.575074] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10629.575157] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10629.575235] RIP: 0010:[<ffffffff815781b3>]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10629.575379] RSP: 0018:ffff88046f263b08  EFLAGS: 00000292
> [10629.575455] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000
> [10629.575529] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88040120af80
> [10629.575603] RBP: ffff88046f263b18 R08: ffff88046f263a88 R09: 0000000000000000
> [10629.575675] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880461585340
> [10629.575748] R13: ffffea000827d500 R14: ffff880461585340 R15: ffff880461489440
> [10629.575820] FS:  0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000
> [10629.575909] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10629.575984] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10629.576062] Stack:
> [10629.576134]  ffff880461585340 0000000000000001 ffff88046f263bc8 ffffffff811640d6
> [10629.576472]  ffff88046f263b58 ffff88040120af80 ffff880168a1fcd8 ffff880209f548c0
> [10629.576808]  ffff880209f548c0 ffff00010827d500 0000000000000292 ffff88046f263b70
> [10629.577140] Call Trace:
> [10629.577210]  <IRQ> 
> [10629.577282]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10629.577430]  [<ffffffff810b71c2>] ? call_rcu_sched+0x12/0x20
> [10629.577512]  [<ffffffff8117baf8>] ? put_object+0x28/0x40
> [10629.577610]  [<ffffffff810a2bf1>] ? debug_check_no_locks_freed+0xb1/0x140
> [10629.577697]  [<ffffffff811d0a12>] ? kiocb_free+0x32/0x40
> [10629.577781]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10629.577870]  [<ffffffff811d0a12>] kiocb_free+0x32/0x40
> [10629.577953]  [<ffffffff811d1b75>] aio_complete+0x1e5/0x3c0
> [10629.578036]  [<ffffffff811bef45>] dio_complete+0x75/0x1d0
> [10629.578118]  [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130
> [10629.578200]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10629.578283]  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
> [10629.578371]  [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod]
> [10629.578465]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10629.578544]  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
> [10629.578629]  [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0
> [10629.578710]  [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610
> [10629.578791]  [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120
> [10629.578872]  [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140
> [10629.578954]  [<ffffffff8128c742>] blk_done_softirq+0x72/0x90
> [10629.579034]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10629.579117]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10629.579197]  [<ffffffff810065a5>] do_IRQ+0x65/0x110
> [10629.579280]  [<ffffffff815793c9>] common_interrupt+0x89/0x89
> [10629.579357]  <EOI> 
> [10629.579428]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10629.579590]  [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
> [10629.579680]  [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
> [10629.579771]  [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
> [10629.579860]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10629.579946]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10629.580032]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10629.580114]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10629.580195]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10629.580276]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10629.580358]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10629.580438] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 
> [10657.563840] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
> [10657.563911] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10657.570523] irq event stamp: 96839370
> [10657.570577] hardirqs last  enabled at (96839369): [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d
> [10657.570688] hardirqs last disabled at (96839370): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10657.570792] softirqs last  enabled at (96839350): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10657.570900] softirqs last disabled at (96839343): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10657.571007] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10657.571066] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10657.571124] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10657.571180] RIP: 0010:[<ffffffff8157816f>]  [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40
> [10657.571277] RSP: 0018:ffff880400fcbd50  EFLAGS: 00000246
> [10657.571328] RAX: ffff88040120af80 RBX: ffff8803fa3a6928 RCX: 0000000000000001
> [10657.571378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80
> [10657.571427] RBP: ffff880400fcbd58 R08: ffff880408393a80 R09: 0000000000000000
> [10657.571485] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000001000
> [10657.571536] R13: ffff8802b3be3c70 R14: ffff88042dbaa358 R15: ffff8803fa3a6900
> [10657.571587] FS:  0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000
> [10657.571641] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10657.571691] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10657.571742] Stack:
> [10657.571791]  ffff8803fa3a6928 ffff880400fcbdb0 ffffffffa010f815 ffff880407cf26c8
> [10657.572021]  ffff880408393a80 0000000000000000 0000000000000000 ffff8802b3be3c28
> [10657.572251]  ffff880037cf2610 ffff88042d0fd668 ffffc900030d5040 ffff8802b3be3c48
> [10657.572485] Call Trace:
> [10657.572540]  [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
> [10657.572613]  [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
> [10657.572685]  [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
> [10657.572753]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10657.572818]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10657.572883]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10657.572943]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10657.573005]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10657.573072]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10657.573132]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10657.573191] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 
> [10693.562792] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [fio:7931]
> [10693.562873] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10693.569186] irq event stamp: 30263533
> [10693.569241] hardirqs last  enabled at (30263532): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10693.569355] hardirqs last disabled at (30263533): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10693.570408] softirqs last  enabled at (7684746): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10693.570522] softirqs last disabled at (7684923): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10693.570633] CPU: 3 PID: 7931 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10693.570693] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10693.570755] task: ffff880064ae2f80 ti: ffff8803f7578000 task.ti: ffff8803f7578000
> [10693.570816] RIP: 0010:[<ffffffff815781b3>]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10693.570932] RSP: 0018:ffff88046f263d78  EFLAGS: 00000292
> [10693.570989] RAX: ffff880064ae2f80 RBX: 0000000000000292 RCX: 0000000000000000
> [10693.571047] RDX: 0000000000000003 RSI: ffff880064ae3718 RDI: ffff880064ae2f80
> [10693.571105] RBP: ffff88046f263d88 R08: ffff88046f263cf8 R09: 0000000000000000
> [10693.571165] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10693.571223] R13: ffffea00069c1e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10693.571280] FS:  00007f8a98e2f700(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000
> [10693.571339] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10693.571397] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10693.571456] Stack:
> [10693.571512]  ffff88017fc01c00 0000000000000001 ffff88046f263e38 ffffffff811640d6
> [10693.571777]  ffff88046f263dc8 ffffffff81001017 0000000100300030 ffff8801a707b7c8
> [10693.572047]  ffff8801a707b7c8 ffff000164ae3718 0000000000000292 ffff880064ae3718
> [10693.572314] Call Trace:
> [10693.572372]  <IRQ> 
> [10693.572434]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10693.572557]  [<ffffffff81001017>] ? trace_hardirqs_on_thunk+0x17/0x19
> [10693.572622]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10693.572690]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10693.572754]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10693.572820]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10693.572884]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10693.572948]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10693.573013]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10693.574184]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10693.574252]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10693.574315]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10693.574379]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10693.574443]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10693.574509]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10693.574573]  <EOI> 
> [10693.574635]  [<ffffffff81286284>] ? blk_flush_plug_list+0x224/0x240
> [10693.574759]  [<ffffffff812865c7>] blk_finish_plug+0x27/0x40
> [10693.574824]  [<ffffffff811d3136>] do_io_submit+0x3c6/0x770
> [10693.574887]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10693.574951]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10693.575014]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10693.575077] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 
> [10700.274209] sd 37:0:0:2: [sdbc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [10700.274285] sd 37:0:0:2: [sdbc] tag#0 Sense Key : Not Ready [current] 
> [10700.274341] sd 37:0:0:2: [sdbc] tag#0 Add. Sense: Logical unit not accessible, target port in standby state
> [10700.274400] sd 37:0:0:2: [sdbc] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
> [10700.274455] blk_update_request: I/O error, dev sdbc, sector 0
> [10700.274586] device-mapper: multipath: Failing path 67:96.
> [10725.561889] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:7937]
> [10725.561895] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [fio:7930]
> [10725.561907] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE
> [10725.561907] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [disk11_0:32753]
> [10725.561909]  nf_nat_masquerade_ipv4
> [10725.561910] Modules linked in:
> [10725.561910]  iptable_nat
> [10725.561911]  ib_srpt(O-)
> [10725.561912]  nf_nat_ipv4
> [10725.561913]  scst_vdisk(O)
> [10725.561914]  nf_nat
> [10725.561914]  libcrc32c
> [10725.561915]  nf_conntrack_ipv4
> [10725.561916]  scst(O)
> [10725.561917]  nf_defrag_ipv4
> [10725.561917]  dlm
> [10725.561920]  xt_conntrack
> [10725.561920] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kdmwork-254:2:358]
> [10725.561922] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [fio:7934]
> [10725.561923] Modules linked in:
> [10725.561924] Modules linked in:
> [10725.561924]  ib_srpt(O-)
> [10725.561925]  ib_srpt(O-)
> [10725.561926]  scst_vdisk(O)
> [10725.561926]  scst_vdisk(O)
> [10725.561927]  libcrc32c
> [10725.561928]  libcrc32c
> [10725.561928]  scst(O)
> [10725.561931]  scst(O)
> [10725.561931] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [fio:7935]
> [10725.561932]  dlm
> [10725.561933]  dlm
> [10725.561933]  netconsole
> [10725.561934]  netconsole
> [10725.561934]  configfs
> [10725.561935]  configfs
> [10725.561936] Modules linked in:
> [10725.561937]  dm_queue_length
> [10725.561938]  ib_srpt(O-)
> [10725.561939]  brd
> [10725.561939]  scst_vdisk(O)
> [10725.561940]  nf_conntrack
> [10725.561943]  netconsole
> [10725.561943] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [fio:7940]
> [10725.561944]  ipt_REJECT
> [10725.561945]  libcrc32c
> [10725.561946]  configfs
> [10725.561947]  xt_CHECKSUM
> [10725.561948] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [fio:7938]
> [10725.561949] Modules linked in:
> [10725.561950] Modules linked in:
> [10725.561950]  ib_srpt(O-)
> [10725.561951]  dm_queue_length
> [10725.561952]  scst_vdisk(O)
> [10725.561953]  dm_queue_length
> [10725.561954]  xt_tcpudp
> [10725.561954]  ib_srpt(O-)
> [10725.561955]  libcrc32c
> [10725.561956]  scst_vdisk(O)
> [10725.561957]  scst(O)
> [10725.561957]  libcrc32c
> [10725.561958]  scst(O)
> [10725.561959]  dlm
> [10725.561960]  dlm
> [10725.561961]  brd
> [10725.561961]  netconsole
> [10725.561962]  tun
> [10725.561963]  netconsole
> [10725.561963]  scst(O)
> [10725.561964]  ebtable_filter
> [10725.561965]  xt_CHECKSUM
> [10725.561966]  dlm
> [10725.561966]  configfs
> [10725.561967]  iptable_mangle
> [10725.561968]  netconsole
> [10725.561969]  configfs
> [10725.561969]  configfs
> [10725.561970]  dm_queue_length
> [10725.561971]  dm_queue_length
> [10725.561971]  ipt_MASQUERADE
> [10725.561972]  dm_queue_length
> [10725.561973]  brd
> [10725.561974]  brd
> [10725.561974]  brd
> [10725.561975]  brd
> [10725.561976]  iptable_mangle
> [10725.561976]  xt_CHECKSUM
> [10725.561977]  ipt_MASQUERADE
> [10725.561978]  xt_CHECKSUM
> [10725.561979]  xt_CHECKSUM
> [10725.561980]  nf_nat_masquerade_ipv4
> [10725.561980]  nf_nat_masquerade_ipv4
> [10725.561981]  iptable_mangle
> [10725.561982]  iptable_nat
> [10725.561982]  iptable_mangle
> [10725.561983]  ebtables
> [10725.561984]  nf_nat_ipv4
> [10725.561985]  ipt_MASQUERADE
> [10725.561986]  iptable_nat
> [10725.561986]  ip6table_filter
> [10725.561987]  nf_nat_ipv4
> [10725.561988]  iptable_mangle
> [10725.561989]  ip6_tables
> [10725.561990]  xt_CHECKSUM
> [10725.561991]  ipt_MASQUERADE
> [10725.561991]  iptable_filter
> [10725.561992]  nf_nat_masquerade_ipv4
> [10725.561993]  nf_nat
> [10725.561993]  ipt_MASQUERADE
> [10725.561994]  iptable_mangle
> [10725.561995]  nf_nat_masquerade_ipv4
> [10725.561996]  ip_tables
> [10725.561996]  nf_nat
> [10725.561997]  nf_nat_masquerade_ipv4
> [10725.561998]  x_tables
> [10725.561999]  ipt_MASQUERADE
> [10725.561999]  iptable_nat
> [10725.562000]  iptable_nat
> [10725.562001]  nf_nat_ipv4
> [10725.562001]  af_packet
> [10725.562002]  nf_conntrack_ipv4
> [10725.562003]  nf_nat_ipv4
> [10725.562004]  nf_defrag_ipv4
> [10725.562004]  iptable_nat
> [10725.562005]  nf_conntrack_ipv4
> [10725.562007]  nf_nat_ipv4
> [10725.562008]  nf_defrag_ipv4
> [10725.562009]  nf_nat_masquerade_ipv4
> [10725.562010]  xt_conntrack
> [10725.562011]  xt_conntrack
> [10725.562011]  bridge
> [10725.562012]  nf_conntrack
> [10725.562013]  iptable_nat
> [10725.562014]  nf_nat
> [10725.562014]  stp
> [10725.562015]  nf_conntrack_ipv4
> [10725.562016]  nf_nat
> [10725.562016]  nf_nat
> [10725.562017]  nf_conntrack
> [10725.562018]  nf_conntrack_ipv4
> [10725.562019]  ipt_REJECT
> [10725.562019]  nf_defrag_ipv4
> [10725.562020]  xt_tcpudp
> [10725.562021]  ipt_REJECT
> [10725.562021]  nf_conntrack_ipv4
> [10725.562022]  llc
> [10725.562023]  xt_conntrack
> [10725.562024]  tun
> [10725.562024]  nf_conntrack
> [10725.562025]  ebtable_filter
> [10725.562026]  nf_nat_ipv4
> [10725.562026]  iscsi_ibft
> [10725.562027]  ebtables
> [10725.562028]  nf_nat
> [10725.562029]  nf_defrag_ipv4
> [10725.562029]  ip6table_filter
> [10725.562030]  ipt_REJECT
> [10725.562031]  ip6_tables
> [10725.562031]  xt_tcpudp
> [10725.562032]  iptable_filter
> [10725.562033]  tun
> [10725.562033]  iscsi_boot_sysfs
> [10725.562034]  ebtable_filter
> [10725.562035]  ip_tables
> [10725.562036]  xt_tcpudp
> [10725.562036]  nf_conntrack_ipv4
> [10725.562037]  xt_conntrack
> [10725.562038]  ebtables
> [10725.562039]  x_tables
> [10725.562039]  nf_conntrack
> [10725.562040]  ib_srp
> [10725.562041]  ipt_REJECT
> [10725.562042]  tun
> [10725.562042]  af_packet
> [10725.562043]  ip6table_filter
> [10725.562044]  xt_tcpudp
> [10725.562045]  scsi_transport_srp
> [10725.562045]  nf_defrag_ipv4
> [10725.562046]  ib_ipoib
> [10725.562047]  xt_conntrack
> [10725.562048]  ip6_tables
> [10725.562048]  bridge
> [10725.562049]  iptable_filter
> [10725.562050]  rdma_ucm
> [10725.562051]  tun
> [10725.562051]  ib_ucm
> [10725.562052]  stp
> [10725.562053]  ebtable_filter
> [10725.562054]  llc
> [10725.562054]  ip_tables
> [10725.562055]  iscsi_ibft
> [10725.562056]  x_tables
> [10725.562056]  iscsi_boot_sysfs
> [10725.562057]  ebtables
> [10725.562058]  ib_srp
> [10725.562058]  af_packet
> [10725.562059]  scsi_transport_srp
> [10725.562060]  bridge
> [10725.562061]  ib_ipoib
> [10725.562061]  ebtable_filter
> [10725.562062]  rdma_ucm
> [10725.562063]  stp
> [10725.562063]  ib_ucm
> [10725.562064]  llc
> [10725.562064]  nf_conntrack
> [10725.562065]  nf_defrag_ipv4
> [10725.562066]  ip6table_filter
> [10725.562067]  iscsi_ibft
> [10725.562067]  ib_uverbs
> [10725.562068]  iscsi_boot_sysfs
> [10725.562069]  ib_uverbs
> [10725.562069]  xt_conntrack
> [10725.562070]  msr
> [10725.562071]  ip6_tables
> [10725.562072]  ebtables
> [10725.562073]  ipt_REJECT
> [10725.562074]  ip6table_filter
> [10725.562074]  ib_umad
> [10725.562075]  iptable_filter
> [10725.562076]  nf_conntrack
> [10725.562076]  xt_tcpudp
> [10725.562077]  ipt_REJECT
> [10725.562078]  tun
> [10725.562079]  rdma_cm
> [10725.562079]  ib_srp
> [10725.562080]  msr
> [10725.562081]  scsi_transport_srp
> [10725.562081]  ib_umad
> [10725.562082]  ib_ipoib
> [10725.562083]  rdma_cm
> [10725.562083]  ebtable_filter
> [10725.562084]  xt_tcpudp
> [10725.562085]  ib_cm
> [10725.562087]  ebtables
> [10725.562087]  ip_tables
> [10725.562088]  iw_cm
> [10725.562089]  x_tables
> [10725.562089]  ib_cm
> [10725.562090]  ip6_tables
> [10725.562091]  af_packet
> [10725.562092]  tun
> [10725.562092]  ip6table_filter
> [10725.562093]  mlx4_ib
> [10725.562093]  rdma_ucm
> [10725.562094]  ib_sa
> [10725.562095]  iw_cm
> [10725.562096]  ib_ucm
> [10725.562096]  ib_mad
> [10725.562097]  iptable_filter
> [10725.562098]  ebtable_filter
> [10725.562099]  bridge
> [10725.562099]  ip6_tables
> [10725.562100]  ebtables
> [10725.562101]  iptable_filter
> [10725.562101]  ip6table_filter
> [10725.562102]  ip_tables
> [10725.562103]  ip6_tables
> [10725.562103]  ib_uverbs
> [10725.562104]  mlx4_ib
> [10725.562105]  x_tables
> [10725.562106]  iptable_filter
> [10725.562106]  af_packet
> [10725.562107]  ip_tables
> [10725.562107]  bridge
> [10725.562108]  x_tables
> [10725.562109]  stp
> [10725.562109]  ib_sa
> [10725.562110]  msr
> [10725.562111]  stp
> [10725.562112]  af_packet
> [10725.562112]  llc
> [10725.562113]  bridge
> [10725.562114]  iscsi_ibft
> [10725.562115]  ip_tables
> [10725.562115]  ib_umad
> [10725.562116]  ib_core
> [10725.562116]  rdma_cm
> [10725.562117]  ib_addr
> [10725.562118]  ib_mad
> [10725.562119]  x86_pkg_temp_thermal
> [10725.562120]  x_tables
> [10725.562120]  intel_powerclamp
> [10725.562121]  ib_cm
> [10725.562121]  coretemp
> [10725.562122]  stp
> [10725.562123]  llc
> [10725.562123]  kvm_intel
> [10725.562124]  iw_cm
> [10725.562125]  kvm
> [10725.562125]  mlx4_ib
> [10725.562126]  irqbypass
> [10725.562127]  ib_core
> [10725.562127]  crct10dif_pclmul
> [10725.562128]  ib_sa
> [10725.562129]  crc32_pclmul
> [10725.562129]  ib_mad
> [10725.562130]  crc32c_intel
> [10725.562131]  ib_core
> [10725.562132]  af_packet
> [10725.562132]  iscsi_ibft
> [10725.562133]  ib_addr
> [10725.562134]  ib_addr
> [10725.562134]  aesni_intel
> [10725.562135]  x86_pkg_temp_thermal
> [10725.562135]  tg3
> [10725.562136]  intel_powerclamp
> [10725.562137]  iTCO_wdt
> [10725.562137]  x86_pkg_temp_thermal
> [10725.562138]  coretemp
> [10725.562139]  aes_x86_64
> [10725.562140]  iscsi_boot_sysfs
> [10725.562140]  bridge
> [10725.562141]  lrw
> [10725.562141]  kvm_intel
> [10725.562142]  ipmi_ssif
> [10725.562143]  kvm
> [10725.562143]  ipmi_devintf
> [10725.562144]  irqbypass
> [10725.562145]  iscsi_boot_sysfs
> [10725.562145]  llc
> [10725.562146]  ib_srp
> [10725.562147]  ib_srp
> [10725.562147]  ptp
> [10725.562148]  crct10dif_pclmul
> [10725.562148]  iTCO_vendor_support
> [10725.562149]  crc32_pclmul
> [10725.562150]  gf128mul
> [10725.562150]  crc32c_intel
> [10725.562151]  stp
> [10725.562152]  scsi_transport_srp
> [10725.562153]  scsi_transport_srp
> [10725.562153]  iscsi_ibft
> [10725.562154]  ib_ipoib
> [10725.562155]  intel_powerclamp
> [10725.562156]  ib_ipoib
> [10725.562156]  iscsi_boot_sysfs
> [10725.562157]  rdma_ucm
> [10725.562158]  ib_srp
> [10725.562158]  rdma_ucm
> [10725.562159]  scsi_transport_srp
> [10725.562160]  glue_helper
> [10725.562161]  ib_ucm
> [10725.562161]  ib_ipoib
> [10725.562162]  ib_ucm
> [10725.562163]  coretemp
> [10725.562164]  mlx4_core
> [10725.562164]  llc
> [10725.562165]  ib_uverbs
> [10725.562166]  kvm_intel
> [10725.562167]  aesni_intel
> [10725.562167]  dcdbas
> [10725.562168]  tg3
> [10725.562168]  ablk_helper
> [10725.562169]  iTCO_wdt
> [10725.562169]  pps_core
> [10725.562171]  aes_x86_64
> [10725.562172]  kvm
> [10725.562173]  lrw
> [10725.562173]  cryptd
> [10725.562174]  msr
> [10725.562175]  rdma_ucm
> [10725.562176]  libphy
> [10725.562176]  irqbypass
> [10725.562177]  ib_uverbs
> [10725.562178]  ib_ucm
> [10725.562178]  msr
> [10725.562179]  ib_uverbs
> [10725.562180]  ib_umad
> [10725.562181]  crct10dif_pclmul
> [10725.562181]  ib_umad
> [10725.562182]  crc32_pclmul
> [10725.562183]  ipmi_ssif
> [10725.562183]  fjes
> [10725.562184]  ipmi_devintf
> [10725.562185]  8250_fintek
> [10725.562185]  ptp
> [10725.562186]  sb_edac
> [10725.562186]  iTCO_vendor_support
> [10725.562187]  pcspkr
> [10725.562188]  rdma_cm
> [10725.562189]  crc32c_intel
> [10725.562189]  rdma_cm
> [10725.562190]  iscsi_ibft
> [10725.562191]  aesni_intel
> [10725.562192]  ib_cm
> [10725.562192]  msr
> [10725.562193]  ib_cm
> [10725.562194]  ib_umad
> [10725.562194]  iw_cm
> [10725.562195]  rdma_cm
> [10725.562196]  mlx4_ib
> [10725.562196]  iscsi_boot_sysfs
> [10725.562197]  iw_cm
> [10725.562198]  tg3
> [10725.562199]  gf128mul
> [10725.562199]  ipmi_si
> [10725.562200]  ib_sa
> [10725.562201]  mlx4_ib
> [10725.562202]  ib_srp
> [10725.562202]  edac_core
> [10725.562203]  glue_helper
> [10725.562204]  ipmi_msghandler
> [10725.562204]  mlx4_core
> [10725.562205]  acpi_power_meter
> [10725.562206]  ib_cm
> [10725.562206]  ib_mad
> [10725.562207]  iw_cm
> [10725.562208]  scsi_transport_srp
> [10725.562208]  tpm_tis
> [10725.562209]  dcdbas
> [10725.562210]  iTCO_wdt
> [10725.562211]  ib_sa
> [10725.562211]  aes_x86_64
> [10725.562212]  ib_mad
> [10725.562213]  ib_ipoib
> [10725.562214]  lrw
> [10725.562214]  ib_core
> [10725.562215]  tpm
> [10725.562216]  ablk_helper
> [10725.562216]  mlx4_ib
> [10725.562217]  ib_core
> [10725.562218]  ib_sa
> [10725.562219]  ib_addr
> [10725.562219]  pps_core
> [10725.562220]  lpc_ich
> [10725.562221]  rdma_ucm
> [10725.562221]  ib_mad
> [10725.562222]  ib_addr
> [10725.562223]  ib_core
> [10725.562223]  x86_pkg_temp_thermal
> [10725.562224]  cryptd
> [10725.562225]  ipmi_ssif
> [10725.562226]  libphy
> [10725.562226]  button
> [10725.562227]  fjes
> [10725.562228]  intel_powerclamp
> [10725.562228]  ib_addr
> [10725.562229]  coretemp
> [10725.562230]  x86_pkg_temp_thermal
> [10725.562231]  shpchp
> [10725.562231]  x86_pkg_temp_thermal
> [10725.562232]  ipmi_devintf
> [10725.562233]  intel_powerclamp
> [10725.562234]  ib_ucm
> [10725.562234]  mei_me
> [10725.562235]  8250_fintek
> [10725.562236]  ptp
> [10725.562237]  kvm_intel
> [10725.562238]  iTCO_vendor_support
> [10725.562239]  coretemp
> [10725.562239]  intel_powerclamp
> [10725.562240]  mei
> [10725.562241]  sb_edac
> [10725.562241]  mfd_core
> [10725.562242]  pcspkr
> [10725.562243]  kvm_intel
> [10725.562244]  ipmi_si
> [10725.562244]  wmi
> [10725.562245]  edac_core
> [10725.562245]  dm_multipath
> [10725.562246]  ipmi_msghandler
> [10725.562247]  dm_mod
> [10725.562247]  acpi_power_meter
> [10725.562248]  hid_generic
> [10725.562248]  tpm_tis
> [10725.562249]  usbhid
> [10725.562249]  tpm
> [10725.562250]  mgag200
> [10725.562252]  lpc_ich
> [10725.562253]  ib_uverbs
> [10725.562253]  kvm
> [10725.562254]  coretemp
> [10725.562255]  kvm
> [10725.562255]  kvm_intel
> [10725.562256]  gf128mul
> [10725.562257]  irqbypass
> [10725.562258]  glue_helper
> [10725.562258]  crct10dif_pclmul
> [10725.562259]  button
> [10725.562260]  i2c_algo_bit
> [10725.562260]  shpchp
> [10725.562261]  crc32_pclmul
> [10725.562262]  kvm
> [10725.562262]  irqbypass
> [10725.562263]  irqbypass
> [10725.562264]  crct10dif_pclmul
> [10725.562264]  crc32c_intel
> [10725.562265]  msr
> [10725.562266]  aesni_intel
> [10725.562267]  ib_umad
> [10725.562268]  tg3
> [10725.562268]  mei_me
> [10725.562269]  iTCO_wdt
> [10725.562270]  mei
> [10725.562270]  drm_kms_helper
> [10725.562271]  mfd_core
> [10725.562271]  syscopyarea
> [10725.562272]  wmi
> [10725.562273]  sysfillrect
> [10725.562274]  rdma_cm
> [10725.562274]  mlx4_core
> [10725.562275]  ib_cm
> [10725.562276]  aes_x86_64
> [10725.562276]  iw_cm
> [10725.562277]  dm_multipath
> [10725.562278]  lrw
> [10725.562279]  dcdbas
> [10725.562279]  dm_mod
> [10725.562280]  sysimgblt
> [10725.562281]  crc32_pclmul
> [10725.562282]  crct10dif_pclmul
> [10725.562282]  ipmi_ssif
> [10725.562283]  crc32_pclmul
> [10725.562284]  crc32c_intel
> [10725.562285]  ablk_helper
> [10725.562285]  aesni_intel
> [10725.562286]  crc32c_intel
> [10725.562287]  tg3
> [10725.562287]  ipmi_devintf
> [10725.562288]  hid_generic
> [10725.562289]  fb_sys_fops
> [10725.562289]  usbhid
> [10725.562290]  pps_core
> [10725.562291]  mgag200
> [10725.562291]  ptp
> [10725.562292]  i2c_algo_bit
> [10725.562293]  ttm
> [10725.562293]  aesni_intel
> [10725.562294]  iTCO_wdt
> [10725.562295]  tg3
> [10725.562295]  aes_x86_64
> [10725.562296]  drm
> [10725.562297]  drm_kms_helper
> [10725.562297]  ehci_pci
> [10725.562298]  syscopyarea
> [10725.562299]  sr_mod
> [10725.562299]  iTCO_vendor_support
> [10725.562300]  cryptd
> [10725.562301]  cdrom
> [10725.562301]  gf128mul
> [10725.562302]  mlx4_ib
> [10725.562303]  ehci_hcd
> [10725.562304]  sysfillrect
> [10725.562304]  ib_sa
> [10725.562305]  iTCO_wdt
> [10725.562306]  lrw
> [10725.562307]  sysimgblt
> [10725.562307]  usbcore
> [10725.562308]  ipmi_ssif
> [10725.562309]  aes_x86_64
> [10725.562309]  ipmi_devintf
> [10725.562310]  lrw
> [10725.562311]  ptp
> [10725.562311]  libphy
> [10725.562312]  ib_mad
> [10725.562313]  iTCO_vendor_support
> [10725.562313]  ipmi_ssif
> [10725.562314]  gf128mul
> [10725.562315]  ipmi_devintf
> [10725.562315]  glue_helper
> [10725.562316]  ptp
> [10725.562317]  glue_helper
> [10725.562318]  iTCO_vendor_support
> [10725.562318]  mlx4_core
> [10725.562319]  gf128mul
> [10725.562319]  dcdbas
> [10725.562320]  glue_helper
> [10725.562321]  ablk_helper
> [10725.562321]  mlx4_core
> [10725.562322]  pps_core
> [10725.562322]  dcdbas
> [10725.562323]  cryptd
> [10725.562325]  fb_sys_fops
> [10725.562326]  fjes
> [10725.562326]  mlx4_core
> [10725.562327]  ttm
> [10725.562328]  usb_common
> [10725.562328]  drm
> [10725.562329]  sg
> [10725.562329]  ehci_pci
> [10725.562329] 
> [10725.562331]  8250_fintek
> [10725.562332]  dcdbas
> [10725.562332] irq event stamp: 11245005
> [10725.562333]  libphy
> [10725.562334]  sb_edac
> [10725.562335]  ib_core
> [10725.562335]  fjes
> [10725.562336]  ablk_helper
> [10725.562337]  8250_fintek
> [10725.562337]  pps_core
> [10725.562338]  sr_mod
> [10725.562339]  ablk_helper
> [10725.562340]  cryptd
> [10725.562340]  pcspkr
> [10725.562341]  cdrom
> [10725.562342] hardirqs last  enabled at (11245004): 
> [10725.562343]  ehci_hcd
> [10725.562343]  sb_edac
> [10725.562344]  libphy
> [10725.562350]  pcspkr
> [10725.562350] [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d
> [10725.562351]  usbcore
> [10725.562352] hardirqs last disabled at (11245005): 
> [10725.562354]  usb_common
> [10725.562354] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10725.562355]  sg
> [10725.562356]  ipmi_si
> [10725.562357] softirqs last  enabled at (8566716): 
> [10725.562357] 
> [10725.562358]  fjes
> [10725.562359]  ipmi_si
> [10725.562359]  8250_fintek
> [10725.562360]  edac_core
> [10725.562361]  sb_edac
> [10725.562361]  ipmi_msghandler
> [10725.562362]  ib_addr
> [10725.562363]  pps_core
> [10725.562363]  pcspkr
> [10725.562364]  acpi_power_meter
> [10725.562367]  ipmi_si
> [10725.562367] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10725.562368]  cryptd
> [10725.562369]  edac_core
> [10725.562370] softirqs last disabled at (8566949): 
> [10725.562370] irq event stamp: 122720091
> [10725.562372] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562373] hardirqs last  enabled at (122720090): 
> [10725.562375]  x86_pkg_temp_thermal
> [10725.562376] CPU: 7 PID: 7934 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10725.562378] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10725.562379]  ipmi_msghandler
> [10725.562380] hardirqs last disabled at (122720091): 
> [10725.562381]  libphy
> [10725.562381]  edac_core
> [10725.562382]  tpm_tis
> [10725.562383]  intel_powerclamp
> [10725.562383] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10725.562385]  tpm
> [10725.562385] task: ffff8804426faf80 ti: ffff8803f7598000 task.ti: ffff8803f7598000
> [10725.562387] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10725.562389] RIP: 0010:[<ffffffff815781b3>] 
> [10725.562391] softirqs last  enabled at (119833606): 
> [10725.562391]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10725.562392]  lpc_ich
> [10725.562393]  ipmi_msghandler
> [10725.562394]  button
> [10725.562394]  fjes
> [10725.562396]  acpi_power_meter
> [10725.562396] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10725.562398] RSP: 0018:ffff88046f2e3d78  EFLAGS: 00000292
> [10725.562399] softirqs last disabled at (119835011): 
> [10725.562402]  tpm_tis tpm
> [10725.562402] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562404]  8250_fintek
> [10725.562404] CPU: 1 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10725.562405]  shpchp
> [10725.562406]  acpi_power_meter
> [10725.562406]  mei_me
> [10725.562407]  tpm_tis
> [10725.562407]  mei
> [10725.562408]  tpm
> [10725.562409]  sb_edac
> [10725.562410]  lpc_ich
> [10725.562411]  pcspkr
> [10725.562412]  button
> [10725.562412] RAX: ffff8804426faf80 RBX: 0000000000000292 RCX: 0000000000000000
> [10725.562414]  ipmi_si
> [10725.562414] RDX: 0000000000000003 RSI: ffff8804426fb718 RDI: ffff8804426faf80
> [10725.562415]  edac_core
> [10725.562417]  shpchp
> [10725.562417] RBP: ffff88046f2e3d88 R08: ffff88046f2e3cf8 R09: 0000000000000000
> [10725.562418] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10725.562419] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10725.562420]  mfd_core
> [10725.562421]  lpc_ich
> [10725.562421]  wmi
> [10725.562422]  coretemp
> [10725.562423]  ipmi_msghandler
> [10725.562424] R13: ffffea0009ac0400 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10725.562425] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10725.562426] FS:  00007f8a9be35700(0000) GS:ffff88046f2e0000(0000) knlGS:0000000000000000
> [10725.562428] RIP: 0010:[<ffffffff815781b3>] 
> [10725.562429]  mei_me
> [10725.562429]  dm_multipath
> [10725.562430]  button
> [10725.562431]  dm_mod
> [10725.562432]  kvm_intel
> [10725.562433]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10725.562434] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10725.562435] RSP: 0018:ffff88046f223d78  EFLAGS: 00000292
> [10725.562436]  kvm
> [10725.562437]  mei
> [10725.562437]  hid_generic
> [10725.562438]  shpchp
> [10725.562439]  usbhid
> [10725.562439]  mei_me
> [10725.562440]  mgag200
> [10725.562441]  irqbypass
> [10725.562441] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000
> [10725.562442] CR2: 00007ffeaad51d1c CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10725.562443]  mei
> [10725.562444]  i2c_algo_bit
> [10725.562445]  mfd_core
> [10725.562446]  mfd_core
> [10725.562446]  drm_kms_helper
> [10725.562447]  wmi
> [10725.562448]  syscopyarea
> [10725.562449] RDX: 0000000000000003 RSI: ffff88040120b718 RDI: ffff88040120af80
> [10725.562449] Stack:
> [10725.562450] RBP: ffff88046f223d88 R08: ffff88046f223cf8 R09: 0000000000000000
> [10725.562452]  ffff88017fc01c00
> [10725.562452] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10725.562453]  0000000000000001
> [10725.562453] R13: ffffea000bac6e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10725.562455]  ffff88046f2e3e38
> [10725.562455] FS:  0000000000000000(0000) GS:ffff88046f220000(0000) knlGS:0000000000000000
> [10725.562457]  ffffffff811640d6
> [10725.562457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10725.562458] 
> [10725.562459] CR2: 00007ffadf28aa08 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10725.562460]  acpi_power_meter
> [10725.562461]  wmi
> [10725.562462]  crct10dif_pclmul
> [10725.562463]  tpm_tis
> [10725.562463]  ffff88046f2e3dc8
> [10725.562463] Stack:
> [10725.562464]  0000000000000246
> [10725.562465]  ffff88017fc01c00
> [10725.562466]  ffffffff811859b3
> [10725.562466]  0000000000000001
> [10725.562467]  ffff88026b010aa0
> [10725.562467]  ffff88046f223e38
> [10725.562467] 
> [10725.562469]  crc32_pclmul
> [10725.562470]  tpm
> [10725.562470]  sysfillrect
> [10725.562471]  dm_multipath
> [10725.562472]  lpc_ich
> [10725.562472]  crc32c_intel
> [10725.562473]  dm_mod
> [10725.562474]  button
> [10725.562475]  ffff88026b010aa0
> [10725.562476]  shpchp
> [10725.562478]  dm_multipath dm_mod
> [10725.562478]  mei_me
> [10725.562479]  aesni_intel
> [10725.562480]  ffff0001426fb718
> [10725.562480]  ffffffff811640d6
> [10725.562481]  0000000000000292
> [10725.562482]  sysimgblt
> [10725.562482]  hid_generic
> [10725.562483]  fb_sys_fops
> [10725.562483]  usbhid
> [10725.562484]  ttm
> [10725.562485]  mgag200
> [10725.562485]  drm
> [10725.562486]  i2c_algo_bit
> [10725.562486]  ehci_pci
> [10725.562487]  hid_generic
> [10725.562488]  mei
> [10725.562489]  sr_mod
> [10725.562489] 
> [10725.562490]  ffff8804426fb718
> [10725.562491]  cdrom
> [10725.562492]  drm_kms_helper
> [10725.562492]  ehci_hcd
> [10725.562493]  syscopyarea
> [10725.562493]  usbcore
> [10725.562494]  sysfillrect
> [10725.562495]  usb_common
> [10725.562495]  sysimgblt
> [10725.562496]  sg
> [10725.562497]  mfd_core
> [10725.562497]  fb_sys_fops
> [10725.562497] 
> [10725.562499]  ttm
> [10725.562499] irq event stamp: 11251127
> [10725.562500]  drm
> [10725.562501] hardirqs last  enabled at (11251126): 
> [10725.562501]  ehci_pci
> [10725.562505]  wmi
> [10725.562505] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10725.562506] 
> [10725.562507]  ffff88046f223dc8
> [10725.562507] Call Trace:
> [10725.562508]  usbhid
> [10725.562509]  sr_mod
> [10725.562510] hardirqs last disabled at (11251127): 
> [10725.562511]  tg3
> [10725.562512]  0000000000000246
> [10725.562512]  <IRQ> 
> [10725.562513]  0000000100300030
> [10725.562514]  dm_multipath
> [10725.562516]  cdrom
> [10725.562516] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10725.562520]  ehci_hcd
> [10725.562520]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10725.562524]  ffff8802eb1bf238
> [10725.562524]  [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30
> [10725.562525] 
> [10725.562526]  usbcore
> [10725.562527] softirqs last  enabled at (8589138): 
> [10725.562529]  ffff8802eb1bf238
> [10725.562530]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10725.562533]  ffff00010120b718
> [10725.562533]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562536]  0000000000000292
> [10725.562536] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10725.562537]  usb_common
> [10725.562540] softirqs last disabled at (8589495): 
> [10725.562540]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562541]  dm_mod
> [10725.562544]  mgag200
> [10725.562544]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10725.562545]  ffff88040120b718
> [10725.562547]  i2c_algo_bit drm_kms_helper
> [10725.562547] 
> [10725.562549]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10725.562551]  iTCO_wdt
> [10725.562552]  sg
> [10725.562552] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562553] 
> [10725.562556]  aes_x86_64
> [10725.562556] CPU: 2 PID: 7938 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10725.562557]  hid_generic
> [10725.562558]  syscopyarea
> [10725.562559]  lrw
> [10725.562559] irq event stamp: 11005237
> [10725.562560] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10725.562561]  sysfillrect
> [10725.562562]  ipmi_ssif
> [10725.562563]  sysimgblt
> [10725.562564]  usbhid
> [10725.562564]  fb_sys_fops
> [10725.562565] Call Trace:
> [10725.562567]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10725.562569] hardirqs last  enabled at (11005236): 
> [10725.562569] task: ffff880400708000 ti: ffff8803f7af8000 task.ti: ffff8803f7af8000
> [10725.562571] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10725.562573] RIP: 0010:[<ffffffff815781b3>] 
> [10725.562574] hardirqs last disabled at (11005237): 
> [10725.562575]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10725.562576] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10725.562578]  mgag200
> [10725.562581]  ipmi_devintf
> [10725.562581]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10725.562583]  <IRQ> 
> [10725.562584]  ttm
> [10725.562584] RSP: 0018:ffff88046f243d78  EFLAGS: 00000292
> [10725.562586]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10725.562589]  drm
> [10725.562589]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10725.562591]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10725.562592]  ptp
> [10725.562593]  i2c_algo_bit
> [10725.562594] RAX: ffff880400708000 RBX: 0000000000000292 RCX: 0000000000000000
> [10725.562595] softirqs last  enabled at (8447496): 
> [10725.562596]  ehci_pci
> [10725.562598]  drm_kms_helper
> [10725.562598]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10725.562600]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562602]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562605]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10725.562607]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562609]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10725.562611]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10725.562613]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10725.562614] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10725.562615] RDX: 0000000000000003 RSI: ffff880400708798 RDI: ffff880400708000
> [10725.562617] softirqs last disabled at (8447895): 
> [10725.562617] RBP: ffff88046f243d88 R08: ffff88046f243cf8 R09: 0000000000000000
> [10725.562619] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562621]  syscopyarea
> [10725.562623]  iTCO_vendor_support
> [10725.562623] CPU: 8 PID: 7940 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10725.562624] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10725.562625] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10725.562626] R13: ffffea000beaca00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10725.562628]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10725.562631]  <EOI> 
> [10725.562631]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10725.562633] FS:  00007f8a99e31700(0000) GS:ffff88046f240000(0000) knlGS:0000000000000000
> [10725.562634] task: ffff8803fabddf00 ti: ffff8803f753c000 task.ti: ffff8803f753c000
> [10725.562635] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10725.562637] RIP: 0010:[<ffffffff815781b3>] 
> [10725.562637] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10725.562639]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10725.562640]  gf128mul
> [10725.562641]  sr_mod
> [10725.562642] RSP: 0018:ffff88046f303d78  EFLAGS: 00000292
> [10725.562642] Stack:
> [10725.562644]  sysfillrect
> [10725.562645]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10725.562648]  [<ffffffff81286284>] ? blk_flush_plug_list+0x224/0x240
> [10725.562649]  ffff88017fc01c00
> [10725.562649] RAX: ffff8803fabddf00 RBX: 0000000000000292 RCX: 0000000000000000
> [10725.562650]  0000000000000001
> [10725.562651]  sysimgblt
> [10725.562652]  ffff88046f243e38
> [10725.562653] RDX: 0000000000000003 RSI: ffff8803fabde698 RDI: ffff8803fabddf00
> [10725.562654]  ffffffff811640d6
> [10725.562654] RBP: ffff88046f303d88 R08: ffff88046f303cf8 R09: 0000000000000000
> [10725.562657]  [<ffffffff81286491>] blk_queue_bio+0x1f1/0x300
> [10725.562658]  cdrom
> [10725.562658] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10725.562659] 
> [10725.562660] R13: ffffea0009b52400 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10725.562662]  ffff88046f243dc8
> [10725.562662] FS:  00007f8a96e2b700(0000) GS:ffff88046f300000(0000) knlGS:0000000000000000
> [10725.562664]  glue_helper
> [10725.562664] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10725.562665]  ffff88017fc07600
> [10725.562666] CR2: 00007fe88affadd0 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10725.562669]  fb_sys_fops
> [10725.562669]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10725.562671]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10725.562673]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10725.562674]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10725.562676]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10725.562679]  mlx4_core
> [10725.562679]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562682]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10725.562684]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10725.562685]  ehci_hcd
> [10725.562686]  0000000100300030
> [10725.562687]  usbcore
> [10725.562689]  ffff8802fab2b520
> [10725.562689]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10725.562690]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10725.562692]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10725.562694]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10725.562694] 
> [10725.562695] Stack:
> [10725.562696]  dcdbas
> [10725.562698]  ttm
> [10725.562699]  usb_common
> [10725.562699]  drm
> [10725.562700]  ablk_helper
> [10725.562701]  ehci_pci
> [10725.562702]  sg
> [10725.562702]  sr_mod
> [10725.562703]  pps_core
> [10725.562704]  ffff88017fc01c00
> [10725.562704] 
> [10725.562707]  <EOI> 
> [10725.562707]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10725.562708]  ffff8802fab2b520
> [10725.562709]  0000000000000001
> [10725.562710]  ffff000100708798
> [10725.562711]  cryptd
> [10725.562714]  cdrom
> [10725.562714]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10725.562715]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10725.562717]  ffff88046f303e38
> [10725.562717]  0000000000000292
> [10725.562718]  ffffffff811640d6
> [10725.562721]  libphy
> [10725.562721]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10725.562725]  [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
> [10725.562727]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10725.562729]  [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
> [10725.562731]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10725.562736]  [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
> [10725.562739]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10725.562739] 
> [10725.562741]  ffff880400708798
> [10725.562741]  ehci_hcd
> [10725.562742]  ffff88046f303dc8
> [10725.562742] 
> [10725.562744]  usbcore
> [10725.562745]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10725.562749]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10725.562751]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10725.562754]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10725.562755]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10725.562756] Call Trace:
> [10725.562757]  0000000000000246
> [10725.562758]  usb_common
> [10725.562759]  ffffffff811859b3
> [10725.562759] irq event stamp: 10703453
> [10725.562761]  ffff88026d492a80
> [10725.562762]  <IRQ> 
> [10725.562764]  fjes
> [10725.562764]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10725.562766]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10725.562767] Code: 
> [10725.562769] hardirqs last  enabled at (10703452): 
> [10725.562769]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10725.562773]  sg
> [10725.562773]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10725.562777] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10725.562777] 
> [10725.562779]  8250_fintek
> [10725.562781] fc 
> [10725.562781]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10725.562783] hardirqs last disabled at (10703453): 
> [10725.562784]  ffff88026d492a80
> [10725.562785] 48 
> [10725.562786]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10725.562788] 83 
> [10725.562788]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10725.562789] c7 
> [10725.562792]  sb_edac
> [10725.562792] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10725.562792] 
> [10725.562794] Code: 
> [10725.562796] 18 
> [10725.562796]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10725.562797]  ffff0001fabde698
> [10725.562798] 48 
> [10725.562801]  0000000000000292
> [10725.562801]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562802] irq event stamp: 9765723
> [10725.562803]  pcspkr
> [10725.562806]  ffff8803fabde698
> [10725.562806]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562807]  ipmi_si
> [10725.562807] 
> [10725.562809]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10725.562810] Call Trace:
> [10725.562812]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10725.562815]  <IRQ> 
> [10725.562815]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10725.562817]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10725.562818] 8b 
> [10725.562819] hardirqs last  enabled at (9765722): 
> [10725.562820]  edac_core
> [10725.562820] 55 
> [10725.562821] fc 
> [10725.562823]  ipmi_msghandler
> [10725.562824]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10725.562827]  acpi_power_meter
> [10725.562827]  [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30
> [10725.562828] 08 
> [10725.562829] 48 
> [10725.562830] 53 
> [10725.562830] 83 
> [10725.562831] 48 
> [10725.562834] c7 
> [10725.562834] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10725.562835] 89 
> [10725.562836] 18 
> [10725.562837]  tpm_tis
> [10725.562838] hardirqs last disabled at (9765723): 
> [10725.562839]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10725.562841]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10725.562842] 48 
> [10725.562843] f3 
> [10725.562843] 8b 
> [10725.562846] be 
> [10725.562846]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562848]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10725.562850]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562852] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10725.562853] softirqs last  enabled at (8080444): 
> [10725.562856] softirqs last  enabled at (8596058): 
> [10725.562856] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10725.562858] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10725.562859] softirqs last disabled at (8080721): 
> [10725.562860] softirqs last disabled at (8596459): 
> [10725.562862]  tpm
> [10725.562862] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562863] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562866] CPU: 6 PID: 7930 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10725.562868] CPU: 4 PID: 7935 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10725.562869] 55 
> [10725.562869] 01 
> [10725.562870] 08 
> [10725.562871] 00 
> [10725.562871] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10725.562872] 53 
> [10725.562873]  lpc_ich
> [10725.562874] 48 
> [10725.562874] task: ffff8803fbdd8000 ti: ffff8803f7524000 task.ti: ffff8803f7524000
> [10725.562875] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10725.562878] RIP: 0010:[<ffffffff815781b3>] 
> [10725.562878] task: ffff880064ae4740 ti: ffff8803f031c000 task.ti: ffff8803f031c000
> [10725.562880]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10725.562882] 89 
> [10725.562882]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10725.562884] 00 
> [10725.562884] RSP: 0018:ffff88046f283d78  EFLAGS: 00000292
> [10725.562886] RIP: 0010:[<ffffffff815781b3>] 
> [10725.562887] 00 
> [10725.562887] f3 
> [10725.562888] e8 
> [10725.562888] be 
> [10725.562889] e1 
> [10725.562889] 01 
> [10725.562890] ce 
> [10725.562891] RAX: ffff8803fbdd8000 RBX: 0000000000000292 RCX: 0000000000000000
> [10725.562892] 00 
> [10725.562892] b2 
> [10725.562893] 00 
> [10725.562894]  button
> [10725.562895] RDX: 0000000000000003 RSI: ffff8803fbdd8798 RDI: ffff8803fbdd8000
> [10725.562896] 00 
> [10725.562896] RBP: ffff88046f283d88 R08: ffff88046f283cf8 R09: 0000000000000000
> [10725.562899]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.562900]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10725.562903]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10725.562905]  shpchp
> [10725.562905]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10725.562906] e8 
> [10725.562908] ff 
> [10725.562908]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10725.562910]  <EOI> 
> [10725.562911] RSP: 0018:ffff88046f2c3d78  EFLAGS: 00000292
> [10725.562912] e1 
> [10725.562912] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10725.562914] RAX: ffff880064ae4740 RBX: 0000000000000292 RCX: 0000000000000000
> [10725.562915] 4c 
> [10725.562915] ce 
> [10725.562916] 89 
> [10725.562916] b2 
> [10725.562917] e7 
> [10725.562918] ff 
> [10725.562920] e8 
> [10725.562920]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10725.562922]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10725.562923] R13: ffffea000e1e9e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10725.562925] RDX: 0000000000000003 RSI: ffff880064ae4ed8 RDI: ffff880064ae4740
> [10725.562926] 4c 
> [10725.562927] 49 
> [10725.562928] 89 
> [10725.562928] FS:  00007f8a9b634700(0000) GS:ffff88046f280000(0000) knlGS:0000000000000000
> [10725.562932]  mei_me
> [10725.562932]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10725.562933] e7 
> [10725.562934] 00 
> [10725.562934] e8 
> [10725.562936] b3 
> [10725.562937]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10725.562939]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10725.562941]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10725.562943]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10725.562947]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10725.562948]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10725.562951]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10725.562952]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10725.562955]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10725.562956] RBP: ffff88046f2c3d88 R08: ffff88046f2c3cf8 R09: 0000000000000000
> [10725.562957] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10725.562959]  mei
> [10725.562959] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10725.562960] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10725.562961] Stack:
> [10725.562963]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10725.562965]  mfd_core
> [10725.562965] R13: ffffea0009dcea00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10725.562966] ff 
> [10725.562967] 49 
> [10725.562967] f6 
> [10725.562968] 00 
> [10725.562970]  wmi
> [10725.562970]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.562972]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10725.562973]  ffff88017fc01c00
> [10725.562974] b3 
> [10725.562976]  dm_multipath
> [10725.562976] FS:  00007f8a9862e700(0000) GS:ffff88046f2c0000(0000) knlGS:0000000000000000
> [10725.562979]  0000000000000001
> [10725.562980]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10725.562981] c7 
> [10725.562983] ff 
> [10725.562983]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10725.562984]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10725.562986]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10725.562988]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10725.562989]  ffff88046f283e38
> [10725.562990]  dm_mod
> [10725.562990] f6 
> [10725.562991] 02 
> [10725.562991] c7 
> [10725.562992] 74 
> [10725.562993] 02 
> [10725.562993] 13 
> [10725.562994]  ffffffff811640d6
> [10725.562995] e8 
> [10725.562997] 74 
> [10725.562997]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10725.562999]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10725.563001]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10725.563004]  <EOI> 
> [10725.563004]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10725.563006]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10725.563007] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10725.563007] 
> [10725.563009] CR2: 00005633132c3fe8 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10725.563012]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10725.563013] 13 
> [10725.563013] 0f 
> [10725.563014]  ffff88046f283dc8
> [10725.563015] a8 
> [10725.563015]  0000000000000246
> [10725.563016]  hid_generic
> [10725.563016] Stack:
> [10725.563018]  ffffffff811859b3
> [10725.563018] e8 
> [10725.563019] b2 
> [10725.563019] 0f 
> [10725.563020] ff 
> [10725.563021]  usbhid
> [10725.563022]  ffff880387a78d48
> [10725.563022]  mgag200
> [10725.563023] 53 
> [10725.563023] a8 
> [10725.563024] 9d 
> [10725.563025] b2 
> [10725.563026] <65> 
> [10725.563027]  i2c_algo_bit
> [10725.563029]  ffff88017fc01c00
> [10725.563029]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10725.563030] ff 
> [10725.563031] ff 
> [10725.563031] 53 
> [10725.563033] 0d 
> [10725.563033]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10725.563034]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10725.563036]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10725.563039]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10725.563041]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10725.563043]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10725.563045]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10725.563047]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10725.563049]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10725.563051]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10725.563054]  0000000000000001
> [10725.563054]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10725.563056]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10725.563056] 
> [10725.563057] 56 
> [10725.563058]  ffff880387a78d48
> [10725.563060]  ffff88046f2c3e38
> [10725.563060]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10725.563062]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10725.563063] Code: 
> [10725.563064]  ffffffff811640d6
> [10725.563065]  drm_kms_helper
> [10725.563067] fc 
> [10725.563067]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10725.563068] 48 
> [10725.563069]  syscopyarea
> [10725.563069] 
> [10725.563070] 2b 
> [10725.563070] 9d 
> [10725.563071] a9 
> [10725.563073] <65> 
> [10725.563073]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10725.563075] 83 
> [10725.563075]  ffff88046f2c3dc8
> [10725.563076]  sysfillrect
> [10725.563077] ff 
> [10725.563077] 7e 
> [10725.563078] 0d 
> [10725.563079]  ffff0001fbdd8798
> [10725.563079] 56 
> [10725.563081]  0000000000000292
> [10725.563081]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10725.563084] c7 
> [10725.563084]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10725.563087] 18 
> [10725.563087]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10725.563089] 48 
> [10725.563089]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10725.563091] 8b 
> [10725.563091]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10725.563093] 5b 
> [10725.563094]  sysimgblt
> [10725.563096]  0000000000000246 0000000100300030
> [10725.563096] 41 
> [10725.563097] 2b 
> [10725.563098] 5c 
> [10725.563098] a9 
> [10725.563099] 5d 
> [10725.563101]  ffff8802773ab7c8
> [10725.563101]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10725.563102] 
> [10725.563103]  fb_sys_fops
> [10725.563104] 55 
> [10725.563105]  ffff8802773ab7c8
> [10725.563105]  ttm
> [10725.563106]  ffff000164ae4ed8
> [10725.563107]  ffff8803fbdd8798
> [10725.563108] 08 
> [10725.563109]  0000000000000292
> [10725.563109]  drm
> [10725.563109] 
> [10725.563111]  ffff880064ae4ed8
> [10725.563111] Call Trace:
> [10725.563112]  ehci_pci
> [10725.563113]  <IRQ> 
> [10725.563115] 53 
> [10725.563115]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10725.563117]  sr_mod
> [10725.563117]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10725.563118] 
> [10725.563119] c3 
> [10725.563120] Call Trace:
> [10725.563121]  cdrom
> [10725.563121] 53 
> [10725.563122] 7e 
> [10725.563125] 9d 
> [10725.563125]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10725.563126] 5b 
> [10725.563126] e8 
> [10725.563127] 41 
> [10725.563127] 6a 
> [10725.563128] 5c 
> [10725.563128] 79 
> [10725.563131] 5d 
> [10725.563131]  [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30
> [10725.563133]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10725.563135]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10725.563137]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10725.563138] 48 
> [10725.563139] c3 
> [10725.563140] b2 
> [10725.563140] 53 
> [10725.563141] ff 
> [10725.563141] 89 
> [10725.563142] Code: 
> [10725.563143] f3 
> [10725.563145] fc 
> [10725.563145]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.563147]  ehci_hcd
> [10725.563148] 48 
> [10725.563148]  <IRQ> 
> [10725.563149] 9d 
> [10725.563149] eb 
> [10725.563150] e8 
> [10725.563151] eb 
> [10725.563151] 6a 
> [10725.563151] 
> [10725.563154] 79 
> [10725.563154]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.563155]  usbcore
> [10725.563156] 83 
> [10725.563156] be 
> [10725.563157] c7 
> [10725.563158] b2 
> [10725.563161]  usb_common
> [10725.563161]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10725.563162] 18 
> [10725.563163]  sg
> [10725.563165] ff 
> [10725.563165]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10725.563166] 48 
> [10725.563167] 01 
> [10725.563167] 8b 
> [10725.563168] 00 
> [10725.563168] 55 
> [10725.563169] 00 
> [10725.563172] 08 
> [10725.563172]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10725.563175]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10725.563177]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.563178] 
> [10725.563179] 00 
> [10725.563180] 53 
> [10725.563181] e8 
> [10725.563181] irq event stamp: 11920625
> [10725.563182] eb 
> [10725.563183] hardirqs last  enabled at (11920624): 
> [10725.563184] 48 
> [10725.563186] e1 
> [10725.563186]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.563187] eb 
> [10725.563188] 89 
> [10725.563189] ce 
> [10725.563189] 
> [10725.563190] b2 
> [10725.563190] f3 
> [10725.563191] ff 
> [10725.563192] be 
> [10725.563192] 4c 
> [10725.563193] 01 
> [10725.563193] 89 
> [10725.563194] 00 
> [10725.563194] e7 
> [10725.563195] 00 
> [10725.563195] e8 
> [10725.563196] 00 
> [10725.563197] 49 
> [10725.563197] e8 
> [10725.563198] 00 
> [10725.563198] e1 
> [10725.563199] b3 
> [10725.563201] ce 
> [10725.563201]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10725.563204] b2 
> [10725.563204]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10725.563207] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10725.563210]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10725.563212]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10725.563214]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10725.563215] ff 
> [10725.563216] ff 
> [10725.563217] f6 
> [10725.563219] hardirqs last disabled at (11920625): 
> [10725.563219]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10725.563220] c7 
> [10725.563222] 4c 
> [10725.563222]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10725.563224]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10725.563226]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10725.563227] 02 
> [10725.563227] 89 
> [10725.563228] 74 
> [10725.563230] e7 
> [10725.563230] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10725.563232]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.563233] e8 
> [10725.563235] 13 
> [10725.563235]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10725.563237] softirqs last  enabled at (10133236): 
> [10725.563237]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10725.563240] e8 
> [10725.563240]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10725.563241] 49 
> [10725.563243] 0f 
> [10725.563243]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10725.563244] 00 
> [10725.563245] a8 
> [10725.563245] b3 
> [10725.563246]  <EOI> 
> [10725.563247] ff 
> [10725.563247] b2 
> [10725.563249] f6 
> [10725.563249]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10725.563251] c7 
> [10725.563251]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.563280] [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst]
> [10725.563281] 02 
> [10725.563281] ff 
> [10725.563282] 74 
> [10725.563283] 53 
> [10725.563283] 13 
> [10725.563284] 9d 
> [10725.563284] e8 
> [10725.563287] <65> 
> [10725.563287]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10725.563289]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10725.563291] softirqs last disabled at (10133267): 
> [10725.563292]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10725.563293] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.563296] 0f 
> [10725.563296]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10725.563298]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10725.563299] ff 
> [10725.563300] a8 
> [10725.563302] 0d 
> [10725.563302]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10725.563305]  <EOI> 
> [10725.563305]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10725.563307] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G           O L  4.4.6-debug+ #2
> [10725.563308] 56 
> [10725.563309] b2 
> [10725.563311] 2b 
> [10725.563311]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10725.563312]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10725.563315]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10725.563317]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10725.563318] ff 
> [10725.563319] a9 
> [10725.563319] 53 
> [10725.563320] 7e 
> [10725.563322] 9d 
> [10725.563322]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10725.563323] <65> 
> [10725.563323] 5b 
> [10725.563324] ff 
> [10725.563324] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10725.563326]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10725.563327] 41 
> [10725.563328] 0d 
> [10725.563328] 5c 
> [10725.563329] 56 
> [10725.563329] 5d 
> [10725.563330] 2b 
> [10725.563331] c3 
> [10725.563331] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000
> [10725.563332] 53 
> [10725.563333] a9 
> [10725.563335] 9d 
> [10725.563335]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10725.563337] RIP: 0010:[<ffffffff815781b3>] 
> [10725.563339] 7e 
> [10725.563339]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10725.563341]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10725.563342] RSP: 0018:ffff88046f363d68  EFLAGS: 00000292
> [10725.563344]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10725.563345] 5b 
> [10725.563348] e8 
> [10725.563348]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10725.563349]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10725.563350] 41 
> [10725.563351] 6a 
> [10725.563353] 5c 
> [10725.563353]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10725.563355] 5d 
> [10725.563355]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10725.563357] c3 
> [10725.563359] 79 
> [10725.563359]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10725.563361]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10725.563364]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10725.563365] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000
> [10725.563367]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10725.563368] 53 
> [10725.563368] b2 
> [10725.563369] 9d 
> [10725.563369] ff 
> [10725.563370] e8 
> [10725.563371] eb 
> [10725.563373] 6a 
> [10725.563373]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10725.563374] eb 
> [10725.563374] 79 
> [10725.563375] 
> [10725.563377] b2 
> [10725.563377]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10725.563379]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10725.563381]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10725.563382] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00
> [10725.563384]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10725.563386]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10725.563388]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10725.563390]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10725.563391] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000
> [10725.563393]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10725.563394]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10725.563396]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10725.563399] ff 
> [10725.563399]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10725.563400] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10725.563401] eb 
> [10725.563403] Code: 
> [10725.563403]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10725.563405] fc 
> [10725.563405] R13: ffffea000f204e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10725.563408]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10725.563409] FS:  0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000
> [10725.563410] 48 
> [10725.563411] eb 
> [10725.563413] 83 
> [10725.563413]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10725.563415] c7 
> [10725.563415] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10725.563417]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10725.563419] 18 
> [10725.563419]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10725.563420] 48 
> [10725.563421] 
> [10725.563422] CR2: 00005633130dd1a8 CR3: 0000000401468000 CR4: 00000000001406e0
> [10725.563424] 8b 
> [10725.563424]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10725.563425] 55 
> [10725.563425] Stack:
> [10725.563427] 08 
> [10725.563427]  ffff88017fc01c00
> [10725.563429] 53 
> [10725.563429]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10725.563430] 48 
> [10725.563431]  0000000000000001
> [10725.563433] 89 
> [10725.563433]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10725.563434] f3 
> [10725.563435]  ffff88046f363e30
> [10725.563437] be 
> [10725.563437]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10725.563438] 01 
> [10725.563440]  ffffffff811640d6
> [10725.563440]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10725.563441] 00 
> [10725.563441] 
> [10725.563442] 00 
> [10725.563443] Code: 
> [10725.563444] 00 
> [10725.563445] fc 
> [10725.563445]  ffff88046f363db8
> [10725.563446] e8 
> [10725.563447] 48 
> [10725.563447] e1 
> [10725.563448]  ffff88016d2e0000
> [10725.563449] 83 
> [10725.563449] ce 
> [10725.563450] c7 
> [10725.563451] b2 
> [10725.563451]  0000000000000000
> [10725.563452] ff 
> [10725.563452] 18 
> [10725.563453] 4c 
> [10725.563454] 48 
> [10725.563454] 89 
> [10725.563455] 8b 
> [10725.563456]  ffff8803c813ef90
> [10725.563456] e7 
> [10725.563457] 55 
> [10725.563457] e8 
> [10725.563458] 08 
> [10725.563459] 49 
> [10725.563459] 53 
> [10725.563460] 00 
> [10725.563460] 48 
> [10725.563461] 
> [10725.563462] b3 
> [10725.563462]  ffff8803c813ef90
> [10725.563463] ff 
> [10725.563464] 89 
> [10725.563464] f6 
> [10725.563465] f3 
> [10725.563466] c7 
> [10725.563466]  000000018157940f
> [10725.563467] be 
> [10725.563467]  0000000000000292
> [10725.563468] 02 
> [10725.563469] 01 
> [10725.563469] 74 
> [10725.563470]  ffff880439376698
> [10725.563470] 13 
> [10725.563471] 
> [10725.563472] 00 
> [10725.563472] e8 
> [10725.563472] Call Trace:
> [10725.563473] 0f 
> [10725.563474] 00 
> [10725.563475] a8 
> [10725.563475]  <IRQ> 
> [10725.563478] b2 
> [10725.563478]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10725.563479] ff 
> [10725.563480] 00 
> [10725.563482] 53 
> [10725.563482]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10725.563485] 9d 
> [10725.563485]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.563488] e8 
> [10725.563488]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10725.563489] e1 
> [10725.563490] <65> 
> [10725.563492] ce 
> [10725.563492]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10725.563494] b2 
> [10725.563494] ff 
> [10725.563497] ff 
> [10725.563497]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10725.563498] 4c 
> [10725.563500] 0d 
> [10725.563500]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10725.563503] 56 
> [10725.563503]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10725.563505] 89 
> [10725.563505]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10725.563506] 2b 
> [10725.563508] e7 
> [10725.563508]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10725.563510] a9 
> [10725.563510]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10725.563511] 7e 
> [10725.563512] e8 
> [10725.563512] 5b 
> [10725.563515] 49 
> [10725.563515]  [<ffffffff810065a5>] do_IRQ+0x65/0x110
> [10725.563518] 41 
> [10725.563518]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10725.563519] 5c 
> [10725.563521] 00 
> [10725.563521]  [<ffffffff815793c9>] common_interrupt+0x89/0x89
> [10725.563523] b3 
> [10725.563523]  <EOI> 
> [10725.563524] 5d 
> [10725.563526] ff 
> [10725.563526]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10725.563527] f6 
> [10725.563530] c3 
> [10725.563530]  [<ffffffff8117bad7>] ? put_object+0x7/0x40
> [10725.563531] c7 
> [10725.563533] 53 
> [10725.563533]  [<ffffffff8117bb48>] ? __delete_object+0x38/0x70
> [10725.563536] 9d 
> [10725.563536]  [<ffffffff8117bb98>] delete_object_full+0x18/0x20
> [10725.563537] e8 
> [10725.563540] 02 
> [10725.563540]  [<ffffffff8156dab1>] kmemleak_free+0x31/0x80
> [10725.563541] 74 
> [10725.563542] 6a 
> [10725.563544] 13 
> [10725.563544]  [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160
> [10725.563545] 79 
> [10725.563546] e8 
> [10725.563548] b2 
> [10725.563548]  [<ffffffff81113282>] mempool_free_slab+0x12/0x20
> [10725.563550] 0f 
> [10725.563552] ff 
> [10725.563552]  [<ffffffff81113406>] mempool_free+0x26/0x80
> [10725.563553] eb 
> [10725.563555] a8 
> [10725.563555]  [<ffffffff8127e8c1>] bvec_free+0x31/0x40
> [10725.563556] eb 
> [10725.563556] b2 
> [10725.563557] 
> [10725.563559] ff 
> [10725.563559]  [<ffffffff8127ee54>] bio_free+0x34/0x60
> [10725.563561] 53 
> [10725.563561]  [<ffffffff8127ee9e>] bio_put+0x1e/0x30
> [10725.563568] 9d 
> [10725.563568]  [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk]
> [10725.563570] <65> 
> [10725.563570]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10725.563573] ff 
> [10725.563573]  [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd]
> [10725.563576] 0d 
> [10725.563576]  [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630
> [10725.563579] 56 
> [10725.563579]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10725.563581] 2b 
> [10725.563581]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10725.563587] a9 7e 
> [10725.563587]  [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk]
> [10725.563593] 5b 41 5c 
> [10725.563593]  [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk]
> [10725.563598] 5d 
> [10725.563598]  [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk]
> [10725.563602] c3 
> [10725.563602]  [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk]
> [10725.563607] 53 9d 
> [10725.563607]  [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk]
> [10725.563610] e8 
> [10725.563610]  [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0
> [10725.563632] 6a 
> [10725.563632]  [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst]
> [10725.563649] 79 b2 ff 
> [10725.563649]  [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst]
> [10725.563664] eb 
> [10725.563664]  [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst]
> [10725.563679] eb 
> [10725.563679]  [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst]
> [10725.563679] 
> [10725.563694]  [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst]
> [10725.563709]  [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst]
> [10725.563711]  [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220
> [10725.563713]  [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220
> [10725.563728]  [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst]
> [10725.563730]  [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10
> [10725.563744]  [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst]
> [10725.563759]  [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst]
> [10725.563762]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10725.563776]  [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst]
> [10725.563779]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10725.563782]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10725.563784]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10725.563785]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10725.563814] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 
> [10726.319704] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10726.380711] irq event stamp: 16370765
> [10726.381076] hardirqs last  enabled at (16370764): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10726.382350] hardirqs last disabled at (16370765): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10726.383582] softirqs last  enabled at (13717672): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10726.384329] softirqs last disabled at (13717773): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10726.384996] CPU: 0 PID: 7937 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10726.385483] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10726.385685] task: ffff88040070df00 ti: ffff8803f1678000 task.ti: ffff8803f1678000
> [10726.386036] RIP: 0010:[<ffffffff815781b3>]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10726.386575] RSP: 0018:ffff88046f203d78  EFLAGS: 00000292
> [10726.387228] RAX: ffff88040070df00 RBX: 0000000000000292 RCX: 0000000000000000
> [10726.387689] RDX: 0000000000000003 RSI: ffff88040070e698 RDI: ffff88040070df00
> [10726.388315] RBP: ffff88046f203d88 R08: ffff88046f203cf8 R09: 0000000000000000
> [10726.388558] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10726.389279] R13: ffffea000bef5800 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10726.389728] FS:  00007f8a9a632700(0000) GS:ffff88046f200000(0000) knlGS:0000000000000000
> [10726.390012] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10726.390373] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406f0
> [10726.390881] Stack:
> [10726.391294]  ffff88017fc01c00 0000000000000001 ffff88046f203e38 ffffffff811640d6
> [10726.393205]  ffff88046f203dc8 0000000000000246 ffffffff811859b3 ffff8802fbd64d08
> [10726.394507]  ffff8802fbd64d08 ffff00010070e698 0000000000000292 ffff88040070e698
> [10726.397125] Call Trace:
> [10726.397494]  <IRQ> 
> [10726.397750]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10726.398542]  [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30
> [10726.398908]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10726.399384]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10726.399674]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10726.399952]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10726.400402]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10726.401001]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10726.401417]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10726.402363]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10726.402673]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10726.403097]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10726.403658]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10726.403861]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10726.404301]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10726.404600]  <EOI> 
> [10726.405112]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10726.405768]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10726.405980]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10726.407030]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10726.407348]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10726.407859]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10726.408537]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10726.409417]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10726.409897]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10726.410140]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10726.410347]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10726.410703]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10726.411311]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10726.412019]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10726.412290]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10726.412748]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10726.413086]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10726.413349]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10726.413782]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10726.414050]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10726.414463]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10726.414818]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10726.415555] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 
> [10753.561101] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753]
> [10753.561110] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [fio:7935]
> [10753.561118] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs
> [10753.561119] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [fio:7938]
> [10753.561120]  dm_queue_length
> [10753.561121] Modules linked in:
> [10753.561121]  brd
> [10753.561122]  ib_srpt(O-)
> [10753.561123]  xt_CHECKSUM
> [10753.561125]  scst_vdisk(O)
> [10753.561126] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kdmwork-254:2:358]
> [10753.561127]  libcrc32c
> [10753.561128]  iptable_mangle
> [10753.561128]  scst(O)
> [10753.561129]  ipt_MASQUERADE
> [10753.561130]  dlm
> [10753.561130] Modules linked in:
> [10753.561131]  netconsole
> [10753.561132]  ib_srpt(O-)
> [10753.561133]  nf_nat_masquerade_ipv4
> [10753.561133]  configfs
> [10753.561134]  iptable_nat
> [10753.561135]  scst_vdisk(O)
> [10753.561136]  nf_nat_ipv4
> [10753.561136]  libcrc32c
> [10753.561137]  nf_nat
> [10753.561138]  scst(O)
> [10753.561138]  nf_conntrack_ipv4
> [10753.561141]  dlm
> [10753.561141] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [fio:7937]
> [10753.561142]  nf_defrag_ipv4
> [10753.561143]  netconsole
> [10753.561143]  dm_queue_length
> [10753.561144] Modules linked in:
> [10753.561145]  xt_conntrack
> [10753.561146]  ib_srpt(O-)
> [10753.561147]  brd
> [10753.561147]  scst_vdisk(O)
> [10753.561148]  nf_conntrack
> [10753.561149]  libcrc32c
> [10753.561150]  xt_CHECKSUM
> [10753.561150]  configfs
> [10753.561151]  ipt_REJECT
> [10753.561152]  scst(O)
> [10753.561152]  xt_tcpudp
> [10753.561153]  dlm
> [10753.561154]  tun
> [10753.561155]  netconsole
> [10753.561155]  dm_queue_length
> [10753.561156]  ebtable_filter
> [10753.561157]  brd
> [10753.561157]  ebtables
> [10753.561158]  iptable_mangle
> [10753.561159]  ip6table_filter
> [10753.561160]  configfs
> [10753.561160]  ipt_MASQUERADE
> [10753.561161]  dm_queue_length
> [10753.561162]  ip6_tables
> [10753.561163]  nf_nat_masquerade_ipv4
> [10753.561163]  iptable_filter
> [10753.561164]  xt_CHECKSUM
> [10753.561165]  ip_tables
> [10753.561165]  brd
> [10753.561166]  iptable_nat
> [10753.561167]  iptable_mangle
> [10753.561168]  nf_nat_ipv4
> [10753.561168]  xt_CHECKSUM
> [10753.561169]  ipt_MASQUERADE
> [10753.561170]  iptable_mangle
> [10753.561171]  x_tables
> [10753.561171]  ipt_MASQUERADE
> [10753.561172]  af_packet
> [10753.561173]  nf_nat_masquerade_ipv4
> [10753.561174]  nf_nat_masquerade_ipv4
> [10753.561174]  nf_nat
> [10753.561175]  iptable_nat
> [10753.561176]  iptable_nat
> [10753.561177]  bridge
> [10753.561177]  nf_nat_ipv4
> [10753.561178]  nf_nat_ipv4
> [10753.561179]  nf_nat
> [10753.561179]  nf_nat
> [10753.561180]  stp
> [10753.561181]  nf_conntrack_ipv4
> [10753.561181]  llc
> [10753.561182]  nf_defrag_ipv4
> [10753.561183]  nf_conntrack_ipv4
> [10753.561184]  iscsi_ibft
> [10753.561184]  nf_defrag_ipv4
> [10753.561185]  iscsi_boot_sysfs
> [10753.561186]  nf_conntrack_ipv4
> [10753.561187]  xt_conntrack
> [10753.561187]  xt_conntrack
> [10753.561188]  nf_defrag_ipv4
> [10753.561189]  ib_srp
> [10753.561190]  nf_conntrack
> [10753.561190]  xt_conntrack
> [10753.561191]  scsi_transport_srp
> [10753.561192]  nf_conntrack
> [10753.561192]  ipt_REJECT
> [10753.561193]  ib_ipoib
> [10753.561194]  nf_conntrack
> [10753.561195]  rdma_ucm
> [10753.561195]  xt_tcpudp
> [10753.561196]  ipt_REJECT
> [10753.561197]  ib_ucm
> [10753.561197]  tun
> [10753.561198]  ipt_REJECT
> [10753.561199]  xt_tcpudp
> [10753.561199]  ebtable_filter
> [10753.561200]  xt_tcpudp
> [10753.561201]  ib_uverbs
> [10753.561201]  tun
> [10753.561202]  ebtables
> [10753.561203]  ebtable_filter
> [10753.561204]  ip6table_filter
> [10753.561204]  ebtables
> [10753.561205]  tun
> [10753.561206]  msr
> [10753.561206]  ip6_tables
> [10753.561207]  ebtable_filter
> [10753.561208]  iptable_filter
> [10753.561209]  ebtables
> [10753.561209]  ib_umad
> [10753.561210]  ip_tables
> [10753.561211]  rdma_cm
> [10753.561211]  ip6table_filter
> [10753.561212]  ib_cm
> [10753.561213]  ip6_tables
> [10753.561213]  x_tables
> [10753.561214]  iptable_filter
> [10753.561215]  iw_cm
> [10753.561215]  af_packet
> [10753.561216]  mlx4_ib
> [10753.561217]  bridge
> [10753.561218]  ip6table_filter
> [10753.561218]  stp
> [10753.561220]  ip_tables
> [10753.561221]  ip6_tables
> [10753.561222]  ib_sa
> [10753.561222]  iptable_filter
> [10753.561223]  llc
> [10753.561224]  ib_mad
> [10753.561224]  ip_tables
> [10753.561225]  x_tables
> [10753.561226]  iscsi_ibft
> [10753.561226]  x_tables
> [10753.561227]  iscsi_boot_sysfs
> [10753.561228]  af_packet
> [10753.561229]  af_packet
> [10753.561229]  ib_srp
> [10753.561230]  bridge
> [10753.561231]  ib_core
> [10753.561231]  scsi_transport_srp
> [10753.561232]  ib_addr
> [10753.561233]  stp
> [10753.561234]  x86_pkg_temp_thermal
> [10753.561234]  llc
> [10753.561235]  ib_ipoib
> [10753.561236]  iscsi_ibft
> [10753.561237]  bridge
> [10753.561237]  iscsi_boot_sysfs
> [10753.561238]  intel_powerclamp
> [10753.561239]  ib_srp
> [10753.561239]  stp
> [10753.561240]  rdma_ucm
> [10753.561241]  llc
> [10753.561241]  coretemp
> [10753.561242]  ib_ucm
> [10753.561243]  kvm_intel
> [10753.561243]  iscsi_ibft
> [10753.561244]  ib_uverbs
> [10753.561245]  scsi_transport_srp
> [10753.561246]  msr
> [10753.561246]  iscsi_boot_sysfs
> [10753.561247]  ib_umad
> [10753.561248]  ib_srp
> [10753.561249]  ib_ipoib
> [10753.561249]  kvm
> [10753.561250]  rdma_ucm
> [10753.561251]  rdma_cm
> [10753.561251]  irqbypass
> [10753.561252]  ib_cm
> [10753.561253]  crct10dif_pclmul
> [10753.561253]  scsi_transport_srp
> [10753.561254]  crc32_pclmul
> [10753.561255]  iw_cm
> [10753.561255]  ib_ipoib
> [10753.561256]  crc32c_intel
> [10753.561257]  rdma_ucm
> [10753.561258]  mlx4_ib
> [10753.561258]  aesni_intel
> [10753.561259]  ib_sa
> [10753.561260]  tg3
> [10753.561260]  ib_mad
> [10753.561261]  iTCO_wdt
> [10753.561262]  ib_ucm
> [10753.561263]  ib_core
> [10753.561263]  aes_x86_64
> [10753.561264]  ib_uverbs
> [10753.561265]  lrw
> [10753.561265]  ib_ucm
> [10753.561266]  ib_addr
> [10753.561267]  ipmi_ssif
> [10753.561268]  x86_pkg_temp_thermal
> [10753.561268]  ipmi_devintf
> [10753.561269]  ib_uverbs
> [10753.561270]  ptp
> [10753.561270]  msr
> [10753.561271]  iTCO_vendor_support
> [10753.561272]  msr
> [10753.561272]  intel_powerclamp
> [10753.561273]  gf128mul
> [10753.561274]  coretemp
> [10753.561275]  ib_umad
> [10753.561275]  kvm_intel
> [10753.561276]  glue_helper
> [10753.561277]  rdma_cm
> [10753.561277]  mlx4_core
> [10753.561278]  ib_umad
> [10753.561279]  kvm
> [10753.561279]  dcdbas
> [10753.561280]  ib_cm
> [10753.561281]  ablk_helper
> [10753.561281]  irqbypass
> [10753.561282]  iw_cm
> [10753.561283]  rdma_cm
> [10753.561283]  pps_core
> [10753.561284]  crct10dif_pclmul
> [10753.561285]  cryptd
> [10753.561285]  crc32_pclmul
> [10753.561286]  mlx4_ib
> [10753.561287]  libphy
> [10753.561287]  ib_sa
> [10753.561288]  crc32c_intel
> [10753.561289]  fjes
> [10753.561290]  ib_mad
> [10753.561290]  aesni_intel
> [10753.561291]  ib_core
> [10753.561292]  8250_fintek
> [10753.561292]  tg3
> [10753.561293]  sb_edac
> [10753.561294]  ib_addr
> [10753.561294]  pcspkr
> [10753.561295]  iTCO_wdt
> [10753.561296]  ib_cm
> [10753.561296]  aes_x86_64
> [10753.561297]  iw_cm
> [10753.561298]  lrw
> [10753.561298]  ipmi_si
> [10753.561299]  ipmi_ssif
> [10753.561300]  x86_pkg_temp_thermal
> [10753.561301]  ipmi_devintf
> [10753.561301]  mlx4_ib
> [10753.561302]  edac_core
> [10753.561303]  ptp
> [10753.561303]  ipmi_msghandler
> [10753.561304]  intel_powerclamp
> [10753.561305]  acpi_power_meter
> [10753.561306]  coretemp
> [10753.561306]  tpm_tis
> [10753.561307]  kvm_intel
> [10753.561308]  tpm
> [10753.561308]  ib_sa
> [10753.561309]  kvm
> [10753.561310]  ib_mad
> [10753.561310]  lpc_ich
> [10753.561312]  ib_core
> [10753.561313]  button
> [10753.561313]  irqbypass
> [10753.561314]  ib_addr
> [10753.561315]  iTCO_vendor_support
> [10753.561315]  shpchp
> [10753.561316]  gf128mul
> [10753.561317]  mei_me
> [10753.561318]  x86_pkg_temp_thermal
> [10753.561318]  glue_helper
> [10753.561319]  intel_powerclamp
> [10753.561320]  mei
> [10753.561320]  mlx4_core
> [10753.561321]  mfd_core
> [10753.561322]  coretemp
> [10753.561322]  wmi
> [10753.561323]  dcdbas
> [10753.561324]  kvm_intel
> [10753.561324]  ablk_helper
> [10753.561325]  kvm
> [10753.561326]  dm_multipath
> [10753.561327]  crct10dif_pclmul
> [10753.561327]  dm_mod
> [10753.561328]  pps_core
> [10753.561329]  hid_generic
> [10753.561329]  cryptd
> [10753.561330]  usbhid
> [10753.561331]  libphy
> [10753.561331]  mgag200
> [10753.561332]  crc32_pclmul
> [10753.561333]  i2c_algo_bit
> [10753.561334]  crc32c_intel
> [10753.561334]  fjes
> [10753.561335]  irqbypass
> [10753.561336]  aesni_intel
> [10753.561336]  drm_kms_helper
> [10753.561337]  8250_fintek
> [10753.561338]  syscopyarea
> [10753.561338]  sb_edac
> [10753.561339]  sysfillrect
> [10753.561340]  pcspkr
> [10753.561340]  tg3
> [10753.561341]  crct10dif_pclmul
> [10753.561342]  ipmi_si
> [10753.561343]  iTCO_wdt
> [10753.561343]  edac_core
> [10753.561344]  crc32_pclmul
> [10753.561345]  ipmi_msghandler
> [10753.561346]  sysimgblt
> [10753.561346]  crc32c_intel
> [10753.561347]  fb_sys_fops
> [10753.561348]  aes_x86_64
> [10753.561348]  ttm
> [10753.561349]  acpi_power_meter
> [10753.561350]  drm
> [10753.561350]  tpm_tis
> [10753.561351]  aesni_intel
> [10753.561352]  ehci_pci
> [10753.561352]  tpm
> [10753.561353]  lrw
> [10753.561354]  tg3
> [10753.561354]  sr_mod
> [10753.561355]  lpc_ich
> [10753.561356]  cdrom
> [10753.561356]  iTCO_wdt
> [10753.561357]  button
> [10753.561358]  ehci_hcd
> [10753.561359]  ipmi_ssif
> [10753.561359]  aes_x86_64
> [10753.561360]  shpchp
> [10753.561361]  usbcore
> [10753.561361]  ipmi_devintf
> [10753.561362]  usb_common
> [10753.561363]  lrw
> [10753.561364]  sg
> [10753.561364]  mei_me
> [10753.561365]  ipmi_ssif
> [10753.561365] 
> [10753.561366]  ptp
> [10753.561367] irq event stamp: 10969989
> [10753.561368]  mei
> [10753.561369] hardirqs last  enabled at (10969988): 
> [10753.561374]  mfd_core
> [10753.561375] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10753.561376]  ipmi_devintf
> [10753.561377]  wmi
> [10753.561377]  iTCO_vendor_support
> [10753.561378] hardirqs last disabled at (10969989): 
> [10753.561380]  gf128mul
> [10753.561381] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10753.561382]  dm_multipath
> [10753.561383] softirqs last  enabled at (8596058): 
> [10753.561385]  dm_mod
> [10753.561386] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10753.561387]  ptp
> [10753.561388]  glue_helper
> [10753.561388] softirqs last disabled at (8596459): 
> [10753.561390]  mlx4_core
> [10753.561390] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561391]  hid_generic
> [10753.561392]  dcdbas
> [10753.561392]  usbhid
> [10753.561394]  ablk_helper
> [10753.561395] CPU: 4 PID: 7935 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10753.561396]  mgag200
> [10753.561397]  pps_core
> [10753.561397] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10753.561398]  iTCO_vendor_support
> [10753.561399]  i2c_algo_bit
> [10753.561400]  cryptd
> [10753.561400] task: ffff8803fbdd8000 ti: ffff8803f7524000 task.ti: ffff8803f7524000
> [10753.561401]  libphy
> [10753.561402]  gf128mul
> [10753.561403]  drm_kms_helper
> [10753.561404] RIP: 0010:[<ffffffff815781b3>] 
> [10753.561405]  fjes
> [10753.561406]  glue_helper
> [10753.561406]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10753.561408]  syscopyarea
> [10753.561408] RSP: 0018:ffff88046f283d78  EFLAGS: 00000292
> [10753.561410]  8250_fintek
> [10753.561410] RAX: ffff8803fbdd8000 RBX: 0000000000000292 RCX: 0000000000000000
> [10753.561412]  sb_edac
> [10753.561412]  sysfillrect
> [10753.561414]  mlx4_core
> [10753.561414] RDX: 0000000000000003 RSI: ffff8803fbdd8798 RDI: ffff8803fbdd8000
> [10753.561415]  dcdbas
> [10753.561416] RBP: ffff88046f283d88 R08: ffff88046f283cf8 R09: 0000000000000000
> [10753.561418]  ablk_helper
> [10753.561418] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10753.561419]  pcspkr
> [10753.561420]  sysimgblt
> [10753.561421]  ipmi_si
> [10753.561421]  fb_sys_fops
> [10753.561422]  pps_core
> [10753.561423]  ttm
> [10753.561424]  edac_core
> [10753.561424] R13: ffffea000c5d8000 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10753.561425]  drm
> [10753.561426]  ipmi_msghandler
> [10753.561426]  ehci_pci
> [10753.561428]  acpi_power_meter
> [10753.561428] FS:  00007f8a9b634700(0000) GS:ffff88046f280000(0000) knlGS:0000000000000000
> [10753.561430]  sr_mod
> [10753.561430] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10753.561431]  cdrom
> [10753.561432]  cryptd
> [10753.561432] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10753.561434]  tpm_tis
> [10753.561434]  ehci_hcd
> [10753.561435]  tpm
> [10753.561436]  usbcore
> [10753.561436] Stack:
> [10753.561437]  libphy
> [10753.561438]  usb_common
> [10753.561438]  ffff88017fc01c00
> [10753.561439]  sg
> [10753.561440]  lpc_ich
> [10753.561440]  0000000000000001
> [10753.561441] 
> [10753.561442]  ffff88046f283e38
> [10753.561442] irq event stamp: 19364635
> [10753.561443]  fjes
> [10753.561444]  ffffffff811640d6
> [10753.561445]  button
> [10753.561445] 
> [10753.561446] hardirqs last  enabled at (19364634): 
> [10753.561447]  ffff88046f283dc8
> [10753.561448]  8250_fintek
> [10753.561448]  shpchp
> [10753.561452]  ffff880319820078
> [10753.561452] [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d
> [10753.561453]  sb_edac
> [10753.561454] hardirqs last disabled at (19364635): 
> [10753.561456]  ffff88017fc1ba80
> [10753.561456] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10753.561457]  ffff880317607cd8
> [10753.561458] softirqs last  enabled at (13717672): 
> [10753.561458] 
> [10753.561459]  pcspkr
> [10753.561460]  mei_me
> [10753.561462]  ffff880317607cd8
> [10753.561462] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10753.561464]  ffff0001fbdd8798
> [10753.561464]  ipmi_si
> [10753.561465]  0000000000000292
> [10753.561466] softirqs last disabled at (13717773): 
> [10753.561467]  ffff8803fbdd8798
> [10753.561468] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561468] 
> [10753.561469]  mei
> [10753.561470]  edac_core
> [10753.561470] Call Trace:
> [10753.561472] CPU: 0 PID: 7937 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10753.561473]  <IRQ> 
> [10753.561474] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10753.561477]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10753.561480]  mfd_core
> [10753.561480]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10753.561482] task: ffff88040070df00 ti: ffff8803f1678000 task.ti: ffff8803f1678000
> [10753.561485]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561487] RIP: 0010:[<ffffffff815781b3>] 
> [10753.561489]  wmi
> [10753.561489]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561492]  dm_multipath
> [10753.561492]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10753.561494]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10753.561496]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10753.561499]  dm_mod
> [10753.561499]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10753.561500] RSP: 0018:ffff88046f203d78  EFLAGS: 00000292
> [10753.561503]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10753.561504] RAX: ffff88040070df00 RBX: 0000000000000292 RCX: 0000000000000000
> [10753.561506]  ipmi_msghandler
> [10753.561506]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10753.561507] RDX: 0000000000000003 RSI: ffff88040070e698 RDI: ffff88040070df00
> [10753.561509]  acpi_power_meter
> [10753.561510]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10753.561511]  tpm_tis
> [10753.561511] RBP: ffff88046f203d88 R08: ffff88046f203cf8 R09: 0000000000000000
> [10753.561514]  hid_generic
> [10753.561514] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10753.561516]  usbhid
> [10753.561516] R13: ffffea0007054c00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10753.561518]  tpm
> [10753.561518]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561520] FS:  00007f8a9a632700(0000) GS:ffff88046f200000(0000) knlGS:0000000000000000
> [10753.561521]  mgag200
> [10753.561522] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10753.561524]  i2c_algo_bit
> [10753.561525]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10753.561526] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406f0
> [10753.561528]  drm_kms_helper
> [10753.561528]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10753.561530]  syscopyarea
> [10753.561531]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10753.561531] Stack:
> [10753.561532]  lpc_ich
> [10753.561533]  <EOI> 
> [10753.561535]  sysfillrect
> [10753.561535]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10753.561536]  button
> [10753.561539]  ffff88017fc01c00
> [10753.561539]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10753.561542]  sysimgblt
> [10753.561542]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10753.561543]  fb_sys_fops
> [10753.561545]  0000000000000001
> [10753.561546]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10753.561547]  ffff88046f203e38
> [10753.561551]  ttm
> [10753.561551]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10753.561552]  ffffffff811640d6
> [10753.561554]  drm
> [10753.561554]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10753.561555] 
> [10753.561557]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10753.561558]  shpchp
> [10753.561560]  ffff88046f203dc8
> [10753.561560]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10753.561562]  ehci_pci
> [10753.561562]  mei_me
> [10753.561565]  0000000000000001
> [10753.561565]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10753.561569]  0000000000000001
> [10753.561569]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10753.561570]  sr_mod
> [10753.561572]  mei
> [10753.561572]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10753.561573]  cdrom
> [10753.561574]  ffff8801c1537a30
> [10753.561576]  mfd_core
> [10753.561576]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10753.561579]  wmi
> [10753.561579]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10753.561580]  dm_multipath
> [10753.561580] 
> [10753.561584]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10753.561585]  ffff8801c1537a30
> [10753.561587]  dm_mod
> [10753.561588]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10753.561590]  ffff00010070e698
> [10753.561590]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10753.561592]  0000000000000292
> [10753.561592]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10753.561595]  ehci_hcd
> [10753.561596]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10753.561597]  ffff88040070e698
> [10753.561599]  usbcore
> [10753.561599]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10753.561599] 
> [10753.561601]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10753.561602]  hid_generic
> [10753.561603]  usb_common
> [10753.561605]  usbhid
> [10753.561606]  sg
> [10753.561606] Call Trace:
> [10753.561606] 
> [10753.561608]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10753.561610]  <IRQ> 
> [10753.561610] irq event stamp: 14391137
> [10753.561613]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10753.561615]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10753.561618]  mgag200
> [10753.561618]  [<ffffffff810b9331>] ? rcu_irq_exit+0x41/0x50
> [10753.561619]  i2c_algo_bit
> [10753.561621] Code: 
> [10753.561621]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10753.561623] hardirqs last  enabled at (14391136): 
> [10753.561625] fc 
> [10753.561625] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10753.561628]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561629]  drm_kms_helper
> [10753.561630] hardirqs last disabled at (14391137): 
> [10753.561632] 48 
> [10753.561632] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10753.561635]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561636]  syscopyarea
> [10753.561638] 83 
> [10753.561638]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10753.561639] c7 
> [10753.561640]  sysfillrect
> [10753.561643] 18 
> [10753.561643]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10753.561645]  sysimgblt
> [10753.561646]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10753.561648] 48 
> [10753.561648]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10753.561651] softirqs last  enabled at (8589138): 
> [10753.561651]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10753.561653] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10753.561654]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10753.561656] softirqs last disabled at (8589495): 
> [10753.561657]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561659]  fb_sys_fops
> [10753.561659]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10753.561661] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561663]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10753.561665] 8b 
> [10753.561665] CPU: 2 PID: 7938 Comm: fio Tainted: G           O L  4.4.6-debug+ #2
> [10753.561666] 55 
> [10753.561667]  ttm
> [10753.561667] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10753.561669]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10753.561670] 08 
> [10753.561671]  drm
> [10753.561672] 53 
> [10753.561672]  ehci_pci
> [10753.561673] 48 
> [10753.561674]  sr_mod
> [10753.561675] task: ffff880400708000 ti: ffff8803f7af8000 task.ti: ffff8803f7af8000
> [10753.561676] 89 
> [10753.561677]  <EOI> 
> [10753.561677] f3 
> [10753.561679]  cdrom
> [10753.561679]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10753.561680] be 
> [10753.561683]  ehci_hcd
> [10753.561683]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10753.561684] 01 
> [10753.561685]  usbcore
> [10753.561686] RIP: 0010:[<ffffffff815781b3>] 
> [10753.561688] 00 
> [10753.561688]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10753.561689] 00 
> [10753.561690] RSP: 0018:ffff88046f243d78  EFLAGS: 00000292
> [10753.561691] 00 
> [10753.561691]  usb_common
> [10753.561693] e8 
> [10753.561694]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10753.561696] e1 
> [10753.561696]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10753.561697]  sg
> [10753.561698] ce 
> [10753.561698] 
> [10753.561702]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10753.561703] b2 
> [10753.561703] irq event stamp: 126134521
> [10753.561705]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10753.561706] ff 
> [10753.561707] hardirqs last  enabled at (126134520): 
> [10753.561709] 4c 
> [10753.561709]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10753.561712] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10753.561714] 89 
> [10753.561714] RAX: ffff880400708000 RBX: 0000000000000292 RCX: 0000000000000000
> [10753.561716] e7 
> [10753.561716] RDX: 0000000000000003 RSI: ffff880400708798 RDI: ffff880400708000
> [10753.561718] e8 
> [10753.561718]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10753.561719] 49 
> [10753.561721] hardirqs last disabled at (126134521): 
> [10753.561722]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10753.561723] RBP: ffff88046f243d88 R08: ffff88046f243cf8 R09: 0000000000000000
> [10753.561725] 00 
> [10753.561726]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10753.561728] b3 
> [10753.561728] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10753.561729] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10753.561731]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10753.561732] ff 
> [10753.561732] R13: ffffea0008065000 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10753.561734] f6 
> [10753.561734] FS:  00007f8a99e31700(0000) GS:ffff88046f240000(0000) knlGS:0000000000000000
> [10753.561736]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10753.561738] c7 
> [10753.561739]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10753.561742] 02 
> [10753.561742]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10753.561744] 74 
> [10753.561744]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10753.561746] 13 
> [10753.561747] softirqs last  enabled at (119833606): 
> [10753.561748]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10753.561750] e8 
> [10753.561750]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10753.561751] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10753.561754] 0f 
> [10753.561754]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10753.561756] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> [10753.561758]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10753.561760] a8 
> [10753.561760]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10753.561762] b2 
> [10753.561762] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0
> [10753.561764] ff 
> [10753.561764]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10753.561766] 53 
> [10753.561766] softirqs last disabled at (119835011): 
> [10753.561766] Stack:
> [10753.561768] [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561770]  ffff88017fc01c00
> [10753.561771] CPU: 1 PID: 358 Comm: kdmwork-254:2 Tainted: G           O L  4.4.6-debug+ #2
> [10753.561772]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10753.561774] 9d 
> [10753.561774]  0000000000000001
> [10753.561775] <65> 
> [10753.561776] Code: 
> [10753.561777] ff 
> [10753.561777]  ffff88046f243e38
> [10753.561778] fc 
> [10753.561779] 0d 
> [10753.561780]  ffffffff811640d6
> [10753.561780] 48 
> [10753.561781] 56 
> [10753.561782] 83 
> [10753.561782] 
> [10753.561783] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10753.561784] 2b 
> [10753.561784] c7 
> [10753.561786] a9 
> [10753.561786] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000
> [10753.561787] 7e 
> [10753.561788] 18 
> [10753.561788] 5b 
> [10753.561789] 48 
> [10753.561790] 41 
> [10753.561790] 8b 
> [10753.561791] 5c 
> [10753.561792] 55 
> [10753.561793] RIP: 0010:[<ffffffff815781b3>] 
> [10753.561794] 08 
> [10753.561794] 5d 
> [10753.561795] 53 
> [10753.561795] c3 
> [10753.561796] 48 
> [10753.561797] 53 
> [10753.561799] 89 f3 
> [10753.561799] 9d 
> [10753.561801] be 
> [10753.561801]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10753.561803] e8 
> [10753.561803]  ffff88046f243dc8
> [10753.561804] 6a 
> [10753.561804]  0000000000000000
> [10753.561805] 79 
> [10753.561806] 01 
> [10753.561806]  ffff880400708000
> [10753.561807] b2 
> [10753.561808]  ffff880201943278
> [10753.561809] 00 
> [10753.561809] RSP: 0018:ffff88046f223d78  EFLAGS: 00000292
> [10753.561811] ff 
> [10753.561811] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000
> [10753.561812] eb 
> [10753.561813] 00 
> [10753.561814] eb 
> [10753.561814] RDX: 0000000000000003 RSI: ffff88040120b718 RDI: ffff88040120af80
> [10753.561814] 
> [10753.561815] RBP: ffff88046f223d88 R08: ffff88046f223cf8 R09: 0000000000000000
> [10753.561816] 
> [10753.561817] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10753.561818] 00 
> [10753.561819]  ffff880201943278
> [10753.561820] e8 
> [10753.561820] R13: ffffea00060e8600 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10753.561821]  ffff000100708798
> [10753.561823] e1 
> [10753.561823] FS:  0000000000000000(0000) GS:ffff88046f220000(0000) knlGS:0000000000000000
> [10753.561824] ce 
> [10753.561824]  0000000000000292
> [10753.561826] b2 
> [10753.561826] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10753.561827] ff 
> [10753.561828]  ffff880400708798
> [10753.561829] 4c 
> [10753.561829] CR2: 00007ffadf28aa08 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10753.561830] 89 
> [10753.561830] Stack:
> [10753.561831] e7 
> [10753.561832]  ffff88017fc01c00
> [10753.561833] e8 
> [10753.561833]  0000000000000001
> [10753.561834] 49 
> [10753.561835]  ffff88046f223e38
> [10753.561835] 00 
> [10753.561836]  ffffffff811640d6
> [10753.561836] 
> [10753.561837] b3 
> [10753.561837] 
> [10753.561838] ff 
> [10753.561839]  ffff88046f223dc8
> [10753.561840] f6 
> [10753.561840]  0000000000000246
> [10753.561841] c7 
> [10753.561842]  ffffffff811859b3
> [10753.561842] 02 
> [10753.561843] Call Trace:
> [10753.561844] 74 
> [10753.561844]  <IRQ> 
> [10753.561845] 13 
> [10753.561846]  ffff880183a1d258
> [10753.561846] e8 
> [10753.561846] 
> [10753.561848] 0f 
> [10753.561848]  ffff880183a1d258
> [10753.561851] a8 
> [10753.561851]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10753.561854] b2 
> [10753.561854]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10753.561857] ff 
> [10753.561857]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10753.561860] 53 
> [10753.561860]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561861]  ffff00010120b718
> [10753.561862] 9d 
> [10753.561862]  0000000000000292
> [10753.561863] <65> 
> [10753.561864]  ffff88040120b718
> [10753.561864] ff 
> [10753.561864] 
> [10753.561866] 0d 
> [10753.561866] Call Trace:
> [10753.561868] 56 
> [10753.561868]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561871] 2b 
> [10753.561871]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10753.561874] a9 
> [10753.561874]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10753.561875]  <IRQ> 
> [10753.561878] 7e 
> [10753.561878]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10753.561881]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10753.561883] 5b 
> [10753.561883]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10753.561886]  [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30
> [10753.561888] 41 
> [10753.561888]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10753.561890]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10753.561892] 5c 
> [10753.561893]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10753.561894] 5d 
> [10753.561895]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561897] c3 
> [10753.561897]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10753.561900] 53 
> [10753.561900]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561902]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10753.561904] 9d 
> [10753.561904]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10753.561906]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.561909] e8 
> [10753.561909]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10753.561910]  <EOI> 
> [10753.561914] 6a 
> [10753.561914]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10753.561916] 79 
> [10753.561916]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10753.561919] b2 
> [10753.561919]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10753.561921]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10753.561923] ff 
> [10753.561923]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10753.561926] eb 
> [10753.561926]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10753.561928] eb 
> [10753.561928]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10753.561928] 
> [10753.561930]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.561932]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10753.561934]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10753.561935]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10753.561937]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10753.561940]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10753.561941]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10753.561943]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10753.561946]  <EOI> 
> [10753.561946]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10753.561948]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10753.561950]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10753.561953]  [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
> [10753.561955]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10753.561957]  [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
> [10753.561960]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10753.561965]  [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
> [10753.561966]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10753.561971]  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
> [10753.561973]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10753.561975]  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
> [10753.561977]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10753.561978]  [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60
> [10753.561981]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10753.561983]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10753.561985]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10753.561986]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10753.561988]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10753.561990]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10753.561991]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10753.561993]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10753.561995]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10753.561998] Code: 
> [10753.561998]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10753.562000] fc 
> [10753.562000]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10753.562003] 48 83 
> [10753.562003]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10753.562006] c7 
> [10753.562006]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10753.562007] 18 48 
> [10753.562008] Code: 
> [10753.562009] 8b 
> [10753.562009] fc 
> [10753.562010] 55 
> [10753.562011] 48 
> [10753.562012] 08 
> [10753.562013] 83 
> [10753.562013] 53 
> [10753.562014] c7 
> [10753.562014] 48 
> [10753.562015] 18 
> [10753.562016] 89 
> [10753.562016] 48 
> [10753.562017] f3 
> [10753.562017] 8b 
> [10753.562018] be 
> [10753.562019] 55 
> [10753.562019] 01 
> [10753.562020] 08 
> [10753.562020] 00 
> [10753.562021] 53 
> [10753.562022] 00 
> [10753.562022] 48 
> [10753.562023] 00 
> [10753.562023] 89 
> [10753.562024] e8 
> [10753.562025] f3 
> [10753.562025] e1 
> [10753.562026] be 
> [10753.562026] ce 
> [10753.562027] 01 
> [10753.562028] b2 
> [10753.562028] 00 
> [10753.562029] ff 
> [10753.562029] 00 
> [10753.562030] 4c 
> [10753.562030] 00 
> [10753.562031] 89 
> [10753.562032] e8 
> [10753.562032] e7 
> [10753.562033] e1 
> [10753.562033] e8 
> [10753.562034] ce 
> [10753.562034] 49 
> [10753.562035] b2 
> [10753.562036] 00 
> [10753.562036] ff 
> [10753.562037] b3 
> [10753.562037] 4c 
> [10753.562038] ff 
> [10753.562038] 89 
> [10753.562039] f6 
> [10753.562040] e7 
> [10753.562040] c7 
> [10753.562041] e8 
> [10753.562041] 02 
> [10753.562042] 49 
> [10753.562042] 74 
> [10753.562043] 00 
> [10753.562044] 13 
> [10753.562044] b3 
> [10753.562045] e8 
> [10753.562045] ff 
> [10753.562046] 0f 
> [10753.562047] f6 
> [10753.562047] a8 
> [10753.562048] c7 
> [10753.562048] b2 
> [10753.562049] 02 
> [10753.562049] ff 
> [10753.562050] 74 
> [10753.562051] 53 
> [10753.562051] 13 
> [10753.562052] 9d 
> [10753.562052] e8 
> [10753.562053] <65> 
> [10753.562054] 0f 
> [10753.562054] ff 
> [10753.562055] a8 
> [10753.562055] 0d 
> [10753.562056] b2 
> [10753.562056] 56 
> [10753.562057] ff 
> [10753.562058] 2b 
> [10753.562058] 53 
> [10753.562059] a9 
> [10753.562059] 9d 
> [10753.562060] 7e 
> [10753.562061] <65> 
> [10753.562061] 5b 
> [10753.562062] ff 
> [10753.562062] 41 
> [10753.562063] 0d 
> [10753.562064] 5c 
> [10753.562064] 56 
> [10753.562065] 5d 
> [10753.562065] 2b 
> [10753.562066] c3 
> [10753.562066] a9 
> [10753.562067] 53 
> [10753.562068] 7e 
> [10753.562068] 9d 
> [10753.562069] 5b 
> [10753.562069] e8 
> [10753.562070] 41 
> [10753.562071] 6a 
> [10753.562071] 5c 
> [10753.562072] 79 
> [10753.562072] 5d 
> [10753.562073] b2 
> [10753.562073] c3 
> [10753.562074] ff 
> [10753.562075] 53 
> [10753.562075] eb 
> [10753.562076] 9d 
> [10753.562076] eb 
> [10753.562077] e8 
> [10753.562077] 
> [10753.562080] 6a 79 b2 ff eb eb 
> [10753.718766] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10753.737483] irq event stamp: 13918891
> [10753.737667] hardirqs last  enabled at (13918890): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10753.738020] hardirqs last disabled at (13918891): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10753.738364] softirqs last  enabled at (10133236): [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst]
> [10753.738765] softirqs last disabled at (10133267): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.739063] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G           O L  4.4.6-debug+ #2
> [10753.739256] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10753.739473] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000
> [10753.739599] RIP: 0010:[<ffffffff815781b3>]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10753.739990] RSP: 0018:ffff88046f363d68  EFLAGS: 00000292
> [10753.740149] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000
> [10753.740321] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00
> [10753.740439] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000
> [10753.740664] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10753.740772] R13: ffffea000e725600 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10753.740991] FS:  0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000
> [10753.741189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10753.741321] CR2: 00005633130dd1a8 CR3: 0000000401468000 CR4: 00000000001406e0
> [10753.741465] Stack:
> [10753.741613]  ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6
> [10753.742334]  ffff88046f363db8 ffffffff8157940f 0000000100300001 ffff88039c95bfc0
> [10753.743130]  ffff88039c95bfc0 0000000100000282 0000000000000292 ffff880439376698
> [10753.743870] Call Trace:
> [10753.744031]  <IRQ> 
> [10753.744151]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10753.744464]  [<ffffffff8157940f>] ? retint_kernel+0x10/0x10
> [10753.744618]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10753.744803]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.744935]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10753.745075]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10753.745209]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10753.745387]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10753.745585]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10753.745769]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10753.745920]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10753.746063]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10753.746262]  [<ffffffff810065a5>] do_IRQ+0x65/0x110
> [10753.746437]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10753.746653]  [<ffffffff815793c9>] common_interrupt+0x89/0x89
> [10753.746803]  <EOI> 
> [10753.746973]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10753.747278]  [<ffffffff8117bad7>] ? put_object+0x7/0x40
> [10753.747453]  [<ffffffff8117bb48>] ? __delete_object+0x38/0x70
> [10753.747650]  [<ffffffff8117bb98>] delete_object_full+0x18/0x20
> [10753.747797]  [<ffffffff8156dab1>] kmemleak_free+0x31/0x80
> [10753.747987]  [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160
> [10753.748137]  [<ffffffff81113282>] mempool_free_slab+0x12/0x20
> [10753.748284]  [<ffffffff81113406>] mempool_free+0x26/0x80
> [10753.748423]  [<ffffffff8127e8c1>] bvec_free+0x31/0x40
> [10753.748551]  [<ffffffff8127ee54>] bio_free+0x34/0x60
> [10753.748737]  [<ffffffff8127ee9e>] bio_put+0x1e/0x30
> [10753.748865]  [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk]
> [10753.749111]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10753.749289]  [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd]
> [10753.749431]  [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630
> [10753.749628]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10753.749762]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10753.749942]  [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk]
> [10753.750123]  [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk]
> [10753.750325]  [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk]
> [10753.750472]  [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk]
> [10753.750631]  [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk]
> [10753.750759]  [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0
> [10753.750930]  [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst]
> [10753.751195]  [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst]
> [10753.751372]  [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst]
> [10753.751602]  [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst]
> [10753.751802]  [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst]
> [10753.751983]  [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst]
> [10753.752136]  [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220
> [10753.752307]  [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220
> [10753.752548]  [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst]
> [10753.752696]  [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10
> [10753.752875]  [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst]
> [10753.753142]  [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst]
> [10753.753311]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10753.753524]  [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst]
> [10753.753681]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10753.753836]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10753.753992]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10753.754165]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10753.754315] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 
> [10764.112230] scsi host50: SRP abort called
> [10764.112455] scsi host50: Sending SRP abort for tag 0x0
> [10764.116228] [32754]: scst: scst_rx_mgmt_fn:7350:TM fn ABORT_TASK/0 (mcmd ffff88006892e788, initiator fe80:0000:0000:0000:e41d:2d03:000a:85b1, target fe80:0000:0000:0000:e41d:2d03:000a:6d51)
> [10764.116401] [32754]: scst_rx_mgmt_fn:7364:sess=ffff88041067f400, tag_set 1, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880454310548
> [10764.116635] [32754]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e788 to active mgmt cmd list
> [10764.117014] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e788 from active cmd list
> [10764.117333] [31019]: scst_abort_task:6762:Aborting task (cmd ffff8804543140a8, sn -20, set 1, tag 0, queue_type 1)
> [10764.117488] [31019]: scst: scst_abort_cmd:5849:Aborting cmd ffff8804543140a8 (tag 0, op READ(10))
> [10764.117908] [31019]: scst_abort_cmd:5934:cmd ffff8804543140a8 (tag 0) is being executed
> [10764.118220] [31019]: scst: scst_abort_cmd:5970:cmd ffff8804543140a8 (tag 0, sn 4294967276) being executed/xmitted (state EXEC_WAIT, op READ(10), proc time 63 sec., timeout 60 sec.), deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1, internal 0, mcmd fn 0 (mcmd ffff88006892e788), initiator fe80:0000:0000:0000:e41d:2d03:000a:85b1, target fe80:0000:0000:0000:e41d:2d03:000a:6d51)
> [10764.118408] [31019]: scst: scst_set_mcmd_next_state:6011:cmd_done_wait_count(1) not 0, preparing to wait
> [10765.270812] INFO: rcu_sched self-detected stall on CPU
> [10765.271293] 	4-...: (6499 ticks this GP) idle=ca3/140000000000002/0 softirq=1007789/1007789 fqs=1001 
> [10765.271442] 	 (t=6500 jiffies g=219292 c=219291 q=13711541)
> [10765.271809] Task dump for CPU 0:
> [10765.272032] fio             R  running task        0  7937   7744 0x00000008
> [10765.272691]  ffff8803f167b898 ffff8804536bd140 0000000000000000 0000000000000000
> [10765.273812]  0000000000000000 0000000000000000 ffff88040070df00 0000000000000001
> [10765.274809]  0000000000000000 0000000000000001 ffff88040070df00 ffffffffffffff10
> [10765.275910] Call Trace:
> [10765.276149]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10765.276362]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10765.276514]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10765.276828]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10765.277080]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10765.277216]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10765.277479]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10765.277651]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10765.277899]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10765.278145]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10765.278401]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10765.278610]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10765.278777]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10765.278941]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10765.279138]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10765.279261]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10765.279486]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10765.279661]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10765.279829]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10765.280114]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.280307]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10765.280564]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.280766] INFO: rcu_sched self-detected stall on CPU
> [10765.280770] 	0-...: (6412 ticks this GP) idle=911/140000000000002/0 softirq=1235409/1235409 fqs=1001 
> [10765.280773] 	 (t=6500 jiffies g=219292 c=219291 q=13711939)
> [10765.280775] INFO: rcu_sched self-detected stall on CPU
> [10765.280779] 	2-...: (6500 ticks this GP) idle=203/140000000000002/0 softirq=1498700/1498700 fqs=1002 
> [10765.280781] 	 (t=6500 jiffies g=219292 c=219291 q=13711939)
> [10765.281492] Task dump for CPU 2:
> [10765.281636] fio             R  running task        0  7938   7744 0x00000008
> [10765.281940]  ffff8803f7afb898 ffff8804536bd140 0000000000000000 0000000000000000
> [10765.282551]  0000000000000000 0000000000000000 ffff880400708000 0000000000000001
> [10765.283093]  0000000000000000 0000000000000001 ffff880400708000 ffffffffffffff10
> [10765.283618] Call Trace:
> [10765.283746]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10765.283849]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10765.283977]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10765.284102]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10765.284238]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10765.284368]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10765.284486]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10765.284616]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10765.284722]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10765.284842]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10765.284985]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10765.285085]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10765.285202]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10765.285313]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10765.285428]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10765.285558]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10765.285688]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10765.285812]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10765.285932]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10765.286048]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.286193]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10765.286317]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.286415] Task dump for CPU 4:
> [10765.286525] fio             R  running task        0  7935   7744 0x00000008
> [10765.286964]  0000000000001e40 ffff88046f283ab8 ffffffff8108195f 0000000000000004
> [10765.287498]  ffffffff81a40a40 ffff88046f283ad0 ffffffff81084742 0000000000000005
> [10765.288065]  ffff88046f283b00 ffffffff810b5e79 ffff88046f295400 ffffffff81a40a40
> [10765.288674] Call Trace:
> [10765.288821]  <IRQ>  [<ffffffff8108195f>] sched_show_task+0xbf/0x150
> [10765.289152]  [<ffffffff81084742>] dump_cpu_task+0x32/0x40
> [10765.289292]  [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0
> [10765.289432]  [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730
> [10765.289588]  [<ffffffff810bc1c4>] update_process_times+0x34/0x60
> [10765.289730]  [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50
> [10765.289896]  [<ffffffff810cb148>] tick_sched_timer+0x38/0x70
> [10765.290036]  [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30
> [10765.290173]  [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0
> [10765.290390]  [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0
> [10765.290529]  [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10
> [10765.290631]  [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90
> [10765.290744]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10765.290855]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10765.290982]  [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50
> [10765.291096]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10765.291208]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10765.291322]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10765.291449]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10765.291561]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10765.291687]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10765.291802]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10765.291915]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10765.292040]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10765.292204]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10765.292330]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10765.292492]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10765.292615]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10765.292751]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10765.292871]  <EOI>  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10765.293139]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10765.293259]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10765.293392]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10765.293538]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10765.293670]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10765.293790]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10765.293910]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10765.294041]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10765.294151]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10765.294295]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10765.294446]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10765.294613]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10765.294747]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10765.294914]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10765.295062]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10765.295183]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10765.295317]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10765.295451]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10765.295576]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.295698]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10765.295830]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.295963] Task dump for CPU 0:
> [10765.296100] fio             R  running task        0  7937   7744 0x00000008
> [10765.296734]  ffff8803f167b898 ffff8804536bd140 0000000000000000 0000000000000000
> [10765.297665]  0000000000000000 0000000000000000 ffff88040070df00 0000000000000001
> [10765.298695]  0000000000000000 0000000000000001 ffff88040070df00 ffffffffffffff10
> [10765.299591] Call Trace:
> [10765.299782]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10765.299953]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10765.300118]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10765.300363]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10765.300506]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10765.300650]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10765.300887]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10765.301094]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10765.301393]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10765.301577]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10765.301817]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10765.301951]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10765.302143]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10765.302359]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10765.302512]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10765.302664]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10765.302848]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10765.302981]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10765.303128]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10765.303255]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.303427]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10765.303656]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.303787] Task dump for CPU 2:
> [10765.303918] fio             R  running task        0  7938   7744 0x00000008
> [10765.304354]  0000000000001e40 ffff88046f243ab8 ffffffff8108195f 0000000000000002
> [10765.305171]  ffffffff81a40a40 ffff88046f243ad0 ffffffff81084742 0000000000000003
> [10765.306025]  ffff88046f243b00 ffffffff810b5e79 ffff88046f255400 ffffffff81a40a40
> [10765.306874] Call Trace:
> [10765.307035]  <IRQ>  [<ffffffff8108195f>] sched_show_task+0xbf/0x150
> [10765.307436]  [<ffffffff81084742>] dump_cpu_task+0x32/0x40
> [10765.307729]  [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0
> [10765.308054]  [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730
> [10765.308207]  [<ffffffff810bc1c4>] update_process_times+0x34/0x60
> [10765.308381]  [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50
> [10765.308539]  [<ffffffff810cb148>] tick_sched_timer+0x38/0x70
> [10765.308816]  [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30
> [10765.309055]  [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0
> [10765.309246]  [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0
> [10765.309401]  [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10
> [10765.309643]  [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90
> [10765.309759]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10765.309909]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10765.310054]  [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50
> [10765.310226]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10765.310381]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10765.310522]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10765.310667]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10765.310860]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10765.311075]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10765.311285]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10765.311467]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10765.311660]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10765.311842]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10765.311991]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10765.312170]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10765.312381]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10765.312502]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10765.312632]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10765.312826]  <EOI>  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10765.313135]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10765.313266]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10765.313424]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10765.313607]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10765.313738]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10765.313954]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10765.314094]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10765.314275]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10765.314388]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10765.314516]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10765.314738]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10765.314917]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10765.315116]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10765.315283]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10765.315415]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10765.315566]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10765.315707]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10765.315868]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10765.316010]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.316198]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10765.316322]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.316530] Task dump for CPU 4:
> [10765.316737] fio             R  running task        0  7935   7744 0x00000008
> [10765.317203]  0000000000000001 00007f8a58031000 0000000000001000 000000000000001e
> [10765.317947]  ffff8803f7527de8 ffff8803fbdd8000 0000000000000246 ffff8803f7437300
> [10765.318778]  000000000000001e 0000000000000246 ffff8803f7527e38 ffff8803fbdd8000
> [10765.319525] Call Trace:
> [10765.319669]  [<ffffffff811d3066>] ? do_io_submit+0x2f6/0x770
> [10765.319853]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.319993]  [<ffffffff811d34eb>] ? SyS_io_submit+0xb/0x10
> [10765.320150]  [<ffffffff815788db>] ? entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.320304] Task dump for CPU 0:
> [10765.320555] fio             R  running task        0  7937   7744 0x00000008
> [10765.321211]  0000000000001e40 ffff88046f203ab8 ffffffff8108195f 0000000000000000
> [10765.322400]  ffffffff81a40a40 ffff88046f203ad0 ffffffff81084742 0000000000000001
> [10765.323303]  ffff88046f203b00 ffffffff810b5e79 ffff88046f215400 ffffffff81a40a40
> [10765.324130] Call Trace:
> [10765.324330]  <IRQ>  [<ffffffff8108195f>] sched_show_task+0xbf/0x150
> [10765.324936]  [<ffffffff81084742>] dump_cpu_task+0x32/0x40
> [10765.325164]  [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0
> [10765.325309]  [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730
> [10765.325476]  [<ffffffff810bc1c4>] update_process_times+0x34/0x60
> [10765.325659]  [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50
> [10765.325980]  [<ffffffff810cb148>] tick_sched_timer+0x38/0x70
> [10765.326220]  [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30
> [10765.326459]  [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0
> [10765.326699]  [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0
> [10765.326895]  [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10
> [10765.327086]  [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90
> [10765.327224]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10765.327407]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10765.327622]  [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50
> [10765.327760]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10765.327916]  [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30
> [10765.328133]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10765.328434]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10765.328835]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10765.329043]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10765.329243]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10765.329487]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10765.329649]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10765.329865]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10765.330097]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10765.330272]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10765.330456]  [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90
> [10765.330696]  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
> [10765.330836]  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
> [10765.330993]  <EOI>  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10765.331359]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10765.331554]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10765.331757]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10765.332129]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10765.332320]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10765.332594]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10765.332749]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10765.333071]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10765.333244]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10765.333426]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10765.333644]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10765.333810]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10765.334020]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10765.334203]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10765.334348]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10765.334474]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10765.334656]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10765.334972]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10765.335179]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.335430]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10765.335757]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.335928] Task dump for CPU 2:
> [10765.336319] fio             R  running task        0  7938   7744 0x00000008
> [10765.337078]  ffff8803f7afb898 ffff8804536bd140 0000000000000000 0000000000000000
> [10765.338168]  0000000000000000 0000000000000000 ffff880400708000 0000000000000001
> [10765.339181]  0000000000000000 0000000000000001 ffff880400708000 ffffffffffffff10
> [10765.340163] Call Trace:
> [10765.340325]  [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40
> [10765.340515]  [<ffffffff81283717>] get_request+0x447/0x8e0
> [10765.340721]  [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270
> [10765.340843]  [<ffffffff812b2401>] ? __delay+0x1/0x10
> [10765.341044]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10765.341210]  [<ffffffff812863b1>] blk_queue_bio+0x111/0x300
> [10765.341364]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10765.341552]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10765.341776]  [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60
> [10765.341929]  [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0
> [10765.342070]  [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10
> [10765.342378]  [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40
> [10765.342599]  [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80
> [10765.342798]  [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0
> [10765.343017]  [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0
> [10765.343241]  [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100
> [10765.343401]  [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40
> [10765.343562]  [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0
> [10765.343696]  [<ffffffff811d3066>] do_io_submit+0x2f6/0x770
> [10765.343873]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.344044]  [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10
> [10765.344231]  [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f
> [10765.344385] Task dump for CPU 4:
> [10765.344560] fio             R  running task        0  7935   7744 0x00000008
> [10765.345208]  0000000000000001 00007f8a58031000 0000000000001000 000000000000001e
> [10765.345981]  ffff8803f7527de8 ffff8803fbdd8000 0000000000000246 ffff8803f7437300
> [10765.347051]  000000000000001e 0000000000000246 ffff8803f7527e38 ffff8803fbdd8000
> [10765.347947] Call Trace:
> [10765.348145]  [<ffffffff811d3066>] ? do_io_submit+0x2f6/0x770
> [10765.348313]  [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770
> [10765.348448]  [<ffffffff811d34eb>] ? SyS_io_submit+0xb/0x10
> [10765.348880]  [<ffffffff815788db>] ? entry_SYSCALL_64_fastpath+0x16/0x6f
> [10777.682813] systemd-journald[8985]: File /var/log/journal/6d2d7eb462015b615a423a1056c6183d/system.journal corrupted or uncleanly shut down, renaming and replacing.
> [10778.058082] systemd-coredump[8980]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.fd2308d1f77847cf9c32b20535a1d348.499.1458676877000000.xz.
> [10781.560307] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753]
> [10781.560440] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10781.573088] irq event stamp: 20161671
> [10781.573201] hardirqs last  enabled at (20161670): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10781.573436] hardirqs last disabled at (20161671): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10781.573698] softirqs last  enabled at (10133236): [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst]
> [10781.573951] softirqs last disabled at (10133267): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10781.574196] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G           O L  4.4.6-debug+ #2
> [10781.574316] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10781.574425] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000
> [10781.574531] RIP: 0010:[<ffffffff815781b3>]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10781.574742] RSP: 0018:ffff88046f363d68  EFLAGS: 00000292
> [10781.574854] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000
> [10781.574969] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00
> [10781.575105] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000
> [10781.575209] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10781.575310] R13: ffffea0001036200 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10781.575429] FS:  0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000
> [10781.575542] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10781.575676] CR2: 00005633130dd1a8 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10781.575786] Stack:
> [10781.575896]  ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6
> [10781.576422]  ffff88046f363db8 ffffffff811640c0 0000000100300001 ffff880040d8b7c8
> [10781.576944]  ffff880040d8b7c8 0000000113d14e00 0000000000000292 ffff880439376698
> [10781.577475] Call Trace:
> [10781.577585]  <IRQ> 
> [10781.577690]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10781.577918]  [<ffffffff811640c0>] ? __slab_free+0xb0/0x270
> [10781.578050]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10781.578176]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10781.578323]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10781.578447]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10781.578584]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10781.578697]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10781.578805]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10781.578927]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10781.579038]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10781.579164]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10781.579276]  [<ffffffff810065a5>] do_IRQ+0x65/0x110
> [10781.579407]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10781.579518]  [<ffffffff815793c9>] common_interrupt+0x89/0x89
> [10781.579629]  <EOI> 
> [10781.579746]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10781.579977]  [<ffffffff8117bad7>] ? put_object+0x7/0x40
> [10781.580090]  [<ffffffff8117bb48>] ? __delete_object+0x38/0x70
> [10781.580212]  [<ffffffff8117bb98>] delete_object_full+0x18/0x20
> [10781.580328]  [<ffffffff8156dab1>] kmemleak_free+0x31/0x80
> [10781.580430]  [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160
> [10781.580553]  [<ffffffff81113282>] mempool_free_slab+0x12/0x20
> [10781.580672]  [<ffffffff81113406>] mempool_free+0x26/0x80
> [10781.580798]  [<ffffffff8127e8c1>] bvec_free+0x31/0x40
> [10781.580916]  [<ffffffff8127ee54>] bio_free+0x34/0x60
> [10781.581037]  [<ffffffff8127ee9e>] bio_put+0x1e/0x30
> [10781.581162]  [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk]
> [10781.581287]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10781.581404]  [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd]
> [10781.581521]  [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630
> [10781.581631]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10781.581746]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10781.581871]  [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk]
> [10781.581992]  [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk]
> [10781.582106]  [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk]
> [10781.582260]  [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk]
> [10781.582398]  [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk]
> [10781.582514]  [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0
> [10781.582652]  [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst]
> [10781.582804]  [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst]
> [10781.582952]  [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst]
> [10781.583109]  [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst]
> [10781.583245]  [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst]
> [10781.583387]  [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst]
> [10781.583520]  [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220
> [10781.583646]  [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220
> [10781.583777]  [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst]
> [10781.583920]  [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10
> [10781.584054]  [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst]
> [10781.584193]  [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst]
> [10781.584316]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10781.584458]  [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst]
> [10781.584590]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10781.584706]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10781.584827]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10781.584948]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10781.585086] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 
> [10788.304454] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-611 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.304852] scsi host60: ib_srp: DREQ received - connection closed
> [10788.305727] ib_srpt: RDMA t 5 for idx 5 failed with status 5. If this has not been triggered by a cable pull, please check the involved IB HCA's and cables.
> [10788.305926] ib_srpt: sending response for idx 5 failed with status 5
> [10788.306061] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-611.
> [10788.328967] [560]: scst_unregister_session:7901:Unregistering session ffff880416bf0000 (wait 0)
> [10788.329005] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-617 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.329770] [560]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d810)
> [10788.329839] scsi host60: ib_srp: DREQ received - connection closed
> [10788.330132] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-623 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.330698] [574]: scst_unregister_session:7901:Unregistering session ffff880416bf0e80 (wait 0)
> [10788.330921] [560]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0000, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.331187] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-617.
> [10788.331547] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-629 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.332133] [574]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e5d0)
> [10788.332434] [560]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d810 to active mgmt cmd list
> [10788.332490] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-610 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.332907] [574]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0e80, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.333332] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d810 from active cmd list
> [10788.333516] [586]: scst_unregister_session:7901:Unregistering session ffff880416bf1d00 (wait 0)
> [10788.333707] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0000 (mcmd ffff88006892d810)
> [10788.333977] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-616 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.334379] [574]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e5d0 to active mgmt cmd list
> [10788.334553] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.334780] [586]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d9c8)
> [10788.335366] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.335483] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-621 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.335955] [586]: scst_rx_mgmt_fn:7364:sess=ffff880416bf1d00, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.336287] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d810) finished, status 0
> [10788.336554] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-627 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.337339] [586]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d9c8 to active mgmt cmd list
> [10788.337480] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e5d0 from active cmd list
> [10788.337707] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-613 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.337926] [603]: scst_unregister_session:7901:Unregistering session ffff880416bf4880 (wait 0)
> [10788.338031] [554]: scst_unregister_session:7901:Unregistering session ffff880400faba00 (wait 0)
> [10788.338437] [603]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d810)
> [10788.339050] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-619 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.339367] [570]: scst_unregister_session:7901:Unregistering session ffff880416bf6cc0 (wait 0)
> [10788.339774] [555]: scst: scst_cmd_thread:5499:Processing thread disk134_0 finished
> [10788.339973] [570]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e418)
> [10788.340435] [603]: scst_rx_mgmt_fn:7364:sess=ffff880416bf4880, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.340624] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0e80 (mcmd ffff88006892e5d0)
> [10788.340759] [554]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.340994] [603]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d810 to active mgmt cmd list
> [10788.341354] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.341536] [554]: scst_rx_mgmt_fn:7364:sess=ffff880400faba00, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.342115] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.342324] [593]: scst_unregister_session:7901:Unregistering session ffff880416bf4fc0 (wait 0)
> [10788.342613] [554]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.342920] [593]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892dd38)
> [10788.343926] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e5d0) finished, status 0
> [10788.344411] [593]: scst_rx_mgmt_fn:7364:sess=ffff880416bf4fc0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.344785] [570]: scst_rx_mgmt_fn:7364:sess=ffff880416bf6cc0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.345000] [593]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892dd38 to active mgmt cmd list
> [10788.345266] [570]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e418 to active mgmt cmd list
> [10788.345403] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d9c8 from active cmd list
> [10788.345667] [566]: scst_unregister_session:7901:Unregistering session ffff880416bf0740 (wait 0)
> [10788.345840] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf1d00 (mcmd ffff88006892d9c8)
> [10788.346315] [566]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e5d0)
> [10788.346665] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.346879] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.347005] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d9c8) finished, status 0
> [10788.347253] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d810 from active cmd list
> [10788.347462] [566]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0740, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.347810] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf4880 (mcmd ffff88006892d810)
> [10788.347858] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-624 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.348180] [566]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e5d0 to active mgmt cmd list
> [10788.348448] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.348697] [579]: scst_unregister_session:7901:Unregistering session ffff880416bf15c0 (wait 0)
> [10788.349339] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.349487] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d810) finished, status 0
> [10788.349699] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-630 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.349946] [579]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d9c8)
> [10788.350246] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.350560] [579]: scst_rx_mgmt_fn:7364:sess=ffff880416bf15c0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.350823] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880400faba00 (mcmd ffff88006892db80)
> [10788.350953] scsi host58: ib_srp: DREQ received - connection closed
> [10788.351102] [579]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d9c8 to active mgmt cmd list
> [10788.351139] scsi host58: ib_srp: DREQ received - connection closed
> [10788.351454] scsi host58: ib_srp: DREQ received - connection closed
> [10788.351938] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.352233] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.352645] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.352944] [588]: scst_unregister_session:7901:Unregistering session ffff880416bf5700 (wait 0)
> [10788.353527] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e418 from active cmd list
> [10788.353540] scsi host58: ib_srp: DREQ received - connection closed
> [10788.353933] scsi host61: ib_srp: DREQ received - connection closed
> [10788.354074] scsi host61: ib_srp: DREQ received - connection closed
> [10788.354416] scsi host61: ib_srp: DREQ received - connection closed
> [10788.354762] [588]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.354868] scsi host61: ib_srp: DREQ received - connection closed
> [10788.355077] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-610.
> [10788.355101] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-616.
> [10788.355141] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-621.
> [10788.357117] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-627.
> [10788.357120] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf6cc0 (mcmd ffff88006892e418)
> [10788.357183] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.357185] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.357190] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e418) finished, status 0
> [10788.357202] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892dd38 from active cmd list
> [10788.357276] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf4fc0 (mcmd ffff88006892dd38)
> [10788.357337] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.357339] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.357343] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892dd38) finished, status 0
> [10788.357353] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e5d0 from active cmd list
> [10788.357390] [588]: scst_rx_mgmt_fn:7364:sess=ffff880416bf5700, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.357393] [588]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.357445] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0740 (mcmd ffff88006892e5d0)
> [10788.357502] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.357504] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.357508] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e5d0) finished, status 0
> [10788.357517] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d9c8 from active cmd list
> [10788.357748] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf15c0 (mcmd ffff88006892d9c8)
> [10788.357806] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.357808] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.357811] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d9c8) finished, status 0
> [10788.357821] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.357901] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf5700 (mcmd ffff88006892db80)
> [10788.357958] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.357960] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.357964] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.357980] scsi host60: ib_srp: DREQ received - connection closed
> [10788.358287] scsi host60: ib_srp: DREQ received - connection closed
> [10788.359468] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-623.
> [10788.359491] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-629.
> [10788.360123] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-612 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.360815] scsi host59: ib_srp: DREQ received - connection closed
> [10788.361642] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-618 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.361991] [564]: scst_unregister_session:7901:Unregistering session ffff880416bf7400 (wait 0)
> [10788.362038] [564]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.362042] [564]: scst_rx_mgmt_fn:7364:sess=ffff880416bf7400, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.362045] [564]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.362094] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-612.
> [10788.362118] scsi host59: ib_srp: DREQ received - connection closed
> [10788.362616] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.362838] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf7400 (mcmd ffff88006892db80)
> [10788.362899] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.362901] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.362906] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.364202] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-618.
> [10788.370985] [576]: scst_unregister_session:7901:Unregistering session ffff880416bf6580 (wait 0)
> [10788.371047] [576]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.371050] [576]: scst_rx_mgmt_fn:7364:sess=ffff880416bf6580, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.371053] [576]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.371169] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-625 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.371592] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.371644] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf6580 (mcmd ffff88006892db80)
> [10788.371705] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.371707] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.371711] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.386343] [601]: scst_unregister_session:7901:Unregistering session ffff880416bf2b80 (wait 0)
> [10788.386502] [601]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.386505] [601]: scst_rx_mgmt_fn:7364:sess=ffff880416bf2b80, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.386508] [601]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.386522] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.386590] scsi host59: ib_srp: DREQ received - connection closed
> [10788.386724] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf2b80 (mcmd ffff88006892db80)
> [10788.386785] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.386787] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.386791] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.387386] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-625.
> [10788.396186] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-613.
> [10788.396511] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-619.
> [10788.396758] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-624.
> [10788.397245] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-630.
> [10788.397967] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-631 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.399003] scsi host59: ib_srp: DREQ received - connection closed
> [10788.399155] [590]: scst_unregister_session:7901:Unregistering session ffff880416bf2440 (wait 0)
> [10788.399349] [590]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.399353] [590]: scst_rx_mgmt_fn:7364:sess=ffff880416bf2440, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.399356] [590]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.399369] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.399437] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf2440 (mcmd ffff88006892db80)
> [10788.399501] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.399504] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.399508] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.409181] ib_srpt: Closing channel fe80:0000:0000:0000:0002:c903:00a0:2dc2-657 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.409978] [606]: scst_unregister_session:7901:Unregistering session ffff880416bf32c0 (wait 0)
> [10788.410171] [606]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.410174] [606]: scst_rx_mgmt_fn:7364:sess=ffff880416bf32c0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.410177] [606]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.410190] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.410296] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf32c0 (mcmd ffff88006892db80)
> [10788.410359] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.410361] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.410365] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.418821] ib_srpt: Closing channel fe80:0000:0000:0000:0002:c903:00a0:2dc1-658 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled
> [10788.419326] [2408]: scst_unregister_session:7901:Unregistering session ffff8803f8f17400 (wait 0)
> [10788.419798] [2408]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.419801] [2408]: scst_rx_mgmt_fn:7364:sess=ffff8803f8f17400, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.419804] [2408]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.419815] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.419981] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff8803f8f17400 (mcmd ffff88006892db80)
> [10788.420042] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.420045] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.420049] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.427954] [2410]: scst_unregister_session:7901:Unregistering session ffff8803f8f10740 (wait 0)
> [10788.428188] [2410]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.428192] [2410]: scst_rx_mgmt_fn:7364:sess=ffff8803f8f10740, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.428195] [2410]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.428208] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.428513] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff8803f8f10740 (mcmd ffff88006892db80)
> [10788.428576] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.428579] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.428583] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.479546] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10788.479914] [573]: scst: scst_cmd_thread:5499:Processing thread disk138_0 finished
> [10788.484823] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10788.486170] [585]: scst: scst_cmd_thread:5499:Processing thread disk142_0 finished
> [10788.490385] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-618.
> [10788.490785] scsi host59: ib_srp: connection closed
> [10788.490788] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:0002:c903:00a0:2dc2-657.
> [10788.490850] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:0002:c903:00a0:2dc1-658.
> [10788.490894] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-631.
> [10788.493602] scsi host59: ib_srp: connection closed
> [10788.493990] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-612.
> [10788.520887] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10788.521524] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-625.
> [10788.522148] scsi host59: ib_srp: connection closed
> [10788.526290] [602]: scst: scst_cmd_thread:5499:Processing thread disk147_0 finished
> [10788.532024] [553]: scst: scst_cmd_thread:5499:Processing thread disk133_0 finished
> [10788.547718] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10788.548481] [569]: scst: scst_cmd_thread:5499:Processing thread disk137_0 finished
> [10788.551854] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10788.600121] scsi host59: ib_srp: connection closed
> [10788.620114] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-631.
> [10788.670189] [581]: scst_unregister_session:7901:Unregistering session ffff880416bf5e40 (wait 0)
> [10788.671892] [581]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80)
> [10788.673417] [581]: scst_rx_mgmt_fn:7364:sess=ffff880416bf5e40, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv           (null)
> [10788.673860] [581]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list
> [10788.674072] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list
> [10788.674441] [592]: scst: scst_cmd_thread:5499:Processing thread disk145_0 finished
> [10788.675123] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf5e40 (mcmd ffff88006892db80)
> [10788.676436] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10)
> [10788.676676] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned
> [10788.676914] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0
> [10788.698994] [565]: scst: scst_cmd_thread:5499:Processing thread disk136_0 finished
> [10788.703046] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10788.703357] [578]: scst: scst_cmd_thread:5499:Processing thread disk140_0 finished
> [10788.708079] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10788.708352] [587]: scst: scst_cmd_thread:5499:Processing thread disk143_0 finished
> [10788.712911] [563]: scst: scst_cmd_thread:5499:Processing thread disk135_0 finished
> [10788.716807] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10790.350077] scsi host60: ib_srp: connection closed
> [10790.370069] scsi host60: ib_srp: connection closed
> [10790.400086] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-621.
> [10790.400090] scsi host60: ib_srp: connection closed
> [10790.400103] scsi host60: ib_srp: connection closed
> [10790.400685] scsi host61: ib_srp: connection closed
> [10790.400701] scsi host61: ib_srp: connection closed
> [10790.400715] scsi host61: ib_srp: connection closed
> [10790.400729] scsi host61: ib_srp: connection closed
> [10790.402597] scsi host58: ib_srp: connection closed
> [10790.403206] scsi host58: ib_srp: connection closed
> [10790.403613] scsi host58: ib_srp: connection closed
> [10790.404086] scsi host58: ib_srp: connection closed
> [10790.440067] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-630.
> [10790.540711] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:0002:c903:00a0:2dc1-658.
> [10790.541056] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:0002:c903:00a0:2dc2-657.
> [10793.419992] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ...
> [10793.421559] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress
> [10793.422461] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d51-630: state disconnected; 0 commands in progress
> [10793.422928] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-618: state disconnected; 0 commands in progress
> [10793.424120] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-625: state disconnected; 0 commands in progress
> [10793.424912] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-631: state disconnected; 0 commands in progress
> [10793.425992] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc2-657: state disconnected; 0 commands in progress
> [10793.427069] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc1-658: state disconnected; 0 commands in progress
> [10798.310258] scsi host60: ib_srp: Sending CM DREQ failed
> [10798.310968] scsi host60: ib_srp: Sending CM DREQ failed
> [10798.311772] scsi host60: ib_srp: Sending CM DREQ failed
> [10798.312243] scsi host60: ib_srp: Sending CM DREQ failed
> [10798.369850] scsi host58: ib_srp: Sending CM DREQ failed
> [10798.369851] scsi host61: ib_srp: Sending CM DREQ failed
> [10798.369854] scsi host61: ib_srp: Sending CM DREQ failed
> [10798.369856] scsi host61: ib_srp: Sending CM DREQ failed
> [10798.369858] scsi host61: ib_srp: Sending CM DREQ failed
> [10798.371422] scsi host58: ib_srp: Sending CM DREQ failed
> [10798.372037] scsi host58: ib_srp: Sending CM DREQ failed
> [10798.372303] scsi host58: ib_srp: Sending CM DREQ failed
> [10798.389849] scsi host59: ib_srp: Sending CM DREQ failed
> [10798.390128] scsi host59: ib_srp: Sending CM DREQ failed
> [10798.390336] scsi host59: ib_srp: Sending CM DREQ failed
> [10798.390551] scsi host59: ib_srp: Sending CM DREQ failed
> [10798.429858] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ...
> [10798.430095] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress
> [10798.430326] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d51-630: state disconnected; 0 commands in progress
> [10798.430632] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-618: state disconnected; 0 commands in progress
> [10798.431060] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-625: state disconnected; 0 commands in progress
> [10798.431522] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-631: state disconnected; 0 commands in progress
> [10798.432378] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc2-657: state disconnected; 0 commands in progress
> [10798.432607] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc1-658: state disconnected; 0 commands in progress
> [10798.593380] scsi host60: ib_srp: REJ received
> [10798.594137] scsi host60:   REJ reason 0x8
> [10798.595188] scsi host60: reconnect attempt 1 failed (-104)
> [10798.609908] scsi host61: ib_srp: REJ received
> [10798.610291] scsi host61:   REJ reason 0x8
> [10798.610828] scsi host61: reconnect attempt 1 failed (-104)
> [10798.612989] scsi host58: ib_srp: REJ received
> [10798.613366] scsi host58:   REJ reason 0x8
> [10798.613752] scsi host58: reconnect attempt 1 failed (-104)
> [10798.629912] [575]: scst: scst_cmd_thread:5499:Processing thread disk139_0 finished
> [10798.633770] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10798.634435] [600]: scst: scst_cmd_thread:5499:Processing thread disk146_0 finished
> [10798.661741] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10798.661942] [589]: scst: scst_cmd_thread:5499:Processing thread disk144_0 finished
> [10798.665387] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10798.665829] [604]: scst: scst_cmd_thread:5499:Processing thread disk148_0 finished
> [10798.677166] [2407]: scst: scst_cmd_thread:5499:Processing thread disk169_0 finished
> [10798.680754] [2409]: scst: scst_cmd_thread:5499:Processing thread disk170_0 finished
> [10798.683539] scsi host59: ib_srp: REJ received
> [10798.683665] scsi host59:   REJ reason 0x8
> [10798.683743] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0
> [10798.684732] scsi host59: reconnect attempt 1 failed (-104)
> [10803.350085] fast_io_fail_tmo expired for SRP port-60:1 / host60.
> [10803.354685] sd 60:0:0:0: rejecting I/O to offline device
> [10803.358943] sd 60:0:0:0: [sdbx] killing request
> [10803.359803] sd 60:0:0:0: rejecting I/O to offline device
> [10803.360324] sd 60:0:0:0: [sdbx] killing request
> [10803.361332] sd 60:0:0:0: rejecting I/O to offline device
> [10803.362291] sd 60:0:0:0: rejecting I/O to offline device
> [10803.362295] device-mapper: multipath: Failing path 68:176.
> [10803.363197] sd 60:0:0:0: rejecting I/O to offline device
> [10803.363526] sd 60:0:0:0: rejecting I/O to offline device
> [10803.363812] sd 60:0:0:0: rejecting I/O to offline device
> [10803.363983] sd 60:0:0:0: rejecting I/O to offline device
> [10803.364277] sd 60:0:0:0: rejecting I/O to offline device
> [10803.364472] sd 60:0:0:0: [sdbx] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
> [10803.364762] sd 60:0:0:0: [sdbx] CDB: Read(10) 28 00 00 00 1a 58 00 00 08 00
> [10803.365071] blk_update_request: I/O error, dev sdbx, sector 6744
> [10803.365345] sd 60:0:0:0: [sdbx] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
> [10803.365650] sd 60:0:0:0: [sdbx] CDB: Read(10) 28 00 00 00 5c 38 00 00 08 00
> [10803.365915] blk_update_request: I/O error, dev sdbx, sector 23608
> [10803.399704] fast_io_fail_tmo expired for SRP port-59:1 / host59.
> [10803.399708] fast_io_fail_tmo expired for SRP port-58:1 / host58.
> [10803.400381] sd 59:0:0:0: rejecting I/O to offline device
> [10803.400514] fast_io_fail_tmo expired for SRP port-61:1 / host61.
> [10803.401006] sd 59:0:0:0: rejecting I/O to offline device
> [10803.401008] device-mapper: multipath: Failing path 68:192.
> [10803.401448] sd 59:0:0:0: rejecting I/O to offline device
> [10803.401660] sd 59:0:0:0: rejecting I/O to offline device
> [10803.401808] sd 59:0:0:0: rejecting I/O to offline device
> [10803.401999] sd 59:0:0:0: rejecting I/O to offline device
> [10803.402200] sd 59:0:0:0: rejecting I/O to offline device
> [10803.402387] sd 59:0:0:0: rejecting I/O to offline device
> [10803.402586] sd 59:0:0:0: rejecting I/O to offline device
> [10803.429706] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ...
> [10803.430138] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress
> [10803.904798] device-mapper: multipath: Failing path 68:160.
> [10803.909315] device-mapper: multipath: Failing path 68:208.
> [10808.429596] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ...
> [10808.430002] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress
> [10808.609582] scsi host60: ib_srp: Sending CM DREQ failed
> [10808.609770] scsi host60: ib_srp: Sending CM DREQ failed
> [10808.609938] scsi host60: ib_srp: Sending CM DREQ failed
> [10808.610378] scsi host60: ib_srp: Sending CM DREQ failed
> [10808.639569] scsi host58: ib_srp: Sending CM DREQ failed
> [10808.639765] scsi host58: ib_srp: Sending CM DREQ failed
> [10808.640049] scsi host58: ib_srp: Sending CM DREQ failed
> [10808.640217] scsi host58: ib_srp: Sending CM DREQ failed
> [10808.660821] scsi host61: ib_srp: Sending CM DREQ failed
> [10808.661050] scsi host61: ib_srp: Sending CM DREQ failed
> [10808.661794] scsi host61: ib_srp: Sending CM DREQ failed
> [10808.661961] scsi host61: ib_srp: Sending CM DREQ failed
> [10808.690042] [580]: scst: scst_cmd_thread:5499:Processing thread disk141_0 finished
> [10808.693028] [8149]: scst_suspend_activity:965:suspend_count 0
> [10808.693136] [8149]: scst: scst_suspend_activity:990:Waiting for 10 active commands to complete...
> [10808.709633] scsi host59: ib_srp: Sending CM DREQ failed
> [10808.709729] scsi host59: ib_srp: Sending CM DREQ failed
> [10808.709807] scsi host59: ib_srp: Sending CM DREQ failed
> [10808.709892] scsi host59: ib_srp: Sending CM DREQ failed
> [10808.730162] scsi host60: ib_srp: REJ received
> [10808.730248] scsi host60:   REJ reason 0x8
> [10808.730374] scsi host60: reconnect attempt 2 failed (-104)
> [10808.755642] scsi host61: ib_srp: REJ received
> [10808.755727] scsi host61:   REJ reason 0x8
> [10808.755874] scsi host61: reconnect attempt 2 failed (-104)
> [10808.757781] scsi host58: ib_srp: REJ received
> [10808.757861] scsi host58:   REJ reason 0x8
> [10808.757991] scsi host58: reconnect attempt 2 failed (-104)
> [10808.789571] scsi host59: ib_srp: REJ received
> [10808.789652] scsi host59:   REJ reason 0x8
> [10808.789734] scsi host59: reconnect attempt 2 failed (-104)
> [10809.402402] [550]: scst_translate_lun:4957:FLAG SUSPENDED set, skipping
> [10809.559516] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753]
> [10809.559635] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg
> [10809.570166] irq event stamp: 29993531
> [10809.570263] hardirqs last  enabled at (29993530): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50
> [10809.570452] hardirqs last disabled at (29993531): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> [10809.570640] softirqs last  enabled at (10133236): [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst]
> [10809.570851] softirqs last disabled at (10133267): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10809.571039] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G           O L  4.4.6-debug+ #2
> [10809.571140] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> [10809.571239] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000
> [10809.571338] RIP: 0010:[<ffffffff815781b3>]  [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50
> [10809.571530] RSP: 0018:ffff88046f363d68  EFLAGS: 00000292
> [10809.571626] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000
> [10809.571722] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00
> [10809.571818] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000
> [10809.571914] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00
> [10809.572010] R13: ffffea0008392800 R14: ffff88017fc01c00 R15: ffff88017fc1ba80
> [10809.572106] FS:  0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000
> [10809.572204] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10809.572298] CR2: 00005633130dd1a8 CR3: 0000000001a0a000 CR4: 00000000001406e0
> [10809.572392] Stack:
> [10809.572488]  ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6
> [10809.572943]  ffff88046f363db8 ffffffff8157940f 0000000100300001 ffff88020e4a3fc0
> [10809.573396]  ffff88020e4a3fc0 0000000100000282 0000000000000292 ffff880439376698
> [10809.573845] Call Trace:
> [10809.573938]  <IRQ> 
> [10809.574034]  [<ffffffff811640d6>] __slab_free+0xc6/0x270
> [10809.574236]  [<ffffffff8157940f>] ? retint_kernel+0x10/0x10
> [10809.574338]  [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160
> [10809.574441]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10809.574543]  [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90
> [10809.574644]  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
> [10809.574746]  [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0
> [10809.574847]  [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90
> [10809.574948]  [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0
> [10809.575049]  [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0
> [10809.575149]  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
> [10809.575248]  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> [10809.575349]  [<ffffffff810065a5>] do_IRQ+0x65/0x110
> [10809.575450]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10809.575549]  [<ffffffff815793c9>] common_interrupt+0x89/0x89
> [10809.575650]  <EOI> 
> [10809.575748]  [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20
> [10809.575949]  [<ffffffff8117bad7>] ? put_object+0x7/0x40
> [10809.576051]  [<ffffffff8117bb48>] ? __delete_object+0x38/0x70
> [10809.576153]  [<ffffffff8117bb98>] delete_object_full+0x18/0x20
> [10809.576254]  [<ffffffff8156dab1>] kmemleak_free+0x31/0x80
> [10809.576355]  [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160
> [10809.576456]  [<ffffffff81113282>] mempool_free_slab+0x12/0x20
> [10809.576558]  [<ffffffff81113406>] mempool_free+0x26/0x80
> [10809.576659]  [<ffffffff8127e8c1>] bvec_free+0x31/0x40
> [10809.576758]  [<ffffffff8127ee54>] bio_free+0x34/0x60
> [10809.576858]  [<ffffffff8127ee9e>] bio_put+0x1e/0x30
> [10809.576962]  [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk]
> [10809.577067]  [<ffffffff812800da>] bio_endio+0x3a/0x60
> [10809.577168]  [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd]
> [10809.577273]  [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630
> [10809.577378]  [<ffffffff81284342>] generic_make_request+0xd2/0x290
> [10809.577482]  [<ffffffff81284562>] submit_bio+0x62/0x140
> [10809.577587]  [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk]
> [10809.577697]  [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk]
> [10809.577807]  [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk]
> [10809.577915]  [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk]
> [10809.578024]  [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk]
> [10809.578130]  [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0
> [10809.578253]  [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst]
> [10809.578374]  [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst]
> [10809.578500]  [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst]
> [10809.578629]  [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst]
> [10809.578750]  [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst]
> [10809.578876]  [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst]
> [10809.578982]  [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220
> [10809.579084]  [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220
> [10809.579202]  [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst]
> [10809.579319]  [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10
> [10809.579435]  [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst]
> [10809.579560]  [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst]
> [10809.579680]  [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60
> [10809.579797]  [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst]
> [10809.579906]  [<ffffffff8107591a>] kthread+0xea/0x100
> [10809.580009]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10809.580114]  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> [10809.580219]  [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0
> [10809.580321] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb 

Re: RCU stall

[Bart Van Assche]

On 03/22/2016 01:45 PM, Paul E. McKenney wrote:
> You are getting a soft lockup as well as an RCU CPU stall warning, so
> it looks like something is taking a very long time in blk_done_softirq().
>
> You have multiple occurrences at different times, so it looks to be
> a long time as opposed to an infinite time.  Are you perhaps doing
> something that would make a huge amount of work for blk_done_softirq()?
>
> See Documentation/RCU/stallwarn.txt in the kernel source tree for more
> info on how to debug this sort of thing.

Hello Paul,

None of the drivers involved in the test I ran contain RCU code that has 
been changed recently. The block and SCSI subsystems processes I/O 
completions in softirq context but until last week I hadn't seen any RCU 
lockup complaints when I ran an SRP test against a kernel with lockdep 
and several other kernel debugging options enabled. This is why I sent 
an e-mail to you. I have read Documentation/RCU/stallwarn.txt after I 
received your reply but this didn't provide me any clue about where to 
look for the root cause. Any further help would be appreciated.

Thanks,

Bart.

Re: RCU stall

[Paul E. McKenney]

On Tue, Mar 22, 2016 at 04:53:26PM -0700, Bart Van Assche wrote:
> On 03/22/2016 01:45 PM, Paul E. McKenney wrote:
> >You are getting a soft lockup as well as an RCU CPU stall warning, so
> >it looks like something is taking a very long time in blk_done_softirq().
> >
> >You have multiple occurrences at different times, so it looks to be
> >a long time as opposed to an infinite time.  Are you perhaps doing
> >something that would make a huge amount of work for blk_done_softirq()?
> >
> >See Documentation/RCU/stallwarn.txt in the kernel source tree for more
> >info on how to debug this sort of thing.
> 
> Hello Paul,
> 
> None of the drivers involved in the test I ran contain RCU code that
> has been changed recently. The block and SCSI subsystems processes
> I/O completions in softirq context but until last week I hadn't seen
> any RCU lockup complaints when I ran an SRP test against a kernel
> with lockdep and several other kernel debugging options enabled.
> This is why I sent an e-mail to you. I have read
> Documentation/RCU/stallwarn.txt after I received your reply but this
> didn't provide me any clue about where to look for the root cause.
> Any further help would be appreciated.

My suggestion would be to check the block/SCSI softirq handler for
event traces.  If there are some, enable them and see what the loop
is doing.  Documentation/trace/ftrace.txt describes how to enable
existing event tracing.

If there is no event tracing, consider adding some in your local
view.  Failing that, there is always printk().  ;-)

Or perhaps you have some sort of debug setup.

Either way, the next step is to work out why that CPU is spending
so much time in that loop.

							Thanx, Paul

Re: RCU stall

[Paul E. McKenney]

On Tue, Mar 22, 2016 at 06:59:32PM -0700, Paul E. McKenney wrote:
> On Tue, Mar 22, 2016 at 04:53:26PM -0700, Bart Van Assche wrote:
> > On 03/22/2016 01:45 PM, Paul E. McKenney wrote:
> > >You are getting a soft lockup as well as an RCU CPU stall warning, so
> > >it looks like something is taking a very long time in blk_done_softirq().
> > >
> > >You have multiple occurrences at different times, so it looks to be
> > >a long time as opposed to an infinite time.  Are you perhaps doing
> > >something that would make a huge amount of work for blk_done_softirq()?
> > >
> > >See Documentation/RCU/stallwarn.txt in the kernel source tree for more
> > >info on how to debug this sort of thing.
> > 
> > Hello Paul,
> > 
> > None of the drivers involved in the test I ran contain RCU code that
> > has been changed recently. The block and SCSI subsystems processes
> > I/O completions in softirq context but until last week I hadn't seen
> > any RCU lockup complaints when I ran an SRP test against a kernel
> > with lockdep and several other kernel debugging options enabled.
> > This is why I sent an e-mail to you. I have read
> > Documentation/RCU/stallwarn.txt after I received your reply but this
> > didn't provide me any clue about where to look for the root cause.
> > Any further help would be appreciated.
> 
> My suggestion would be to check the block/SCSI softirq handler for
> event traces.  If there are some, enable them and see what the loop
> is doing.  Documentation/trace/ftrace.txt describes how to enable
> existing event tracing.
> 
> If there is no event tracing, consider adding some in your local
> view.  Failing that, there is always printk().  ;-)
> 
> Or perhaps you have some sort of debug setup.
> 
> Either way, the next step is to work out why that CPU is spending
> so much time in that loop.

And the dmesg leading up to the stall might have some clues.

Note that a soft lockup triggered at 10509.568010, well before the RCU
CPU stall warning..  And you have a second soft lockup at 10537.567212,
with the same funtion scsi_request_fn() at the top of the stack in both
stack traces.  That function has a nice big "for (;;)" loop that does
not appear to have any iteration-limiting mechanism.  (Though perhaps
there is such a mechanism implemented in one of the called functions,
but that would be something for you to look into.)  As you saw when
reading stallwarn.txt, having a too-long loop in the kernel is a good
way to get RCU CPU stall warnings.

Also, before the soft lockups, you have a bunch of FAIL indications
and other nasty-looking error messages.  Might you have some sort of
configuration or hardware problem?

							Thanx, Paul

Re: RCU stall

[Bart Van Assche]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

On 03/22/2016 07:29 PM, Paul E. McKenney wrote:
> Note that a soft lockup triggered at 10509.568010, well before the RCU
> CPU stall warning..  And you have a second soft lockup at 10537.567212,
> with the same funtion scsi_request_fn() at the top of the stack in both
> stack traces.  That function has a nice big "for (;;)" loop that does
> not appear to have any iteration-limiting mechanism.

Hello Paul,

Your feedback is really appreciated. I have started with testing the 
four attached patches. With the tests I ran so far these patches were 
sufficient to avoid any soft lockup or RCU stall complaints. I will 
submit these patches to the appropriate maintainers once I have finished 
testing these patches.

Bart.

[-- Attachment #2: 0001-IB-cm-Fix-a-recently-introduced-locking-bug.patch --]
[-- Type: text/x-patch, Size: 3020 bytes --]

>From 0f72b28329342346980ae99c69d19b7adb0123bc Mon Sep 17 00:00:00 2001
From: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Thu, 24 Mar 2016 11:01:14 -0700
Subject: [PATCH 1/4] IB/cm: Fix a recently introduced locking bug

ib_cm_notify() can be called from interrupt context. Hence do not
reenable interrupts unconditionally in cm_establish().

This patch avoids that lockdep reports the following warning:

WARNING: CPU: 0 PID: 23317 at kernel/locking/lockdep.c:2624 trace _hardirqs_on_caller+0x112/0x1b0
DEBUG_LOCKS_WARN_ON(current->hardirq_context)
Call Trace:
 <IRQ>  [<ffffffff812bd0e5>] dump_stack+0x67/0x92
 [<ffffffff81056f21>] __warn+0xc1/0xe0
 [<ffffffff81056f8a>] warn_slowpath_fmt+0x4a/0x50
 [<ffffffff810a5932>] trace_hardirqs_on_caller+0x112/0x1b0
 [<ffffffff810a59dd>] trace_hardirqs_on+0xd/0x10
 [<ffffffff815992c7>] _raw_spin_unlock_irq+0x27/0x40
 [<ffffffffa0382e9c>] ib_cm_notify+0x25c/0x290 [ib_cm]
 [<ffffffffa068fbc1>] srpt_qp_event+0xa1/0xf0 [ib_srpt]
 [<ffffffffa04efb97>] mlx4_ib_qp_event+0x67/0xd0 [mlx4_ib]
 [<ffffffffa034ec0a>] mlx4_qp_event+0x5a/0xc0 [mlx4_core]
 [<ffffffffa03365f8>] mlx4_eq_int+0x3d8/0xcf0 [mlx4_core]
 [<ffffffffa0336f9c>] mlx4_msi_x_interrupt+0xc/0x20 [mlx4_core]
 [<ffffffff810b0914>] handle_irq_event_percpu+0x64/0x100
 [<ffffffff810b09e4>] handle_irq_event+0x34/0x60
 [<ffffffff810b3a6a>] handle_edge_irq+0x6a/0x150
 [<ffffffff8101ad05>] handle_irq+0x15/0x20
 [<ffffffff8101a66c>] do_IRQ+0x5c/0x110
 [<ffffffff8159a2c9>] common_interrupt+0x89/0x89
 [<ffffffff81297a17>] blk_run_queue_async+0x37/0x40
 [<ffffffffa0163e53>] rq_completed+0x43/0x70 [dm_mod]
 [<ffffffffa0164896>] dm_softirq_done+0x176/0x280 [dm_mod]
 [<ffffffff812a26c2>] blk_done_softirq+0x52/0x90
 [<ffffffff8105bc1f>] __do_softirq+0x10f/0x230
 [<ffffffff8105bec8>] irq_exit+0xa8/0xb0
 [<ffffffff8103653e>] smp_trace_call_function_single_interrupt+0x2e/0x30
 [<ffffffff81036549>] smp_call_function_single_interrupt+0x9/0x10
 [<ffffffff8159a959>] call_function_single_interrupt+0x89/0x90
 <EOI>

Fixes: commit be4b499323bf ("IB/cm: Do not queue work to a device that's going away")
Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Erez Shitrit <erezsh@mellanox.com>
Cc: Doug Ledford <dledford@redhat.com>
Cc: stable <stable@vger.kernel.org> # v4.2+
---
 drivers/infiniband/core/cm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
index 1d92e09..c995255 100644
--- a/drivers/infiniband/core/cm.c
+++ b/drivers/infiniband/core/cm.c
@@ -3452,14 +3452,14 @@ static int cm_establish(struct ib_cm_id *cm_id)
 	work->cm_event.event = IB_CM_USER_ESTABLISHED;
 
 	/* Check if the device started its remove_one */
-	spin_lock_irq(&cm.lock);
+	spin_lock_irqsave(&cm.lock, flags);
 	if (!cm_dev->going_down) {
 		queue_delayed_work(cm.wq, &work->work, 0);
 	} else {
 		kfree(work);
 		ret = -ENODEV;
 	}
-	spin_unlock_irq(&cm.lock);
+	spin_unlock_irqrestore(&cm.lock, flags);
 
 out:
 	return ret;
-- 
2.7.3


[-- Attachment #3: 0002-kernel-kthread.c-Avoid-CPU-lockups.patch --]
[-- Type: text/x-patch, Size: 1765 bytes --]

>From 5fd6aedadc04d102cd261507ff61071071455fb6 Mon Sep 17 00:00:00 2001
From: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Thu, 24 Mar 2016 12:04:01 -0700
Subject: [PATCH 2/4] kernel/kthread.c: Avoid CPU lockups

Avoid that complaints similar to the one below are reported against
a debug kernel:

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kdmwork-25 4:2:23313]
irq event stamp: 16320042
hardirqs last  enabled at (16320041): [<ffffffff815992c7>] _raw_spin_unlock_irq+0x27/0x40
hardirqs last disabled at (16320042): [<ffff8803ffbe3cd8>] 0xffff8803ffbe3cd8
softirqs last  enabled at (16319960): [<ffffffff8105bcdb>] __do_softirq+0x1cb/0x230
softirqs last disabled at (16319715): [<ffffffff8105bec8>] irq_exit+0xa8/0xb0
CPU: 1 PID: 23313 Comm: kdmwork-254:2
RIP: 0010:[<ffffffff815992cf>]  [<ffffffff815992cf>] _raw_spin_unlock_irq+0x2f/0x40
Call Trace:
 [<ffffffff813f702f>] scsi_request_fn+0x11f/0x630
 [<ffffffff812977fe>] __blk_run_queue+0x2e/0x40
 [<ffffffff81297065>] __elv_add_request+0x75/0x1f0
 [<ffffffff8129ba91>] blk_insert_cloned_request+0x101/0x190
 [<ffffffffa0189f1a>] map_request+0x16a/0x1b0 [dm_mod]
 [<ffffffffa0189f7d>] map_tio_request+0x1d/0x40 [dm_mod]
 [<ffffffff81077812>] kthread_worker_fn+0x82/0x1a0
 [<ffffffff8107771a>] kthread+0xea/0x100
 [<ffffffff81599b72>] ret_from_fork+0x22/0x40

Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
---
 kernel/kthread.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/kthread.c b/kernel/kthread.c
index 9ff173d..516ca6b 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -593,6 +593,7 @@ repeat:
 	if (work) {
 		__set_current_state(TASK_RUNNING);
 		work->func(work);
+		cond_resched_rcu_qs();
 	} else if (!freezing(current))
 		schedule();
 
-- 
2.7.3


[-- Attachment #4: 0003-block-Limit-work-processed-in-softirq-context.patch --]
[-- Type: text/x-patch, Size: 2854 bytes --]

>From 44985e4b2f3124bf87e84a4c7572efa00ac28d3b Mon Sep 17 00:00:00 2001
From: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Wed, 23 Mar 2016 17:14:57 -0700
Subject: [PATCH 3/4] block: Limit work processed in softirq context

Avoid that complaints like the one below are reported against a
debug kernel:

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [disk11_0:2708]
irq event stamp: 17120809
hardirqs last  enabled at (17120808): [<ffffffff81599191>] _raw_spin_unlock_irqrestore+0x31/0x50
hardirqs last disabled at (17120809): [<ffff88046f223bd0>] 0xffff88046f223bd0
softirqs last  enabled at (17120794): [<ffffffffa060aa67>] scst_check_blocked_dev+0x77/0x1c0 [scst]
softirqs last disabled at (17120795): [<ffffffff8159acbc>] do_softirq_own_stack+0x1c/0x30
RIP: 0010:[<ffffffff81599193>]  [<ffffffff81599193>] _raw_spin_unlock_irqrestore+0x33/0x50
Call Trace:
 <IRQ>
 [<ffffffff81171450>] free_debug_processing+0x270/0x3a0
 [<ffffffff8117277a>] __slab_free+0x17a/0x2c0
 [<ffffffff81172a74>] kmem_cache_free+0x1b4/0x1d0
 [<ffffffff8111a6c2>] mempool_free_slab+0x12/0x20
 [<ffffffff8111a846>] mempool_free+0x26/0x80
 [<ffffffff81294cb9>] bio_free+0x49/0x60
 [<ffffffff81294cee>] bio_put+0x1e/0x30
 [<ffffffffa0199d31>] end_clone_bio+0x21/0x70 [dm_mod]
 [<ffffffff81294d52>] bio_endio+0x52/0x60
 [<ffffffff8129aaec>] blk_update_request+0x7c/0x2a0
 [<ffffffff813f4a8e>] scsi_end_request+0x2e/0x1d0
 [<ffffffff813f7674>] scsi_io_completion+0xb4/0x610
 [<ffffffff813ee79a>] scsi_finish_command+0xca/0x120
 [<ffffffff813f6ef0>] scsi_softirq_done+0x120/0x140
 [<ffffffff812a26e6>] blk_done_softirq+0x76/0x90
 [<ffffffff8105bc1f>] __do_softirq+0x10f/0x230
 [<ffffffff8159acbc>] do_softirq_own_stack+0x1c/0x30
 <EOI>
---
 block/blk-softirq.c | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/block/blk-softirq.c b/block/blk-softirq.c
index 53b1737..d739949 100644
--- a/block/blk-softirq.c
+++ b/block/blk-softirq.c
@@ -20,20 +20,26 @@ static DEFINE_PER_CPU(struct list_head, blk_cpu_done);
  */
 static void blk_done_softirq(struct softirq_action *h)
 {
-	struct list_head *cpu_list, local_list;
+	struct list_head *cpu_list = this_cpu_ptr(&blk_cpu_done);
+	struct request *rq;
+	int i;
 
 	local_irq_disable();
-	cpu_list = this_cpu_ptr(&blk_cpu_done);
-	list_replace_init(cpu_list, &local_list);
-	local_irq_enable();
-
-	while (!list_empty(&local_list)) {
-		struct request *rq;
-
-		rq = list_entry(local_list.next, struct request, ipi_list);
+	for (i = 64; i > 0; i--) {
+		if (list_empty(cpu_list))
+			goto done;
+		rq = list_first_entry(cpu_list, struct request, ipi_list);
 		list_del_init(&rq->ipi_list);
+		local_irq_enable();
+
 		rq->q->softirq_done_fn(rq);
+
+		local_irq_disable();
 	}
+	raise_softirq_irqoff(BLOCK_SOFTIRQ);
+
+done:
+	local_irq_enable();
 }
 
 #ifdef CONFIG_SMP
-- 
2.7.3


[-- Attachment #5: 0004-Avoid-that-I-O-completion-processing-triggers-lockup.patch --]
[-- Type: text/x-patch, Size: 4862 bytes --]

>From a73fdf710b98922fa02d464af96b499ea2740832 Mon Sep 17 00:00:00 2001
From: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Wed, 23 Mar 2016 14:38:13 -0700
Subject: [PATCH 4/4] Avoid that I/O completion processing triggers lockup
 complaints

Avoid that I/O completion processing triggers the following complaints
if kernel debug options that slow down the kernel significantly are
enabled:

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
irq event stamp: 15233868
hardirqs last  enabled at (15233867): [<ffffffff81578167>] _raw_spin_unlock_irq+0x27/0x40
hardirqs last disabled at (15233868): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
softirqs last  enabled at (15233850): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
softirqs last disabled at (15233743): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
CPU: 3 PID: 358 Comm: kdmwork-254:2
RIP: 0010:[<ffffffff8157816f>]  [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40
Call Trace:
 [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600
 [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40
 [<ffffffff81281175>] __elv_add_request+0x75/0x1f0
 [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190
 [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod]
 [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
 [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
 [<ffffffff8107591a>] kthread+0xea/0x100
 [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70

INFO: rcu_sched self-detected stall on CPU
 3-...: (6497 ticks this GP) idle=fb9/140000000000002/0 softirq=2044956/2045037 fqs=5414
  (t=6500 jiffies g=219289 c=219288 q=7233211)
Task dump for CPU 3:
kdmwork-254:2   R  running task        0   358      2 0x00000008
Call Trace:
 <IRQ>  [<ffffffff8108195f>] sched_show_task+0xbf/0x150
 [<ffffffff81084742>] dump_cpu_task+0x32/0x40
 [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0
 [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730
 [<ffffffff810bc1c4>] update_process_times+0x34/0x60
 [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50
 [<ffffffff810cb148>] tick_sched_timer+0x38/0x70
 [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0
 [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0
 [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90
 [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
 [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
 [<ffffffff811640d6>] __slab_free+0xc6/0x270
 [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
 [<ffffffff811d0a12>] kiocb_free+0x32/0x40
 [<ffffffff811d1b75>] aio_complete+0x1e5/0x3c0
 [<ffffffff811bef45>] dio_complete+0x75/0x1d0
 [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130
 [<ffffffff812800da>] bio_endio+0x3a/0x60
 [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
 [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod]
 [<ffffffff812800da>] bio_endio+0x3a/0x60
 [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
 [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0
 [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610
 [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120
 [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140
 [<ffffffff8128c742>] blk_done_softirq+0x72/0x90
 [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
 [<ffffffff8105a748>] irq_exit+0xa8/0xb0
 [<ffffffff810065a5>] do_IRQ+0x65/0x110
 [<ffffffff815793c9>] common_interrupt+0x89/0x89
 <EOI>
 [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
 [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
 [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
 [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
 [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
 [<ffffffff8107591a>] kthread+0xea/0x100
 [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70

Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 drivers/scsi/scsi_lib.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 8106515..8f264a0 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1770,13 +1770,14 @@ static void scsi_request_fn(struct request_queue *q)
 	struct Scsi_Host *shost;
 	struct scsi_cmnd *cmd;
 	struct request *req;
+	int i;
 
 	/*
-	 * To start with, we keep looping until the queue is empty, or until
-	 * the host is no longer able to accept any more requests.
+	 * Loop until the queue is empty, until the host is no longer able to
+	 * accept any more requests or until 64 requests have been processed.
 	 */
 	shost = sdev->host;
-	for (;;) {
+	for (i = 64; i > 0; i--) {
 		int rtn;
 		/*
 		 * get next queueable request.  We do this early to make sure
@@ -1861,6 +1862,9 @@ static void scsi_request_fn(struct request_queue *q)
 		spin_lock_irq(q->queue_lock);
 	}
 
+	if (unlikely(i == 0))
+		blk_delay_queue(q, 0);
+
 	return;
 
  host_not_ready:
-- 
2.7.3

Re: RCU stall

[Paul E. McKenney]

On Thu, Mar 24, 2016 at 01:24:02PM -0700, Bart Van Assche wrote:
> On 03/22/2016 07:29 PM, Paul E. McKenney wrote:
> >Note that a soft lockup triggered at 10509.568010, well before the RCU
> >CPU stall warning..  And you have a second soft lockup at 10537.567212,
> >with the same funtion scsi_request_fn() at the top of the stack in both
> >stack traces.  That function has a nice big "for (;;)" loop that does
> >not appear to have any iteration-limiting mechanism.
> 
> Hello Paul,
> 
> Your feedback is really appreciated. I have started with testing the
> four attached patches. With the tests I ran so far these patches
> were sufficient to avoid any soft lockup or RCU stall complaints. I
> will submit these patches to the appropriate maintainers once I have
> finished testing these patches.

Hello, Bart,

Glad it helped!

The patches look sane to me, but that should be taken with a large
quantity of salt given that I know very little about Linux's storage
subsystem.

							Thanx, Paul

> Bart.

> >From 0f72b28329342346980ae99c69d19b7adb0123bc Mon Sep 17 00:00:00 2001
> From: Bart Van Assche <bart.vanassche@sandisk.com>
> Date: Thu, 24 Mar 2016 11:01:14 -0700
> Subject: [PATCH 1/4] IB/cm: Fix a recently introduced locking bug
> 
> ib_cm_notify() can be called from interrupt context. Hence do not
> reenable interrupts unconditionally in cm_establish().
> 
> This patch avoids that lockdep reports the following warning:
> 
> WARNING: CPU: 0 PID: 23317 at kernel/locking/lockdep.c:2624 trace _hardirqs_on_caller+0x112/0x1b0
> DEBUG_LOCKS_WARN_ON(current->hardirq_context)
> Call Trace:
>  <IRQ>  [<ffffffff812bd0e5>] dump_stack+0x67/0x92
>  [<ffffffff81056f21>] __warn+0xc1/0xe0
>  [<ffffffff81056f8a>] warn_slowpath_fmt+0x4a/0x50
>  [<ffffffff810a5932>] trace_hardirqs_on_caller+0x112/0x1b0
>  [<ffffffff810a59dd>] trace_hardirqs_on+0xd/0x10
>  [<ffffffff815992c7>] _raw_spin_unlock_irq+0x27/0x40
>  [<ffffffffa0382e9c>] ib_cm_notify+0x25c/0x290 [ib_cm]
>  [<ffffffffa068fbc1>] srpt_qp_event+0xa1/0xf0 [ib_srpt]
>  [<ffffffffa04efb97>] mlx4_ib_qp_event+0x67/0xd0 [mlx4_ib]
>  [<ffffffffa034ec0a>] mlx4_qp_event+0x5a/0xc0 [mlx4_core]
>  [<ffffffffa03365f8>] mlx4_eq_int+0x3d8/0xcf0 [mlx4_core]
>  [<ffffffffa0336f9c>] mlx4_msi_x_interrupt+0xc/0x20 [mlx4_core]
>  [<ffffffff810b0914>] handle_irq_event_percpu+0x64/0x100
>  [<ffffffff810b09e4>] handle_irq_event+0x34/0x60
>  [<ffffffff810b3a6a>] handle_edge_irq+0x6a/0x150
>  [<ffffffff8101ad05>] handle_irq+0x15/0x20
>  [<ffffffff8101a66c>] do_IRQ+0x5c/0x110
>  [<ffffffff8159a2c9>] common_interrupt+0x89/0x89
>  [<ffffffff81297a17>] blk_run_queue_async+0x37/0x40
>  [<ffffffffa0163e53>] rq_completed+0x43/0x70 [dm_mod]
>  [<ffffffffa0164896>] dm_softirq_done+0x176/0x280 [dm_mod]
>  [<ffffffff812a26c2>] blk_done_softirq+0x52/0x90
>  [<ffffffff8105bc1f>] __do_softirq+0x10f/0x230
>  [<ffffffff8105bec8>] irq_exit+0xa8/0xb0
>  [<ffffffff8103653e>] smp_trace_call_function_single_interrupt+0x2e/0x30
>  [<ffffffff81036549>] smp_call_function_single_interrupt+0x9/0x10
>  [<ffffffff8159a959>] call_function_single_interrupt+0x89/0x90
>  <EOI>
> 
> Fixes: commit be4b499323bf ("IB/cm: Do not queue work to a device that's going away")
> Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Cc: Erez Shitrit <erezsh@mellanox.com>
> Cc: Doug Ledford <dledford@redhat.com>
> Cc: stable <stable@vger.kernel.org> # v4.2+
> ---
>  drivers/infiniband/core/cm.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
> index 1d92e09..c995255 100644
> --- a/drivers/infiniband/core/cm.c
> +++ b/drivers/infiniband/core/cm.c
> @@ -3452,14 +3452,14 @@ static int cm_establish(struct ib_cm_id *cm_id)
>  	work->cm_event.event = IB_CM_USER_ESTABLISHED;
> 
>  	/* Check if the device started its remove_one */
> -	spin_lock_irq(&cm.lock);
> +	spin_lock_irqsave(&cm.lock, flags);
>  	if (!cm_dev->going_down) {
>  		queue_delayed_work(cm.wq, &work->work, 0);
>  	} else {
>  		kfree(work);
>  		ret = -ENODEV;
>  	}
> -	spin_unlock_irq(&cm.lock);
> +	spin_unlock_irqrestore(&cm.lock, flags);
> 
>  out:
>  	return ret;
> -- 
> 2.7.3
> 

> >From 5fd6aedadc04d102cd261507ff61071071455fb6 Mon Sep 17 00:00:00 2001
> From: Bart Van Assche <bart.vanassche@sandisk.com>
> Date: Thu, 24 Mar 2016 12:04:01 -0700
> Subject: [PATCH 2/4] kernel/kthread.c: Avoid CPU lockups
> 
> Avoid that complaints similar to the one below are reported against
> a debug kernel:
> 
> NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kdmwork-25 4:2:23313]
> irq event stamp: 16320042
> hardirqs last  enabled at (16320041): [<ffffffff815992c7>] _raw_spin_unlock_irq+0x27/0x40
> hardirqs last disabled at (16320042): [<ffff8803ffbe3cd8>] 0xffff8803ffbe3cd8
> softirqs last  enabled at (16319960): [<ffffffff8105bcdb>] __do_softirq+0x1cb/0x230
> softirqs last disabled at (16319715): [<ffffffff8105bec8>] irq_exit+0xa8/0xb0
> CPU: 1 PID: 23313 Comm: kdmwork-254:2
> RIP: 0010:[<ffffffff815992cf>]  [<ffffffff815992cf>] _raw_spin_unlock_irq+0x2f/0x40
> Call Trace:
>  [<ffffffff813f702f>] scsi_request_fn+0x11f/0x630
>  [<ffffffff812977fe>] __blk_run_queue+0x2e/0x40
>  [<ffffffff81297065>] __elv_add_request+0x75/0x1f0
>  [<ffffffff8129ba91>] blk_insert_cloned_request+0x101/0x190
>  [<ffffffffa0189f1a>] map_request+0x16a/0x1b0 [dm_mod]
>  [<ffffffffa0189f7d>] map_tio_request+0x1d/0x40 [dm_mod]
>  [<ffffffff81077812>] kthread_worker_fn+0x82/0x1a0
>  [<ffffffff8107771a>] kthread+0xea/0x100
>  [<ffffffff81599b72>] ret_from_fork+0x22/0x40
> 
> Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
> ---
>  kernel/kthread.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/kernel/kthread.c b/kernel/kthread.c
> index 9ff173d..516ca6b 100644
> --- a/kernel/kthread.c
> +++ b/kernel/kthread.c
> @@ -593,6 +593,7 @@ repeat:
>  	if (work) {
>  		__set_current_state(TASK_RUNNING);
>  		work->func(work);
> +		cond_resched_rcu_qs();
>  	} else if (!freezing(current))
>  		schedule();
> 
> -- 
> 2.7.3
> 

> >From 44985e4b2f3124bf87e84a4c7572efa00ac28d3b Mon Sep 17 00:00:00 2001
> From: Bart Van Assche <bart.vanassche@sandisk.com>
> Date: Wed, 23 Mar 2016 17:14:57 -0700
> Subject: [PATCH 3/4] block: Limit work processed in softirq context
> 
> Avoid that complaints like the one below are reported against a
> debug kernel:
> 
> NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [disk11_0:2708]
> irq event stamp: 17120809
> hardirqs last  enabled at (17120808): [<ffffffff81599191>] _raw_spin_unlock_irqrestore+0x31/0x50
> hardirqs last disabled at (17120809): [<ffff88046f223bd0>] 0xffff88046f223bd0
> softirqs last  enabled at (17120794): [<ffffffffa060aa67>] scst_check_blocked_dev+0x77/0x1c0 [scst]
> softirqs last disabled at (17120795): [<ffffffff8159acbc>] do_softirq_own_stack+0x1c/0x30
> RIP: 0010:[<ffffffff81599193>]  [<ffffffff81599193>] _raw_spin_unlock_irqrestore+0x33/0x50
> Call Trace:
>  <IRQ>
>  [<ffffffff81171450>] free_debug_processing+0x270/0x3a0
>  [<ffffffff8117277a>] __slab_free+0x17a/0x2c0
>  [<ffffffff81172a74>] kmem_cache_free+0x1b4/0x1d0
>  [<ffffffff8111a6c2>] mempool_free_slab+0x12/0x20
>  [<ffffffff8111a846>] mempool_free+0x26/0x80
>  [<ffffffff81294cb9>] bio_free+0x49/0x60
>  [<ffffffff81294cee>] bio_put+0x1e/0x30
>  [<ffffffffa0199d31>] end_clone_bio+0x21/0x70 [dm_mod]
>  [<ffffffff81294d52>] bio_endio+0x52/0x60
>  [<ffffffff8129aaec>] blk_update_request+0x7c/0x2a0
>  [<ffffffff813f4a8e>] scsi_end_request+0x2e/0x1d0
>  [<ffffffff813f7674>] scsi_io_completion+0xb4/0x610
>  [<ffffffff813ee79a>] scsi_finish_command+0xca/0x120
>  [<ffffffff813f6ef0>] scsi_softirq_done+0x120/0x140
>  [<ffffffff812a26e6>] blk_done_softirq+0x76/0x90
>  [<ffffffff8105bc1f>] __do_softirq+0x10f/0x230
>  [<ffffffff8159acbc>] do_softirq_own_stack+0x1c/0x30
>  <EOI>
> ---
>  block/blk-softirq.c | 24 +++++++++++++++---------
>  1 file changed, 15 insertions(+), 9 deletions(-)
> 
> diff --git a/block/blk-softirq.c b/block/blk-softirq.c
> index 53b1737..d739949 100644
> --- a/block/blk-softirq.c
> +++ b/block/blk-softirq.c
> @@ -20,20 +20,26 @@ static DEFINE_PER_CPU(struct list_head, blk_cpu_done);
>   */
>  static void blk_done_softirq(struct softirq_action *h)
>  {
> -	struct list_head *cpu_list, local_list;
> +	struct list_head *cpu_list = this_cpu_ptr(&blk_cpu_done);
> +	struct request *rq;
> +	int i;
> 
>  	local_irq_disable();
> -	cpu_list = this_cpu_ptr(&blk_cpu_done);
> -	list_replace_init(cpu_list, &local_list);
> -	local_irq_enable();
> -
> -	while (!list_empty(&local_list)) {
> -		struct request *rq;
> -
> -		rq = list_entry(local_list.next, struct request, ipi_list);
> +	for (i = 64; i > 0; i--) {
> +		if (list_empty(cpu_list))
> +			goto done;
> +		rq = list_first_entry(cpu_list, struct request, ipi_list);
>  		list_del_init(&rq->ipi_list);
> +		local_irq_enable();
> +
>  		rq->q->softirq_done_fn(rq);
> +
> +		local_irq_disable();
>  	}
> +	raise_softirq_irqoff(BLOCK_SOFTIRQ);
> +
> +done:
> +	local_irq_enable();
>  }
> 
>  #ifdef CONFIG_SMP
> -- 
> 2.7.3
> 

> >From a73fdf710b98922fa02d464af96b499ea2740832 Mon Sep 17 00:00:00 2001
> From: Bart Van Assche <bart.vanassche@sandisk.com>
> Date: Wed, 23 Mar 2016 14:38:13 -0700
> Subject: [PATCH 4/4] Avoid that I/O completion processing triggers lockup
>  complaints
> 
> Avoid that I/O completion processing triggers the following complaints
> if kernel debug options that slow down the kernel significantly are
> enabled:
> 
> NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358]
> irq event stamp: 15233868
> hardirqs last  enabled at (15233867): [<ffffffff81578167>] _raw_spin_unlock_irq+0x27/0x40
> hardirqs last disabled at (15233868): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90
> softirqs last  enabled at (15233850): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230
> softirqs last disabled at (15233743): [<ffffffff8105a748>] irq_exit+0xa8/0xb0
> CPU: 3 PID: 358 Comm: kdmwork-254:2
> RIP: 0010:[<ffffffff8157816f>]  [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40
> Call Trace:
>  [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600
>  [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40
>  [<ffffffff81281175>] __elv_add_request+0x75/0x1f0
>  [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190
>  [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod]
>  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
>  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
>  [<ffffffff8107591a>] kthread+0xea/0x100
>  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> 
> INFO: rcu_sched self-detected stall on CPU
>  3-...: (6497 ticks this GP) idle=fb9/140000000000002/0 softirq=2044956/2045037 fqs=5414
>   (t=6500 jiffies g=219289 c=219288 q=7233211)
> Task dump for CPU 3:
> kdmwork-254:2   R  running task        0   358      2 0x00000008
> Call Trace:
>  <IRQ>  [<ffffffff8108195f>] sched_show_task+0xbf/0x150
>  [<ffffffff81084742>] dump_cpu_task+0x32/0x40
>  [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0
>  [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730
>  [<ffffffff810bc1c4>] update_process_times+0x34/0x60
>  [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50
>  [<ffffffff810cb148>] tick_sched_timer+0x38/0x70
>  [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0
>  [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0
>  [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90
>  [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10
>  [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90
>  [<ffffffff811640d6>] __slab_free+0xc6/0x270
>  [<ffffffff811643d9>] kmem_cache_free+0x159/0x160
>  [<ffffffff811d0a12>] kiocb_free+0x32/0x40
>  [<ffffffff811d1b75>] aio_complete+0x1e5/0x3c0
>  [<ffffffff811bef45>] dio_complete+0x75/0x1d0
>  [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130
>  [<ffffffff812800da>] bio_endio+0x3a/0x60
>  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
>  [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod]
>  [<ffffffff812800da>] bio_endio+0x3a/0x60
>  [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0
>  [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0
>  [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610
>  [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120
>  [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140
>  [<ffffffff8128c742>] blk_done_softirq+0x72/0x90
>  [<ffffffff8105a49f>] __do_softirq+0x10f/0x230
>  [<ffffffff8105a748>] irq_exit+0xa8/0xb0
>  [<ffffffff810065a5>] do_IRQ+0x65/0x110
>  [<ffffffff815793c9>] common_interrupt+0x89/0x89
>  <EOI>
>  [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath]
>  [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath]
>  [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod]
>  [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod]
>  [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0
>  [<ffffffff8107591a>] kthread+0xea/0x100
>  [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70
> 
> Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  drivers/scsi/scsi_lib.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 8106515..8f264a0 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -1770,13 +1770,14 @@ static void scsi_request_fn(struct request_queue *q)
>  	struct Scsi_Host *shost;
>  	struct scsi_cmnd *cmd;
>  	struct request *req;
> +	int i;
> 
>  	/*
> -	 * To start with, we keep looping until the queue is empty, or until
> -	 * the host is no longer able to accept any more requests.
> +	 * Loop until the queue is empty, until the host is no longer able to
> +	 * accept any more requests or until 64 requests have been processed.
>  	 */
>  	shost = sdev->host;
> -	for (;;) {
> +	for (i = 64; i > 0; i--) {
>  		int rtn;
>  		/*
>  		 * get next queueable request.  We do this early to make sure
> @@ -1861,6 +1862,9 @@ static void scsi_request_fn(struct request_queue *q)
>  		spin_lock_irq(q->queue_lock);
>  	}
> 
> +	if (unlikely(i == 0))
> +		blk_delay_queue(q, 0);
> +
>  	return;
> 
>   host_not_ready:
> -- 
> 2.7.3
> 

Re: rcu stall.

[Paul E. McKenney]

On Wed, Apr 20, 2011 at 10:36:16AM +0200, Ingo Molnar wrote:
> 
> * Dave Jones <davej@redhat.com> wrote:
> 
> > Machine was under heavy load (300 or so running processes
> > calling random system calls). The rcu stall detector kicked in,
> > spewed this, and then the machine completely locked up.
> 
> Without having looked at it in detail, isnt this a lockup somewhere in the 
> wireless code:
> 
> >   [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
> >   [<ffffffff81082e4f>] ? arch_local_irq_restore+0x6/0xd
> >   [<ffffffff81084df8>] lock_acquired+0x20f/0x21e
> >   [<ffffffff814be9cc>] _raw_spin_lock+0x62/0x6a
> >   [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
> >   [<ffffffff814bf215>] ? _raw_spin_unlock+0x28/0x2c
> >   [<ffffffff8114f842>] simple_release_fs+0x22/0x57
> >   [<ffffffff811f53e9>] debugfs_remove_recursive+0x11f/0x16b
> >   [<ffffffffa037adf3>] ieee80211_debugfs_key_remove+0x1f/0x2e [mac80211]
> >   [<ffffffffa0373e7a>] __ieee80211_key_destroy+0x61/0x6d [mac80211]
> >   [<ffffffffa0374250>] ieee80211_key_link+0x12c/0x165 [mac80211]
> >   [<ffffffffa036b90e>] ieee80211_add_key+0xfb/0x133 [mac80211]
> >   [<ffffffffa0277ff4>] nl80211_new_key+0xe5/0x106 [cfg80211]
> >   [<ffffffffa026d2c5>] ? cfg80211_get_dev_from_ifindex+0x72/0x7a [cfg80211]
> >   [<ffffffff81422244>] genl_rcv_msg+0x1dc/0x207
> >   [<ffffffff81422068>] ? genl_rcv+0x2d/0x2d
> >   [<ffffffff81421c69>] netlink_rcv_skb+0x43/0x8f
> >   [<ffffffff81422061>] genl_rcv+0x26/0x2d
> >   [<ffffffff8142176a>] netlink_unicast+0xec/0x156
> >   [<ffffffff81421a53>] netlink_sendmsg+0x27f/0x2c0
> >   [<ffffffff813ed78c>] __sock_sendmsg+0x69/0x75
> >   [<ffffffff813ed905>] sock_sendmsg+0xa1/0xb6
> >   [<ffffffff81086c30>] ? lock_release+0x181/0x18e
> >   [<ffffffff81100de0>] ? might_fault+0xa5/0xac
> >   [<ffffffff81100d97>] ? might_fault+0x5c/0xac
> >   [<ffffffff813ec8e4>] ? copy_from_user+0x2f/0x31
> >   [<ffffffff813f707a>] ? copy_from_user+0x2f/0x31
> >   [<ffffffff813f7370>] ? verify_iovec+0x52/0xa6
> >   [<ffffffff813eece3>] sys_sendmsg+0x23a/0x2b8
> >   [<ffffffff81086d29>] ? lock_acquire+0xec/0xfb
> >   [<ffffffff81086c30>] ? lock_release+0x181/0x18e
> >   [<ffffffff8114b7d7>] ? mntput+0x26/0x28
> >   [<ffffffff811343bc>] ? fput+0x1e6/0x1f5
> >   [<ffffffff8113ba95>] ? path_put+0x1f/0x23
> >   [<ffffffff810a9f23>] ? audit_syscall_entry+0x11c/0x148
> >   [<ffffffff81255e4e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> >   [<ffffffff814c5d82>] system_call_fastpath+0x16/0x1b
> 
> RCU stall detector is simply the first thing that noticed the hang. Enabling 
> the regular lockup detector would probably have resulted in a similar looking 
> hang.

Hello, Dave,

In case the lockup detector would get you better information, you can
prevent RCU from checking for stall by setting the rcu_cpu_stall_suppress
module parameter to 1, either as a boot parameter or via sysfs.

							Thanx, Paul

Re: rcu stall.

[Ingo Molnar]

* Dave Jones <davej@redhat.com> wrote:

> Machine was under heavy load (300 or so running processes
> calling random system calls). The rcu stall detector kicked in,
> spewed this, and then the machine completely locked up.

Without having looked at it in detail, isnt this a lockup somewhere in the 
wireless code:

>   [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
>   [<ffffffff81082e4f>] ? arch_local_irq_restore+0x6/0xd
>   [<ffffffff81084df8>] lock_acquired+0x20f/0x21e
>   [<ffffffff814be9cc>] _raw_spin_lock+0x62/0x6a
>   [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
>   [<ffffffff814bf215>] ? _raw_spin_unlock+0x28/0x2c
>   [<ffffffff8114f842>] simple_release_fs+0x22/0x57
>   [<ffffffff811f53e9>] debugfs_remove_recursive+0x11f/0x16b
>   [<ffffffffa037adf3>] ieee80211_debugfs_key_remove+0x1f/0x2e [mac80211]
>   [<ffffffffa0373e7a>] __ieee80211_key_destroy+0x61/0x6d [mac80211]
>   [<ffffffffa0374250>] ieee80211_key_link+0x12c/0x165 [mac80211]
>   [<ffffffffa036b90e>] ieee80211_add_key+0xfb/0x133 [mac80211]
>   [<ffffffffa0277ff4>] nl80211_new_key+0xe5/0x106 [cfg80211]
>   [<ffffffffa026d2c5>] ? cfg80211_get_dev_from_ifindex+0x72/0x7a [cfg80211]
>   [<ffffffff81422244>] genl_rcv_msg+0x1dc/0x207
>   [<ffffffff81422068>] ? genl_rcv+0x2d/0x2d
>   [<ffffffff81421c69>] netlink_rcv_skb+0x43/0x8f
>   [<ffffffff81422061>] genl_rcv+0x26/0x2d
>   [<ffffffff8142176a>] netlink_unicast+0xec/0x156
>   [<ffffffff81421a53>] netlink_sendmsg+0x27f/0x2c0
>   [<ffffffff813ed78c>] __sock_sendmsg+0x69/0x75
>   [<ffffffff813ed905>] sock_sendmsg+0xa1/0xb6
>   [<ffffffff81086c30>] ? lock_release+0x181/0x18e
>   [<ffffffff81100de0>] ? might_fault+0xa5/0xac
>   [<ffffffff81100d97>] ? might_fault+0x5c/0xac
>   [<ffffffff813ec8e4>] ? copy_from_user+0x2f/0x31
>   [<ffffffff813f707a>] ? copy_from_user+0x2f/0x31
>   [<ffffffff813f7370>] ? verify_iovec+0x52/0xa6
>   [<ffffffff813eece3>] sys_sendmsg+0x23a/0x2b8
>   [<ffffffff81086d29>] ? lock_acquire+0xec/0xfb
>   [<ffffffff81086c30>] ? lock_release+0x181/0x18e
>   [<ffffffff8114b7d7>] ? mntput+0x26/0x28
>   [<ffffffff811343bc>] ? fput+0x1e6/0x1f5
>   [<ffffffff8113ba95>] ? path_put+0x1f/0x23
>   [<ffffffff810a9f23>] ? audit_syscall_entry+0x11c/0x148
>   [<ffffffff81255e4e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
>   [<ffffffff814c5d82>] system_call_fastpath+0x16/0x1b

RCU stall detector is simply the first thing that noticed the hang. Enabling 
the regular lockup detector would probably have resulted in a similar looking 
hang.

Thanks,

	Ingo

rcu stall.

[Dave Jones]

Machine was under heavy load (300 or so running processes
calling random system calls). The rcu stall detector kicked in,
spewed this, and then the machine completely locked up.

	Dave

 INFO: rcu_sched_state detected stall on CPU 0 (t=65000 jiffies)
 sending NMI to all CPUs:
 NMI backtrace for cpu 0
 CPU 0 
 Modules linked in: snd_seq_dummy ip6_queue nfnetlink scsi_transport_iscsi ip_queue ipt_ULOG can_raw hidp inet_diag tun can_bcm sctp libcrc32c bnep rfcomm cmtp kernelcapi ipx p8022 p8023 af_key rose ax25 phonet appletalk psnap llc can rds pppoe pppox ppp_generic slhc decnet irda crc_ccitt af_802154 atm fuse nfsd lockd nfs_acl auth_rpcgss sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 snd_hda_codec_realtek iwlagn snd_hda_intel snd_hda_codec snd_hwdep snd_seq mac80211 snd_seq_device snd_pcm uvcvideo snd_timer btusb bluetooth snd e1000e videodev cfg80211 microcode joydev v4l2_compat_ioctl32 iTCO_wdt pcspkr soundcore iTCO_vendor_support i2c_i801 snd_page_alloc sony_laptop rfkill tpm_infineon wmi uinput ipv6 sdhci_pci sdhci mmc_core firewire_ohci firewire_core yenta_socket crc_itu_t nouveau i915 ttm drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
 
 Pid: 983, comm: wpa_supplicant Not tainted 2.6.39-rc4+ #2 Sony Corporation VGN-Z540N/VAIO
 RIP: 0010:[<ffffffff81256fea>]  [<ffffffff81256fea>] __bitmap_empty+0x56/0x58
 RSP: 0018:ffff8800baa03dc8  EFLAGS: 00000046
 RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000040
 RDX: 0000000000000001 RSI: 0000000000000200 RDI: ffffffff81b5fa50
 RBP: ffff8800baa03dc8 R08: 0000000000000002 R09: 0000000000000000
 R10: 0000ffff00066c0a R11: 0000000000000001 R12: ffffffff81a32000
 R13: ffffffff81a32100 R14: ffff8800baa03f50 R15: ffffffff810819c4
 FS:  00007f17cd9ac7e0(0000) GS:ffff8800baa00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007fbf04ce3010 CR3: 00000000a1427000 CR4: 00000000000006f0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 Process wpa_supplicant (pid: 983, threadinfo ffff8800a6fbe000, task ffff8800a6c20000)
 Stack:
  ffff8800baa03de8 ffffffff810218a0 0000000000000000 ffff8800babd0680
  ffff8800baa03e38 ffffffff810bbf77 0000000000000096 0000000000000000
  ffff8800baa03e18 0000000000000000 0000000000000000 0000000000000000
 Call Trace:
  <IRQ> 
  [<ffffffff810218a0>] arch_trigger_all_cpu_backtrace+0x68/0x88
  [<ffffffff810bbf77>] __rcu_pending+0x8c/0x321
  [<ffffffff810819c4>] ? tick_nohz_handler+0xdf/0xdf
  [<ffffffff810bc668>] rcu_check_callbacks+0x88/0xb9
  [<ffffffff81064ad8>] update_process_times+0x3f/0x75
  [<ffffffff81081a39>] tick_sched_timer+0x75/0x9e
  [<ffffffff8107653f>] __run_hrtimer+0xcf/0x15a
  [<ffffffff81076cef>] hrtimer_interrupt+0xe1/0x1c2
  [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
  [<ffffffff814c794e>] smp_apic_timer_interrupt+0x79/0x8c
  [<ffffffff814c67d3>] apic_timer_interrupt+0x13/0x20
  <EOI> 
  [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
  [<ffffffff81082e4f>] ? arch_local_irq_restore+0x6/0xd
  [<ffffffff81084df8>] lock_acquired+0x20f/0x21e
  [<ffffffff814be9cc>] _raw_spin_lock+0x62/0x6a
  [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
  [<ffffffff814bf215>] ? _raw_spin_unlock+0x28/0x2c
  [<ffffffff8114f842>] simple_release_fs+0x22/0x57
  [<ffffffff811f53e9>] debugfs_remove_recursive+0x11f/0x16b
  [<ffffffffa037adf3>] ieee80211_debugfs_key_remove+0x1f/0x2e [mac80211]
  [<ffffffffa0373e7a>] __ieee80211_key_destroy+0x61/0x6d [mac80211]
  [<ffffffffa0374250>] ieee80211_key_link+0x12c/0x165 [mac80211]
  [<ffffffffa036b90e>] ieee80211_add_key+0xfb/0x133 [mac80211]
  [<ffffffffa0277ff4>] nl80211_new_key+0xe5/0x106 [cfg80211]
  [<ffffffffa026d2c5>] ? cfg80211_get_dev_from_ifindex+0x72/0x7a [cfg80211]
  [<ffffffff81422244>] genl_rcv_msg+0x1dc/0x207
  [<ffffffff81422068>] ? genl_rcv+0x2d/0x2d
  [<ffffffff81421c69>] netlink_rcv_skb+0x43/0x8f
  [<ffffffff81422061>] genl_rcv+0x26/0x2d
  [<ffffffff8142176a>] netlink_unicast+0xec/0x156
  [<ffffffff81421a53>] netlink_sendmsg+0x27f/0x2c0
  [<ffffffff813ed78c>] __sock_sendmsg+0x69/0x75
  [<ffffffff813ed905>] sock_sendmsg+0xa1/0xb6
  [<ffffffff81086c30>] ? lock_release+0x181/0x18e
  [<ffffffff81100de0>] ? might_fault+0xa5/0xac
  [<ffffffff81100d97>] ? might_fault+0x5c/0xac
  [<ffffffff813ec8e4>] ? copy_from_user+0x2f/0x31
  [<ffffffff813f707a>] ? copy_from_user+0x2f/0x31
  [<ffffffff813f7370>] ? verify_iovec+0x52/0xa6
  [<ffffffff813eece3>] sys_sendmsg+0x23a/0x2b8
  [<ffffffff81086d29>] ? lock_acquire+0xec/0xfb
  [<ffffffff81086c30>] ? lock_release+0x181/0x18e
  [<ffffffff8114b7d7>] ? mntput+0x26/0x28
  [<ffffffff811343bc>] ? fput+0x1e6/0x1f5
  [<ffffffff8113ba95>] ? path_put+0x1f/0x23
  [<ffffffff810a9f23>] ? audit_syscall_entry+0x11c/0x148
  [<ffffffff81255e4e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff814c5d82>] system_call_fastpath+0x16/0x1b
 Code: 2a 89 f0 4c 63 c2 41 b9 40 00 00 00 99 41 f7 f9 b8 01 00 00 00 88 d1 48 d3 e0 48 ff c8 4a 85 04 c7 0f 94 c0 0f b6 c0 eb 02 31 c0 <5d> c3 89 f0 55 b9 40 00 00 00 99 f7 f9 48 89 e5 31 d2 eb 0b 48 
 Call Trace:
  <IRQ>  [<ffffffff810218a0>] arch_trigger_all_cpu_backtrace+0x68/0x88
  [<ffffffff810bbf77>] __rcu_pending+0x8c/0x321
  [<ffffffff810819c4>] ? tick_nohz_handler+0xdf/0xdf
  [<ffffffff810bc668>] rcu_check_callbacks+0x88/0xb9
  [<ffffffff81064ad8>] update_process_times+0x3f/0x75
  [<ffffffff81081a39>] tick_sched_timer+0x75/0x9e
  [<ffffffff8107653f>] __run_hrtimer+0xcf/0x15a
  [<ffffffff81076cef>] hrtimer_interrupt+0xe1/0x1c2
  [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
  [<ffffffff814c794e>] smp_apic_timer_interrupt+0x79/0x8c
  [<ffffffff814c67d3>] apic_timer_interrupt+0x13/0x20
  <EOI>  [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
  [<ffffffff81082e4f>] ? arch_local_irq_restore+0x6/0xd
  [<ffffffff81084df8>] lock_acquired+0x20f/0x21e
  [<ffffffff814be9cc>] _raw_spin_lock+0x62/0x6a
  [<ffffffff8114f842>] ? simple_release_fs+0x22/0x57
  [<ffffffff814bf215>] ? _raw_spin_unlock+0x28/0x2c
  [<ffffffff8114f842>] simple_release_fs+0x22/0x57
  [<ffffffff811f53e9>] debugfs_remove_recursive+0x11f/0x16b
  [<ffffffffa037adf3>] ieee80211_debugfs_key_remove+0x1f/0x2e [mac80211]
  [<ffffffffa0373e7a>] __ieee80211_key_destroy+0x61/0x6d [mac80211]
  [<ffffffffa0374250>] ieee80211_key_link+0x12c/0x165 [mac80211]
  [<ffffffffa036b90e>] ieee80211_add_key+0xfb/0x133 [mac80211]
  [<ffffffffa0277ff4>] nl80211_new_key+0xe5/0x106 [cfg80211]
  [<ffffffffa026d2c5>] ? cfg80211_get_dev_from_ifindex+0x72/0x7a [cfg80211]
  [<ffffffff81422244>] genl_rcv_msg+0x1dc/0x207
  [<ffffffff81422068>] ? genl_rcv+0x2d/0x2d
  [<ffffffff81421c69>] netlink_rcv_skb+0x43/0x8f
  [<ffffffff81422061>] genl_rcv+0x26/0x2d
  [<ffffffff8142176a>] netlink_unicast+0xec/0x156
  [<ffffffff81421a53>] netlink_sendmsg+0x27f/0x2c0
  [<ffffffff813ed78c>] __sock_sendmsg+0x69/0x75
  [<ffffffff813ed905>] sock_sendmsg+0xa1/0xb6
  [<ffffffff81086c30>] ? lock_release+0x181/0x18e
  [<ffffffff81100de0>] ? might_fault+0xa5/0xac
  [<ffffffff81100d97>] ? might_fault+0x5c/0xac
  [<ffffffff813ec8e4>] ? copy_from_user+0x2f/0x31
  [<ffffffff813f707a>] ? copy_from_user+0x2f/0x31
  [<ffffffff813f7370>] ? verify_iovec+0x52/0xa6
  [<ffffffff813eece3>] sys_sendmsg+0x23a/0x2b8
  [<ffffffff81086d29>] ? lock_acquire+0xec/0xfb
  [<ffffffff81086c30>] ? lock_release+0x181/0x18e
  [<ffffffff8114b7d7>] ? mntput+0x26/0x28
  [<ffffffff811343bc>] ? fput+0x1e6/0x1f5
  [<ffffffff8113ba95>] ? path_put+0x1f/0x23
  [<ffffffff810a9f23>] ? audit_syscall_entry+0x11c/0x148
  [<ffffffff81255e4e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff814c5d82>] system_call_fastpath+0x16/0x1b
 NMI backtrace for cpu 1
 CPU 1 
 Modules linked in: snd_seq_dummy ip6_queue nfnetlink scsi_transport_iscsi ip_queue ipt_ULOG can_raw hidp inet_diag tun can_bcm sctp libcrc32c bnep rfcomm cmtp kernelcapi ipx p8022 p8023 af_key rose ax25 phonet appletalk psnap llc can rds pppoe pppox ppp_generic slhc decnet irda crc_ccitt af_802154 atm fuse nfsd lockd nfs_acl auth_rpcgss sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 snd_hda_codec_realtek iwlagn snd_hda_intel snd_hda_codec snd_hwdep snd_seq mac80211 snd_seq_device snd_pcm uvcvideo snd_timer btusb bluetooth snd e1000e videodev cfg80211 microcode joydev v4l2_compat_ioctl32 iTCO_wdt pcspkr soundcore iTCO_vendor_support i2c_i801 snd_page_alloc sony_laptop rfkill tpm_infineon wmi uinput ipv6 sdhci_pci sdhci mmc_core firewire_ohci firewire_core yenta_socket crc_itu_t nouveau i915 ttm drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
 
 Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #2 Sony Corporation VGN-Z540N/VAIO
 RIP: 0010:[<ffffffff8124c28c>]  [<ffffffff8124c28c>] cpumask_next_and+0x2c/0x39
 RSP: 0018:ffff8800bac03b80  EFLAGS: 00000202
 RAX: 0000000000000001 RBX: ffff8800bac0fc80 RCX: 0000000000000000
 RDX: 0000000000000000 RSI: 0000000000000200 RDI: 0000000000000200
 RBP: ffff8800bac03b90 R08: 0000000000000000 R09: ffff8800bac0f848
 R10: 0000000000706071 R11: ffff8800b57fc760 R12: ffff8800bac0f848
 R13: 0000000000000001 R14: ffff8800bac0f830 R15: 00000000ffffffff
 FS:  0000000000000000(0000) GS:ffff8800bac00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
 CR2: 00007f5cd7f66010 CR3: 00000000a17aa000 CR4: 00000000000006e0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 Process kworker/0:0 (pid: 0, threadinfo ffff8800b5004000, task ffff8800b57fc760)
 Stack:
  ffff8800bac0f890 00000000ffffffff ffff8800bac03d50 ffffffff8105106d
  0000000000000000 0000001e3229a026 ffff8800bac03e00 00000000001d4340
  00000000001d4340 ffff8800bac0fc80 0000000000000000 0000000000000002
 Call Trace:
  <IRQ> 
  [<ffffffff8105106d>] find_busiest_group+0x256/0x8bc
  [<ffffffff8105175c>] load_balance+0x89/0x654
  [<ffffffff81086c30>] ? lock_release+0x181/0x18e
  [<ffffffff81042ace>] ? rcu_read_unlock+0x21/0x23
  [<ffffffff81051e19>] rebalance_domains+0xf2/0x168
  [<ffffffff812525f6>] ? timerqueue_add+0x86/0xa8
  [<ffffffff8107bf84>] ? timekeeping_get_ns+0x18/0x3a
  [<ffffffff81051ed5>] run_rebalance_domains+0x46/0x108
  [<ffffffff8105ccff>] __do_softirq+0xf4/0x1da
  [<ffffffff8108161f>] ? tick_program_event+0x1f/0x21
  [<ffffffff814c701c>] call_softirq+0x1c/0x30
  [<ffffffff8100abc9>] do_softirq+0x4b/0xa2
  [<ffffffff8105cff5>] irq_exit+0x5d/0xa8
  [<ffffffff814c7953>] smp_apic_timer_interrupt+0x7e/0x8c
  [<ffffffff814c67d3>] apic_timer_interrupt+0x13/0x20
  <EOI> 
  [<ffffffff8104303e>] ? set_next_entity+0x46/0x9c
  [<ffffffff812cd370>] ? acpi_idle_enter_c1+0x9b/0xbe
  [<ffffffff812cc7e2>] ? arch_local_irq_enable+0xb/0xd
  [<ffffffff81087128>] ? trace_hardirqs_on+0xd/0xf
  [<ffffffff812cd375>] acpi_idle_enter_c1+0xa0/0xbe
  [<ffffffff813d1882>] cpuidle_idle_call+0xf0/0x173
  [<ffffffff81008303>] cpu_idle+0xaa/0xe4
  [<ffffffff814ac579>] start_secondary+0x232/0x234
 Code: 89 f8 48 89 e5 41 54 49 89 f4 53 48 89 d3 eb 09 0f a3 03 19 d2 85 d2 75 1a ff c0 be 00 02 00 00 4c 89 e7 48 63 d0 e8 7c 02 00 00 <3b> 05 72 5c 91 00 7c dd 5b 41 5c 5d c3 55 ff c7 48 63 d7 48 89 
 Call Trace:
  <IRQ>  [<ffffffff8105106d>] find_busiest_group+0x256/0x8bc
  [<ffffffff8105175c>] load_balance+0x89/0x654
  [<ffffffff81086c30>] ? lock_release+0x181/0x18e
  [<ffffffff81042ace>] ? rcu_read_unlock+0x21/0x23
  [<ffffffff81051e19>] rebalance_domains+0xf2/0x168
  [<ffffffff812525f6>] ? timerqueue_add+0x86/0xa8
  [<ffffffff8107bf84>] ? timekeeping_get_ns+0x18/0x3a
  [<ffffffff81051ed5>] run_rebalance_domains+0x46/0x108
  [<ffffffff8105ccff>] __do_softirq+0xf4/0x1da
  [<ffffffff8108161f>] ? tick_program_event+0x1f/0x21
  [<ffffffff814c701c>] call_softirq+0x1c/0x30
  [<ffffffff8100abc9>] do_softirq+0x4b/0xa2
  [<ffffffff8105cff5>] irq_exit+0x5d/0xa8
  [<ffffffff814c7953>] smp_apic_timer_interrupt+0x7e/0x8c
  [<ffffffff814c67d3>] apic_timer_interrupt+0x13/0x20
  <EOI>  [<ffffffff8104303e>] ? set_next_entity+0x46/0x9c
  [<ffffffff812cd370>] ? acpi_idle_enter_c1+0x9b/0xbe
  [<ffffffff812cc7e2>] ? arch_local_irq_enable+0xb/0xd
  [<ffffffff81087128>] ? trace_hardirqs_on+0xd/0xf
  [<ffffffff812cd375>] acpi_idle_enter_c1+0xa0/0xbe
  [<ffffffff813d1882>] cpuidle_idle_call+0xf0/0x173
  [<ffffffff81008303>] cpu_idle+0xaa/0xe4
  [<ffffffff814ac579>] start_secondary+0x232/0x234
 INFO: rcu_sched_state detected stalls on CPUs/tasks: { 0} (detected by 1, t=65002 jiffies)