* dm-rq queue stalls
@ 2018-01-17 18:05 Bart Van Assche
2018-01-17 18:18 ` Mike Snitzer
0 siblings, 1 reply; 12+ messages in thread
From: Bart Van Assche @ 2018-01-17 18:05 UTC (permalink / raw)
To: snitzer; +Cc: axboe, dm-devel
[-- Attachment #1: Type: text/plain, Size: 764 bytes --]
Hello Mike,
With Jens' latest for-next branch (commit 6c4f43dfecae) and with commit
6077c2d70609 ("dm rq: Avoid that request processing stalls sporadically")
reverted I see again queue stalls every now and then. An example:
# srp-test/run_tests -c -d -r 10 -t 02-mq -e kyber
Running test /home/bart/software/infiniband/srp-test/tests/02-mq ...
Test file I/O on top of multipath concurrently with logout and login (0 min; mq)
/dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000: not found
Test /home/bart/software/infiniband/srp-test/tests/02-mq failed
I have attached the output of the following command to this e-mail:
(cd /sys/kernel/debug/block && find -type f | xargs grep -aH '')
Can you have a look at this?
Thanks,
Bart.
[-- Attachment #2: blk-state --]
[-- Type: text/plain, Size: 48386 bytes --]
./dm-2/hctx0/cpu3/completed:0 0
./dm-2/hctx0/cpu3/merged:0
./dm-2/hctx0/cpu3/dispatched:0 0
./dm-2/hctx0/cpu2/completed:0 0
./dm-2/hctx0/cpu2/merged:0
./dm-2/hctx0/cpu2/dispatched:0 0
./dm-2/hctx0/cpu1/completed:25 0
./dm-2/hctx0/cpu1/merged:0
./dm-2/hctx0/cpu1/dispatched:26 0
./dm-2/hctx0/cpu0/completed:0 0
./dm-2/hctx0/cpu0/merged:0
./dm-2/hctx0/cpu0/dispatched:0 0
./dm-2/hctx0/active:0
./dm-2/hctx0/run:27
./dm-2/hctx0/queued:26
./dm-2/hctx0/dispatched: 0 6
./dm-2/hctx0/dispatched: 1 25
./dm-2/hctx0/dispatched: 2 0
./dm-2/hctx0/dispatched: 4 0
./dm-2/hctx0/dispatched: 8 0
./dm-2/hctx0/dispatched: 16 0
./dm-2/hctx0/dispatched: 32+ 0
./dm-2/hctx0/io_poll:considered=0
./dm-2/hctx0/io_poll:invoked=0
./dm-2/hctx0/io_poll:success=0
./dm-2/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0020
./dm-2/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/sched_tags:nr_tags=256
./dm-2/hctx0/sched_tags:nr_reserved_tags=0
./dm-2/hctx0/sched_tags:active_queues=0
./dm-2/hctx0/sched_tags:
./dm-2/hctx0/sched_tags:bitmap_tags:
./dm-2/hctx0/sched_tags:depth=256
./dm-2/hctx0/sched_tags:busy=1
./dm-2/hctx0/sched_tags:bits_per_word=64
./dm-2/hctx0/sched_tags:map_nr=4
./dm-2/hctx0/sched_tags:alloc_hint={83, 124, 234, 58}
./dm-2/hctx0/sched_tags:wake_batch=8
./dm-2/hctx0/sched_tags:wake_index=0
./dm-2/hctx0/sched_tags:ws={
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags:}
./dm-2/hctx0/sched_tags:round_robin=0
./dm-2/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags:nr_tags=2048
./dm-2/hctx0/tags:nr_reserved_tags=0
./dm-2/hctx0/tags:active_queues=0
./dm-2/hctx0/tags:
./dm-2/hctx0/tags:bitmap_tags:
./dm-2/hctx0/tags:depth=2048
./dm-2/hctx0/tags:busy=0
./dm-2/hctx0/tags:bits_per_word=64
./dm-2/hctx0/tags:map_nr=32
./dm-2/hctx0/tags:alloc_hint={1060, 81, 1050, 844}
./dm-2/hctx0/tags:wake_batch=8
./dm-2/hctx0/tags:wake_index=0
./dm-2/hctx0/tags:ws={
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags:}
./dm-2/hctx0/tags:round_robin=0
./dm-2/hctx0/ctx_map:00000000: 00
./dm-2/hctx0/dispatch:00000000a7224aed {.op=READ, .cmd_flags=, .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=125}
./dm-2/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./dm-2/hctx0/state:SCHED_RESTART
./dm-2/sched/starved:0
./dm-2/sched/batching:5
./dm-2/write_hints:hint0: 0
./dm-2/write_hints:hint1: 0
./dm-2/write_hints:hint2: 0
./dm-2/write_hints:hint3: 0
./dm-2/write_hints:hint4: 0
./dm-2/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
./dm-2/poll_stat:read (512 Bytes): samples=0
./dm-2/poll_stat:write (512 Bytes): samples=0
./dm-2/poll_stat:read (1024 Bytes): samples=0
./dm-2/poll_stat:write (1024 Bytes): samples=0
./dm-2/poll_stat:read (2048 Bytes): samples=0
./dm-2/poll_stat:write (2048 Bytes): samples=0
./dm-2/poll_stat:read (4096 Bytes): samples=0
./dm-2/poll_stat:write (4096 Bytes): samples=0
./dm-2/poll_stat:read (8192 Bytes): samples=0
./dm-2/poll_stat:write (8192 Bytes): samples=0
./dm-2/poll_stat:read (16384 Bytes): samples=0
./dm-2/poll_stat:write (16384 Bytes): samples=0
./dm-2/poll_stat:read (32768 Bytes): samples=0
./dm-2/poll_stat:write (32768 Bytes): samples=0
./dm-2/poll_stat:read (65536 Bytes): samples=0
./dm-2/poll_stat:write (65536 Bytes): samples=0
./dm-1/hctx0/cpu3/completed:26 0
./dm-1/hctx0/cpu3/merged:0
./dm-1/hctx0/cpu3/dispatched:26 0
./dm-1/hctx0/cpu2/completed:2 0
./dm-1/hctx0/cpu2/merged:0
./dm-1/hctx0/cpu2/dispatched:2 0
./dm-1/hctx0/cpu1/completed:2 0
./dm-1/hctx0/cpu1/merged:0
./dm-1/hctx0/cpu1/dispatched:2 0
./dm-1/hctx0/cpu0/completed:2 0
./dm-1/hctx0/cpu0/merged:0
./dm-1/hctx0/cpu0/dispatched:2 0
./dm-1/hctx0/active:0
./dm-1/hctx0/run:34
./dm-1/hctx0/queued:32
./dm-1/hctx0/dispatched: 0 7
./dm-1/hctx0/dispatched: 1 32
./dm-1/hctx0/dispatched: 2 0
./dm-1/hctx0/dispatched: 4 0
./dm-1/hctx0/dispatched: 8 0
./dm-1/hctx0/dispatched: 16 0
./dm-1/hctx0/dispatched: 32+ 0
./dm-1/hctx0/io_poll:considered=0
./dm-1/hctx0/io_poll:invoked=0
./dm-1/hctx0/io_poll:success=0
./dm-1/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/sched_tags:nr_tags=256
./dm-1/hctx0/sched_tags:nr_reserved_tags=0
./dm-1/hctx0/sched_tags:active_queues=0
./dm-1/hctx0/sched_tags:
./dm-1/hctx0/sched_tags:bitmap_tags:
./dm-1/hctx0/sched_tags:depth=256
./dm-1/hctx0/sched_tags:busy=0
./dm-1/hctx0/sched_tags:bits_per_word=64
./dm-1/hctx0/sched_tags:map_nr=4
./dm-1/hctx0/sched_tags:alloc_hint={125, 55, 142, 82}
./dm-1/hctx0/sched_tags:wake_batch=8
./dm-1/hctx0/sched_tags:wake_index=0
./dm-1/hctx0/sched_tags:ws={
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags:}
./dm-1/hctx0/sched_tags:round_robin=0
./dm-1/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags:nr_tags=2048
./dm-1/hctx0/tags:nr_reserved_tags=0
./dm-1/hctx0/tags:active_queues=0
./dm-1/hctx0/tags:
./dm-1/hctx0/tags:bitmap_tags:
./dm-1/hctx0/tags:depth=2048
./dm-1/hctx0/tags:busy=0
./dm-1/hctx0/tags:bits_per_word=64
./dm-1/hctx0/tags:map_nr=32
./dm-1/hctx0/tags:alloc_hint={478, 1303, 102, 127}
./dm-1/hctx0/tags:wake_batch=8
./dm-1/hctx0/tags:wake_index=0
./dm-1/hctx0/tags:ws={
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags:}
./dm-1/hctx0/tags:round_robin=0
./dm-1/hctx0/ctx_map:00000000: 00
./dm-1/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./dm-1/hctx0/state:
./dm-1/sched/starved:0
./dm-1/sched/batching:1
./dm-1/write_hints:hint0: 0
./dm-1/write_hints:hint1: 0
./dm-1/write_hints:hint2: 0
./dm-1/write_hints:hint3: 0
./dm-1/write_hints:hint4: 0
./dm-1/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
./dm-1/poll_stat:read (512 Bytes): samples=0
./dm-1/poll_stat:write (512 Bytes): samples=0
./dm-1/poll_stat:read (1024 Bytes): samples=0
./dm-1/poll_stat:write (1024 Bytes): samples=0
./dm-1/poll_stat:read (2048 Bytes): samples=0
./dm-1/poll_stat:write (2048 Bytes): samples=0
./dm-1/poll_stat:read (4096 Bytes): samples=0
./dm-1/poll_stat:write (4096 Bytes): samples=0
./dm-1/poll_stat:read (8192 Bytes): samples=0
./dm-1/poll_stat:write (8192 Bytes): samples=0
./dm-1/poll_stat:read (16384 Bytes): samples=0
./dm-1/poll_stat:write (16384 Bytes): samples=0
./dm-1/poll_stat:read (32768 Bytes): samples=0
./dm-1/poll_stat:write (32768 Bytes): samples=0
./dm-1/poll_stat:read (65536 Bytes): samples=0
./dm-1/poll_stat:write (65536 Bytes): samples=0
./dm-0/hctx0/cpu3/completed:13 0
./dm-0/hctx0/cpu3/merged:0
./dm-0/hctx0/cpu3/dispatched:14 0
./dm-0/hctx0/cpu2/completed:0 0
./dm-0/hctx0/cpu2/merged:0
./dm-0/hctx0/cpu2/dispatched:0 0
./dm-0/hctx0/cpu1/completed:0 0
./dm-0/hctx0/cpu1/merged:0
./dm-0/hctx0/cpu1/dispatched:0 0
./dm-0/hctx0/cpu0/completed:0 0
./dm-0/hctx0/cpu0/merged:0
./dm-0/hctx0/cpu0/dispatched:0 0
./dm-0/hctx0/active:0
./dm-0/hctx0/run:15
./dm-0/hctx0/queued:14
./dm-0/hctx0/dispatched: 0 2
./dm-0/hctx0/dispatched: 1 13
./dm-0/hctx0/dispatched: 2 0
./dm-0/hctx0/dispatched: 4 0
./dm-0/hctx0/dispatched: 8 0
./dm-0/hctx0/dispatched: 16 0
./dm-0/hctx0/dispatched: 32+ 0
./dm-0/hctx0/io_poll:considered=0
./dm-0/hctx0/io_poll:invoked=0
./dm-0/hctx0/io_poll:success=0
./dm-0/hctx0/sched_tags_bitmap:00000000: 0001 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/sched_tags:nr_tags=256
./dm-0/hctx0/sched_tags:nr_reserved_tags=0
./dm-0/hctx0/sched_tags:active_queues=0
./dm-0/hctx0/sched_tags:
./dm-0/hctx0/sched_tags:bitmap_tags:
./dm-0/hctx0/sched_tags:depth=256
./dm-0/hctx0/sched_tags:busy=1
./dm-0/hctx0/sched_tags:bits_per_word=64
./dm-0/hctx0/sched_tags:map_nr=4
./dm-0/hctx0/sched_tags:alloc_hint={11, 18, 205, 9}
./dm-0/hctx0/sched_tags:wake_batch=8
./dm-0/hctx0/sched_tags:wake_index=0
./dm-0/hctx0/sched_tags:ws={
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags:}
./dm-0/hctx0/sched_tags:round_robin=0
./dm-0/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags:nr_tags=2048
./dm-0/hctx0/tags:nr_reserved_tags=0
./dm-0/hctx0/tags:active_queues=0
./dm-0/hctx0/tags:
./dm-0/hctx0/tags:bitmap_tags:
./dm-0/hctx0/tags:depth=2048
./dm-0/hctx0/tags:busy=0
./dm-0/hctx0/tags:bits_per_word=64
./dm-0/hctx0/tags:map_nr=32
./dm-0/hctx0/tags:alloc_hint={1392, 843, 879, 1030}
./dm-0/hctx0/tags:wake_batch=8
./dm-0/hctx0/tags:wake_index=0
./dm-0/hctx0/tags:ws={
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags:}
./dm-0/hctx0/tags:round_robin=0
./dm-0/hctx0/ctx_map:00000000: 00
./dm-0/hctx0/dispatch:000000002f291aa3 {.op=READ, .cmd_flags=, .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=8}
./dm-0/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./dm-0/hctx0/state:SCHED_RESTART
./dm-0/sched/starved:0
./dm-0/sched/batching:1
./dm-0/write_hints:hint0: 0
./dm-0/write_hints:hint1: 0
./dm-0/write_hints:hint2: 0
./dm-0/write_hints:hint3: 0
./dm-0/write_hints:hint4: 0
./dm-0/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED
./dm-0/poll_stat:read (512 Bytes): samples=0
./dm-0/poll_stat:write (512 Bytes): samples=0
./dm-0/poll_stat:read (1024 Bytes): samples=0
./dm-0/poll_stat:write (1024 Bytes): samples=0
./dm-0/poll_stat:read (2048 Bytes): samples=0
./dm-0/poll_stat:write (2048 Bytes): samples=0
./dm-0/poll_stat:read (4096 Bytes): samples=0
./dm-0/poll_stat:write (4096 Bytes): samples=0
./dm-0/poll_stat:read (8192 Bytes): samples=0
./dm-0/poll_stat:write (8192 Bytes): samples=0
./dm-0/poll_stat:read (16384 Bytes): samples=0
./dm-0/poll_stat:write (16384 Bytes): samples=0
./dm-0/poll_stat:read (32768 Bytes): samples=0
./dm-0/poll_stat:write (32768 Bytes): samples=0
./dm-0/poll_stat:read (65536 Bytes): samples=0
./dm-0/poll_stat:write (65536 Bytes): samples=0
./sde/hctx3/cpu3/completed:58 12
./sde/hctx3/cpu3/merged:0
./sde/hctx3/cpu3/dispatched:58 12
./sde/hctx3/active:0
./sde/hctx3/run:39
./sde/hctx3/queued:70
./sde/hctx3/dispatched: 0 2
./sde/hctx3/dispatched: 1 36
./sde/hctx3/dispatched: 2 0
./sde/hctx3/dispatched: 4 0
./sde/hctx3/dispatched: 8 0
./sde/hctx3/dispatched: 16 0
./sde/hctx3/dispatched: 32+ 0
./sde/hctx3/io_poll:considered=0
./sde/hctx3/io_poll:invoked=0
./sde/hctx3/io_poll:success=0
./sde/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx3/tags:nr_tags=62
./sde/hctx3/tags:nr_reserved_tags=0
./sde/hctx3/tags:active_queues=0
./sde/hctx3/tags:
./sde/hctx3/tags:bitmap_tags:
./sde/hctx3/tags:depth=62
./sde/hctx3/tags:busy=0
./sde/hctx3/tags:bits_per_word=8
./sde/hctx3/tags:map_nr=8
./sde/hctx3/tags:alloc_hint={42, 53, 24, 50}
./sde/hctx3/tags:wake_batch=7
./sde/hctx3/tags:wake_index=0
./sde/hctx3/tags:ws={
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags:}
./sde/hctx3/tags:round_robin=0
./sde/hctx3/ctx_map:00000000: 00
./sde/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx3/state:
./sde/hctx2/cpu2/completed:2 3
./sde/hctx2/cpu2/merged:0
./sde/hctx2/cpu2/dispatched:2 3
./sde/hctx2/active:0
./sde/hctx2/run:3
./sde/hctx2/queued:5
./sde/hctx2/dispatched: 0 0
./sde/hctx2/dispatched: 1 3
./sde/hctx2/dispatched: 2 0
./sde/hctx2/dispatched: 4 0
./sde/hctx2/dispatched: 8 0
./sde/hctx2/dispatched: 16 0
./sde/hctx2/dispatched: 32+ 0
./sde/hctx2/io_poll:considered=0
./sde/hctx2/io_poll:invoked=0
./sde/hctx2/io_poll:success=0
./sde/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx2/tags:nr_tags=62
./sde/hctx2/tags:nr_reserved_tags=0
./sde/hctx2/tags:active_queues=0
./sde/hctx2/tags:
./sde/hctx2/tags:bitmap_tags:
./sde/hctx2/tags:depth=62
./sde/hctx2/tags:busy=0
./sde/hctx2/tags:bits_per_word=8
./sde/hctx2/tags:map_nr=8
./sde/hctx2/tags:alloc_hint={24, 37, 46, 30}
./sde/hctx2/tags:wake_batch=7
./sde/hctx2/tags:wake_index=0
./sde/hctx2/tags:ws={
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags:}
./sde/hctx2/tags:round_robin=0
./sde/hctx2/ctx_map:00000000: 00
./sde/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx2/state:
./sde/hctx1/cpu1/completed:31 47
./sde/hctx1/cpu1/merged:0
./sde/hctx1/cpu1/dispatched:31 47
./sde/hctx1/active:0
./sde/hctx1/run:76
./sde/hctx1/queued:78
./sde/hctx1/dispatched: 0 3
./sde/hctx1/dispatched: 1 71
./sde/hctx1/dispatched: 2 0
./sde/hctx1/dispatched: 4 0
./sde/hctx1/dispatched: 8 0
./sde/hctx1/dispatched: 16 0
./sde/hctx1/dispatched: 32+ 0
./sde/hctx1/io_poll:considered=0
./sde/hctx1/io_poll:invoked=0
./sde/hctx1/io_poll:success=0
./sde/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx1/tags:nr_tags=62
./sde/hctx1/tags:nr_reserved_tags=0
./sde/hctx1/tags:active_queues=0
./sde/hctx1/tags:
./sde/hctx1/tags:bitmap_tags:
./sde/hctx1/tags:depth=62
./sde/hctx1/tags:busy=0
./sde/hctx1/tags:bits_per_word=8
./sde/hctx1/tags:map_nr=8
./sde/hctx1/tags:alloc_hint={29, 0, 21, 37}
./sde/hctx1/tags:wake_batch=7
./sde/hctx1/tags:wake_index=0
./sde/hctx1/tags:ws={
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags:}
./sde/hctx1/tags:round_robin=0
./sde/hctx1/ctx_map:00000000: 00
./sde/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx1/state:
./sde/hctx0/cpu0/completed:4 6
./sde/hctx0/cpu0/merged:0
./sde/hctx0/cpu0/dispatched:4 6
./sde/hctx0/active:0
./sde/hctx0/run:6
./sde/hctx0/queued:10
./sde/hctx0/dispatched: 0 0
./sde/hctx0/dispatched: 1 6
./sde/hctx0/dispatched: 2 0
./sde/hctx0/dispatched: 4 0
./sde/hctx0/dispatched: 8 0
./sde/hctx0/dispatched: 16 0
./sde/hctx0/dispatched: 32+ 0
./sde/hctx0/io_poll:considered=0
./sde/hctx0/io_poll:invoked=0
./sde/hctx0/io_poll:success=0
./sde/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx0/tags:nr_tags=62
./sde/hctx0/tags:nr_reserved_tags=0
./sde/hctx0/tags:active_queues=0
./sde/hctx0/tags:
./sde/hctx0/tags:bitmap_tags:
./sde/hctx0/tags:depth=62
./sde/hctx0/tags:busy=0
./sde/hctx0/tags:bits_per_word=8
./sde/hctx0/tags:map_nr=8
./sde/hctx0/tags:alloc_hint={27, 47, 46, 38}
./sde/hctx0/tags:wake_batch=7
./sde/hctx0/tags:wake_index=0
./sde/hctx0/tags:ws={
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags:}
./sde/hctx0/tags:round_robin=0
./sde/hctx0/ctx_map:00000000: 00
./sde/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx0/state:
./sde/write_hints:hint0: 0
./sde/write_hints:hint1: 0
./sde/write_hints:hint2: 0
./sde/write_hints:hint3: 0
./sde/write_hints:hint4: 0
./sde/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
./sde/poll_stat:read (512 Bytes): samples=0
./sde/poll_stat:write (512 Bytes): samples=0
./sde/poll_stat:read (1024 Bytes): samples=0
./sde/poll_stat:write (1024 Bytes): samples=0
./sde/poll_stat:read (2048 Bytes): samples=0
./sde/poll_stat:write (2048 Bytes): samples=0
./sde/poll_stat:read (4096 Bytes): samples=0
./sde/poll_stat:write (4096 Bytes): samples=0
./sde/poll_stat:read (8192 Bytes): samples=0
./sde/poll_stat:write (8192 Bytes): samples=0
./sde/poll_stat:read (16384 Bytes): samples=0
./sde/poll_stat:write (16384 Bytes): samples=0
./sde/poll_stat:read (32768 Bytes): samples=0
./sde/poll_stat:write (32768 Bytes): samples=0
./sde/poll_stat:read (65536 Bytes): samples=0
./sde/poll_stat:write (65536 Bytes): samples=0
./sdd/hctx3/cpu3/completed:17 7
./sdd/hctx3/cpu3/merged:0
./sdd/hctx3/cpu3/dispatched:17 7
./sdd/hctx3/active:0
./sdd/hctx3/run:13
./sdd/hctx3/queued:24
./sdd/hctx3/dispatched: 0 4
./sdd/hctx3/dispatched: 1 9
./sdd/hctx3/dispatched: 2 0
./sdd/hctx3/dispatched: 4 0
./sdd/hctx3/dispatched: 8 0
./sdd/hctx3/dispatched: 16 0
./sdd/hctx3/dispatched: 32+ 0
./sdd/hctx3/io_poll:considered=0
./sdd/hctx3/io_poll:invoked=0
./sdd/hctx3/io_poll:success=0
./sdd/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
./sdd/hctx3/tags:nr_tags=62
./sdd/hctx3/tags:nr_reserved_tags=0
./sdd/hctx3/tags:active_queues=0
./sdd/hctx3/tags:
./sdd/hctx3/tags:bitmap_tags:
./sdd/hctx3/tags:depth=62
./sdd/hctx3/tags:busy=0
./sdd/hctx3/tags:bits_per_word=8
./sdd/hctx3/tags:map_nr=8
./sdd/hctx3/tags:alloc_hint={42, 53, 24, 50}
./sdd/hctx3/tags:wake_batch=7
./sdd/hctx3/tags:wake_index=0
./sdd/hctx3/tags:ws={
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx3/tags:}
./sdd/hctx3/tags:round_robin=0
./sdd/hctx3/ctx_map:00000000: 00
./sdd/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdd/hctx3/state:
./sdd/hctx2/cpu2/completed:20 37
./sdd/hctx2/cpu2/merged:0
./sdd/hctx2/cpu2/dispatched:20 37
./sdd/hctx2/active:0
./sdd/hctx2/run:57
./sdd/hctx2/queued:57
./sdd/hctx2/dispatched: 0 5
./sdd/hctx2/dispatched: 1 52
./sdd/hctx2/dispatched: 2 0
./sdd/hctx2/dispatched: 4 0
./sdd/hctx2/dispatched: 8 0
./sdd/hctx2/dispatched: 16 0
./sdd/hctx2/dispatched: 32+ 0
./sdd/hctx2/io_poll:considered=0
./sdd/hctx2/io_poll:invoked=0
./sdd/hctx2/io_poll:success=0
./sdd/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
./sdd/hctx2/tags:nr_tags=62
./sdd/hctx2/tags:nr_reserved_tags=0
./sdd/hctx2/tags:active_queues=0
./sdd/hctx2/tags:
./sdd/hctx2/tags:bitmap_tags:
./sdd/hctx2/tags:depth=62
./sdd/hctx2/tags:busy=0
./sdd/hctx2/tags:bits_per_word=8
./sdd/hctx2/tags:map_nr=8
./sdd/hctx2/tags:alloc_hint={24, 37, 46, 30}
./sdd/hctx2/tags:wake_batch=7
./sdd/hctx2/tags:wake_index=0
./sdd/hctx2/tags:ws={
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx2/tags:}
./sdd/hctx2/tags:round_robin=0
./sdd/hctx2/ctx_map:00000000: 00
./sdd/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdd/hctx2/state:
./sdd/hctx1/cpu1/completed:5 15
./sdd/hctx1/cpu1/merged:0
./sdd/hctx1/cpu1/dispatched:5 15
./sdd/hctx1/active:0
./sdd/hctx1/run:26
./sdd/hctx1/queued:20
./sdd/hctx1/dispatched: 0 7
./sdd/hctx1/dispatched: 1 19
./sdd/hctx1/dispatched: 2 0
./sdd/hctx1/dispatched: 4 0
./sdd/hctx1/dispatched: 8 0
./sdd/hctx1/dispatched: 16 0
./sdd/hctx1/dispatched: 32+ 0
./sdd/hctx1/io_poll:considered=0
./sdd/hctx1/io_poll:invoked=0
./sdd/hctx1/io_poll:success=0
./sdd/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
./sdd/hctx1/tags:nr_tags=62
./sdd/hctx1/tags:nr_reserved_tags=0
./sdd/hctx1/tags:active_queues=0
./sdd/hctx1/tags:
./sdd/hctx1/tags:bitmap_tags:
./sdd/hctx1/tags:depth=62
./sdd/hctx1/tags:busy=0
./sdd/hctx1/tags:bits_per_word=8
./sdd/hctx1/tags:map_nr=8
./sdd/hctx1/tags:alloc_hint={29, 0, 21, 37}
./sdd/hctx1/tags:wake_batch=7
./sdd/hctx1/tags:wake_index=0
./sdd/hctx1/tags:ws={
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx1/tags:}
./sdd/hctx1/tags:round_robin=0
./sdd/hctx1/ctx_map:00000000: 00
./sdd/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdd/hctx1/state:
./sdd/hctx0/cpu0/completed:34 8
./sdd/hctx0/cpu0/merged:0
./sdd/hctx0/cpu0/dispatched:34 8
./sdd/hctx0/active:0
./sdd/hctx0/run:43
./sdd/hctx0/queued:42
./sdd/hctx0/dispatched: 0 4
./sdd/hctx0/dispatched: 1 39
./sdd/hctx0/dispatched: 2 0
./sdd/hctx0/dispatched: 4 0
./sdd/hctx0/dispatched: 8 0
./sdd/hctx0/dispatched: 16 0
./sdd/hctx0/dispatched: 32+ 0
./sdd/hctx0/io_poll:considered=0
./sdd/hctx0/io_poll:invoked=0
./sdd/hctx0/io_poll:success=0
./sdd/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
./sdd/hctx0/tags:nr_tags=62
./sdd/hctx0/tags:nr_reserved_tags=0
./sdd/hctx0/tags:active_queues=0
./sdd/hctx0/tags:
./sdd/hctx0/tags:bitmap_tags:
./sdd/hctx0/tags:depth=62
./sdd/hctx0/tags:busy=0
./sdd/hctx0/tags:bits_per_word=8
./sdd/hctx0/tags:map_nr=8
./sdd/hctx0/tags:alloc_hint={27, 47, 46, 38}
./sdd/hctx0/tags:wake_batch=7
./sdd/hctx0/tags:wake_index=0
./sdd/hctx0/tags:ws={
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdd/hctx0/tags:}
./sdd/hctx0/tags:round_robin=0
./sdd/hctx0/ctx_map:00000000: 00
./sdd/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdd/hctx0/state:
./sdd/write_hints:hint0: 0
./sdd/write_hints:hint1: 0
./sdd/write_hints:hint2: 0
./sdd/write_hints:hint3: 0
./sdd/write_hints:hint4: 0
./sdd/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
./sdd/poll_stat:read (512 Bytes): samples=0
./sdd/poll_stat:write (512 Bytes): samples=0
./sdd/poll_stat:read (1024 Bytes): samples=0
./sdd/poll_stat:write (1024 Bytes): samples=0
./sdd/poll_stat:read (2048 Bytes): samples=0
./sdd/poll_stat:write (2048 Bytes): samples=0
./sdd/poll_stat:read (4096 Bytes): samples=0
./sdd/poll_stat:write (4096 Bytes): samples=0
./sdd/poll_stat:read (8192 Bytes): samples=0
./sdd/poll_stat:write (8192 Bytes): samples=0
./sdd/poll_stat:read (16384 Bytes): samples=0
./sdd/poll_stat:write (16384 Bytes): samples=0
./sdd/poll_stat:read (32768 Bytes): samples=0
./sdd/poll_stat:write (32768 Bytes): samples=0
./sdd/poll_stat:read (65536 Bytes): samples=0
./sdd/poll_stat:write (65536 Bytes): samples=0
./sdc/hctx3/cpu3/completed:1 11
./sdc/hctx3/cpu3/merged:0
./sdc/hctx3/cpu3/dispatched:1 11
./sdc/hctx3/active:0
./sdc/hctx3/run:15
./sdc/hctx3/queued:12
./sdc/hctx3/dispatched: 0 2
./sdc/hctx3/dispatched: 1 12
./sdc/hctx3/dispatched: 2 0
./sdc/hctx3/dispatched: 4 0
./sdc/hctx3/dispatched: 8 0
./sdc/hctx3/dispatched: 16 0
./sdc/hctx3/dispatched: 32+ 0
./sdc/hctx3/io_poll:considered=0
./sdc/hctx3/io_poll:invoked=0
./sdc/hctx3/io_poll:success=0
./sdc/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx3/tags:nr_tags=62
./sdc/hctx3/tags:nr_reserved_tags=0
./sdc/hctx3/tags:active_queues=0
./sdc/hctx3/tags:
./sdc/hctx3/tags:bitmap_tags:
./sdc/hctx3/tags:depth=62
./sdc/hctx3/tags:busy=0
./sdc/hctx3/tags:bits_per_word=8
./sdc/hctx3/tags:map_nr=8
./sdc/hctx3/tags:alloc_hint={42, 53, 24, 50}
./sdc/hctx3/tags:wake_batch=7
./sdc/hctx3/tags:wake_index=0
./sdc/hctx3/tags:ws={
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags:}
./sdc/hctx3/tags:round_robin=0
./sdc/hctx3/ctx_map:00000000: 00
./sdc/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx3/state:
./sdc/hctx2/cpu2/completed:16 37
./sdc/hctx2/cpu2/merged:0
./sdc/hctx2/cpu2/dispatched:16 37
./sdc/hctx2/active:0
./sdc/hctx2/run:73
./sdc/hctx2/queued:53
./sdc/hctx2/dispatched: 0 22
./sdc/hctx2/dispatched: 1 52
./sdc/hctx2/dispatched: 2 0
./sdc/hctx2/dispatched: 4 0
./sdc/hctx2/dispatched: 8 0
./sdc/hctx2/dispatched: 16 0
./sdc/hctx2/dispatched: 32+ 0
./sdc/hctx2/io_poll:considered=0
./sdc/hctx2/io_poll:invoked=0
./sdc/hctx2/io_poll:success=0
./sdc/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx2/tags:nr_tags=62
./sdc/hctx2/tags:nr_reserved_tags=0
./sdc/hctx2/tags:active_queues=0
./sdc/hctx2/tags:
./sdc/hctx2/tags:bitmap_tags:
./sdc/hctx2/tags:depth=62
./sdc/hctx2/tags:busy=0
./sdc/hctx2/tags:bits_per_word=8
./sdc/hctx2/tags:map_nr=8
./sdc/hctx2/tags:alloc_hint={24, 37, 46, 30}
./sdc/hctx2/tags:wake_batch=7
./sdc/hctx2/tags:wake_index=0
./sdc/hctx2/tags:ws={
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags:}
./sdc/hctx2/tags:round_robin=0
./sdc/hctx2/ctx_map:00000000: 00
./sdc/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx2/state:
./sdc/hctx1/cpu1/completed:32 8
./sdc/hctx1/cpu1/merged:0
./sdc/hctx1/cpu1/dispatched:32 8
./sdc/hctx1/active:0
./sdc/hctx1/run:10
./sdc/hctx1/queued:40
./sdc/hctx1/dispatched: 0 2
./sdc/hctx1/dispatched: 1 8
./sdc/hctx1/dispatched: 2 0
./sdc/hctx1/dispatched: 4 0
./sdc/hctx1/dispatched: 8 0
./sdc/hctx1/dispatched: 16 0
./sdc/hctx1/dispatched: 32+ 0
./sdc/hctx1/io_poll:considered=0
./sdc/hctx1/io_poll:invoked=0
./sdc/hctx1/io_poll:success=0
./sdc/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx1/tags:nr_tags=62
./sdc/hctx1/tags:nr_reserved_tags=0
./sdc/hctx1/tags:active_queues=0
./sdc/hctx1/tags:
./sdc/hctx1/tags:bitmap_tags:
./sdc/hctx1/tags:depth=62
./sdc/hctx1/tags:busy=0
./sdc/hctx1/tags:bits_per_word=8
./sdc/hctx1/tags:map_nr=8
./sdc/hctx1/tags:alloc_hint={29, 0, 21, 37}
./sdc/hctx1/tags:wake_batch=7
./sdc/hctx1/tags:wake_index=0
./sdc/hctx1/tags:ws={
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags:}
./sdc/hctx1/tags:round_robin=0
./sdc/hctx1/ctx_map:00000000: 00
./sdc/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx1/state:
./sdc/hctx0/cpu0/completed:39 12
./sdc/hctx0/cpu0/merged:0
./sdc/hctx0/cpu0/dispatched:39 12
./sdc/hctx0/active:0
./sdc/hctx0/run:48
./sdc/hctx0/queued:51
./sdc/hctx0/dispatched: 0 0
./sdc/hctx0/dispatched: 1 48
./sdc/hctx0/dispatched: 2 0
./sdc/hctx0/dispatched: 4 0
./sdc/hctx0/dispatched: 8 0
./sdc/hctx0/dispatched: 16 0
./sdc/hctx0/dispatched: 32+ 0
./sdc/hctx0/io_poll:considered=0
./sdc/hctx0/io_poll:invoked=0
./sdc/hctx0/io_poll:success=0
./sdc/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx0/tags:nr_tags=62
./sdc/hctx0/tags:nr_reserved_tags=0
./sdc/hctx0/tags:active_queues=0
./sdc/hctx0/tags:
./sdc/hctx0/tags:bitmap_tags:
./sdc/hctx0/tags:depth=62
./sdc/hctx0/tags:busy=0
./sdc/hctx0/tags:bits_per_word=8
./sdc/hctx0/tags:map_nr=8
./sdc/hctx0/tags:alloc_hint={27, 47, 46, 38}
./sdc/hctx0/tags:wake_batch=7
./sdc/hctx0/tags:wake_index=0
./sdc/hctx0/tags:ws={
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags:}
./sdc/hctx0/tags:round_robin=0
./sdc/hctx0/ctx_map:00000000: 00
./sdc/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx0/state:
./sdc/write_hints:hint0: 0
./sdc/write_hints:hint1: 0
./sdc/write_hints:hint2: 0
./sdc/write_hints:hint3: 0
./sdc/write_hints:hint4: 0
./sdc/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
./sdc/poll_stat:read (512 Bytes): samples=0
./sdc/poll_stat:write (512 Bytes): samples=0
./sdc/poll_stat:read (1024 Bytes): samples=0
./sdc/poll_stat:write (1024 Bytes): samples=0
./sdc/poll_stat:read (2048 Bytes): samples=0
./sdc/poll_stat:write (2048 Bytes): samples=0
./sdc/poll_stat:read (4096 Bytes): samples=0
./sdc/poll_stat:write (4096 Bytes): samples=0
./sdc/poll_stat:read (8192 Bytes): samples=0
./sdc/poll_stat:write (8192 Bytes): samples=0
./sdc/poll_stat:read (16384 Bytes): samples=0
./sdc/poll_stat:write (16384 Bytes): samples=0
./sdc/poll_stat:read (32768 Bytes): samples=0
./sdc/poll_stat:write (32768 Bytes): samples=0
./sdc/poll_stat:read (65536 Bytes): samples=0
./sdc/poll_stat:write (65536 Bytes): samples=0
./sdb/hctx0/cpu3/completed:42 6
./sdb/hctx0/cpu3/merged:0
./sdb/hctx0/cpu3/dispatched:42 6
./sdb/hctx0/cpu2/completed:110 13
./sdb/hctx0/cpu2/merged:0
./sdb/hctx0/cpu2/dispatched:110 13
./sdb/hctx0/cpu1/completed:35 39
./sdb/hctx0/cpu1/merged:0
./sdb/hctx0/cpu1/dispatched:35 39
./sdb/hctx0/cpu0/completed:35 26
./sdb/hctx0/cpu0/merged:0
./sdb/hctx0/cpu0/dispatched:35 26
./sdb/hctx0/active:0
./sdb/hctx0/run:300
./sdb/hctx0/queued:306
./sdb/hctx0/dispatched: 0 0
./sdb/hctx0/dispatched: 1 306
./sdb/hctx0/dispatched: 2 0
./sdb/hctx0/dispatched: 4 0
./sdb/hctx0/dispatched: 8 0
./sdb/hctx0/dispatched: 16 0
./sdb/hctx0/dispatched: 32+ 0
./sdb/hctx0/io_poll:considered=0
./sdb/hctx0/io_poll:invoked=0
./sdb/hctx0/io_poll:success=0
./sdb/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./sdb/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./sdb/hctx0/sched_tags:nr_tags=256
./sdb/hctx0/sched_tags:nr_reserved_tags=0
./sdb/hctx0/sched_tags:active_queues=0
./sdb/hctx0/sched_tags:
./sdb/hctx0/sched_tags:bitmap_tags:
./sdb/hctx0/sched_tags:depth=256
./sdb/hctx0/sched_tags:busy=0
./sdb/hctx0/sched_tags:bits_per_word=64
./sdb/hctx0/sched_tags:map_nr=4
./sdb/hctx0/sched_tags:alloc_hint={68, 116, 75, 172}
./sdb/hctx0/sched_tags:wake_batch=8
./sdb/hctx0/sched_tags:wake_index=0
./sdb/hctx0/sched_tags:ws={
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags:}
./sdb/hctx0/sched_tags:round_robin=0
./sdb/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./sdb/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000
./sdb/hctx0/tags:nr_tags=192
./sdb/hctx0/tags:nr_reserved_tags=0
./sdb/hctx0/tags:active_queues=0
./sdb/hctx0/tags:
./sdb/hctx0/tags:bitmap_tags:
./sdb/hctx0/tags:depth=192
./sdb/hctx0/tags:busy=0
./sdb/hctx0/tags:bits_per_word=32
./sdb/hctx0/tags:map_nr=6
./sdb/hctx0/tags:alloc_hint={5, 24, 47, 126}
./sdb/hctx0/tags:wake_batch=8
./sdb/hctx0/tags:wake_index=0
./sdb/hctx0/tags:ws={
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags:}
./sdb/hctx0/tags:round_robin=0
./sdb/hctx0/ctx_map:00000000: 00
./sdb/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./sdb/hctx0/state:
./sdb/sched/starved:0
./sdb/sched/batching:1
./sdb/write_hints:hint0: 0
./sdb/write_hints:hint1: 0
./sdb/write_hints:hint2: 0
./sdb/write_hints:hint3: 0
./sdb/write_hints:hint4: 0
./sdb/state:BIDI|SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
./sdb/poll_stat:read (512 Bytes): samples=0
./sdb/poll_stat:write (512 Bytes): samples=0
./sdb/poll_stat:read (1024 Bytes): samples=0
./sdb/poll_stat:write (1024 Bytes): samples=0
./sdb/poll_stat:read (2048 Bytes): samples=0
./sdb/poll_stat:write (2048 Bytes): samples=0
./sdb/poll_stat:read (4096 Bytes): samples=0
./sdb/poll_stat:write (4096 Bytes): samples=0
./sdb/poll_stat:read (8192 Bytes): samples=0
./sdb/poll_stat:write (8192 Bytes): samples=0
./sdb/poll_stat:read (16384 Bytes): samples=0
./sdb/poll_stat:write (16384 Bytes): samples=0
./sdb/poll_stat:read (32768 Bytes): samples=0
./sdb/poll_stat:write (32768 Bytes): samples=0
./sdb/poll_stat:read (65536 Bytes): samples=0
./sdb/poll_stat:write (65536 Bytes): samples=0
./sda/hctx0/cpu3/completed:46 3
./sda/hctx0/cpu3/merged:0
./sda/hctx0/cpu3/dispatched:46 3
./sda/hctx0/cpu2/completed:41 27
./sda/hctx0/cpu2/merged:0
./sda/hctx0/cpu2/dispatched:41 27
./sda/hctx0/cpu1/completed:2 8
./sda/hctx0/cpu1/merged:0
./sda/hctx0/cpu1/dispatched:2 8
./sda/hctx0/cpu0/completed:1 43
./sda/hctx0/cpu0/merged:0
./sda/hctx0/cpu0/dispatched:1 43
./sda/hctx0/active:0
./sda/hctx0/run:144
./sda/hctx0/queued:171
./sda/hctx0/dispatched: 0 0
./sda/hctx0/dispatched: 1 171
./sda/hctx0/dispatched: 2 0
./sda/hctx0/dispatched: 4 0
./sda/hctx0/dispatched: 8 0
./sda/hctx0/dispatched: 16 0
./sda/hctx0/dispatched: 32+ 0
./sda/hctx0/io_poll:considered=0
./sda/hctx0/io_poll:invoked=0
./sda/hctx0/io_poll:success=0
./sda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./sda/hctx0/tags:nr_tags=128
./sda/hctx0/tags:nr_reserved_tags=0
./sda/hctx0/tags:active_queues=0
./sda/hctx0/tags:
./sda/hctx0/tags:bitmap_tags:
./sda/hctx0/tags:depth=128
./sda/hctx0/tags:busy=0
./sda/hctx0/tags:bits_per_word=32
./sda/hctx0/tags:map_nr=4
./sda/hctx0/tags:alloc_hint={111, 80, 44, 68}
./sda/hctx0/tags:wake_batch=8
./sda/hctx0/tags:wake_index=0
./sda/hctx0/tags:ws={
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags:}
./sda/hctx0/tags:round_robin=0
./sda/hctx0/ctx_map:00000000: 00
./sda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./sda/hctx0/state:
./sda/write_hints:hint0: 0
./sda/write_hints:hint1: 0
./sda/write_hints:hint2: 0
./sda/write_hints:hint3: 0
./sda/write_hints:hint4: 0
./sda/state:SAME_COMP|IO_STAT|DISCARD|INIT_DONE|POLL|WC|REGISTERED|SCSI_PASSTHROUGH
./sda/poll_stat:read (512 Bytes): samples=0
./sda/poll_stat:write (512 Bytes): samples=0
./sda/poll_stat:read (1024 Bytes): samples=0
./sda/poll_stat:write (1024 Bytes): samples=0
./sda/poll_stat:read (2048 Bytes): samples=0
./sda/poll_stat:write (2048 Bytes): samples=0
./sda/poll_stat:read (4096 Bytes): samples=0
./sda/poll_stat:write (4096 Bytes): samples=0
./sda/poll_stat:read (8192 Bytes): samples=0
./sda/poll_stat:write (8192 Bytes): samples=0
./sda/poll_stat:read (16384 Bytes): samples=0
./sda/poll_stat:write (16384 Bytes): samples=0
./sda/poll_stat:read (32768 Bytes): samples=0
./sda/poll_stat:write (32768 Bytes): samples=0
./sda/poll_stat:read (65536 Bytes): samples=0
./sda/poll_stat:write (65536 Bytes): samples=0
./vda/hctx0/cpu3/completed:1707 27
./vda/hctx0/cpu3/merged:0
./vda/hctx0/cpu3/dispatched:1707 27
./vda/hctx0/cpu2/completed:1027 12
./vda/hctx0/cpu2/merged:0
./vda/hctx0/cpu2/dispatched:1027 12
./vda/hctx0/cpu1/completed:1138 76
./vda/hctx0/cpu1/merged:4
./vda/hctx0/cpu1/dispatched:1138 76
./vda/hctx0/cpu0/completed:2497 44
./vda/hctx0/cpu0/merged:2
./vda/hctx0/cpu0/dispatched:2497 44
./vda/hctx0/active:0
./vda/hctx0/run:6412
./vda/hctx0/queued:6528
./vda/hctx0/dispatched: 0 0
./vda/hctx0/dispatched: 1 6361
./vda/hctx0/dispatched: 2 20
./vda/hctx0/dispatched: 4 13
./vda/hctx0/dispatched: 8 8
./vda/hctx0/dispatched: 16 8
./vda/hctx0/dispatched: 32+ 0
./vda/hctx0/io_poll:considered=0
./vda/hctx0/io_poll:invoked=0
./vda/hctx0/io_poll:success=0
./vda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./vda/hctx0/tags:nr_tags=128
./vda/hctx0/tags:nr_reserved_tags=0
./vda/hctx0/tags:active_queues=0
./vda/hctx0/tags:
./vda/hctx0/tags:bitmap_tags:
./vda/hctx0/tags:depth=128
./vda/hctx0/tags:busy=0
./vda/hctx0/tags:bits_per_word=32
./vda/hctx0/tags:map_nr=4
./vda/hctx0/tags:alloc_hint={28, 24, 114, 37}
./vda/hctx0/tags:wake_batch=8
./vda/hctx0/tags:wake_index=0
./vda/hctx0/tags:ws={
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags:}
./vda/hctx0/tags:round_robin=0
./vda/hctx0/ctx_map:00000000: 00
./vda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE
./vda/hctx0/state:
./vda/write_hints:hint0: 0
./vda/write_hints:hint1: 0
./vda/write_hints:hint2: 0
./vda/write_hints:hint3: 0
./vda/write_hints:hint4: 0
./vda/state:SAME_COMP|IO_STAT|INIT_DONE|NO_SG_MERGE|POLL|WC|REGISTERED
./vda/poll_stat:read (512 Bytes): samples=0
./vda/poll_stat:write (512 Bytes): samples=0
./vda/poll_stat:read (1024 Bytes): samples=0
./vda/poll_stat:write (1024 Bytes): samples=0
./vda/poll_stat:read (2048 Bytes): samples=0
./vda/poll_stat:write (2048 Bytes): samples=0
./vda/poll_stat:read (4096 Bytes): samples=0
./vda/poll_stat:write (4096 Bytes): samples=0
./vda/poll_stat:read (8192 Bytes): samples=0
./vda/poll_stat:write (8192 Bytes): samples=0
./vda/poll_stat:read (16384 Bytes): samples=0
./vda/poll_stat:write (16384 Bytes): samples=0
./vda/poll_stat:read (32768 Bytes): samples=0
./vda/poll_stat:write (32768 Bytes): samples=0
./vda/poll_stat:read (65536 Bytes): samples=0
./vda/poll_stat:write (65536 Bytes): samples=0
./sr0/hctx0/cpu3/completed:0 22
./sr0/hctx0/cpu3/merged:0
./sr0/hctx0/cpu3/dispatched:0 22
./sr0/hctx0/cpu2/completed:46 107
./sr0/hctx0/cpu2/merged:0
./sr0/hctx0/cpu2/dispatched:46 107
./sr0/hctx0/cpu1/completed:0 10
./sr0/hctx0/cpu1/merged:0
./sr0/hctx0/cpu1/dispatched:0 10
./sr0/hctx0/cpu0/completed:0 18
./sr0/hctx0/cpu0/merged:0
./sr0/hctx0/cpu0/dispatched:0 18
./sr0/hctx0/active:0
./sr0/hctx0/run:203
./sr0/hctx0/queued:203
./sr0/hctx0/dispatched: 0 0
./sr0/hctx0/dispatched: 1 203
./sr0/hctx0/dispatched: 2 0
./sr0/hctx0/dispatched: 4 0
./sr0/hctx0/dispatched: 8 0
./sr0/hctx0/dispatched: 16 0
./sr0/hctx0/dispatched: 32+ 0
./sr0/hctx0/io_poll:considered=0
./sr0/hctx0/io_poll:invoked=0
./sr0/hctx0/io_poll:success=0
./sr0/hctx0/tags_bitmap:00000000: 00
./sr0/hctx0/tags:nr_tags=1
./sr0/hctx0/tags:nr_reserved_tags=0
./sr0/hctx0/tags:active_queues=0
./sr0/hctx0/tags:
./sr0/hctx0/tags:bitmap_tags:
./sr0/hctx0/tags:depth=1
./sr0/hctx0/tags:busy=0
./sr0/hctx0/tags:bits_per_word=64
./sr0/hctx0/tags:map_nr=1
./sr0/hctx0/tags:alloc_hint={0, 0, 0, 0}
./sr0/hctx0/tags:wake_batch=1
./sr0/hctx0/tags:wake_index=7
./sr0/hctx0/tags:ws={
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags:}
./sr0/hctx0/tags:round_robin=1
./sr0/hctx0/ctx_map:00000000: 00
./sr0/hctx0/flags:alloc_policy=RR SHOULD_MERGE|SG_MERGE
./sr0/hctx0/state:
./sr0/write_hints:hint0: 0
./sr0/write_hints:hint1: 0
./sr0/write_hints:hint2: 0
./sr0/write_hints:hint3: 0
./sr0/write_hints:hint4: 0
./sr0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|FLUSH_NQ|REGISTERED|SCSI_PASSTHROUGH
./sr0/poll_stat:read (512 Bytes): samples=0
./sr0/poll_stat:write (512 Bytes): samples=0
./sr0/poll_stat:read (1024 Bytes): samples=0
./sr0/poll_stat:write (1024 Bytes): samples=0
./sr0/poll_stat:read (2048 Bytes): samples=0
./sr0/poll_stat:write (2048 Bytes): samples=0
./sr0/poll_stat:read (4096 Bytes): samples=0
./sr0/poll_stat:write (4096 Bytes): samples=0
./sr0/poll_stat:read (8192 Bytes): samples=0
./sr0/poll_stat:write (8192 Bytes): samples=0
./sr0/poll_stat:read (16384 Bytes): samples=0
./sr0/poll_stat:write (16384 Bytes): samples=0
./sr0/poll_stat:read (32768 Bytes): samples=0
./sr0/poll_stat:write (32768 Bytes): samples=0
./sr0/poll_stat:read (65536 Bytes): samples=0
./sr0/poll_stat:write (65536 Bytes): samples=0
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 18:05 dm-rq queue stalls Bart Van Assche
@ 2018-01-17 18:18 ` Mike Snitzer
2018-01-17 18:27 ` Bart Van Assche
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: Mike Snitzer @ 2018-01-17 18:18 UTC (permalink / raw)
To: Bart Van Assche; +Cc: axboe, dm-devel, Laurence Oberman, Ming Lei
On Wed, Jan 17 2018 at 1:05pm -0500,
Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
> Hello Mike,
>
> With Jens' latest for-next branch (commit 6c4f43dfecae) and with commit
> 6077c2d70609 ("dm rq: Avoid that request processing stalls sporadically")
> reverted I see again queue stalls every now and then. An example:
>
> # srp-test/run_tests -c -d -r 10 -t 02-mq -e kyber
> Running test /home/bart/software/infiniband/srp-test/tests/02-mq ...
> Test file I/O on top of multipath concurrently with logout and login (0 min; mq)
> /dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000: not found
> Test /home/bart/software/infiniband/srp-test/tests/02-mq failed
>
> I have attached the output of the following command to this e-mail:
> (cd /sys/kernel/debug/block && find -type f | xargs grep -aH '')
>
> Can you have a look at this?
Yes, though I need to prepare for a trip that starts on Sunday (so I'll
be distracted through the end of next week).
But Ming Lei should be able to help make sense of this situation too.
Also Laurence may be able to help reproduce your specific srp-test on
his testbed.
Mike
> ./dm-2/hctx0/cpu3/completed:0 0
> ./dm-2/hctx0/cpu3/merged:0
> ./dm-2/hctx0/cpu3/dispatched:0 0
> ./dm-2/hctx0/cpu2/completed:0 0
> ./dm-2/hctx0/cpu2/merged:0
> ./dm-2/hctx0/cpu2/dispatched:0 0
> ./dm-2/hctx0/cpu1/completed:25 0
> ./dm-2/hctx0/cpu1/merged:0
> ./dm-2/hctx0/cpu1/dispatched:26 0
> ./dm-2/hctx0/cpu0/completed:0 0
> ./dm-2/hctx0/cpu0/merged:0
> ./dm-2/hctx0/cpu0/dispatched:0 0
> ./dm-2/hctx0/active:0
> ./dm-2/hctx0/run:27
> ./dm-2/hctx0/queued:26
> ./dm-2/hctx0/dispatched: 0 6
> ./dm-2/hctx0/dispatched: 1 25
> ./dm-2/hctx0/dispatched: 2 0
> ./dm-2/hctx0/dispatched: 4 0
> ./dm-2/hctx0/dispatched: 8 0
> ./dm-2/hctx0/dispatched: 16 0
> ./dm-2/hctx0/dispatched: 32+ 0
> ./dm-2/hctx0/io_poll:considered=0
> ./dm-2/hctx0/io_poll:invoked=0
> ./dm-2/hctx0/io_poll:success=0
> ./dm-2/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0020
> ./dm-2/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/sched_tags:nr_tags=256
> ./dm-2/hctx0/sched_tags:nr_reserved_tags=0
> ./dm-2/hctx0/sched_tags:active_queues=0
> ./dm-2/hctx0/sched_tags:
> ./dm-2/hctx0/sched_tags:bitmap_tags:
> ./dm-2/hctx0/sched_tags:depth=256
> ./dm-2/hctx0/sched_tags:busy=1
> ./dm-2/hctx0/sched_tags:bits_per_word=64
> ./dm-2/hctx0/sched_tags:map_nr=4
> ./dm-2/hctx0/sched_tags:alloc_hint={83, 124, 234, 58}
> ./dm-2/hctx0/sched_tags:wake_batch=8
> ./dm-2/hctx0/sched_tags:wake_index=0
> ./dm-2/hctx0/sched_tags:ws={
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/sched_tags:}
> ./dm-2/hctx0/sched_tags:round_robin=0
> ./dm-2/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-2/hctx0/tags:nr_tags=2048
> ./dm-2/hctx0/tags:nr_reserved_tags=0
> ./dm-2/hctx0/tags:active_queues=0
> ./dm-2/hctx0/tags:
> ./dm-2/hctx0/tags:bitmap_tags:
> ./dm-2/hctx0/tags:depth=2048
> ./dm-2/hctx0/tags:busy=0
> ./dm-2/hctx0/tags:bits_per_word=64
> ./dm-2/hctx0/tags:map_nr=32
> ./dm-2/hctx0/tags:alloc_hint={1060, 81, 1050, 844}
> ./dm-2/hctx0/tags:wake_batch=8
> ./dm-2/hctx0/tags:wake_index=0
> ./dm-2/hctx0/tags:ws={
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-2/hctx0/tags:}
> ./dm-2/hctx0/tags:round_robin=0
> ./dm-2/hctx0/ctx_map:00000000: 00
> ./dm-2/hctx0/dispatch:00000000a7224aed {.op=READ, .cmd_flags=, .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=125}
> ./dm-2/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> ./dm-2/hctx0/state:SCHED_RESTART
> ./dm-2/sched/starved:0
> ./dm-2/sched/batching:5
> ./dm-2/write_hints:hint0: 0
> ./dm-2/write_hints:hint1: 0
> ./dm-2/write_hints:hint2: 0
> ./dm-2/write_hints:hint3: 0
> ./dm-2/write_hints:hint4: 0
> ./dm-2/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
> ./dm-2/poll_stat:read (512 Bytes): samples=0
> ./dm-2/poll_stat:write (512 Bytes): samples=0
> ./dm-2/poll_stat:read (1024 Bytes): samples=0
> ./dm-2/poll_stat:write (1024 Bytes): samples=0
> ./dm-2/poll_stat:read (2048 Bytes): samples=0
> ./dm-2/poll_stat:write (2048 Bytes): samples=0
> ./dm-2/poll_stat:read (4096 Bytes): samples=0
> ./dm-2/poll_stat:write (4096 Bytes): samples=0
> ./dm-2/poll_stat:read (8192 Bytes): samples=0
> ./dm-2/poll_stat:write (8192 Bytes): samples=0
> ./dm-2/poll_stat:read (16384 Bytes): samples=0
> ./dm-2/poll_stat:write (16384 Bytes): samples=0
> ./dm-2/poll_stat:read (32768 Bytes): samples=0
> ./dm-2/poll_stat:write (32768 Bytes): samples=0
> ./dm-2/poll_stat:read (65536 Bytes): samples=0
> ./dm-2/poll_stat:write (65536 Bytes): samples=0
> ./dm-1/hctx0/cpu3/completed:26 0
> ./dm-1/hctx0/cpu3/merged:0
> ./dm-1/hctx0/cpu3/dispatched:26 0
> ./dm-1/hctx0/cpu2/completed:2 0
> ./dm-1/hctx0/cpu2/merged:0
> ./dm-1/hctx0/cpu2/dispatched:2 0
> ./dm-1/hctx0/cpu1/completed:2 0
> ./dm-1/hctx0/cpu1/merged:0
> ./dm-1/hctx0/cpu1/dispatched:2 0
> ./dm-1/hctx0/cpu0/completed:2 0
> ./dm-1/hctx0/cpu0/merged:0
> ./dm-1/hctx0/cpu0/dispatched:2 0
> ./dm-1/hctx0/active:0
> ./dm-1/hctx0/run:34
> ./dm-1/hctx0/queued:32
> ./dm-1/hctx0/dispatched: 0 7
> ./dm-1/hctx0/dispatched: 1 32
> ./dm-1/hctx0/dispatched: 2 0
> ./dm-1/hctx0/dispatched: 4 0
> ./dm-1/hctx0/dispatched: 8 0
> ./dm-1/hctx0/dispatched: 16 0
> ./dm-1/hctx0/dispatched: 32+ 0
> ./dm-1/hctx0/io_poll:considered=0
> ./dm-1/hctx0/io_poll:invoked=0
> ./dm-1/hctx0/io_poll:success=0
> ./dm-1/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/sched_tags:nr_tags=256
> ./dm-1/hctx0/sched_tags:nr_reserved_tags=0
> ./dm-1/hctx0/sched_tags:active_queues=0
> ./dm-1/hctx0/sched_tags:
> ./dm-1/hctx0/sched_tags:bitmap_tags:
> ./dm-1/hctx0/sched_tags:depth=256
> ./dm-1/hctx0/sched_tags:busy=0
> ./dm-1/hctx0/sched_tags:bits_per_word=64
> ./dm-1/hctx0/sched_tags:map_nr=4
> ./dm-1/hctx0/sched_tags:alloc_hint={125, 55, 142, 82}
> ./dm-1/hctx0/sched_tags:wake_batch=8
> ./dm-1/hctx0/sched_tags:wake_index=0
> ./dm-1/hctx0/sched_tags:ws={
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/sched_tags:}
> ./dm-1/hctx0/sched_tags:round_robin=0
> ./dm-1/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-1/hctx0/tags:nr_tags=2048
> ./dm-1/hctx0/tags:nr_reserved_tags=0
> ./dm-1/hctx0/tags:active_queues=0
> ./dm-1/hctx0/tags:
> ./dm-1/hctx0/tags:bitmap_tags:
> ./dm-1/hctx0/tags:depth=2048
> ./dm-1/hctx0/tags:busy=0
> ./dm-1/hctx0/tags:bits_per_word=64
> ./dm-1/hctx0/tags:map_nr=32
> ./dm-1/hctx0/tags:alloc_hint={478, 1303, 102, 127}
> ./dm-1/hctx0/tags:wake_batch=8
> ./dm-1/hctx0/tags:wake_index=0
> ./dm-1/hctx0/tags:ws={
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-1/hctx0/tags:}
> ./dm-1/hctx0/tags:round_robin=0
> ./dm-1/hctx0/ctx_map:00000000: 00
> ./dm-1/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> ./dm-1/hctx0/state:
> ./dm-1/sched/starved:0
> ./dm-1/sched/batching:1
> ./dm-1/write_hints:hint0: 0
> ./dm-1/write_hints:hint1: 0
> ./dm-1/write_hints:hint2: 0
> ./dm-1/write_hints:hint3: 0
> ./dm-1/write_hints:hint4: 0
> ./dm-1/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
> ./dm-1/poll_stat:read (512 Bytes): samples=0
> ./dm-1/poll_stat:write (512 Bytes): samples=0
> ./dm-1/poll_stat:read (1024 Bytes): samples=0
> ./dm-1/poll_stat:write (1024 Bytes): samples=0
> ./dm-1/poll_stat:read (2048 Bytes): samples=0
> ./dm-1/poll_stat:write (2048 Bytes): samples=0
> ./dm-1/poll_stat:read (4096 Bytes): samples=0
> ./dm-1/poll_stat:write (4096 Bytes): samples=0
> ./dm-1/poll_stat:read (8192 Bytes): samples=0
> ./dm-1/poll_stat:write (8192 Bytes): samples=0
> ./dm-1/poll_stat:read (16384 Bytes): samples=0
> ./dm-1/poll_stat:write (16384 Bytes): samples=0
> ./dm-1/poll_stat:read (32768 Bytes): samples=0
> ./dm-1/poll_stat:write (32768 Bytes): samples=0
> ./dm-1/poll_stat:read (65536 Bytes): samples=0
> ./dm-1/poll_stat:write (65536 Bytes): samples=0
> ./dm-0/hctx0/cpu3/completed:13 0
> ./dm-0/hctx0/cpu3/merged:0
> ./dm-0/hctx0/cpu3/dispatched:14 0
> ./dm-0/hctx0/cpu2/completed:0 0
> ./dm-0/hctx0/cpu2/merged:0
> ./dm-0/hctx0/cpu2/dispatched:0 0
> ./dm-0/hctx0/cpu1/completed:0 0
> ./dm-0/hctx0/cpu1/merged:0
> ./dm-0/hctx0/cpu1/dispatched:0 0
> ./dm-0/hctx0/cpu0/completed:0 0
> ./dm-0/hctx0/cpu0/merged:0
> ./dm-0/hctx0/cpu0/dispatched:0 0
> ./dm-0/hctx0/active:0
> ./dm-0/hctx0/run:15
> ./dm-0/hctx0/queued:14
> ./dm-0/hctx0/dispatched: 0 2
> ./dm-0/hctx0/dispatched: 1 13
> ./dm-0/hctx0/dispatched: 2 0
> ./dm-0/hctx0/dispatched: 4 0
> ./dm-0/hctx0/dispatched: 8 0
> ./dm-0/hctx0/dispatched: 16 0
> ./dm-0/hctx0/dispatched: 32+ 0
> ./dm-0/hctx0/io_poll:considered=0
> ./dm-0/hctx0/io_poll:invoked=0
> ./dm-0/hctx0/io_poll:success=0
> ./dm-0/hctx0/sched_tags_bitmap:00000000: 0001 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/sched_tags:nr_tags=256
> ./dm-0/hctx0/sched_tags:nr_reserved_tags=0
> ./dm-0/hctx0/sched_tags:active_queues=0
> ./dm-0/hctx0/sched_tags:
> ./dm-0/hctx0/sched_tags:bitmap_tags:
> ./dm-0/hctx0/sched_tags:depth=256
> ./dm-0/hctx0/sched_tags:busy=1
> ./dm-0/hctx0/sched_tags:bits_per_word=64
> ./dm-0/hctx0/sched_tags:map_nr=4
> ./dm-0/hctx0/sched_tags:alloc_hint={11, 18, 205, 9}
> ./dm-0/hctx0/sched_tags:wake_batch=8
> ./dm-0/hctx0/sched_tags:wake_index=0
> ./dm-0/hctx0/sched_tags:ws={
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/sched_tags:}
> ./dm-0/hctx0/sched_tags:round_robin=0
> ./dm-0/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
> ./dm-0/hctx0/tags:nr_tags=2048
> ./dm-0/hctx0/tags:nr_reserved_tags=0
> ./dm-0/hctx0/tags:active_queues=0
> ./dm-0/hctx0/tags:
> ./dm-0/hctx0/tags:bitmap_tags:
> ./dm-0/hctx0/tags:depth=2048
> ./dm-0/hctx0/tags:busy=0
> ./dm-0/hctx0/tags:bits_per_word=64
> ./dm-0/hctx0/tags:map_nr=32
> ./dm-0/hctx0/tags:alloc_hint={1392, 843, 879, 1030}
> ./dm-0/hctx0/tags:wake_batch=8
> ./dm-0/hctx0/tags:wake_index=0
> ./dm-0/hctx0/tags:ws={
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./dm-0/hctx0/tags:}
> ./dm-0/hctx0/tags:round_robin=0
> ./dm-0/hctx0/ctx_map:00000000: 00
> ./dm-0/hctx0/dispatch:000000002f291aa3 {.op=READ, .cmd_flags=, .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=8}
> ./dm-0/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> ./dm-0/hctx0/state:SCHED_RESTART
> ./dm-0/sched/starved:0
> ./dm-0/sched/batching:1
> ./dm-0/write_hints:hint0: 0
> ./dm-0/write_hints:hint1: 0
> ./dm-0/write_hints:hint2: 0
> ./dm-0/write_hints:hint3: 0
> ./dm-0/write_hints:hint4: 0
> ./dm-0/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED
> ./dm-0/poll_stat:read (512 Bytes): samples=0
> ./dm-0/poll_stat:write (512 Bytes): samples=0
> ./dm-0/poll_stat:read (1024 Bytes): samples=0
> ./dm-0/poll_stat:write (1024 Bytes): samples=0
> ./dm-0/poll_stat:read (2048 Bytes): samples=0
> ./dm-0/poll_stat:write (2048 Bytes): samples=0
> ./dm-0/poll_stat:read (4096 Bytes): samples=0
> ./dm-0/poll_stat:write (4096 Bytes): samples=0
> ./dm-0/poll_stat:read (8192 Bytes): samples=0
> ./dm-0/poll_stat:write (8192 Bytes): samples=0
> ./dm-0/poll_stat:read (16384 Bytes): samples=0
> ./dm-0/poll_stat:write (16384 Bytes): samples=0
> ./dm-0/poll_stat:read (32768 Bytes): samples=0
> ./dm-0/poll_stat:write (32768 Bytes): samples=0
> ./dm-0/poll_stat:read (65536 Bytes): samples=0
> ./dm-0/poll_stat:write (65536 Bytes): samples=0
> ./sde/hctx3/cpu3/completed:58 12
> ./sde/hctx3/cpu3/merged:0
> ./sde/hctx3/cpu3/dispatched:58 12
> ./sde/hctx3/active:0
> ./sde/hctx3/run:39
> ./sde/hctx3/queued:70
> ./sde/hctx3/dispatched: 0 2
> ./sde/hctx3/dispatched: 1 36
> ./sde/hctx3/dispatched: 2 0
> ./sde/hctx3/dispatched: 4 0
> ./sde/hctx3/dispatched: 8 0
> ./sde/hctx3/dispatched: 16 0
> ./sde/hctx3/dispatched: 32+ 0
> ./sde/hctx3/io_poll:considered=0
> ./sde/hctx3/io_poll:invoked=0
> ./sde/hctx3/io_poll:success=0
> ./sde/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sde/hctx3/tags:nr_tags=62
> ./sde/hctx3/tags:nr_reserved_tags=0
> ./sde/hctx3/tags:active_queues=0
> ./sde/hctx3/tags:
> ./sde/hctx3/tags:bitmap_tags:
> ./sde/hctx3/tags:depth=62
> ./sde/hctx3/tags:busy=0
> ./sde/hctx3/tags:bits_per_word=8
> ./sde/hctx3/tags:map_nr=8
> ./sde/hctx3/tags:alloc_hint={42, 53, 24, 50}
> ./sde/hctx3/tags:wake_batch=7
> ./sde/hctx3/tags:wake_index=0
> ./sde/hctx3/tags:ws={
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx3/tags:}
> ./sde/hctx3/tags:round_robin=0
> ./sde/hctx3/ctx_map:00000000: 00
> ./sde/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sde/hctx3/state:
> ./sde/hctx2/cpu2/completed:2 3
> ./sde/hctx2/cpu2/merged:0
> ./sde/hctx2/cpu2/dispatched:2 3
> ./sde/hctx2/active:0
> ./sde/hctx2/run:3
> ./sde/hctx2/queued:5
> ./sde/hctx2/dispatched: 0 0
> ./sde/hctx2/dispatched: 1 3
> ./sde/hctx2/dispatched: 2 0
> ./sde/hctx2/dispatched: 4 0
> ./sde/hctx2/dispatched: 8 0
> ./sde/hctx2/dispatched: 16 0
> ./sde/hctx2/dispatched: 32+ 0
> ./sde/hctx2/io_poll:considered=0
> ./sde/hctx2/io_poll:invoked=0
> ./sde/hctx2/io_poll:success=0
> ./sde/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sde/hctx2/tags:nr_tags=62
> ./sde/hctx2/tags:nr_reserved_tags=0
> ./sde/hctx2/tags:active_queues=0
> ./sde/hctx2/tags:
> ./sde/hctx2/tags:bitmap_tags:
> ./sde/hctx2/tags:depth=62
> ./sde/hctx2/tags:busy=0
> ./sde/hctx2/tags:bits_per_word=8
> ./sde/hctx2/tags:map_nr=8
> ./sde/hctx2/tags:alloc_hint={24, 37, 46, 30}
> ./sde/hctx2/tags:wake_batch=7
> ./sde/hctx2/tags:wake_index=0
> ./sde/hctx2/tags:ws={
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx2/tags:}
> ./sde/hctx2/tags:round_robin=0
> ./sde/hctx2/ctx_map:00000000: 00
> ./sde/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sde/hctx2/state:
> ./sde/hctx1/cpu1/completed:31 47
> ./sde/hctx1/cpu1/merged:0
> ./sde/hctx1/cpu1/dispatched:31 47
> ./sde/hctx1/active:0
> ./sde/hctx1/run:76
> ./sde/hctx1/queued:78
> ./sde/hctx1/dispatched: 0 3
> ./sde/hctx1/dispatched: 1 71
> ./sde/hctx1/dispatched: 2 0
> ./sde/hctx1/dispatched: 4 0
> ./sde/hctx1/dispatched: 8 0
> ./sde/hctx1/dispatched: 16 0
> ./sde/hctx1/dispatched: 32+ 0
> ./sde/hctx1/io_poll:considered=0
> ./sde/hctx1/io_poll:invoked=0
> ./sde/hctx1/io_poll:success=0
> ./sde/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sde/hctx1/tags:nr_tags=62
> ./sde/hctx1/tags:nr_reserved_tags=0
> ./sde/hctx1/tags:active_queues=0
> ./sde/hctx1/tags:
> ./sde/hctx1/tags:bitmap_tags:
> ./sde/hctx1/tags:depth=62
> ./sde/hctx1/tags:busy=0
> ./sde/hctx1/tags:bits_per_word=8
> ./sde/hctx1/tags:map_nr=8
> ./sde/hctx1/tags:alloc_hint={29, 0, 21, 37}
> ./sde/hctx1/tags:wake_batch=7
> ./sde/hctx1/tags:wake_index=0
> ./sde/hctx1/tags:ws={
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx1/tags:}
> ./sde/hctx1/tags:round_robin=0
> ./sde/hctx1/ctx_map:00000000: 00
> ./sde/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sde/hctx1/state:
> ./sde/hctx0/cpu0/completed:4 6
> ./sde/hctx0/cpu0/merged:0
> ./sde/hctx0/cpu0/dispatched:4 6
> ./sde/hctx0/active:0
> ./sde/hctx0/run:6
> ./sde/hctx0/queued:10
> ./sde/hctx0/dispatched: 0 0
> ./sde/hctx0/dispatched: 1 6
> ./sde/hctx0/dispatched: 2 0
> ./sde/hctx0/dispatched: 4 0
> ./sde/hctx0/dispatched: 8 0
> ./sde/hctx0/dispatched: 16 0
> ./sde/hctx0/dispatched: 32+ 0
> ./sde/hctx0/io_poll:considered=0
> ./sde/hctx0/io_poll:invoked=0
> ./sde/hctx0/io_poll:success=0
> ./sde/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sde/hctx0/tags:nr_tags=62
> ./sde/hctx0/tags:nr_reserved_tags=0
> ./sde/hctx0/tags:active_queues=0
> ./sde/hctx0/tags:
> ./sde/hctx0/tags:bitmap_tags:
> ./sde/hctx0/tags:depth=62
> ./sde/hctx0/tags:busy=0
> ./sde/hctx0/tags:bits_per_word=8
> ./sde/hctx0/tags:map_nr=8
> ./sde/hctx0/tags:alloc_hint={27, 47, 46, 38}
> ./sde/hctx0/tags:wake_batch=7
> ./sde/hctx0/tags:wake_index=0
> ./sde/hctx0/tags:ws={
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sde/hctx0/tags:}
> ./sde/hctx0/tags:round_robin=0
> ./sde/hctx0/ctx_map:00000000: 00
> ./sde/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sde/hctx0/state:
> ./sde/write_hints:hint0: 0
> ./sde/write_hints:hint1: 0
> ./sde/write_hints:hint2: 0
> ./sde/write_hints:hint3: 0
> ./sde/write_hints:hint4: 0
> ./sde/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
> ./sde/poll_stat:read (512 Bytes): samples=0
> ./sde/poll_stat:write (512 Bytes): samples=0
> ./sde/poll_stat:read (1024 Bytes): samples=0
> ./sde/poll_stat:write (1024 Bytes): samples=0
> ./sde/poll_stat:read (2048 Bytes): samples=0
> ./sde/poll_stat:write (2048 Bytes): samples=0
> ./sde/poll_stat:read (4096 Bytes): samples=0
> ./sde/poll_stat:write (4096 Bytes): samples=0
> ./sde/poll_stat:read (8192 Bytes): samples=0
> ./sde/poll_stat:write (8192 Bytes): samples=0
> ./sde/poll_stat:read (16384 Bytes): samples=0
> ./sde/poll_stat:write (16384 Bytes): samples=0
> ./sde/poll_stat:read (32768 Bytes): samples=0
> ./sde/poll_stat:write (32768 Bytes): samples=0
> ./sde/poll_stat:read (65536 Bytes): samples=0
> ./sde/poll_stat:write (65536 Bytes): samples=0
> ./sdd/hctx3/cpu3/completed:17 7
> ./sdd/hctx3/cpu3/merged:0
> ./sdd/hctx3/cpu3/dispatched:17 7
> ./sdd/hctx3/active:0
> ./sdd/hctx3/run:13
> ./sdd/hctx3/queued:24
> ./sdd/hctx3/dispatched: 0 4
> ./sdd/hctx3/dispatched: 1 9
> ./sdd/hctx3/dispatched: 2 0
> ./sdd/hctx3/dispatched: 4 0
> ./sdd/hctx3/dispatched: 8 0
> ./sdd/hctx3/dispatched: 16 0
> ./sdd/hctx3/dispatched: 32+ 0
> ./sdd/hctx3/io_poll:considered=0
> ./sdd/hctx3/io_poll:invoked=0
> ./sdd/hctx3/io_poll:success=0
> ./sdd/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdd/hctx3/tags:nr_tags=62
> ./sdd/hctx3/tags:nr_reserved_tags=0
> ./sdd/hctx3/tags:active_queues=0
> ./sdd/hctx3/tags:
> ./sdd/hctx3/tags:bitmap_tags:
> ./sdd/hctx3/tags:depth=62
> ./sdd/hctx3/tags:busy=0
> ./sdd/hctx3/tags:bits_per_word=8
> ./sdd/hctx3/tags:map_nr=8
> ./sdd/hctx3/tags:alloc_hint={42, 53, 24, 50}
> ./sdd/hctx3/tags:wake_batch=7
> ./sdd/hctx3/tags:wake_index=0
> ./sdd/hctx3/tags:ws={
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx3/tags:}
> ./sdd/hctx3/tags:round_robin=0
> ./sdd/hctx3/ctx_map:00000000: 00
> ./sdd/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdd/hctx3/state:
> ./sdd/hctx2/cpu2/completed:20 37
> ./sdd/hctx2/cpu2/merged:0
> ./sdd/hctx2/cpu2/dispatched:20 37
> ./sdd/hctx2/active:0
> ./sdd/hctx2/run:57
> ./sdd/hctx2/queued:57
> ./sdd/hctx2/dispatched: 0 5
> ./sdd/hctx2/dispatched: 1 52
> ./sdd/hctx2/dispatched: 2 0
> ./sdd/hctx2/dispatched: 4 0
> ./sdd/hctx2/dispatched: 8 0
> ./sdd/hctx2/dispatched: 16 0
> ./sdd/hctx2/dispatched: 32+ 0
> ./sdd/hctx2/io_poll:considered=0
> ./sdd/hctx2/io_poll:invoked=0
> ./sdd/hctx2/io_poll:success=0
> ./sdd/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdd/hctx2/tags:nr_tags=62
> ./sdd/hctx2/tags:nr_reserved_tags=0
> ./sdd/hctx2/tags:active_queues=0
> ./sdd/hctx2/tags:
> ./sdd/hctx2/tags:bitmap_tags:
> ./sdd/hctx2/tags:depth=62
> ./sdd/hctx2/tags:busy=0
> ./sdd/hctx2/tags:bits_per_word=8
> ./sdd/hctx2/tags:map_nr=8
> ./sdd/hctx2/tags:alloc_hint={24, 37, 46, 30}
> ./sdd/hctx2/tags:wake_batch=7
> ./sdd/hctx2/tags:wake_index=0
> ./sdd/hctx2/tags:ws={
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx2/tags:}
> ./sdd/hctx2/tags:round_robin=0
> ./sdd/hctx2/ctx_map:00000000: 00
> ./sdd/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdd/hctx2/state:
> ./sdd/hctx1/cpu1/completed:5 15
> ./sdd/hctx1/cpu1/merged:0
> ./sdd/hctx1/cpu1/dispatched:5 15
> ./sdd/hctx1/active:0
> ./sdd/hctx1/run:26
> ./sdd/hctx1/queued:20
> ./sdd/hctx1/dispatched: 0 7
> ./sdd/hctx1/dispatched: 1 19
> ./sdd/hctx1/dispatched: 2 0
> ./sdd/hctx1/dispatched: 4 0
> ./sdd/hctx1/dispatched: 8 0
> ./sdd/hctx1/dispatched: 16 0
> ./sdd/hctx1/dispatched: 32+ 0
> ./sdd/hctx1/io_poll:considered=0
> ./sdd/hctx1/io_poll:invoked=0
> ./sdd/hctx1/io_poll:success=0
> ./sdd/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdd/hctx1/tags:nr_tags=62
> ./sdd/hctx1/tags:nr_reserved_tags=0
> ./sdd/hctx1/tags:active_queues=0
> ./sdd/hctx1/tags:
> ./sdd/hctx1/tags:bitmap_tags:
> ./sdd/hctx1/tags:depth=62
> ./sdd/hctx1/tags:busy=0
> ./sdd/hctx1/tags:bits_per_word=8
> ./sdd/hctx1/tags:map_nr=8
> ./sdd/hctx1/tags:alloc_hint={29, 0, 21, 37}
> ./sdd/hctx1/tags:wake_batch=7
> ./sdd/hctx1/tags:wake_index=0
> ./sdd/hctx1/tags:ws={
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx1/tags:}
> ./sdd/hctx1/tags:round_robin=0
> ./sdd/hctx1/ctx_map:00000000: 00
> ./sdd/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdd/hctx1/state:
> ./sdd/hctx0/cpu0/completed:34 8
> ./sdd/hctx0/cpu0/merged:0
> ./sdd/hctx0/cpu0/dispatched:34 8
> ./sdd/hctx0/active:0
> ./sdd/hctx0/run:43
> ./sdd/hctx0/queued:42
> ./sdd/hctx0/dispatched: 0 4
> ./sdd/hctx0/dispatched: 1 39
> ./sdd/hctx0/dispatched: 2 0
> ./sdd/hctx0/dispatched: 4 0
> ./sdd/hctx0/dispatched: 8 0
> ./sdd/hctx0/dispatched: 16 0
> ./sdd/hctx0/dispatched: 32+ 0
> ./sdd/hctx0/io_poll:considered=0
> ./sdd/hctx0/io_poll:invoked=0
> ./sdd/hctx0/io_poll:success=0
> ./sdd/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdd/hctx0/tags:nr_tags=62
> ./sdd/hctx0/tags:nr_reserved_tags=0
> ./sdd/hctx0/tags:active_queues=0
> ./sdd/hctx0/tags:
> ./sdd/hctx0/tags:bitmap_tags:
> ./sdd/hctx0/tags:depth=62
> ./sdd/hctx0/tags:busy=0
> ./sdd/hctx0/tags:bits_per_word=8
> ./sdd/hctx0/tags:map_nr=8
> ./sdd/hctx0/tags:alloc_hint={27, 47, 46, 38}
> ./sdd/hctx0/tags:wake_batch=7
> ./sdd/hctx0/tags:wake_index=0
> ./sdd/hctx0/tags:ws={
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdd/hctx0/tags:}
> ./sdd/hctx0/tags:round_robin=0
> ./sdd/hctx0/ctx_map:00000000: 00
> ./sdd/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdd/hctx0/state:
> ./sdd/write_hints:hint0: 0
> ./sdd/write_hints:hint1: 0
> ./sdd/write_hints:hint2: 0
> ./sdd/write_hints:hint3: 0
> ./sdd/write_hints:hint4: 0
> ./sdd/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
> ./sdd/poll_stat:read (512 Bytes): samples=0
> ./sdd/poll_stat:write (512 Bytes): samples=0
> ./sdd/poll_stat:read (1024 Bytes): samples=0
> ./sdd/poll_stat:write (1024 Bytes): samples=0
> ./sdd/poll_stat:read (2048 Bytes): samples=0
> ./sdd/poll_stat:write (2048 Bytes): samples=0
> ./sdd/poll_stat:read (4096 Bytes): samples=0
> ./sdd/poll_stat:write (4096 Bytes): samples=0
> ./sdd/poll_stat:read (8192 Bytes): samples=0
> ./sdd/poll_stat:write (8192 Bytes): samples=0
> ./sdd/poll_stat:read (16384 Bytes): samples=0
> ./sdd/poll_stat:write (16384 Bytes): samples=0
> ./sdd/poll_stat:read (32768 Bytes): samples=0
> ./sdd/poll_stat:write (32768 Bytes): samples=0
> ./sdd/poll_stat:read (65536 Bytes): samples=0
> ./sdd/poll_stat:write (65536 Bytes): samples=0
> ./sdc/hctx3/cpu3/completed:1 11
> ./sdc/hctx3/cpu3/merged:0
> ./sdc/hctx3/cpu3/dispatched:1 11
> ./sdc/hctx3/active:0
> ./sdc/hctx3/run:15
> ./sdc/hctx3/queued:12
> ./sdc/hctx3/dispatched: 0 2
> ./sdc/hctx3/dispatched: 1 12
> ./sdc/hctx3/dispatched: 2 0
> ./sdc/hctx3/dispatched: 4 0
> ./sdc/hctx3/dispatched: 8 0
> ./sdc/hctx3/dispatched: 16 0
> ./sdc/hctx3/dispatched: 32+ 0
> ./sdc/hctx3/io_poll:considered=0
> ./sdc/hctx3/io_poll:invoked=0
> ./sdc/hctx3/io_poll:success=0
> ./sdc/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdc/hctx3/tags:nr_tags=62
> ./sdc/hctx3/tags:nr_reserved_tags=0
> ./sdc/hctx3/tags:active_queues=0
> ./sdc/hctx3/tags:
> ./sdc/hctx3/tags:bitmap_tags:
> ./sdc/hctx3/tags:depth=62
> ./sdc/hctx3/tags:busy=0
> ./sdc/hctx3/tags:bits_per_word=8
> ./sdc/hctx3/tags:map_nr=8
> ./sdc/hctx3/tags:alloc_hint={42, 53, 24, 50}
> ./sdc/hctx3/tags:wake_batch=7
> ./sdc/hctx3/tags:wake_index=0
> ./sdc/hctx3/tags:ws={
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx3/tags:}
> ./sdc/hctx3/tags:round_robin=0
> ./sdc/hctx3/ctx_map:00000000: 00
> ./sdc/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdc/hctx3/state:
> ./sdc/hctx2/cpu2/completed:16 37
> ./sdc/hctx2/cpu2/merged:0
> ./sdc/hctx2/cpu2/dispatched:16 37
> ./sdc/hctx2/active:0
> ./sdc/hctx2/run:73
> ./sdc/hctx2/queued:53
> ./sdc/hctx2/dispatched: 0 22
> ./sdc/hctx2/dispatched: 1 52
> ./sdc/hctx2/dispatched: 2 0
> ./sdc/hctx2/dispatched: 4 0
> ./sdc/hctx2/dispatched: 8 0
> ./sdc/hctx2/dispatched: 16 0
> ./sdc/hctx2/dispatched: 32+ 0
> ./sdc/hctx2/io_poll:considered=0
> ./sdc/hctx2/io_poll:invoked=0
> ./sdc/hctx2/io_poll:success=0
> ./sdc/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdc/hctx2/tags:nr_tags=62
> ./sdc/hctx2/tags:nr_reserved_tags=0
> ./sdc/hctx2/tags:active_queues=0
> ./sdc/hctx2/tags:
> ./sdc/hctx2/tags:bitmap_tags:
> ./sdc/hctx2/tags:depth=62
> ./sdc/hctx2/tags:busy=0
> ./sdc/hctx2/tags:bits_per_word=8
> ./sdc/hctx2/tags:map_nr=8
> ./sdc/hctx2/tags:alloc_hint={24, 37, 46, 30}
> ./sdc/hctx2/tags:wake_batch=7
> ./sdc/hctx2/tags:wake_index=0
> ./sdc/hctx2/tags:ws={
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx2/tags:}
> ./sdc/hctx2/tags:round_robin=0
> ./sdc/hctx2/ctx_map:00000000: 00
> ./sdc/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdc/hctx2/state:
> ./sdc/hctx1/cpu1/completed:32 8
> ./sdc/hctx1/cpu1/merged:0
> ./sdc/hctx1/cpu1/dispatched:32 8
> ./sdc/hctx1/active:0
> ./sdc/hctx1/run:10
> ./sdc/hctx1/queued:40
> ./sdc/hctx1/dispatched: 0 2
> ./sdc/hctx1/dispatched: 1 8
> ./sdc/hctx1/dispatched: 2 0
> ./sdc/hctx1/dispatched: 4 0
> ./sdc/hctx1/dispatched: 8 0
> ./sdc/hctx1/dispatched: 16 0
> ./sdc/hctx1/dispatched: 32+ 0
> ./sdc/hctx1/io_poll:considered=0
> ./sdc/hctx1/io_poll:invoked=0
> ./sdc/hctx1/io_poll:success=0
> ./sdc/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdc/hctx1/tags:nr_tags=62
> ./sdc/hctx1/tags:nr_reserved_tags=0
> ./sdc/hctx1/tags:active_queues=0
> ./sdc/hctx1/tags:
> ./sdc/hctx1/tags:bitmap_tags:
> ./sdc/hctx1/tags:depth=62
> ./sdc/hctx1/tags:busy=0
> ./sdc/hctx1/tags:bits_per_word=8
> ./sdc/hctx1/tags:map_nr=8
> ./sdc/hctx1/tags:alloc_hint={29, 0, 21, 37}
> ./sdc/hctx1/tags:wake_batch=7
> ./sdc/hctx1/tags:wake_index=0
> ./sdc/hctx1/tags:ws={
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx1/tags:}
> ./sdc/hctx1/tags:round_robin=0
> ./sdc/hctx1/ctx_map:00000000: 00
> ./sdc/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdc/hctx1/state:
> ./sdc/hctx0/cpu0/completed:39 12
> ./sdc/hctx0/cpu0/merged:0
> ./sdc/hctx0/cpu0/dispatched:39 12
> ./sdc/hctx0/active:0
> ./sdc/hctx0/run:48
> ./sdc/hctx0/queued:51
> ./sdc/hctx0/dispatched: 0 0
> ./sdc/hctx0/dispatched: 1 48
> ./sdc/hctx0/dispatched: 2 0
> ./sdc/hctx0/dispatched: 4 0
> ./sdc/hctx0/dispatched: 8 0
> ./sdc/hctx0/dispatched: 16 0
> ./sdc/hctx0/dispatched: 32+ 0
> ./sdc/hctx0/io_poll:considered=0
> ./sdc/hctx0/io_poll:invoked=0
> ./sdc/hctx0/io_poll:success=0
> ./sdc/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> ./sdc/hctx0/tags:nr_tags=62
> ./sdc/hctx0/tags:nr_reserved_tags=0
> ./sdc/hctx0/tags:active_queues=0
> ./sdc/hctx0/tags:
> ./sdc/hctx0/tags:bitmap_tags:
> ./sdc/hctx0/tags:depth=62
> ./sdc/hctx0/tags:busy=0
> ./sdc/hctx0/tags:bits_per_word=8
> ./sdc/hctx0/tags:map_nr=8
> ./sdc/hctx0/tags:alloc_hint={27, 47, 46, 38}
> ./sdc/hctx0/tags:wake_batch=7
> ./sdc/hctx0/tags:wake_index=0
> ./sdc/hctx0/tags:ws={
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> ./sdc/hctx0/tags:}
> ./sdc/hctx0/tags:round_robin=0
> ./sdc/hctx0/ctx_map:00000000: 00
> ./sdc/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> ./sdc/hctx0/state:
> ./sdc/write_hints:hint0: 0
> ./sdc/write_hints:hint1: 0
> ./sdc/write_hints:hint2: 0
> ./sdc/write_hints:hint3: 0
> ./sdc/write_hints:hint4: 0
> ./sdc/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
> ./sdc/poll_stat:read (512 Bytes): samples=0
> ./sdc/poll_stat:write (512 Bytes): samples=0
> ./sdc/poll_stat:read (1024 Bytes): samples=0
> ./sdc/poll_stat:write (1024 Bytes): samples=0
> ./sdc/poll_stat:read (2048 Bytes): samples=0
> ./sdc/poll_stat:write (2048 Bytes): samples=0
> ./sdc/poll_stat:read (4096 Bytes): samples=0
> ./sdc/poll_stat:write (4096 Bytes): samples=0
> ./sdc/poll_stat:read (8192 Bytes): samples=0
> ./sdc/poll_stat:write (8192 Bytes): samples=0
> ./sdc/poll_stat:read (16384 Bytes): samples=0
> ./sdc/poll_stat:write (16384 Bytes): samples=0
> ./sdc/poll_stat:read (32768 Bytes): samples=0
> ./sdc/poll_stat:write (32768 Bytes): samples=0
> ./sdc/poll_stat:read (65536 Bytes): samples=0
> ./sdc/poll_stat:write (65536 Bytes): samples=0
> ./sdb/hctx0/cpu3/completed:42 6
> ./sdb/hctx0/cpu3/merged:0
> ./sdb/hctx0/cpu3/dispatched:42 6
> ./sdb/hctx0/cpu2/completed:110 13
> ./sdb/hctx0/cpu2/merged:0
> ./sdb/hctx0/cpu2/dispatched:110 13
> ./sdb/hctx0/cpu1/completed:35 39
> ./sdb/hctx0/cpu1/merged:0
> ./sdb/hctx0/cpu1/dispatched:35 39
> ./sdb/hctx0/cpu0/completed:35 26
> ./sdb/hctx0/cpu0/merged:0
> ./sdb/hctx0/cpu0/dispatched:35 26
> ./sdb/hctx0/active:0
> ./sdb/hctx0/run:300
> ./sdb/hctx0/queued:306
> ./sdb/hctx0/dispatched: 0 0
> ./sdb/hctx0/dispatched: 1 306
> ./sdb/hctx0/dispatched: 2 0
> ./sdb/hctx0/dispatched: 4 0
> ./sdb/hctx0/dispatched: 8 0
> ./sdb/hctx0/dispatched: 16 0
> ./sdb/hctx0/dispatched: 32+ 0
> ./sdb/hctx0/io_poll:considered=0
> ./sdb/hctx0/io_poll:invoked=0
> ./sdb/hctx0/io_poll:success=0
> ./sdb/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./sdb/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> ./sdb/hctx0/sched_tags:nr_tags=256
> ./sdb/hctx0/sched_tags:nr_reserved_tags=0
> ./sdb/hctx0/sched_tags:active_queues=0
> ./sdb/hctx0/sched_tags:
> ./sdb/hctx0/sched_tags:bitmap_tags:
> ./sdb/hctx0/sched_tags:depth=256
> ./sdb/hctx0/sched_tags:busy=0
> ./sdb/hctx0/sched_tags:bits_per_word=64
> ./sdb/hctx0/sched_tags:map_nr=4
> ./sdb/hctx0/sched_tags:alloc_hint={68, 116, 75, 172}
> ./sdb/hctx0/sched_tags:wake_batch=8
> ./sdb/hctx0/sched_tags:wake_index=0
> ./sdb/hctx0/sched_tags:ws={
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/sched_tags:}
> ./sdb/hctx0/sched_tags:round_robin=0
> ./sdb/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./sdb/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000
> ./sdb/hctx0/tags:nr_tags=192
> ./sdb/hctx0/tags:nr_reserved_tags=0
> ./sdb/hctx0/tags:active_queues=0
> ./sdb/hctx0/tags:
> ./sdb/hctx0/tags:bitmap_tags:
> ./sdb/hctx0/tags:depth=192
> ./sdb/hctx0/tags:busy=0
> ./sdb/hctx0/tags:bits_per_word=32
> ./sdb/hctx0/tags:map_nr=6
> ./sdb/hctx0/tags:alloc_hint={5, 24, 47, 126}
> ./sdb/hctx0/tags:wake_batch=8
> ./sdb/hctx0/tags:wake_index=0
> ./sdb/hctx0/tags:ws={
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sdb/hctx0/tags:}
> ./sdb/hctx0/tags:round_robin=0
> ./sdb/hctx0/ctx_map:00000000: 00
> ./sdb/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> ./sdb/hctx0/state:
> ./sdb/sched/starved:0
> ./sdb/sched/batching:1
> ./sdb/write_hints:hint0: 0
> ./sdb/write_hints:hint1: 0
> ./sdb/write_hints:hint2: 0
> ./sdb/write_hints:hint3: 0
> ./sdb/write_hints:hint4: 0
> ./sdb/state:BIDI|SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
> ./sdb/poll_stat:read (512 Bytes): samples=0
> ./sdb/poll_stat:write (512 Bytes): samples=0
> ./sdb/poll_stat:read (1024 Bytes): samples=0
> ./sdb/poll_stat:write (1024 Bytes): samples=0
> ./sdb/poll_stat:read (2048 Bytes): samples=0
> ./sdb/poll_stat:write (2048 Bytes): samples=0
> ./sdb/poll_stat:read (4096 Bytes): samples=0
> ./sdb/poll_stat:write (4096 Bytes): samples=0
> ./sdb/poll_stat:read (8192 Bytes): samples=0
> ./sdb/poll_stat:write (8192 Bytes): samples=0
> ./sdb/poll_stat:read (16384 Bytes): samples=0
> ./sdb/poll_stat:write (16384 Bytes): samples=0
> ./sdb/poll_stat:read (32768 Bytes): samples=0
> ./sdb/poll_stat:write (32768 Bytes): samples=0
> ./sdb/poll_stat:read (65536 Bytes): samples=0
> ./sdb/poll_stat:write (65536 Bytes): samples=0
> ./sda/hctx0/cpu3/completed:46 3
> ./sda/hctx0/cpu3/merged:0
> ./sda/hctx0/cpu3/dispatched:46 3
> ./sda/hctx0/cpu2/completed:41 27
> ./sda/hctx0/cpu2/merged:0
> ./sda/hctx0/cpu2/dispatched:41 27
> ./sda/hctx0/cpu1/completed:2 8
> ./sda/hctx0/cpu1/merged:0
> ./sda/hctx0/cpu1/dispatched:2 8
> ./sda/hctx0/cpu0/completed:1 43
> ./sda/hctx0/cpu0/merged:0
> ./sda/hctx0/cpu0/dispatched:1 43
> ./sda/hctx0/active:0
> ./sda/hctx0/run:144
> ./sda/hctx0/queued:171
> ./sda/hctx0/dispatched: 0 0
> ./sda/hctx0/dispatched: 1 171
> ./sda/hctx0/dispatched: 2 0
> ./sda/hctx0/dispatched: 4 0
> ./sda/hctx0/dispatched: 8 0
> ./sda/hctx0/dispatched: 16 0
> ./sda/hctx0/dispatched: 32+ 0
> ./sda/hctx0/io_poll:considered=0
> ./sda/hctx0/io_poll:invoked=0
> ./sda/hctx0/io_poll:success=0
> ./sda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./sda/hctx0/tags:nr_tags=128
> ./sda/hctx0/tags:nr_reserved_tags=0
> ./sda/hctx0/tags:active_queues=0
> ./sda/hctx0/tags:
> ./sda/hctx0/tags:bitmap_tags:
> ./sda/hctx0/tags:depth=128
> ./sda/hctx0/tags:busy=0
> ./sda/hctx0/tags:bits_per_word=32
> ./sda/hctx0/tags:map_nr=4
> ./sda/hctx0/tags:alloc_hint={111, 80, 44, 68}
> ./sda/hctx0/tags:wake_batch=8
> ./sda/hctx0/tags:wake_index=0
> ./sda/hctx0/tags:ws={
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./sda/hctx0/tags:}
> ./sda/hctx0/tags:round_robin=0
> ./sda/hctx0/ctx_map:00000000: 00
> ./sda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> ./sda/hctx0/state:
> ./sda/write_hints:hint0: 0
> ./sda/write_hints:hint1: 0
> ./sda/write_hints:hint2: 0
> ./sda/write_hints:hint3: 0
> ./sda/write_hints:hint4: 0
> ./sda/state:SAME_COMP|IO_STAT|DISCARD|INIT_DONE|POLL|WC|REGISTERED|SCSI_PASSTHROUGH
> ./sda/poll_stat:read (512 Bytes): samples=0
> ./sda/poll_stat:write (512 Bytes): samples=0
> ./sda/poll_stat:read (1024 Bytes): samples=0
> ./sda/poll_stat:write (1024 Bytes): samples=0
> ./sda/poll_stat:read (2048 Bytes): samples=0
> ./sda/poll_stat:write (2048 Bytes): samples=0
> ./sda/poll_stat:read (4096 Bytes): samples=0
> ./sda/poll_stat:write (4096 Bytes): samples=0
> ./sda/poll_stat:read (8192 Bytes): samples=0
> ./sda/poll_stat:write (8192 Bytes): samples=0
> ./sda/poll_stat:read (16384 Bytes): samples=0
> ./sda/poll_stat:write (16384 Bytes): samples=0
> ./sda/poll_stat:read (32768 Bytes): samples=0
> ./sda/poll_stat:write (32768 Bytes): samples=0
> ./sda/poll_stat:read (65536 Bytes): samples=0
> ./sda/poll_stat:write (65536 Bytes): samples=0
> ./vda/hctx0/cpu3/completed:1707 27
> ./vda/hctx0/cpu3/merged:0
> ./vda/hctx0/cpu3/dispatched:1707 27
> ./vda/hctx0/cpu2/completed:1027 12
> ./vda/hctx0/cpu2/merged:0
> ./vda/hctx0/cpu2/dispatched:1027 12
> ./vda/hctx0/cpu1/completed:1138 76
> ./vda/hctx0/cpu1/merged:4
> ./vda/hctx0/cpu1/dispatched:1138 76
> ./vda/hctx0/cpu0/completed:2497 44
> ./vda/hctx0/cpu0/merged:2
> ./vda/hctx0/cpu0/dispatched:2497 44
> ./vda/hctx0/active:0
> ./vda/hctx0/run:6412
> ./vda/hctx0/queued:6528
> ./vda/hctx0/dispatched: 0 0
> ./vda/hctx0/dispatched: 1 6361
> ./vda/hctx0/dispatched: 2 20
> ./vda/hctx0/dispatched: 4 13
> ./vda/hctx0/dispatched: 8 8
> ./vda/hctx0/dispatched: 16 8
> ./vda/hctx0/dispatched: 32+ 0
> ./vda/hctx0/io_poll:considered=0
> ./vda/hctx0/io_poll:invoked=0
> ./vda/hctx0/io_poll:success=0
> ./vda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> ./vda/hctx0/tags:nr_tags=128
> ./vda/hctx0/tags:nr_reserved_tags=0
> ./vda/hctx0/tags:active_queues=0
> ./vda/hctx0/tags:
> ./vda/hctx0/tags:bitmap_tags:
> ./vda/hctx0/tags:depth=128
> ./vda/hctx0/tags:busy=0
> ./vda/hctx0/tags:bits_per_word=32
> ./vda/hctx0/tags:map_nr=4
> ./vda/hctx0/tags:alloc_hint={28, 24, 114, 37}
> ./vda/hctx0/tags:wake_batch=8
> ./vda/hctx0/tags:wake_index=0
> ./vda/hctx0/tags:ws={
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> ./vda/hctx0/tags:}
> ./vda/hctx0/tags:round_robin=0
> ./vda/hctx0/ctx_map:00000000: 00
> ./vda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE
> ./vda/hctx0/state:
> ./vda/write_hints:hint0: 0
> ./vda/write_hints:hint1: 0
> ./vda/write_hints:hint2: 0
> ./vda/write_hints:hint3: 0
> ./vda/write_hints:hint4: 0
> ./vda/state:SAME_COMP|IO_STAT|INIT_DONE|NO_SG_MERGE|POLL|WC|REGISTERED
> ./vda/poll_stat:read (512 Bytes): samples=0
> ./vda/poll_stat:write (512 Bytes): samples=0
> ./vda/poll_stat:read (1024 Bytes): samples=0
> ./vda/poll_stat:write (1024 Bytes): samples=0
> ./vda/poll_stat:read (2048 Bytes): samples=0
> ./vda/poll_stat:write (2048 Bytes): samples=0
> ./vda/poll_stat:read (4096 Bytes): samples=0
> ./vda/poll_stat:write (4096 Bytes): samples=0
> ./vda/poll_stat:read (8192 Bytes): samples=0
> ./vda/poll_stat:write (8192 Bytes): samples=0
> ./vda/poll_stat:read (16384 Bytes): samples=0
> ./vda/poll_stat:write (16384 Bytes): samples=0
> ./vda/poll_stat:read (32768 Bytes): samples=0
> ./vda/poll_stat:write (32768 Bytes): samples=0
> ./vda/poll_stat:read (65536 Bytes): samples=0
> ./vda/poll_stat:write (65536 Bytes): samples=0
> ./sr0/hctx0/cpu3/completed:0 22
> ./sr0/hctx0/cpu3/merged:0
> ./sr0/hctx0/cpu3/dispatched:0 22
> ./sr0/hctx0/cpu2/completed:46 107
> ./sr0/hctx0/cpu2/merged:0
> ./sr0/hctx0/cpu2/dispatched:46 107
> ./sr0/hctx0/cpu1/completed:0 10
> ./sr0/hctx0/cpu1/merged:0
> ./sr0/hctx0/cpu1/dispatched:0 10
> ./sr0/hctx0/cpu0/completed:0 18
> ./sr0/hctx0/cpu0/merged:0
> ./sr0/hctx0/cpu0/dispatched:0 18
> ./sr0/hctx0/active:0
> ./sr0/hctx0/run:203
> ./sr0/hctx0/queued:203
> ./sr0/hctx0/dispatched: 0 0
> ./sr0/hctx0/dispatched: 1 203
> ./sr0/hctx0/dispatched: 2 0
> ./sr0/hctx0/dispatched: 4 0
> ./sr0/hctx0/dispatched: 8 0
> ./sr0/hctx0/dispatched: 16 0
> ./sr0/hctx0/dispatched: 32+ 0
> ./sr0/hctx0/io_poll:considered=0
> ./sr0/hctx0/io_poll:invoked=0
> ./sr0/hctx0/io_poll:success=0
> ./sr0/hctx0/tags_bitmap:00000000: 00
> ./sr0/hctx0/tags:nr_tags=1
> ./sr0/hctx0/tags:nr_reserved_tags=0
> ./sr0/hctx0/tags:active_queues=0
> ./sr0/hctx0/tags:
> ./sr0/hctx0/tags:bitmap_tags:
> ./sr0/hctx0/tags:depth=1
> ./sr0/hctx0/tags:busy=0
> ./sr0/hctx0/tags:bits_per_word=64
> ./sr0/hctx0/tags:map_nr=1
> ./sr0/hctx0/tags:alloc_hint={0, 0, 0, 0}
> ./sr0/hctx0/tags:wake_batch=1
> ./sr0/hctx0/tags:wake_index=7
> ./sr0/hctx0/tags:ws={
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> ./sr0/hctx0/tags:}
> ./sr0/hctx0/tags:round_robin=1
> ./sr0/hctx0/ctx_map:00000000: 00
> ./sr0/hctx0/flags:alloc_policy=RR SHOULD_MERGE|SG_MERGE
> ./sr0/hctx0/state:
> ./sr0/write_hints:hint0: 0
> ./sr0/write_hints:hint1: 0
> ./sr0/write_hints:hint2: 0
> ./sr0/write_hints:hint3: 0
> ./sr0/write_hints:hint4: 0
> ./sr0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|FLUSH_NQ|REGISTERED|SCSI_PASSTHROUGH
> ./sr0/poll_stat:read (512 Bytes): samples=0
> ./sr0/poll_stat:write (512 Bytes): samples=0
> ./sr0/poll_stat:read (1024 Bytes): samples=0
> ./sr0/poll_stat:write (1024 Bytes): samples=0
> ./sr0/poll_stat:read (2048 Bytes): samples=0
> ./sr0/poll_stat:write (2048 Bytes): samples=0
> ./sr0/poll_stat:read (4096 Bytes): samples=0
> ./sr0/poll_stat:write (4096 Bytes): samples=0
> ./sr0/poll_stat:read (8192 Bytes): samples=0
> ./sr0/poll_stat:write (8192 Bytes): samples=0
> ./sr0/poll_stat:read (16384 Bytes): samples=0
> ./sr0/poll_stat:write (16384 Bytes): samples=0
> ./sr0/poll_stat:read (32768 Bytes): samples=0
> ./sr0/poll_stat:write (32768 Bytes): samples=0
> ./sr0/poll_stat:read (65536 Bytes): samples=0
> ./sr0/poll_stat:write (65536 Bytes): samples=0
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 18:18 ` Mike Snitzer
@ 2018-01-17 18:27 ` Bart Van Assche
2018-01-17 18:29 ` Laurence Oberman
2018-01-18 1:53 ` Ming Lei
2 siblings, 0 replies; 12+ messages in thread
From: Bart Van Assche @ 2018-01-17 18:27 UTC (permalink / raw)
To: snitzer; +Cc: axboe, dm-devel, loberman, ming.lei
On Wed, 2018-01-17 at 13:18 -0500, Mike Snitzer wrote:
> But Ming Lei should be able to help make sense of this situation too.
> Also Laurence may be able to help reproduce your specific srp-test on
> his testbed.
Hello Mike,
Something that I should have mentioned in my previous e-mail is that the
command-line option -c for srp-test means "use the rdma_rxe driver over a
local Ethernet interface". So that test can be run in any virtual machine.
The stall also occurs on an IB setup and occurs even more frequent on such
a setup.
Thanks,
Bart.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 18:18 ` Mike Snitzer
2018-01-17 18:27 ` Bart Van Assche
@ 2018-01-17 18:29 ` Laurence Oberman
2018-01-17 19:29 ` Bart Van Assche
2018-01-18 1:53 ` Ming Lei
2 siblings, 1 reply; 12+ messages in thread
From: Laurence Oberman @ 2018-01-17 18:29 UTC (permalink / raw)
To: Mike Snitzer, Bart Van Assche; +Cc: axboe, dm-devel, Ming Lei
On Wed, 2018-01-17 at 13:18 -0500, Mike Snitzer wrote:
> On Wed, Jan 17 2018 at 1:05pm -0500,
> Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
>
> > Hello Mike,
> >
> > With Jens' latest for-next branch (commit 6c4f43dfecae) and with
> > commit
> > 6077c2d70609 ("dm rq: Avoid that request processing stalls
> > sporadically")
> > reverted I see again queue stalls every now and then. An example:
> >
> > # srp-test/run_tests -c -d -r 10 -t 02-mq -e kyber
> > Running test /home/bart/software/infiniband/srp-test/tests/02-mq
> > ...
> > Test file I/O on top of multipath concurrently with logout and
> > login (0 min; mq)
> > /dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000:
> > not found
> > Test /home/bart/software/infiniband/srp-test/tests/02-mq failed
> >
> > I have attached the output of the following command to this e-mail:
> > (cd /sys/kernel/debug/block && find -type f | xargs grep -aH '')
> >
> > Can you have a look at this?
>
> Yes, though I need to prepare for a trip that starts on Sunday (so
> I'll
> be distracted through the end of next week).
>
> But Ming Lei should be able to help make sense of this situation too.
> Also Laurence may be able to help reproduce your specific srp-test on
> his testbed.
>
> Mike
>
>
> > ./dm-2/hctx0/cpu3/completed:0 0
> > ./dm-2/hctx0/cpu3/merged:0
> > ./dm-2/hctx0/cpu3/dispatched:0 0
> > ./dm-2/hctx0/cpu2/completed:0 0
> > ./dm-2/hctx0/cpu2/merged:0
> > ./dm-2/hctx0/cpu2/dispatched:0 0
> > ./dm-2/hctx0/cpu1/completed:25 0
> > ./dm-2/hctx0/cpu1/merged:0
> > ./dm-2/hctx0/cpu1/dispatched:26 0
> > ./dm-2/hctx0/cpu0/completed:0 0
> > ./dm-2/hctx0/cpu0/merged:0
> > ./dm-2/hctx0/cpu0/dispatched:0 0
> > ./dm-2/hctx0/active:0
> > ./dm-2/hctx0/run:27
> > ./dm-2/hctx0/queued:26
> > ./dm-2/hctx0/dispatched: 0 6
> > ./dm-2/hctx0/dispatched: 1 25
> > ./dm-2/hctx0/dispatched: 2 0
> > ./dm-2/hctx0/dispatched: 4 0
> > ./dm-2/hctx0/dispatched: 8 0
> > ./dm-2/hctx0/dispatched: 16 0
> > ./dm-2/hctx0/dispatched: 32+ 0
> > ./dm-2/hctx0/io_poll:considered=0
> > ./dm-2/hctx0/io_poll:invoked=0
> > ./dm-2/hctx0/io_poll:success=0
> > ./dm-2/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000
> > 0000 0000 0020
> > ./dm-2/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000
> > 0000 0000 0000
> > ./dm-2/hctx0/sched_tags:nr_tags=256
> > ./dm-2/hctx0/sched_tags:nr_reserved_tags=0
> > ./dm-2/hctx0/sched_tags:active_queues=0
> > ./dm-2/hctx0/sched_tags:
> > ./dm-2/hctx0/sched_tags:bitmap_tags:
> > ./dm-2/hctx0/sched_tags:depth=256
> > ./dm-2/hctx0/sched_tags:busy=1
> > ./dm-2/hctx0/sched_tags:bits_per_word=64
> > ./dm-2/hctx0/sched_tags:map_nr=4
> > ./dm-2/hctx0/sched_tags:alloc_hint={83, 124, 234, 58}
> > ./dm-2/hctx0/sched_tags:wake_batch=8
> > ./dm-2/hctx0/sched_tags:wake_index=0
> > ./dm-2/hctx0/sched_tags:ws={
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags:}
> > ./dm-2/hctx0/sched_tags:round_robin=0
> > ./dm-2/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-2/hctx0/tags:nr_tags=2048
> > ./dm-2/hctx0/tags:nr_reserved_tags=0
> > ./dm-2/hctx0/tags:active_queues=0
> > ./dm-2/hctx0/tags:
> > ./dm-2/hctx0/tags:bitmap_tags:
> > ./dm-2/hctx0/tags:depth=2048
> > ./dm-2/hctx0/tags:busy=0
> > ./dm-2/hctx0/tags:bits_per_word=64
> > ./dm-2/hctx0/tags:map_nr=32
> > ./dm-2/hctx0/tags:alloc_hint={1060, 81, 1050, 844}
> > ./dm-2/hctx0/tags:wake_batch=8
> > ./dm-2/hctx0/tags:wake_index=0
> > ./dm-2/hctx0/tags:ws={
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags:}
> > ./dm-2/hctx0/tags:round_robin=0
> > ./dm-2/hctx0/ctx_map:00000000: 00
> > ./dm-2/hctx0/dispatch:00000000a7224aed {.op=READ, .cmd_flags=,
> > .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=125}
> > ./dm-2/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./dm-2/hctx0/state:SCHED_RESTART
> > ./dm-2/sched/starved:0
> > ./dm-2/sched/batching:5
> > ./dm-2/write_hints:hint0: 0
> > ./dm-2/write_hints:hint1: 0
> > ./dm-2/write_hints:hint2: 0
> > ./dm-2/write_hints:hint3: 0
> > ./dm-2/write_hints:hint4: 0
> > ./dm-2/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
> > ./dm-2/poll_stat:read (512 Bytes): samples=0
> > ./dm-2/poll_stat:write (512 Bytes): samples=0
> > ./dm-2/poll_stat:read (1024 Bytes): samples=0
> > ./dm-2/poll_stat:write (1024 Bytes): samples=0
> > ./dm-2/poll_stat:read (2048 Bytes): samples=0
> > ./dm-2/poll_stat:write (2048 Bytes): samples=0
> > ./dm-2/poll_stat:read (4096 Bytes): samples=0
> > ./dm-2/poll_stat:write (4096 Bytes): samples=0
> > ./dm-2/poll_stat:read (8192 Bytes): samples=0
> > ./dm-2/poll_stat:write (8192 Bytes): samples=0
> > ./dm-2/poll_stat:read (16384 Bytes): samples=0
> > ./dm-2/poll_stat:write (16384 Bytes): samples=0
> > ./dm-2/poll_stat:read (32768 Bytes): samples=0
> > ./dm-2/poll_stat:write (32768 Bytes): samples=0
> > ./dm-2/poll_stat:read (65536 Bytes): samples=0
> > ./dm-2/poll_stat:write (65536 Bytes): samples=0
> > ./dm-1/hctx0/cpu3/completed:26 0
> > ./dm-1/hctx0/cpu3/merged:0
> > ./dm-1/hctx0/cpu3/dispatched:26 0
> > ./dm-1/hctx0/cpu2/completed:2 0
> > ./dm-1/hctx0/cpu2/merged:0
> > ./dm-1/hctx0/cpu2/dispatched:2 0
> > ./dm-1/hctx0/cpu1/completed:2 0
> > ./dm-1/hctx0/cpu1/merged:0
> > ./dm-1/hctx0/cpu1/dispatched:2 0
> > ./dm-1/hctx0/cpu0/completed:2 0
> > ./dm-1/hctx0/cpu0/merged:0
> > ./dm-1/hctx0/cpu0/dispatched:2 0
> > ./dm-1/hctx0/active:0
> > ./dm-1/hctx0/run:34
> > ./dm-1/hctx0/queued:32
> > ./dm-1/hctx0/dispatched: 0 7
> > ./dm-1/hctx0/dispatched: 1 32
> > ./dm-1/hctx0/dispatched: 2 0
> > ./dm-1/hctx0/dispatched: 4 0
> > ./dm-1/hctx0/dispatched: 8 0
> > ./dm-1/hctx0/dispatched: 16 0
> > ./dm-1/hctx0/dispatched: 32+ 0
> > ./dm-1/hctx0/io_poll:considered=0
> > ./dm-1/hctx0/io_poll:invoked=0
> > ./dm-1/hctx0/io_poll:success=0
> > ./dm-1/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000
> > 0000 0000 0000
> > ./dm-1/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000
> > 0000 0000 0000
> > ./dm-1/hctx0/sched_tags:nr_tags=256
> > ./dm-1/hctx0/sched_tags:nr_reserved_tags=0
> > ./dm-1/hctx0/sched_tags:active_queues=0
> > ./dm-1/hctx0/sched_tags:
> > ./dm-1/hctx0/sched_tags:bitmap_tags:
> > ./dm-1/hctx0/sched_tags:depth=256
> > ./dm-1/hctx0/sched_tags:busy=0
> > ./dm-1/hctx0/sched_tags:bits_per_word=64
> > ./dm-1/hctx0/sched_tags:map_nr=4
> > ./dm-1/hctx0/sched_tags:alloc_hint={125, 55, 142, 82}
> > ./dm-1/hctx0/sched_tags:wake_batch=8
> > ./dm-1/hctx0/sched_tags:wake_index=0
> > ./dm-1/hctx0/sched_tags:ws={
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags:}
> > ./dm-1/hctx0/sched_tags:round_robin=0
> > ./dm-1/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-1/hctx0/tags:nr_tags=2048
> > ./dm-1/hctx0/tags:nr_reserved_tags=0
> > ./dm-1/hctx0/tags:active_queues=0
> > ./dm-1/hctx0/tags:
> > ./dm-1/hctx0/tags:bitmap_tags:
> > ./dm-1/hctx0/tags:depth=2048
> > ./dm-1/hctx0/tags:busy=0
> > ./dm-1/hctx0/tags:bits_per_word=64
> > ./dm-1/hctx0/tags:map_nr=32
> > ./dm-1/hctx0/tags:alloc_hint={478, 1303, 102, 127}
> > ./dm-1/hctx0/tags:wake_batch=8
> > ./dm-1/hctx0/tags:wake_index=0
> > ./dm-1/hctx0/tags:ws={
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags:}
> > ./dm-1/hctx0/tags:round_robin=0
> > ./dm-1/hctx0/ctx_map:00000000: 00
> > ./dm-1/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./dm-1/hctx0/state:
> > ./dm-1/sched/starved:0
> > ./dm-1/sched/batching:1
> > ./dm-1/write_hints:hint0: 0
> > ./dm-1/write_hints:hint1: 0
> > ./dm-1/write_hints:hint2: 0
> > ./dm-1/write_hints:hint3: 0
> > ./dm-1/write_hints:hint4: 0
> > ./dm-1/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
> > ./dm-1/poll_stat:read (512 Bytes): samples=0
> > ./dm-1/poll_stat:write (512 Bytes): samples=0
> > ./dm-1/poll_stat:read (1024 Bytes): samples=0
> > ./dm-1/poll_stat:write (1024 Bytes): samples=0
> > ./dm-1/poll_stat:read (2048 Bytes): samples=0
> > ./dm-1/poll_stat:write (2048 Bytes): samples=0
> > ./dm-1/poll_stat:read (4096 Bytes): samples=0
> > ./dm-1/poll_stat:write (4096 Bytes): samples=0
> > ./dm-1/poll_stat:read (8192 Bytes): samples=0
> > ./dm-1/poll_stat:write (8192 Bytes): samples=0
> > ./dm-1/poll_stat:read (16384 Bytes): samples=0
> > ./dm-1/poll_stat:write (16384 Bytes): samples=0
> > ./dm-1/poll_stat:read (32768 Bytes): samples=0
> > ./dm-1/poll_stat:write (32768 Bytes): samples=0
> > ./dm-1/poll_stat:read (65536 Bytes): samples=0
> > ./dm-1/poll_stat:write (65536 Bytes): samples=0
> > ./dm-0/hctx0/cpu3/completed:13 0
> > ./dm-0/hctx0/cpu3/merged:0
> > ./dm-0/hctx0/cpu3/dispatched:14 0
> > ./dm-0/hctx0/cpu2/completed:0 0
> > ./dm-0/hctx0/cpu2/merged:0
> > ./dm-0/hctx0/cpu2/dispatched:0 0
> > ./dm-0/hctx0/cpu1/completed:0 0
> > ./dm-0/hctx0/cpu1/merged:0
> > ./dm-0/hctx0/cpu1/dispatched:0 0
> > ./dm-0/hctx0/cpu0/completed:0 0
> > ./dm-0/hctx0/cpu0/merged:0
> > ./dm-0/hctx0/cpu0/dispatched:0 0
> > ./dm-0/hctx0/active:0
> > ./dm-0/hctx0/run:15
> > ./dm-0/hctx0/queued:14
> > ./dm-0/hctx0/dispatched: 0 2
> > ./dm-0/hctx0/dispatched: 1 13
> > ./dm-0/hctx0/dispatched: 2 0
> > ./dm-0/hctx0/dispatched: 4 0
> > ./dm-0/hctx0/dispatched: 8 0
> > ./dm-0/hctx0/dispatched: 16 0
> > ./dm-0/hctx0/dispatched: 32+ 0
> > ./dm-0/hctx0/io_poll:considered=0
> > ./dm-0/hctx0/io_poll:invoked=0
> > ./dm-0/hctx0/io_poll:success=0
> > ./dm-0/hctx0/sched_tags_bitmap:00000000: 0001 0000 0000 0000 0000
> > 0000 0000 0000
> > ./dm-0/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000
> > 0000 0000 0000
> > ./dm-0/hctx0/sched_tags:nr_tags=256
> > ./dm-0/hctx0/sched_tags:nr_reserved_tags=0
> > ./dm-0/hctx0/sched_tags:active_queues=0
> > ./dm-0/hctx0/sched_tags:
> > ./dm-0/hctx0/sched_tags:bitmap_tags:
> > ./dm-0/hctx0/sched_tags:depth=256
> > ./dm-0/hctx0/sched_tags:busy=1
> > ./dm-0/hctx0/sched_tags:bits_per_word=64
> > ./dm-0/hctx0/sched_tags:map_nr=4
> > ./dm-0/hctx0/sched_tags:alloc_hint={11, 18, 205, 9}
> > ./dm-0/hctx0/sched_tags:wake_batch=8
> > ./dm-0/hctx0/sched_tags:wake_index=0
> > ./dm-0/hctx0/sched_tags:ws={
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags:}
> > ./dm-0/hctx0/sched_tags:round_robin=0
> > ./dm-0/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./dm-0/hctx0/tags:nr_tags=2048
> > ./dm-0/hctx0/tags:nr_reserved_tags=0
> > ./dm-0/hctx0/tags:active_queues=0
> > ./dm-0/hctx0/tags:
> > ./dm-0/hctx0/tags:bitmap_tags:
> > ./dm-0/hctx0/tags:depth=2048
> > ./dm-0/hctx0/tags:busy=0
> > ./dm-0/hctx0/tags:bits_per_word=64
> > ./dm-0/hctx0/tags:map_nr=32
> > ./dm-0/hctx0/tags:alloc_hint={1392, 843, 879, 1030}
> > ./dm-0/hctx0/tags:wake_batch=8
> > ./dm-0/hctx0/tags:wake_index=0
> > ./dm-0/hctx0/tags:ws={
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags:}
> > ./dm-0/hctx0/tags:round_robin=0
> > ./dm-0/hctx0/ctx_map:00000000: 00
> > ./dm-0/hctx0/dispatch:000000002f291aa3 {.op=READ, .cmd_flags=,
> > .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=8}
> > ./dm-0/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./dm-0/hctx0/state:SCHED_RESTART
> > ./dm-0/sched/starved:0
> > ./dm-0/sched/batching:1
> > ./dm-0/write_hints:hint0: 0
> > ./dm-0/write_hints:hint1: 0
> > ./dm-0/write_hints:hint2: 0
> > ./dm-0/write_hints:hint3: 0
> > ./dm-0/write_hints:hint4: 0
> > ./dm-
> > 0/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED
> > ./dm-0/poll_stat:read (512 Bytes): samples=0
> > ./dm-0/poll_stat:write (512 Bytes): samples=0
> > ./dm-0/poll_stat:read (1024 Bytes): samples=0
> > ./dm-0/poll_stat:write (1024 Bytes): samples=0
> > ./dm-0/poll_stat:read (2048 Bytes): samples=0
> > ./dm-0/poll_stat:write (2048 Bytes): samples=0
> > ./dm-0/poll_stat:read (4096 Bytes): samples=0
> > ./dm-0/poll_stat:write (4096 Bytes): samples=0
> > ./dm-0/poll_stat:read (8192 Bytes): samples=0
> > ./dm-0/poll_stat:write (8192 Bytes): samples=0
> > ./dm-0/poll_stat:read (16384 Bytes): samples=0
> > ./dm-0/poll_stat:write (16384 Bytes): samples=0
> > ./dm-0/poll_stat:read (32768 Bytes): samples=0
> > ./dm-0/poll_stat:write (32768 Bytes): samples=0
> > ./dm-0/poll_stat:read (65536 Bytes): samples=0
> > ./dm-0/poll_stat:write (65536 Bytes): samples=0
> > ./sde/hctx3/cpu3/completed:58 12
> > ./sde/hctx3/cpu3/merged:0
> > ./sde/hctx3/cpu3/dispatched:58 12
> > ./sde/hctx3/active:0
> > ./sde/hctx3/run:39
> > ./sde/hctx3/queued:70
> > ./sde/hctx3/dispatched: 0 2
> > ./sde/hctx3/dispatched: 1 36
> > ./sde/hctx3/dispatched: 2 0
> > ./sde/hctx3/dispatched: 4 0
> > ./sde/hctx3/dispatched: 8 0
> > ./sde/hctx3/dispatched: 16 0
> > ./sde/hctx3/dispatched: 32+ 0
> > ./sde/hctx3/io_poll:considered=0
> > ./sde/hctx3/io_poll:invoked=0
> > ./sde/hctx3/io_poll:success=0
> > ./sde/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx3/tags:nr_tags=62
> > ./sde/hctx3/tags:nr_reserved_tags=0
> > ./sde/hctx3/tags:active_queues=0
> > ./sde/hctx3/tags:
> > ./sde/hctx3/tags:bitmap_tags:
> > ./sde/hctx3/tags:depth=62
> > ./sde/hctx3/tags:busy=0
> > ./sde/hctx3/tags:bits_per_word=8
> > ./sde/hctx3/tags:map_nr=8
> > ./sde/hctx3/tags:alloc_hint={42, 53, 24, 50}
> > ./sde/hctx3/tags:wake_batch=7
> > ./sde/hctx3/tags:wake_index=0
> > ./sde/hctx3/tags:ws={
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags:}
> > ./sde/hctx3/tags:round_robin=0
> > ./sde/hctx3/ctx_map:00000000: 00
> > ./sde/hctx3/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx3/state:
> > ./sde/hctx2/cpu2/completed:2 3
> > ./sde/hctx2/cpu2/merged:0
> > ./sde/hctx2/cpu2/dispatched:2 3
> > ./sde/hctx2/active:0
> > ./sde/hctx2/run:3
> > ./sde/hctx2/queued:5
> > ./sde/hctx2/dispatched: 0 0
> > ./sde/hctx2/dispatched: 1 3
> > ./sde/hctx2/dispatched: 2 0
> > ./sde/hctx2/dispatched: 4 0
> > ./sde/hctx2/dispatched: 8 0
> > ./sde/hctx2/dispatched: 16 0
> > ./sde/hctx2/dispatched: 32+ 0
> > ./sde/hctx2/io_poll:considered=0
> > ./sde/hctx2/io_poll:invoked=0
> > ./sde/hctx2/io_poll:success=0
> > ./sde/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx2/tags:nr_tags=62
> > ./sde/hctx2/tags:nr_reserved_tags=0
> > ./sde/hctx2/tags:active_queues=0
> > ./sde/hctx2/tags:
> > ./sde/hctx2/tags:bitmap_tags:
> > ./sde/hctx2/tags:depth=62
> > ./sde/hctx2/tags:busy=0
> > ./sde/hctx2/tags:bits_per_word=8
> > ./sde/hctx2/tags:map_nr=8
> > ./sde/hctx2/tags:alloc_hint={24, 37, 46, 30}
> > ./sde/hctx2/tags:wake_batch=7
> > ./sde/hctx2/tags:wake_index=0
> > ./sde/hctx2/tags:ws={
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags:}
> > ./sde/hctx2/tags:round_robin=0
> > ./sde/hctx2/ctx_map:00000000: 00
> > ./sde/hctx2/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx2/state:
> > ./sde/hctx1/cpu1/completed:31 47
> > ./sde/hctx1/cpu1/merged:0
> > ./sde/hctx1/cpu1/dispatched:31 47
> > ./sde/hctx1/active:0
> > ./sde/hctx1/run:76
> > ./sde/hctx1/queued:78
> > ./sde/hctx1/dispatched: 0 3
> > ./sde/hctx1/dispatched: 1 71
> > ./sde/hctx1/dispatched: 2 0
> > ./sde/hctx1/dispatched: 4 0
> > ./sde/hctx1/dispatched: 8 0
> > ./sde/hctx1/dispatched: 16 0
> > ./sde/hctx1/dispatched: 32+ 0
> > ./sde/hctx1/io_poll:considered=0
> > ./sde/hctx1/io_poll:invoked=0
> > ./sde/hctx1/io_poll:success=0
> > ./sde/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx1/tags:nr_tags=62
> > ./sde/hctx1/tags:nr_reserved_tags=0
> > ./sde/hctx1/tags:active_queues=0
> > ./sde/hctx1/tags:
> > ./sde/hctx1/tags:bitmap_tags:
> > ./sde/hctx1/tags:depth=62
> > ./sde/hctx1/tags:busy=0
> > ./sde/hctx1/tags:bits_per_word=8
> > ./sde/hctx1/tags:map_nr=8
> > ./sde/hctx1/tags:alloc_hint={29, 0, 21, 37}
> > ./sde/hctx1/tags:wake_batch=7
> > ./sde/hctx1/tags:wake_index=0
> > ./sde/hctx1/tags:ws={
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags:}
> > ./sde/hctx1/tags:round_robin=0
> > ./sde/hctx1/ctx_map:00000000: 00
> > ./sde/hctx1/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx1/state:
> > ./sde/hctx0/cpu0/completed:4 6
> > ./sde/hctx0/cpu0/merged:0
> > ./sde/hctx0/cpu0/dispatched:4 6
> > ./sde/hctx0/active:0
> > ./sde/hctx0/run:6
> > ./sde/hctx0/queued:10
> > ./sde/hctx0/dispatched: 0 0
> > ./sde/hctx0/dispatched: 1 6
> > ./sde/hctx0/dispatched: 2 0
> > ./sde/hctx0/dispatched: 4 0
> > ./sde/hctx0/dispatched: 8 0
> > ./sde/hctx0/dispatched: 16 0
> > ./sde/hctx0/dispatched: 32+ 0
> > ./sde/hctx0/io_poll:considered=0
> > ./sde/hctx0/io_poll:invoked=0
> > ./sde/hctx0/io_poll:success=0
> > ./sde/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx0/tags:nr_tags=62
> > ./sde/hctx0/tags:nr_reserved_tags=0
> > ./sde/hctx0/tags:active_queues=0
> > ./sde/hctx0/tags:
> > ./sde/hctx0/tags:bitmap_tags:
> > ./sde/hctx0/tags:depth=62
> > ./sde/hctx0/tags:busy=0
> > ./sde/hctx0/tags:bits_per_word=8
> > ./sde/hctx0/tags:map_nr=8
> > ./sde/hctx0/tags:alloc_hint={27, 47, 46, 38}
> > ./sde/hctx0/tags:wake_batch=7
> > ./sde/hctx0/tags:wake_index=0
> > ./sde/hctx0/tags:ws={
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags:}
> > ./sde/hctx0/tags:round_robin=0
> > ./sde/hctx0/ctx_map:00000000: 00
> > ./sde/hctx0/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx0/state:
> > ./sde/write_hints:hint0: 0
> > ./sde/write_hints:hint1: 0
> > ./sde/write_hints:hint2: 0
> > ./sde/write_hints:hint3: 0
> > ./sde/write_hints:hint4: 0
> > ./sde/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTH
> > ROUGH
> > ./sde/poll_stat:read (512 Bytes): samples=0
> > ./sde/poll_stat:write (512 Bytes): samples=0
> > ./sde/poll_stat:read (1024 Bytes): samples=0
> > ./sde/poll_stat:write (1024 Bytes): samples=0
> > ./sde/poll_stat:read (2048 Bytes): samples=0
> > ./sde/poll_stat:write (2048 Bytes): samples=0
> > ./sde/poll_stat:read (4096 Bytes): samples=0
> > ./sde/poll_stat:write (4096 Bytes): samples=0
> > ./sde/poll_stat:read (8192 Bytes): samples=0
> > ./sde/poll_stat:write (8192 Bytes): samples=0
> > ./sde/poll_stat:read (16384 Bytes): samples=0
> > ./sde/poll_stat:write (16384 Bytes): samples=0
> > ./sde/poll_stat:read (32768 Bytes): samples=0
> > ./sde/poll_stat:write (32768 Bytes): samples=0
> > ./sde/poll_stat:read (65536 Bytes): samples=0
> > ./sde/poll_stat:write (65536 Bytes): samples=0
> > ./sdd/hctx3/cpu3/completed:17 7
> > ./sdd/hctx3/cpu3/merged:0
> > ./sdd/hctx3/cpu3/dispatched:17 7
> > ./sdd/hctx3/active:0
> > ./sdd/hctx3/run:13
> > ./sdd/hctx3/queued:24
> > ./sdd/hctx3/dispatched: 0 4
> > ./sdd/hctx3/dispatched: 1 9
> > ./sdd/hctx3/dispatched: 2 0
> > ./sdd/hctx3/dispatched: 4 0
> > ./sdd/hctx3/dispatched: 8 0
> > ./sdd/hctx3/dispatched: 16 0
> > ./sdd/hctx3/dispatched: 32+ 0
> > ./sdd/hctx3/io_poll:considered=0
> > ./sdd/hctx3/io_poll:invoked=0
> > ./sdd/hctx3/io_poll:success=0
> > ./sdd/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx3/tags:nr_tags=62
> > ./sdd/hctx3/tags:nr_reserved_tags=0
> > ./sdd/hctx3/tags:active_queues=0
> > ./sdd/hctx3/tags:
> > ./sdd/hctx3/tags:bitmap_tags:
> > ./sdd/hctx3/tags:depth=62
> > ./sdd/hctx3/tags:busy=0
> > ./sdd/hctx3/tags:bits_per_word=8
> > ./sdd/hctx3/tags:map_nr=8
> > ./sdd/hctx3/tags:alloc_hint={42, 53, 24, 50}
> > ./sdd/hctx3/tags:wake_batch=7
> > ./sdd/hctx3/tags:wake_index=0
> > ./sdd/hctx3/tags:ws={
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags:}
> > ./sdd/hctx3/tags:round_robin=0
> > ./sdd/hctx3/ctx_map:00000000: 00
> > ./sdd/hctx3/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx3/state:
> > ./sdd/hctx2/cpu2/completed:20 37
> > ./sdd/hctx2/cpu2/merged:0
> > ./sdd/hctx2/cpu2/dispatched:20 37
> > ./sdd/hctx2/active:0
> > ./sdd/hctx2/run:57
> > ./sdd/hctx2/queued:57
> > ./sdd/hctx2/dispatched: 0 5
> > ./sdd/hctx2/dispatched: 1 52
> > ./sdd/hctx2/dispatched: 2 0
> > ./sdd/hctx2/dispatched: 4 0
> > ./sdd/hctx2/dispatched: 8 0
> > ./sdd/hctx2/dispatched: 16 0
> > ./sdd/hctx2/dispatched: 32+ 0
> > ./sdd/hctx2/io_poll:considered=0
> > ./sdd/hctx2/io_poll:invoked=0
> > ./sdd/hctx2/io_poll:success=0
> > ./sdd/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx2/tags:nr_tags=62
> > ./sdd/hctx2/tags:nr_reserved_tags=0
> > ./sdd/hctx2/tags:active_queues=0
> > ./sdd/hctx2/tags:
> > ./sdd/hctx2/tags:bitmap_tags:
> > ./sdd/hctx2/tags:depth=62
> > ./sdd/hctx2/tags:busy=0
> > ./sdd/hctx2/tags:bits_per_word=8
> > ./sdd/hctx2/tags:map_nr=8
> > ./sdd/hctx2/tags:alloc_hint={24, 37, 46, 30}
> > ./sdd/hctx2/tags:wake_batch=7
> > ./sdd/hctx2/tags:wake_index=0
> > ./sdd/hctx2/tags:ws={
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags:}
> > ./sdd/hctx2/tags:round_robin=0
> > ./sdd/hctx2/ctx_map:00000000: 00
> > ./sdd/hctx2/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx2/state:
> > ./sdd/hctx1/cpu1/completed:5 15
> > ./sdd/hctx1/cpu1/merged:0
> > ./sdd/hctx1/cpu1/dispatched:5 15
> > ./sdd/hctx1/active:0
> > ./sdd/hctx1/run:26
> > ./sdd/hctx1/queued:20
> > ./sdd/hctx1/dispatched: 0 7
> > ./sdd/hctx1/dispatched: 1 19
> > ./sdd/hctx1/dispatched: 2 0
> > ./sdd/hctx1/dispatched: 4 0
> > ./sdd/hctx1/dispatched: 8 0
> > ./sdd/hctx1/dispatched: 16 0
> > ./sdd/hctx1/dispatched: 32+ 0
> > ./sdd/hctx1/io_poll:considered=0
> > ./sdd/hctx1/io_poll:invoked=0
> > ./sdd/hctx1/io_poll:success=0
> > ./sdd/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx1/tags:nr_tags=62
> > ./sdd/hctx1/tags:nr_reserved_tags=0
> > ./sdd/hctx1/tags:active_queues=0
> > ./sdd/hctx1/tags:
> > ./sdd/hctx1/tags:bitmap_tags:
> > ./sdd/hctx1/tags:depth=62
> > ./sdd/hctx1/tags:busy=0
> > ./sdd/hctx1/tags:bits_per_word=8
> > ./sdd/hctx1/tags:map_nr=8
> > ./sdd/hctx1/tags:alloc_hint={29, 0, 21, 37}
> > ./sdd/hctx1/tags:wake_batch=7
> > ./sdd/hctx1/tags:wake_index=0
> > ./sdd/hctx1/tags:ws={
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags:}
> > ./sdd/hctx1/tags:round_robin=0
> > ./sdd/hctx1/ctx_map:00000000: 00
> > ./sdd/hctx1/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx1/state:
> > ./sdd/hctx0/cpu0/completed:34 8
> > ./sdd/hctx0/cpu0/merged:0
> > ./sdd/hctx0/cpu0/dispatched:34 8
> > ./sdd/hctx0/active:0
> > ./sdd/hctx0/run:43
> > ./sdd/hctx0/queued:42
> > ./sdd/hctx0/dispatched: 0 4
> > ./sdd/hctx0/dispatched: 1 39
> > ./sdd/hctx0/dispatched: 2 0
> > ./sdd/hctx0/dispatched: 4 0
> > ./sdd/hctx0/dispatched: 8 0
> > ./sdd/hctx0/dispatched: 16 0
> > ./sdd/hctx0/dispatched: 32+ 0
> > ./sdd/hctx0/io_poll:considered=0
> > ./sdd/hctx0/io_poll:invoked=0
> > ./sdd/hctx0/io_poll:success=0
> > ./sdd/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx0/tags:nr_tags=62
> > ./sdd/hctx0/tags:nr_reserved_tags=0
> > ./sdd/hctx0/tags:active_queues=0
> > ./sdd/hctx0/tags:
> > ./sdd/hctx0/tags:bitmap_tags:
> > ./sdd/hctx0/tags:depth=62
> > ./sdd/hctx0/tags:busy=0
> > ./sdd/hctx0/tags:bits_per_word=8
> > ./sdd/hctx0/tags:map_nr=8
> > ./sdd/hctx0/tags:alloc_hint={27, 47, 46, 38}
> > ./sdd/hctx0/tags:wake_batch=7
> > ./sdd/hctx0/tags:wake_index=0
> > ./sdd/hctx0/tags:ws={
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags:}
> > ./sdd/hctx0/tags:round_robin=0
> > ./sdd/hctx0/ctx_map:00000000: 00
> > ./sdd/hctx0/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx0/state:
> > ./sdd/write_hints:hint0: 0
> > ./sdd/write_hints:hint1: 0
> > ./sdd/write_hints:hint2: 0
> > ./sdd/write_hints:hint3: 0
> > ./sdd/write_hints:hint4: 0
> > ./sdd/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTER
> > ED|SCSI_PASSTHROUGH
> > ./sdd/poll_stat:read (512 Bytes): samples=0
> > ./sdd/poll_stat:write (512 Bytes): samples=0
> > ./sdd/poll_stat:read (1024 Bytes): samples=0
> > ./sdd/poll_stat:write (1024 Bytes): samples=0
> > ./sdd/poll_stat:read (2048 Bytes): samples=0
> > ./sdd/poll_stat:write (2048 Bytes): samples=0
> > ./sdd/poll_stat:read (4096 Bytes): samples=0
> > ./sdd/poll_stat:write (4096 Bytes): samples=0
> > ./sdd/poll_stat:read (8192 Bytes): samples=0
> > ./sdd/poll_stat:write (8192 Bytes): samples=0
> > ./sdd/poll_stat:read (16384 Bytes): samples=0
> > ./sdd/poll_stat:write (16384 Bytes): samples=0
> > ./sdd/poll_stat:read (32768 Bytes): samples=0
> > ./sdd/poll_stat:write (32768 Bytes): samples=0
> > ./sdd/poll_stat:read (65536 Bytes): samples=0
> > ./sdd/poll_stat:write (65536 Bytes): samples=0
> > ./sdc/hctx3/cpu3/completed:1 11
> > ./sdc/hctx3/cpu3/merged:0
> > ./sdc/hctx3/cpu3/dispatched:1 11
> > ./sdc/hctx3/active:0
> > ./sdc/hctx3/run:15
> > ./sdc/hctx3/queued:12
> > ./sdc/hctx3/dispatched: 0 2
> > ./sdc/hctx3/dispatched: 1 12
> > ./sdc/hctx3/dispatched: 2 0
> > ./sdc/hctx3/dispatched: 4 0
> > ./sdc/hctx3/dispatched: 8 0
> > ./sdc/hctx3/dispatched: 16 0
> > ./sdc/hctx3/dispatched: 32+ 0
> > ./sdc/hctx3/io_poll:considered=0
> > ./sdc/hctx3/io_poll:invoked=0
> > ./sdc/hctx3/io_poll:success=0
> > ./sdc/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx3/tags:nr_tags=62
> > ./sdc/hctx3/tags:nr_reserved_tags=0
> > ./sdc/hctx3/tags:active_queues=0
> > ./sdc/hctx3/tags:
> > ./sdc/hctx3/tags:bitmap_tags:
> > ./sdc/hctx3/tags:depth=62
> > ./sdc/hctx3/tags:busy=0
> > ./sdc/hctx3/tags:bits_per_word=8
> > ./sdc/hctx3/tags:map_nr=8
> > ./sdc/hctx3/tags:alloc_hint={42, 53, 24, 50}
> > ./sdc/hctx3/tags:wake_batch=7
> > ./sdc/hctx3/tags:wake_index=0
> > ./sdc/hctx3/tags:ws={
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags:}
> > ./sdc/hctx3/tags:round_robin=0
> > ./sdc/hctx3/ctx_map:00000000: 00
> > ./sdc/hctx3/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx3/state:
> > ./sdc/hctx2/cpu2/completed:16 37
> > ./sdc/hctx2/cpu2/merged:0
> > ./sdc/hctx2/cpu2/dispatched:16 37
> > ./sdc/hctx2/active:0
> > ./sdc/hctx2/run:73
> > ./sdc/hctx2/queued:53
> > ./sdc/hctx2/dispatched: 0 22
> > ./sdc/hctx2/dispatched: 1 52
> > ./sdc/hctx2/dispatched: 2 0
> > ./sdc/hctx2/dispatched: 4 0
> > ./sdc/hctx2/dispatched: 8 0
> > ./sdc/hctx2/dispatched: 16 0
> > ./sdc/hctx2/dispatched: 32+ 0
> > ./sdc/hctx2/io_poll:considered=0
> > ./sdc/hctx2/io_poll:invoked=0
> > ./sdc/hctx2/io_poll:success=0
> > ./sdc/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx2/tags:nr_tags=62
> > ./sdc/hctx2/tags:nr_reserved_tags=0
> > ./sdc/hctx2/tags:active_queues=0
> > ./sdc/hctx2/tags:
> > ./sdc/hctx2/tags:bitmap_tags:
> > ./sdc/hctx2/tags:depth=62
> > ./sdc/hctx2/tags:busy=0
> > ./sdc/hctx2/tags:bits_per_word=8
> > ./sdc/hctx2/tags:map_nr=8
> > ./sdc/hctx2/tags:alloc_hint={24, 37, 46, 30}
> > ./sdc/hctx2/tags:wake_batch=7
> > ./sdc/hctx2/tags:wake_index=0
> > ./sdc/hctx2/tags:ws={
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags:}
> > ./sdc/hctx2/tags:round_robin=0
> > ./sdc/hctx2/ctx_map:00000000: 00
> > ./sdc/hctx2/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx2/state:
> > ./sdc/hctx1/cpu1/completed:32 8
> > ./sdc/hctx1/cpu1/merged:0
> > ./sdc/hctx1/cpu1/dispatched:32 8
> > ./sdc/hctx1/active:0
> > ./sdc/hctx1/run:10
> > ./sdc/hctx1/queued:40
> > ./sdc/hctx1/dispatched: 0 2
> > ./sdc/hctx1/dispatched: 1 8
> > ./sdc/hctx1/dispatched: 2 0
> > ./sdc/hctx1/dispatched: 4 0
> > ./sdc/hctx1/dispatched: 8 0
> > ./sdc/hctx1/dispatched: 16 0
> > ./sdc/hctx1/dispatched: 32+ 0
> > ./sdc/hctx1/io_poll:considered=0
> > ./sdc/hctx1/io_poll:invoked=0
> > ./sdc/hctx1/io_poll:success=0
> > ./sdc/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx1/tags:nr_tags=62
> > ./sdc/hctx1/tags:nr_reserved_tags=0
> > ./sdc/hctx1/tags:active_queues=0
> > ./sdc/hctx1/tags:
> > ./sdc/hctx1/tags:bitmap_tags:
> > ./sdc/hctx1/tags:depth=62
> > ./sdc/hctx1/tags:busy=0
> > ./sdc/hctx1/tags:bits_per_word=8
> > ./sdc/hctx1/tags:map_nr=8
> > ./sdc/hctx1/tags:alloc_hint={29, 0, 21, 37}
> > ./sdc/hctx1/tags:wake_batch=7
> > ./sdc/hctx1/tags:wake_index=0
> > ./sdc/hctx1/tags:ws={
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags:}
> > ./sdc/hctx1/tags:round_robin=0
> > ./sdc/hctx1/ctx_map:00000000: 00
> > ./sdc/hctx1/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx1/state:
> > ./sdc/hctx0/cpu0/completed:39 12
> > ./sdc/hctx0/cpu0/merged:0
> > ./sdc/hctx0/cpu0/dispatched:39 12
> > ./sdc/hctx0/active:0
> > ./sdc/hctx0/run:48
> > ./sdc/hctx0/queued:51
> > ./sdc/hctx0/dispatched: 0 0
> > ./sdc/hctx0/dispatched: 1 48
> > ./sdc/hctx0/dispatched: 2 0
> > ./sdc/hctx0/dispatched: 4 0
> > ./sdc/hctx0/dispatched: 8 0
> > ./sdc/hctx0/dispatched: 16 0
> > ./sdc/hctx0/dispatched: 32+ 0
> > ./sdc/hctx0/io_poll:considered=0
> > ./sdc/hctx0/io_poll:invoked=0
> > ./sdc/hctx0/io_poll:success=0
> > ./sdc/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx0/tags:nr_tags=62
> > ./sdc/hctx0/tags:nr_reserved_tags=0
> > ./sdc/hctx0/tags:active_queues=0
> > ./sdc/hctx0/tags:
> > ./sdc/hctx0/tags:bitmap_tags:
> > ./sdc/hctx0/tags:depth=62
> > ./sdc/hctx0/tags:busy=0
> > ./sdc/hctx0/tags:bits_per_word=8
> > ./sdc/hctx0/tags:map_nr=8
> > ./sdc/hctx0/tags:alloc_hint={27, 47, 46, 38}
> > ./sdc/hctx0/tags:wake_batch=7
> > ./sdc/hctx0/tags:wake_index=0
> > ./sdc/hctx0/tags:ws={
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags:}
> > ./sdc/hctx0/tags:round_robin=0
> > ./sdc/hctx0/ctx_map:00000000: 00
> > ./sdc/hctx0/flags:alloc_policy=FIFO
> > SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx0/state:
> > ./sdc/write_hints:hint0: 0
> > ./sdc/write_hints:hint1: 0
> > ./sdc/write_hints:hint2: 0
> > ./sdc/write_hints:hint3: 0
> > ./sdc/write_hints:hint4: 0
> > ./sdc/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTH
> > ROUGH
> > ./sdc/poll_stat:read (512 Bytes): samples=0
> > ./sdc/poll_stat:write (512 Bytes): samples=0
> > ./sdc/poll_stat:read (1024 Bytes): samples=0
> > ./sdc/poll_stat:write (1024 Bytes): samples=0
> > ./sdc/poll_stat:read (2048 Bytes): samples=0
> > ./sdc/poll_stat:write (2048 Bytes): samples=0
> > ./sdc/poll_stat:read (4096 Bytes): samples=0
> > ./sdc/poll_stat:write (4096 Bytes): samples=0
> > ./sdc/poll_stat:read (8192 Bytes): samples=0
> > ./sdc/poll_stat:write (8192 Bytes): samples=0
> > ./sdc/poll_stat:read (16384 Bytes): samples=0
> > ./sdc/poll_stat:write (16384 Bytes): samples=0
> > ./sdc/poll_stat:read (32768 Bytes): samples=0
> > ./sdc/poll_stat:write (32768 Bytes): samples=0
> > ./sdc/poll_stat:read (65536 Bytes): samples=0
> > ./sdc/poll_stat:write (65536 Bytes): samples=0
> > ./sdb/hctx0/cpu3/completed:42 6
> > ./sdb/hctx0/cpu3/merged:0
> > ./sdb/hctx0/cpu3/dispatched:42 6
> > ./sdb/hctx0/cpu2/completed:110 13
> > ./sdb/hctx0/cpu2/merged:0
> > ./sdb/hctx0/cpu2/dispatched:110 13
> > ./sdb/hctx0/cpu1/completed:35 39
> > ./sdb/hctx0/cpu1/merged:0
> > ./sdb/hctx0/cpu1/dispatched:35 39
> > ./sdb/hctx0/cpu0/completed:35 26
> > ./sdb/hctx0/cpu0/merged:0
> > ./sdb/hctx0/cpu0/dispatched:35 26
> > ./sdb/hctx0/active:0
> > ./sdb/hctx0/run:300
> > ./sdb/hctx0/queued:306
> > ./sdb/hctx0/dispatched: 0 0
> > ./sdb/hctx0/dispatched: 1 306
> > ./sdb/hctx0/dispatched: 2 0
> > ./sdb/hctx0/dispatched: 4 0
> > ./sdb/hctx0/dispatched: 8 0
> > ./sdb/hctx0/dispatched: 16 0
> > ./sdb/hctx0/dispatched: 32+ 0
> > ./sdb/hctx0/io_poll:considered=0
> > ./sdb/hctx0/io_poll:invoked=0
> > ./sdb/hctx0/io_poll:success=0
> > ./sdb/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000
> > 0000 0000 0000
> > ./sdb/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000
> > 0000 0000 0000
> > ./sdb/hctx0/sched_tags:nr_tags=256
> > ./sdb/hctx0/sched_tags:nr_reserved_tags=0
> > ./sdb/hctx0/sched_tags:active_queues=0
> > ./sdb/hctx0/sched_tags:
> > ./sdb/hctx0/sched_tags:bitmap_tags:
> > ./sdb/hctx0/sched_tags:depth=256
> > ./sdb/hctx0/sched_tags:busy=0
> > ./sdb/hctx0/sched_tags:bits_per_word=64
> > ./sdb/hctx0/sched_tags:map_nr=4
> > ./sdb/hctx0/sched_tags:alloc_hint={68, 116, 75, 172}
> > ./sdb/hctx0/sched_tags:wake_batch=8
> > ./sdb/hctx0/sched_tags:wake_index=0
> > ./sdb/hctx0/sched_tags:ws={
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags:}
> > ./sdb/hctx0/sched_tags:round_robin=0
> > ./sdb/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./sdb/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000
> > ./sdb/hctx0/tags:nr_tags=192
> > ./sdb/hctx0/tags:nr_reserved_tags=0
> > ./sdb/hctx0/tags:active_queues=0
> > ./sdb/hctx0/tags:
> > ./sdb/hctx0/tags:bitmap_tags:
> > ./sdb/hctx0/tags:depth=192
> > ./sdb/hctx0/tags:busy=0
> > ./sdb/hctx0/tags:bits_per_word=32
> > ./sdb/hctx0/tags:map_nr=6
> > ./sdb/hctx0/tags:alloc_hint={5, 24, 47, 126}
> > ./sdb/hctx0/tags:wake_batch=8
> > ./sdb/hctx0/tags:wake_index=0
> > ./sdb/hctx0/tags:ws={
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags:}
> > ./sdb/hctx0/tags:round_robin=0
> > ./sdb/hctx0/ctx_map:00000000: 00
> > ./sdb/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./sdb/hctx0/state:
> > ./sdb/sched/starved:0
> > ./sdb/sched/batching:1
> > ./sdb/write_hints:hint0: 0
> > ./sdb/write_hints:hint1: 0
> > ./sdb/write_hints:hint2: 0
> > ./sdb/write_hints:hint3: 0
> > ./sdb/write_hints:hint4: 0
> > ./sdb/state:BIDI|SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REG
> > ISTERED|SCSI_PASSTHROUGH
> > ./sdb/poll_stat:read (512 Bytes): samples=0
> > ./sdb/poll_stat:write (512 Bytes): samples=0
> > ./sdb/poll_stat:read (1024 Bytes): samples=0
> > ./sdb/poll_stat:write (1024 Bytes): samples=0
> > ./sdb/poll_stat:read (2048 Bytes): samples=0
> > ./sdb/poll_stat:write (2048 Bytes): samples=0
> > ./sdb/poll_stat:read (4096 Bytes): samples=0
> > ./sdb/poll_stat:write (4096 Bytes): samples=0
> > ./sdb/poll_stat:read (8192 Bytes): samples=0
> > ./sdb/poll_stat:write (8192 Bytes): samples=0
> > ./sdb/poll_stat:read (16384 Bytes): samples=0
> > ./sdb/poll_stat:write (16384 Bytes): samples=0
> > ./sdb/poll_stat:read (32768 Bytes): samples=0
> > ./sdb/poll_stat:write (32768 Bytes): samples=0
> > ./sdb/poll_stat:read (65536 Bytes): samples=0
> > ./sdb/poll_stat:write (65536 Bytes): samples=0
> > ./sda/hctx0/cpu3/completed:46 3
> > ./sda/hctx0/cpu3/merged:0
> > ./sda/hctx0/cpu3/dispatched:46 3
> > ./sda/hctx0/cpu2/completed:41 27
> > ./sda/hctx0/cpu2/merged:0
> > ./sda/hctx0/cpu2/dispatched:41 27
> > ./sda/hctx0/cpu1/completed:2 8
> > ./sda/hctx0/cpu1/merged:0
> > ./sda/hctx0/cpu1/dispatched:2 8
> > ./sda/hctx0/cpu0/completed:1 43
> > ./sda/hctx0/cpu0/merged:0
> > ./sda/hctx0/cpu0/dispatched:1 43
> > ./sda/hctx0/active:0
> > ./sda/hctx0/run:144
> > ./sda/hctx0/queued:171
> > ./sda/hctx0/dispatched: 0 0
> > ./sda/hctx0/dispatched: 1 171
> > ./sda/hctx0/dispatched: 2 0
> > ./sda/hctx0/dispatched: 4 0
> > ./sda/hctx0/dispatched: 8 0
> > ./sda/hctx0/dispatched: 16 0
> > ./sda/hctx0/dispatched: 32+ 0
> > ./sda/hctx0/io_poll:considered=0
> > ./sda/hctx0/io_poll:invoked=0
> > ./sda/hctx0/io_poll:success=0
> > ./sda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./sda/hctx0/tags:nr_tags=128
> > ./sda/hctx0/tags:nr_reserved_tags=0
> > ./sda/hctx0/tags:active_queues=0
> > ./sda/hctx0/tags:
> > ./sda/hctx0/tags:bitmap_tags:
> > ./sda/hctx0/tags:depth=128
> > ./sda/hctx0/tags:busy=0
> > ./sda/hctx0/tags:bits_per_word=32
> > ./sda/hctx0/tags:map_nr=4
> > ./sda/hctx0/tags:alloc_hint={111, 80, 44, 68}
> > ./sda/hctx0/tags:wake_batch=8
> > ./sda/hctx0/tags:wake_index=0
> > ./sda/hctx0/tags:ws={
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags:}
> > ./sda/hctx0/tags:round_robin=0
> > ./sda/hctx0/ctx_map:00000000: 00
> > ./sda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./sda/hctx0/state:
> > ./sda/write_hints:hint0: 0
> > ./sda/write_hints:hint1: 0
> > ./sda/write_hints:hint2: 0
> > ./sda/write_hints:hint3: 0
> > ./sda/write_hints:hint4: 0
> > ./sda/state:SAME_COMP|IO_STAT|DISCARD|INIT_DONE|POLL|WC|REGISTERED|
> > SCSI_PASSTHROUGH
> > ./sda/poll_stat:read (512 Bytes): samples=0
> > ./sda/poll_stat:write (512 Bytes): samples=0
> > ./sda/poll_stat:read (1024 Bytes): samples=0
> > ./sda/poll_stat:write (1024 Bytes): samples=0
> > ./sda/poll_stat:read (2048 Bytes): samples=0
> > ./sda/poll_stat:write (2048 Bytes): samples=0
> > ./sda/poll_stat:read (4096 Bytes): samples=0
> > ./sda/poll_stat:write (4096 Bytes): samples=0
> > ./sda/poll_stat:read (8192 Bytes): samples=0
> > ./sda/poll_stat:write (8192 Bytes): samples=0
> > ./sda/poll_stat:read (16384 Bytes): samples=0
> > ./sda/poll_stat:write (16384 Bytes): samples=0
> > ./sda/poll_stat:read (32768 Bytes): samples=0
> > ./sda/poll_stat:write (32768 Bytes): samples=0
> > ./sda/poll_stat:read (65536 Bytes): samples=0
> > ./sda/poll_stat:write (65536 Bytes): samples=0
> > ./vda/hctx0/cpu3/completed:1707 27
> > ./vda/hctx0/cpu3/merged:0
> > ./vda/hctx0/cpu3/dispatched:1707 27
> > ./vda/hctx0/cpu2/completed:1027 12
> > ./vda/hctx0/cpu2/merged:0
> > ./vda/hctx0/cpu2/dispatched:1027 12
> > ./vda/hctx0/cpu1/completed:1138 76
> > ./vda/hctx0/cpu1/merged:4
> > ./vda/hctx0/cpu1/dispatched:1138 76
> > ./vda/hctx0/cpu0/completed:2497 44
> > ./vda/hctx0/cpu0/merged:2
> > ./vda/hctx0/cpu0/dispatched:2497 44
> > ./vda/hctx0/active:0
> > ./vda/hctx0/run:6412
> > ./vda/hctx0/queued:6528
> > ./vda/hctx0/dispatched: 0 0
> > ./vda/hctx0/dispatched: 1 6361
> > ./vda/hctx0/dispatched: 2 20
> > ./vda/hctx0/dispatched: 4 13
> > ./vda/hctx0/dispatched: 8 8
> > ./vda/hctx0/dispatched: 16 8
> > ./vda/hctx0/dispatched: 32+ 0
> > ./vda/hctx0/io_poll:considered=0
> > ./vda/hctx0/io_poll:invoked=0
> > ./vda/hctx0/io_poll:success=0
> > ./vda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000
> > 0000 0000
> > ./vda/hctx0/tags:nr_tags=128
> > ./vda/hctx0/tags:nr_reserved_tags=0
> > ./vda/hctx0/tags:active_queues=0
> > ./vda/hctx0/tags:
> > ./vda/hctx0/tags:bitmap_tags:
> > ./vda/hctx0/tags:depth=128
> > ./vda/hctx0/tags:busy=0
> > ./vda/hctx0/tags:bits_per_word=32
> > ./vda/hctx0/tags:map_nr=4
> > ./vda/hctx0/tags:alloc_hint={28, 24, 114, 37}
> > ./vda/hctx0/tags:wake_batch=8
> > ./vda/hctx0/tags:wake_index=0
> > ./vda/hctx0/tags:ws={
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags:}
> > ./vda/hctx0/tags:round_robin=0
> > ./vda/hctx0/ctx_map:00000000: 00
> > ./vda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE
> > ./vda/hctx0/state:
> > ./vda/write_hints:hint0: 0
> > ./vda/write_hints:hint1: 0
> > ./vda/write_hints:hint2: 0
> > ./vda/write_hints:hint3: 0
> > ./vda/write_hints:hint4: 0
> > ./vda/state:SAME_COMP|IO_STAT|INIT_DONE|NO_SG_MERGE|POLL|WC|REGISTE
> > RED
> > ./vda/poll_stat:read (512 Bytes): samples=0
> > ./vda/poll_stat:write (512 Bytes): samples=0
> > ./vda/poll_stat:read (1024 Bytes): samples=0
> > ./vda/poll_stat:write (1024 Bytes): samples=0
> > ./vda/poll_stat:read (2048 Bytes): samples=0
> > ./vda/poll_stat:write (2048 Bytes): samples=0
> > ./vda/poll_stat:read (4096 Bytes): samples=0
> > ./vda/poll_stat:write (4096 Bytes): samples=0
> > ./vda/poll_stat:read (8192 Bytes): samples=0
> > ./vda/poll_stat:write (8192 Bytes): samples=0
> > ./vda/poll_stat:read (16384 Bytes): samples=0
> > ./vda/poll_stat:write (16384 Bytes): samples=0
> > ./vda/poll_stat:read (32768 Bytes): samples=0
> > ./vda/poll_stat:write (32768 Bytes): samples=0
> > ./vda/poll_stat:read (65536 Bytes): samples=0
> > ./vda/poll_stat:write (65536 Bytes): samples=0
> > ./sr0/hctx0/cpu3/completed:0 22
> > ./sr0/hctx0/cpu3/merged:0
> > ./sr0/hctx0/cpu3/dispatched:0 22
> > ./sr0/hctx0/cpu2/completed:46 107
> > ./sr0/hctx0/cpu2/merged:0
> > ./sr0/hctx0/cpu2/dispatched:46 107
> > ./sr0/hctx0/cpu1/completed:0 10
> > ./sr0/hctx0/cpu1/merged:0
> > ./sr0/hctx0/cpu1/dispatched:0 10
> > ./sr0/hctx0/cpu0/completed:0 18
> > ./sr0/hctx0/cpu0/merged:0
> > ./sr0/hctx0/cpu0/dispatched:0 18
> > ./sr0/hctx0/active:0
> > ./sr0/hctx0/run:203
> > ./sr0/hctx0/queued:203
> > ./sr0/hctx0/dispatched: 0 0
> > ./sr0/hctx0/dispatched: 1 203
> > ./sr0/hctx0/dispatched: 2 0
> > ./sr0/hctx0/dispatched: 4 0
> > ./sr0/hctx0/dispatched: 8 0
> > ./sr0/hctx0/dispatched: 16 0
> > ./sr0/hctx0/dispatched: 32+ 0
> > ./sr0/hctx0/io_poll:considered=0
> > ./sr0/hctx0/io_poll:invoked=0
> > ./sr0/hctx0/io_poll:success=0
> > ./sr0/hctx0/tags_bitmap:00000000: 00
> > ./sr0/hctx0/tags:nr_tags=1
> > ./sr0/hctx0/tags:nr_reserved_tags=0
> > ./sr0/hctx0/tags:active_queues=0
> > ./sr0/hctx0/tags:
> > ./sr0/hctx0/tags:bitmap_tags:
> > ./sr0/hctx0/tags:depth=1
> > ./sr0/hctx0/tags:busy=0
> > ./sr0/hctx0/tags:bits_per_word=64
> > ./sr0/hctx0/tags:map_nr=1
> > ./sr0/hctx0/tags:alloc_hint={0, 0, 0, 0}
> > ./sr0/hctx0/tags:wake_batch=1
> > ./sr0/hctx0/tags:wake_index=7
> > ./sr0/hctx0/tags:ws={
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags:}
> > ./sr0/hctx0/tags:round_robin=1
> > ./sr0/hctx0/ctx_map:00000000: 00
> > ./sr0/hctx0/flags:alloc_policy=RR SHOULD_MERGE|SG_MERGE
> > ./sr0/hctx0/state:
> > ./sr0/write_hints:hint0: 0
> > ./sr0/write_hints:hint1: 0
> > ./sr0/write_hints:hint2: 0
> > ./sr0/write_hints:hint3: 0
> > ./sr0/write_hints:hint4: 0
> > ./sr0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|FLUSH_NQ|REGISTERED|SC
> > SI_PASSTHROUGH
> > ./sr0/poll_stat:read (512 Bytes): samples=0
> > ./sr0/poll_stat:write (512 Bytes): samples=0
> > ./sr0/poll_stat:read (1024 Bytes): samples=0
> > ./sr0/poll_stat:write (1024 Bytes): samples=0
> > ./sr0/poll_stat:read (2048 Bytes): samples=0
> > ./sr0/poll_stat:write (2048 Bytes): samples=0
> > ./sr0/poll_stat:read (4096 Bytes): samples=0
> > ./sr0/poll_stat:write (4096 Bytes): samples=0
> > ./sr0/poll_stat:read (8192 Bytes): samples=0
> > ./sr0/poll_stat:write (8192 Bytes): samples=0
> > ./sr0/poll_stat:read (16384 Bytes): samples=0
> > ./sr0/poll_stat:write (16384 Bytes): samples=0
> > ./sr0/poll_stat:read (32768 Bytes): samples=0
> > ./sr0/poll_stat:write (32768 Bytes): samples=0
> > ./sr0/poll_stat:read (65536 Bytes): samples=0
> > ./sr0/poll_stat:write (65536 Bytes): samples=0
>
>
Hello Bart
If you can share your exact fio run scriptparameters I can try this on
my mlx5 test bed using the same kernel you are using.
Ming has access to the same test bed as well.
Is this direct to devices, or on a file system etc. etc.
Are these from the tests directory in the srp test kit of yours.
Regards
Laurence
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 18:29 ` Laurence Oberman
@ 2018-01-17 19:29 ` Bart Van Assche
2018-01-17 19:46 ` Laurence Oberman
0 siblings, 1 reply; 12+ messages in thread
From: Bart Van Assche @ 2018-01-17 19:29 UTC (permalink / raw)
To: loberman, snitzer; +Cc: axboe, dm-devel, ming.lei
On Wed, 2018-01-17 at 13:29 -0500, Laurence Oberman wrote:
> If you can share your exact fio run scriptparameters I can try this on
> my mlx5 test bed using the same kernel you are using.
> Ming has access to the same test bed as well.
> Is this direct to devices, or on a file system etc. etc.
>
> Are these from the tests directory in the srp test kit of yours.
Hello Laurence,
This issue can be reproduced as follows:
* Clone the block-scsi-for-next branch from https://github.com/bvanassche/linux
(commit cd0b6b9f8779).
* Clone the srp-test project from https://github.com/bvanassche/srp-test.
* Modify /etc/multipath.conf as indicated in the README.md file of the srp-test
project and restart multipathd.
If you prefer the rdma_rxe driver, run the following command:
srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq -e kyber
Or if you prefer to run the test against an IB driver (mlx4, mlx5, ..) on an IB
system, run the following command:
srp-test/run_tests -d -r 10 -q 1 -t 02-mq
If running one of the above commands once would not be sufficient to trigger the
lockup, rerun the command a few times.
Bart.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 19:29 ` Bart Van Assche
@ 2018-01-17 19:46 ` Laurence Oberman
0 siblings, 0 replies; 12+ messages in thread
From: Laurence Oberman @ 2018-01-17 19:46 UTC (permalink / raw)
To: Bart Van Assche, snitzer; +Cc: axboe, dm-devel, ming.lei
On Wed, 2018-01-17 at 19:29 +0000, Bart Van Assche wrote:
> On Wed, 2018-01-17 at 13:29 -0500, Laurence Oberman wrote:
> > If you can share your exact fio run scriptparameters I can try this
> > on
> > my mlx5 test bed using the same kernel you are using.
> > Ming has access to the same test bed as well.
> > Is this direct to devices, or on a file system etc. etc.
> >
> > Are these from the tests directory in the srp test kit of yours.
>
> Hello Laurence,
>
> This issue can be reproduced as follows:
> * Clone the block-scsi-for-next branch from https://github.com/bvanas
> sche/linux
> (commit cd0b6b9f8779).
> * Clone the srp-test project from https://github.com/bvanassche/srp-t
> est.
> * Modify /etc/multipath.conf as indicated in the README.md file of
> the srp-test
> project and restart multipathd.
>
> If you prefer the rdma_rxe driver, run the following command:
>
> srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq -e kyber
>
> Or if you prefer to run the test against an IB driver (mlx4, mlx5,
> ..) on an IB
> system, run the following command:
>
> srp-test/run_tests -d -r 10 -q 1 -t 02-mq
>
> If running one of the above commands once would not be sufficient to
> trigger the
> lockup, rerun the command a few times.
>
> Bart.
Thanks
Bart
I will take care of this test and come back to you.
First going to test the scatter/gather fix.
Regards
Laurence
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 18:18 ` Mike Snitzer
2018-01-17 18:27 ` Bart Van Assche
2018-01-17 18:29 ` Laurence Oberman
@ 2018-01-18 1:53 ` Ming Lei
2 siblings, 0 replies; 12+ messages in thread
From: Ming Lei @ 2018-01-18 1:53 UTC (permalink / raw)
To: Mike Snitzer; +Cc: Bart Van Assche, axboe, dm-devel, Laurence Oberman
On Wed, Jan 17, 2018 at 01:18:52PM -0500, Mike Snitzer wrote:
> On Wed, Jan 17 2018 at 1:05pm -0500,
> Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
>
> > Hello Mike,
> >
> > With Jens' latest for-next branch (commit 6c4f43dfecae) and with commit
> > 6077c2d70609 ("dm rq: Avoid that request processing stalls sporadically")
> > reverted I see again queue stalls every now and then. An example:
> >
> > # srp-test/run_tests -c -d -r 10 -t 02-mq -e kyber
> > Running test /home/bart/software/infiniband/srp-test/tests/02-mq ...
> > Test file I/O on top of multipath concurrently with logout and login (0 min; mq)
> > /dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000: not found
> > Test /home/bart/software/infiniband/srp-test/tests/02-mq failed
> >
> > I have attached the output of the following command to this e-mail:
> > (cd /sys/kernel/debug/block && find -type f | xargs grep -aH '')
> >
> > Can you have a look at this?
>
> Yes, though I need to prepare for a trip that starts on Sunday (so I'll
> be distracted through the end of next week).
>
> But Ming Lei should be able to help make sense of this situation too.
> Also Laurence may be able to help reproduce your specific srp-test on
> his testbed.
Hi Mike, Bart and Guys,
I know there is a hole in blk-mq about RESTART, will post a patch soon if
it can be done before my flight, otherwise you should see the patch at
your tomorrow.
And just take a quick look at Bart's debugfs log, my patch should
address this issue, but we need to discuss about the solution a bit, and the
patch will be marked as RFC.
Thank,
Ming
>
> Mike
>
>
> > ./dm-2/hctx0/cpu3/completed:0 0
> > ./dm-2/hctx0/cpu3/merged:0
> > ./dm-2/hctx0/cpu3/dispatched:0 0
> > ./dm-2/hctx0/cpu2/completed:0 0
> > ./dm-2/hctx0/cpu2/merged:0
> > ./dm-2/hctx0/cpu2/dispatched:0 0
> > ./dm-2/hctx0/cpu1/completed:25 0
> > ./dm-2/hctx0/cpu1/merged:0
> > ./dm-2/hctx0/cpu1/dispatched:26 0
> > ./dm-2/hctx0/cpu0/completed:0 0
> > ./dm-2/hctx0/cpu0/merged:0
> > ./dm-2/hctx0/cpu0/dispatched:0 0
> > ./dm-2/hctx0/active:0
> > ./dm-2/hctx0/run:27
> > ./dm-2/hctx0/queued:26
> > ./dm-2/hctx0/dispatched: 0 6
> > ./dm-2/hctx0/dispatched: 1 25
> > ./dm-2/hctx0/dispatched: 2 0
> > ./dm-2/hctx0/dispatched: 4 0
> > ./dm-2/hctx0/dispatched: 8 0
> > ./dm-2/hctx0/dispatched: 16 0
> > ./dm-2/hctx0/dispatched: 32+ 0
> > ./dm-2/hctx0/io_poll:considered=0
> > ./dm-2/hctx0/io_poll:invoked=0
> > ./dm-2/hctx0/io_poll:success=0
> > ./dm-2/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0020
> > ./dm-2/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/sched_tags:nr_tags=256
> > ./dm-2/hctx0/sched_tags:nr_reserved_tags=0
> > ./dm-2/hctx0/sched_tags:active_queues=0
> > ./dm-2/hctx0/sched_tags:
> > ./dm-2/hctx0/sched_tags:bitmap_tags:
> > ./dm-2/hctx0/sched_tags:depth=256
> > ./dm-2/hctx0/sched_tags:busy=1
> > ./dm-2/hctx0/sched_tags:bits_per_word=64
> > ./dm-2/hctx0/sched_tags:map_nr=4
> > ./dm-2/hctx0/sched_tags:alloc_hint={83, 124, 234, 58}
> > ./dm-2/hctx0/sched_tags:wake_batch=8
> > ./dm-2/hctx0/sched_tags:wake_index=0
> > ./dm-2/hctx0/sched_tags:ws={
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/sched_tags:}
> > ./dm-2/hctx0/sched_tags:round_robin=0
> > ./dm-2/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-2/hctx0/tags:nr_tags=2048
> > ./dm-2/hctx0/tags:nr_reserved_tags=0
> > ./dm-2/hctx0/tags:active_queues=0
> > ./dm-2/hctx0/tags:
> > ./dm-2/hctx0/tags:bitmap_tags:
> > ./dm-2/hctx0/tags:depth=2048
> > ./dm-2/hctx0/tags:busy=0
> > ./dm-2/hctx0/tags:bits_per_word=64
> > ./dm-2/hctx0/tags:map_nr=32
> > ./dm-2/hctx0/tags:alloc_hint={1060, 81, 1050, 844}
> > ./dm-2/hctx0/tags:wake_batch=8
> > ./dm-2/hctx0/tags:wake_index=0
> > ./dm-2/hctx0/tags:ws={
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-2/hctx0/tags:}
> > ./dm-2/hctx0/tags:round_robin=0
> > ./dm-2/hctx0/ctx_map:00000000: 00
> > ./dm-2/hctx0/dispatch:00000000a7224aed {.op=READ, .cmd_flags=, .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=125}
> > ./dm-2/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./dm-2/hctx0/state:SCHED_RESTART
> > ./dm-2/sched/starved:0
> > ./dm-2/sched/batching:5
> > ./dm-2/write_hints:hint0: 0
> > ./dm-2/write_hints:hint1: 0
> > ./dm-2/write_hints:hint2: 0
> > ./dm-2/write_hints:hint3: 0
> > ./dm-2/write_hints:hint4: 0
> > ./dm-2/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
> > ./dm-2/poll_stat:read (512 Bytes): samples=0
> > ./dm-2/poll_stat:write (512 Bytes): samples=0
> > ./dm-2/poll_stat:read (1024 Bytes): samples=0
> > ./dm-2/poll_stat:write (1024 Bytes): samples=0
> > ./dm-2/poll_stat:read (2048 Bytes): samples=0
> > ./dm-2/poll_stat:write (2048 Bytes): samples=0
> > ./dm-2/poll_stat:read (4096 Bytes): samples=0
> > ./dm-2/poll_stat:write (4096 Bytes): samples=0
> > ./dm-2/poll_stat:read (8192 Bytes): samples=0
> > ./dm-2/poll_stat:write (8192 Bytes): samples=0
> > ./dm-2/poll_stat:read (16384 Bytes): samples=0
> > ./dm-2/poll_stat:write (16384 Bytes): samples=0
> > ./dm-2/poll_stat:read (32768 Bytes): samples=0
> > ./dm-2/poll_stat:write (32768 Bytes): samples=0
> > ./dm-2/poll_stat:read (65536 Bytes): samples=0
> > ./dm-2/poll_stat:write (65536 Bytes): samples=0
> > ./dm-1/hctx0/cpu3/completed:26 0
> > ./dm-1/hctx0/cpu3/merged:0
> > ./dm-1/hctx0/cpu3/dispatched:26 0
> > ./dm-1/hctx0/cpu2/completed:2 0
> > ./dm-1/hctx0/cpu2/merged:0
> > ./dm-1/hctx0/cpu2/dispatched:2 0
> > ./dm-1/hctx0/cpu1/completed:2 0
> > ./dm-1/hctx0/cpu1/merged:0
> > ./dm-1/hctx0/cpu1/dispatched:2 0
> > ./dm-1/hctx0/cpu0/completed:2 0
> > ./dm-1/hctx0/cpu0/merged:0
> > ./dm-1/hctx0/cpu0/dispatched:2 0
> > ./dm-1/hctx0/active:0
> > ./dm-1/hctx0/run:34
> > ./dm-1/hctx0/queued:32
> > ./dm-1/hctx0/dispatched: 0 7
> > ./dm-1/hctx0/dispatched: 1 32
> > ./dm-1/hctx0/dispatched: 2 0
> > ./dm-1/hctx0/dispatched: 4 0
> > ./dm-1/hctx0/dispatched: 8 0
> > ./dm-1/hctx0/dispatched: 16 0
> > ./dm-1/hctx0/dispatched: 32+ 0
> > ./dm-1/hctx0/io_poll:considered=0
> > ./dm-1/hctx0/io_poll:invoked=0
> > ./dm-1/hctx0/io_poll:success=0
> > ./dm-1/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/sched_tags:nr_tags=256
> > ./dm-1/hctx0/sched_tags:nr_reserved_tags=0
> > ./dm-1/hctx0/sched_tags:active_queues=0
> > ./dm-1/hctx0/sched_tags:
> > ./dm-1/hctx0/sched_tags:bitmap_tags:
> > ./dm-1/hctx0/sched_tags:depth=256
> > ./dm-1/hctx0/sched_tags:busy=0
> > ./dm-1/hctx0/sched_tags:bits_per_word=64
> > ./dm-1/hctx0/sched_tags:map_nr=4
> > ./dm-1/hctx0/sched_tags:alloc_hint={125, 55, 142, 82}
> > ./dm-1/hctx0/sched_tags:wake_batch=8
> > ./dm-1/hctx0/sched_tags:wake_index=0
> > ./dm-1/hctx0/sched_tags:ws={
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/sched_tags:}
> > ./dm-1/hctx0/sched_tags:round_robin=0
> > ./dm-1/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-1/hctx0/tags:nr_tags=2048
> > ./dm-1/hctx0/tags:nr_reserved_tags=0
> > ./dm-1/hctx0/tags:active_queues=0
> > ./dm-1/hctx0/tags:
> > ./dm-1/hctx0/tags:bitmap_tags:
> > ./dm-1/hctx0/tags:depth=2048
> > ./dm-1/hctx0/tags:busy=0
> > ./dm-1/hctx0/tags:bits_per_word=64
> > ./dm-1/hctx0/tags:map_nr=32
> > ./dm-1/hctx0/tags:alloc_hint={478, 1303, 102, 127}
> > ./dm-1/hctx0/tags:wake_batch=8
> > ./dm-1/hctx0/tags:wake_index=0
> > ./dm-1/hctx0/tags:ws={
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-1/hctx0/tags:}
> > ./dm-1/hctx0/tags:round_robin=0
> > ./dm-1/hctx0/ctx_map:00000000: 00
> > ./dm-1/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./dm-1/hctx0/state:
> > ./dm-1/sched/starved:0
> > ./dm-1/sched/batching:1
> > ./dm-1/write_hints:hint0: 0
> > ./dm-1/write_hints:hint1: 0
> > ./dm-1/write_hints:hint2: 0
> > ./dm-1/write_hints:hint3: 0
> > ./dm-1/write_hints:hint4: 0
> > ./dm-1/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
> > ./dm-1/poll_stat:read (512 Bytes): samples=0
> > ./dm-1/poll_stat:write (512 Bytes): samples=0
> > ./dm-1/poll_stat:read (1024 Bytes): samples=0
> > ./dm-1/poll_stat:write (1024 Bytes): samples=0
> > ./dm-1/poll_stat:read (2048 Bytes): samples=0
> > ./dm-1/poll_stat:write (2048 Bytes): samples=0
> > ./dm-1/poll_stat:read (4096 Bytes): samples=0
> > ./dm-1/poll_stat:write (4096 Bytes): samples=0
> > ./dm-1/poll_stat:read (8192 Bytes): samples=0
> > ./dm-1/poll_stat:write (8192 Bytes): samples=0
> > ./dm-1/poll_stat:read (16384 Bytes): samples=0
> > ./dm-1/poll_stat:write (16384 Bytes): samples=0
> > ./dm-1/poll_stat:read (32768 Bytes): samples=0
> > ./dm-1/poll_stat:write (32768 Bytes): samples=0
> > ./dm-1/poll_stat:read (65536 Bytes): samples=0
> > ./dm-1/poll_stat:write (65536 Bytes): samples=0
> > ./dm-0/hctx0/cpu3/completed:13 0
> > ./dm-0/hctx0/cpu3/merged:0
> > ./dm-0/hctx0/cpu3/dispatched:14 0
> > ./dm-0/hctx0/cpu2/completed:0 0
> > ./dm-0/hctx0/cpu2/merged:0
> > ./dm-0/hctx0/cpu2/dispatched:0 0
> > ./dm-0/hctx0/cpu1/completed:0 0
> > ./dm-0/hctx0/cpu1/merged:0
> > ./dm-0/hctx0/cpu1/dispatched:0 0
> > ./dm-0/hctx0/cpu0/completed:0 0
> > ./dm-0/hctx0/cpu0/merged:0
> > ./dm-0/hctx0/cpu0/dispatched:0 0
> > ./dm-0/hctx0/active:0
> > ./dm-0/hctx0/run:15
> > ./dm-0/hctx0/queued:14
> > ./dm-0/hctx0/dispatched: 0 2
> > ./dm-0/hctx0/dispatched: 1 13
> > ./dm-0/hctx0/dispatched: 2 0
> > ./dm-0/hctx0/dispatched: 4 0
> > ./dm-0/hctx0/dispatched: 8 0
> > ./dm-0/hctx0/dispatched: 16 0
> > ./dm-0/hctx0/dispatched: 32+ 0
> > ./dm-0/hctx0/io_poll:considered=0
> > ./dm-0/hctx0/io_poll:invoked=0
> > ./dm-0/hctx0/io_poll:success=0
> > ./dm-0/hctx0/sched_tags_bitmap:00000000: 0001 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/sched_tags:nr_tags=256
> > ./dm-0/hctx0/sched_tags:nr_reserved_tags=0
> > ./dm-0/hctx0/sched_tags:active_queues=0
> > ./dm-0/hctx0/sched_tags:
> > ./dm-0/hctx0/sched_tags:bitmap_tags:
> > ./dm-0/hctx0/sched_tags:depth=256
> > ./dm-0/hctx0/sched_tags:busy=1
> > ./dm-0/hctx0/sched_tags:bits_per_word=64
> > ./dm-0/hctx0/sched_tags:map_nr=4
> > ./dm-0/hctx0/sched_tags:alloc_hint={11, 18, 205, 9}
> > ./dm-0/hctx0/sched_tags:wake_batch=8
> > ./dm-0/hctx0/sched_tags:wake_index=0
> > ./dm-0/hctx0/sched_tags:ws={
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/sched_tags:}
> > ./dm-0/hctx0/sched_tags:round_robin=0
> > ./dm-0/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./dm-0/hctx0/tags:nr_tags=2048
> > ./dm-0/hctx0/tags:nr_reserved_tags=0
> > ./dm-0/hctx0/tags:active_queues=0
> > ./dm-0/hctx0/tags:
> > ./dm-0/hctx0/tags:bitmap_tags:
> > ./dm-0/hctx0/tags:depth=2048
> > ./dm-0/hctx0/tags:busy=0
> > ./dm-0/hctx0/tags:bits_per_word=64
> > ./dm-0/hctx0/tags:map_nr=32
> > ./dm-0/hctx0/tags:alloc_hint={1392, 843, 879, 1030}
> > ./dm-0/hctx0/tags:wake_batch=8
> > ./dm-0/hctx0/tags:wake_index=0
> > ./dm-0/hctx0/tags:ws={
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./dm-0/hctx0/tags:}
> > ./dm-0/hctx0/tags:round_robin=0
> > ./dm-0/hctx0/ctx_map:00000000: 00
> > ./dm-0/hctx0/dispatch:000000002f291aa3 {.op=READ, .cmd_flags=, .rq_flags=STARTED|IO_STAT, complete=0, .tag=-1, .internal_tag=8}
> > ./dm-0/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./dm-0/hctx0/state:SCHED_RESTART
> > ./dm-0/sched/starved:0
> > ./dm-0/sched/batching:1
> > ./dm-0/write_hints:hint0: 0
> > ./dm-0/write_hints:hint1: 0
> > ./dm-0/write_hints:hint2: 0
> > ./dm-0/write_hints:hint3: 0
> > ./dm-0/write_hints:hint4: 0
> > ./dm-0/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED
> > ./dm-0/poll_stat:read (512 Bytes): samples=0
> > ./dm-0/poll_stat:write (512 Bytes): samples=0
> > ./dm-0/poll_stat:read (1024 Bytes): samples=0
> > ./dm-0/poll_stat:write (1024 Bytes): samples=0
> > ./dm-0/poll_stat:read (2048 Bytes): samples=0
> > ./dm-0/poll_stat:write (2048 Bytes): samples=0
> > ./dm-0/poll_stat:read (4096 Bytes): samples=0
> > ./dm-0/poll_stat:write (4096 Bytes): samples=0
> > ./dm-0/poll_stat:read (8192 Bytes): samples=0
> > ./dm-0/poll_stat:write (8192 Bytes): samples=0
> > ./dm-0/poll_stat:read (16384 Bytes): samples=0
> > ./dm-0/poll_stat:write (16384 Bytes): samples=0
> > ./dm-0/poll_stat:read (32768 Bytes): samples=0
> > ./dm-0/poll_stat:write (32768 Bytes): samples=0
> > ./dm-0/poll_stat:read (65536 Bytes): samples=0
> > ./dm-0/poll_stat:write (65536 Bytes): samples=0
> > ./sde/hctx3/cpu3/completed:58 12
> > ./sde/hctx3/cpu3/merged:0
> > ./sde/hctx3/cpu3/dispatched:58 12
> > ./sde/hctx3/active:0
> > ./sde/hctx3/run:39
> > ./sde/hctx3/queued:70
> > ./sde/hctx3/dispatched: 0 2
> > ./sde/hctx3/dispatched: 1 36
> > ./sde/hctx3/dispatched: 2 0
> > ./sde/hctx3/dispatched: 4 0
> > ./sde/hctx3/dispatched: 8 0
> > ./sde/hctx3/dispatched: 16 0
> > ./sde/hctx3/dispatched: 32+ 0
> > ./sde/hctx3/io_poll:considered=0
> > ./sde/hctx3/io_poll:invoked=0
> > ./sde/hctx3/io_poll:success=0
> > ./sde/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx3/tags:nr_tags=62
> > ./sde/hctx3/tags:nr_reserved_tags=0
> > ./sde/hctx3/tags:active_queues=0
> > ./sde/hctx3/tags:
> > ./sde/hctx3/tags:bitmap_tags:
> > ./sde/hctx3/tags:depth=62
> > ./sde/hctx3/tags:busy=0
> > ./sde/hctx3/tags:bits_per_word=8
> > ./sde/hctx3/tags:map_nr=8
> > ./sde/hctx3/tags:alloc_hint={42, 53, 24, 50}
> > ./sde/hctx3/tags:wake_batch=7
> > ./sde/hctx3/tags:wake_index=0
> > ./sde/hctx3/tags:ws={
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx3/tags:}
> > ./sde/hctx3/tags:round_robin=0
> > ./sde/hctx3/ctx_map:00000000: 00
> > ./sde/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx3/state:
> > ./sde/hctx2/cpu2/completed:2 3
> > ./sde/hctx2/cpu2/merged:0
> > ./sde/hctx2/cpu2/dispatched:2 3
> > ./sde/hctx2/active:0
> > ./sde/hctx2/run:3
> > ./sde/hctx2/queued:5
> > ./sde/hctx2/dispatched: 0 0
> > ./sde/hctx2/dispatched: 1 3
> > ./sde/hctx2/dispatched: 2 0
> > ./sde/hctx2/dispatched: 4 0
> > ./sde/hctx2/dispatched: 8 0
> > ./sde/hctx2/dispatched: 16 0
> > ./sde/hctx2/dispatched: 32+ 0
> > ./sde/hctx2/io_poll:considered=0
> > ./sde/hctx2/io_poll:invoked=0
> > ./sde/hctx2/io_poll:success=0
> > ./sde/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx2/tags:nr_tags=62
> > ./sde/hctx2/tags:nr_reserved_tags=0
> > ./sde/hctx2/tags:active_queues=0
> > ./sde/hctx2/tags:
> > ./sde/hctx2/tags:bitmap_tags:
> > ./sde/hctx2/tags:depth=62
> > ./sde/hctx2/tags:busy=0
> > ./sde/hctx2/tags:bits_per_word=8
> > ./sde/hctx2/tags:map_nr=8
> > ./sde/hctx2/tags:alloc_hint={24, 37, 46, 30}
> > ./sde/hctx2/tags:wake_batch=7
> > ./sde/hctx2/tags:wake_index=0
> > ./sde/hctx2/tags:ws={
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx2/tags:}
> > ./sde/hctx2/tags:round_robin=0
> > ./sde/hctx2/ctx_map:00000000: 00
> > ./sde/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx2/state:
> > ./sde/hctx1/cpu1/completed:31 47
> > ./sde/hctx1/cpu1/merged:0
> > ./sde/hctx1/cpu1/dispatched:31 47
> > ./sde/hctx1/active:0
> > ./sde/hctx1/run:76
> > ./sde/hctx1/queued:78
> > ./sde/hctx1/dispatched: 0 3
> > ./sde/hctx1/dispatched: 1 71
> > ./sde/hctx1/dispatched: 2 0
> > ./sde/hctx1/dispatched: 4 0
> > ./sde/hctx1/dispatched: 8 0
> > ./sde/hctx1/dispatched: 16 0
> > ./sde/hctx1/dispatched: 32+ 0
> > ./sde/hctx1/io_poll:considered=0
> > ./sde/hctx1/io_poll:invoked=0
> > ./sde/hctx1/io_poll:success=0
> > ./sde/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx1/tags:nr_tags=62
> > ./sde/hctx1/tags:nr_reserved_tags=0
> > ./sde/hctx1/tags:active_queues=0
> > ./sde/hctx1/tags:
> > ./sde/hctx1/tags:bitmap_tags:
> > ./sde/hctx1/tags:depth=62
> > ./sde/hctx1/tags:busy=0
> > ./sde/hctx1/tags:bits_per_word=8
> > ./sde/hctx1/tags:map_nr=8
> > ./sde/hctx1/tags:alloc_hint={29, 0, 21, 37}
> > ./sde/hctx1/tags:wake_batch=7
> > ./sde/hctx1/tags:wake_index=0
> > ./sde/hctx1/tags:ws={
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx1/tags:}
> > ./sde/hctx1/tags:round_robin=0
> > ./sde/hctx1/ctx_map:00000000: 00
> > ./sde/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx1/state:
> > ./sde/hctx0/cpu0/completed:4 6
> > ./sde/hctx0/cpu0/merged:0
> > ./sde/hctx0/cpu0/dispatched:4 6
> > ./sde/hctx0/active:0
> > ./sde/hctx0/run:6
> > ./sde/hctx0/queued:10
> > ./sde/hctx0/dispatched: 0 0
> > ./sde/hctx0/dispatched: 1 6
> > ./sde/hctx0/dispatched: 2 0
> > ./sde/hctx0/dispatched: 4 0
> > ./sde/hctx0/dispatched: 8 0
> > ./sde/hctx0/dispatched: 16 0
> > ./sde/hctx0/dispatched: 32+ 0
> > ./sde/hctx0/io_poll:considered=0
> > ./sde/hctx0/io_poll:invoked=0
> > ./sde/hctx0/io_poll:success=0
> > ./sde/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sde/hctx0/tags:nr_tags=62
> > ./sde/hctx0/tags:nr_reserved_tags=0
> > ./sde/hctx0/tags:active_queues=0
> > ./sde/hctx0/tags:
> > ./sde/hctx0/tags:bitmap_tags:
> > ./sde/hctx0/tags:depth=62
> > ./sde/hctx0/tags:busy=0
> > ./sde/hctx0/tags:bits_per_word=8
> > ./sde/hctx0/tags:map_nr=8
> > ./sde/hctx0/tags:alloc_hint={27, 47, 46, 38}
> > ./sde/hctx0/tags:wake_batch=7
> > ./sde/hctx0/tags:wake_index=0
> > ./sde/hctx0/tags:ws={
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sde/hctx0/tags:}
> > ./sde/hctx0/tags:round_robin=0
> > ./sde/hctx0/ctx_map:00000000: 00
> > ./sde/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sde/hctx0/state:
> > ./sde/write_hints:hint0: 0
> > ./sde/write_hints:hint1: 0
> > ./sde/write_hints:hint2: 0
> > ./sde/write_hints:hint3: 0
> > ./sde/write_hints:hint4: 0
> > ./sde/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
> > ./sde/poll_stat:read (512 Bytes): samples=0
> > ./sde/poll_stat:write (512 Bytes): samples=0
> > ./sde/poll_stat:read (1024 Bytes): samples=0
> > ./sde/poll_stat:write (1024 Bytes): samples=0
> > ./sde/poll_stat:read (2048 Bytes): samples=0
> > ./sde/poll_stat:write (2048 Bytes): samples=0
> > ./sde/poll_stat:read (4096 Bytes): samples=0
> > ./sde/poll_stat:write (4096 Bytes): samples=0
> > ./sde/poll_stat:read (8192 Bytes): samples=0
> > ./sde/poll_stat:write (8192 Bytes): samples=0
> > ./sde/poll_stat:read (16384 Bytes): samples=0
> > ./sde/poll_stat:write (16384 Bytes): samples=0
> > ./sde/poll_stat:read (32768 Bytes): samples=0
> > ./sde/poll_stat:write (32768 Bytes): samples=0
> > ./sde/poll_stat:read (65536 Bytes): samples=0
> > ./sde/poll_stat:write (65536 Bytes): samples=0
> > ./sdd/hctx3/cpu3/completed:17 7
> > ./sdd/hctx3/cpu3/merged:0
> > ./sdd/hctx3/cpu3/dispatched:17 7
> > ./sdd/hctx3/active:0
> > ./sdd/hctx3/run:13
> > ./sdd/hctx3/queued:24
> > ./sdd/hctx3/dispatched: 0 4
> > ./sdd/hctx3/dispatched: 1 9
> > ./sdd/hctx3/dispatched: 2 0
> > ./sdd/hctx3/dispatched: 4 0
> > ./sdd/hctx3/dispatched: 8 0
> > ./sdd/hctx3/dispatched: 16 0
> > ./sdd/hctx3/dispatched: 32+ 0
> > ./sdd/hctx3/io_poll:considered=0
> > ./sdd/hctx3/io_poll:invoked=0
> > ./sdd/hctx3/io_poll:success=0
> > ./sdd/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx3/tags:nr_tags=62
> > ./sdd/hctx3/tags:nr_reserved_tags=0
> > ./sdd/hctx3/tags:active_queues=0
> > ./sdd/hctx3/tags:
> > ./sdd/hctx3/tags:bitmap_tags:
> > ./sdd/hctx3/tags:depth=62
> > ./sdd/hctx3/tags:busy=0
> > ./sdd/hctx3/tags:bits_per_word=8
> > ./sdd/hctx3/tags:map_nr=8
> > ./sdd/hctx3/tags:alloc_hint={42, 53, 24, 50}
> > ./sdd/hctx3/tags:wake_batch=7
> > ./sdd/hctx3/tags:wake_index=0
> > ./sdd/hctx3/tags:ws={
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx3/tags:}
> > ./sdd/hctx3/tags:round_robin=0
> > ./sdd/hctx3/ctx_map:00000000: 00
> > ./sdd/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx3/state:
> > ./sdd/hctx2/cpu2/completed:20 37
> > ./sdd/hctx2/cpu2/merged:0
> > ./sdd/hctx2/cpu2/dispatched:20 37
> > ./sdd/hctx2/active:0
> > ./sdd/hctx2/run:57
> > ./sdd/hctx2/queued:57
> > ./sdd/hctx2/dispatched: 0 5
> > ./sdd/hctx2/dispatched: 1 52
> > ./sdd/hctx2/dispatched: 2 0
> > ./sdd/hctx2/dispatched: 4 0
> > ./sdd/hctx2/dispatched: 8 0
> > ./sdd/hctx2/dispatched: 16 0
> > ./sdd/hctx2/dispatched: 32+ 0
> > ./sdd/hctx2/io_poll:considered=0
> > ./sdd/hctx2/io_poll:invoked=0
> > ./sdd/hctx2/io_poll:success=0
> > ./sdd/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx2/tags:nr_tags=62
> > ./sdd/hctx2/tags:nr_reserved_tags=0
> > ./sdd/hctx2/tags:active_queues=0
> > ./sdd/hctx2/tags:
> > ./sdd/hctx2/tags:bitmap_tags:
> > ./sdd/hctx2/tags:depth=62
> > ./sdd/hctx2/tags:busy=0
> > ./sdd/hctx2/tags:bits_per_word=8
> > ./sdd/hctx2/tags:map_nr=8
> > ./sdd/hctx2/tags:alloc_hint={24, 37, 46, 30}
> > ./sdd/hctx2/tags:wake_batch=7
> > ./sdd/hctx2/tags:wake_index=0
> > ./sdd/hctx2/tags:ws={
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx2/tags:}
> > ./sdd/hctx2/tags:round_robin=0
> > ./sdd/hctx2/ctx_map:00000000: 00
> > ./sdd/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx2/state:
> > ./sdd/hctx1/cpu1/completed:5 15
> > ./sdd/hctx1/cpu1/merged:0
> > ./sdd/hctx1/cpu1/dispatched:5 15
> > ./sdd/hctx1/active:0
> > ./sdd/hctx1/run:26
> > ./sdd/hctx1/queued:20
> > ./sdd/hctx1/dispatched: 0 7
> > ./sdd/hctx1/dispatched: 1 19
> > ./sdd/hctx1/dispatched: 2 0
> > ./sdd/hctx1/dispatched: 4 0
> > ./sdd/hctx1/dispatched: 8 0
> > ./sdd/hctx1/dispatched: 16 0
> > ./sdd/hctx1/dispatched: 32+ 0
> > ./sdd/hctx1/io_poll:considered=0
> > ./sdd/hctx1/io_poll:invoked=0
> > ./sdd/hctx1/io_poll:success=0
> > ./sdd/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx1/tags:nr_tags=62
> > ./sdd/hctx1/tags:nr_reserved_tags=0
> > ./sdd/hctx1/tags:active_queues=0
> > ./sdd/hctx1/tags:
> > ./sdd/hctx1/tags:bitmap_tags:
> > ./sdd/hctx1/tags:depth=62
> > ./sdd/hctx1/tags:busy=0
> > ./sdd/hctx1/tags:bits_per_word=8
> > ./sdd/hctx1/tags:map_nr=8
> > ./sdd/hctx1/tags:alloc_hint={29, 0, 21, 37}
> > ./sdd/hctx1/tags:wake_batch=7
> > ./sdd/hctx1/tags:wake_index=0
> > ./sdd/hctx1/tags:ws={
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx1/tags:}
> > ./sdd/hctx1/tags:round_robin=0
> > ./sdd/hctx1/ctx_map:00000000: 00
> > ./sdd/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx1/state:
> > ./sdd/hctx0/cpu0/completed:34 8
> > ./sdd/hctx0/cpu0/merged:0
> > ./sdd/hctx0/cpu0/dispatched:34 8
> > ./sdd/hctx0/active:0
> > ./sdd/hctx0/run:43
> > ./sdd/hctx0/queued:42
> > ./sdd/hctx0/dispatched: 0 4
> > ./sdd/hctx0/dispatched: 1 39
> > ./sdd/hctx0/dispatched: 2 0
> > ./sdd/hctx0/dispatched: 4 0
> > ./sdd/hctx0/dispatched: 8 0
> > ./sdd/hctx0/dispatched: 16 0
> > ./sdd/hctx0/dispatched: 32+ 0
> > ./sdd/hctx0/io_poll:considered=0
> > ./sdd/hctx0/io_poll:invoked=0
> > ./sdd/hctx0/io_poll:success=0
> > ./sdd/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdd/hctx0/tags:nr_tags=62
> > ./sdd/hctx0/tags:nr_reserved_tags=0
> > ./sdd/hctx0/tags:active_queues=0
> > ./sdd/hctx0/tags:
> > ./sdd/hctx0/tags:bitmap_tags:
> > ./sdd/hctx0/tags:depth=62
> > ./sdd/hctx0/tags:busy=0
> > ./sdd/hctx0/tags:bits_per_word=8
> > ./sdd/hctx0/tags:map_nr=8
> > ./sdd/hctx0/tags:alloc_hint={27, 47, 46, 38}
> > ./sdd/hctx0/tags:wake_batch=7
> > ./sdd/hctx0/tags:wake_index=0
> > ./sdd/hctx0/tags:ws={
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdd/hctx0/tags:}
> > ./sdd/hctx0/tags:round_robin=0
> > ./sdd/hctx0/ctx_map:00000000: 00
> > ./sdd/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdd/hctx0/state:
> > ./sdd/write_hints:hint0: 0
> > ./sdd/write_hints:hint1: 0
> > ./sdd/write_hints:hint2: 0
> > ./sdd/write_hints:hint3: 0
> > ./sdd/write_hints:hint4: 0
> > ./sdd/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
> > ./sdd/poll_stat:read (512 Bytes): samples=0
> > ./sdd/poll_stat:write (512 Bytes): samples=0
> > ./sdd/poll_stat:read (1024 Bytes): samples=0
> > ./sdd/poll_stat:write (1024 Bytes): samples=0
> > ./sdd/poll_stat:read (2048 Bytes): samples=0
> > ./sdd/poll_stat:write (2048 Bytes): samples=0
> > ./sdd/poll_stat:read (4096 Bytes): samples=0
> > ./sdd/poll_stat:write (4096 Bytes): samples=0
> > ./sdd/poll_stat:read (8192 Bytes): samples=0
> > ./sdd/poll_stat:write (8192 Bytes): samples=0
> > ./sdd/poll_stat:read (16384 Bytes): samples=0
> > ./sdd/poll_stat:write (16384 Bytes): samples=0
> > ./sdd/poll_stat:read (32768 Bytes): samples=0
> > ./sdd/poll_stat:write (32768 Bytes): samples=0
> > ./sdd/poll_stat:read (65536 Bytes): samples=0
> > ./sdd/poll_stat:write (65536 Bytes): samples=0
> > ./sdc/hctx3/cpu3/completed:1 11
> > ./sdc/hctx3/cpu3/merged:0
> > ./sdc/hctx3/cpu3/dispatched:1 11
> > ./sdc/hctx3/active:0
> > ./sdc/hctx3/run:15
> > ./sdc/hctx3/queued:12
> > ./sdc/hctx3/dispatched: 0 2
> > ./sdc/hctx3/dispatched: 1 12
> > ./sdc/hctx3/dispatched: 2 0
> > ./sdc/hctx3/dispatched: 4 0
> > ./sdc/hctx3/dispatched: 8 0
> > ./sdc/hctx3/dispatched: 16 0
> > ./sdc/hctx3/dispatched: 32+ 0
> > ./sdc/hctx3/io_poll:considered=0
> > ./sdc/hctx3/io_poll:invoked=0
> > ./sdc/hctx3/io_poll:success=0
> > ./sdc/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx3/tags:nr_tags=62
> > ./sdc/hctx3/tags:nr_reserved_tags=0
> > ./sdc/hctx3/tags:active_queues=0
> > ./sdc/hctx3/tags:
> > ./sdc/hctx3/tags:bitmap_tags:
> > ./sdc/hctx3/tags:depth=62
> > ./sdc/hctx3/tags:busy=0
> > ./sdc/hctx3/tags:bits_per_word=8
> > ./sdc/hctx3/tags:map_nr=8
> > ./sdc/hctx3/tags:alloc_hint={42, 53, 24, 50}
> > ./sdc/hctx3/tags:wake_batch=7
> > ./sdc/hctx3/tags:wake_index=0
> > ./sdc/hctx3/tags:ws={
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx3/tags:}
> > ./sdc/hctx3/tags:round_robin=0
> > ./sdc/hctx3/ctx_map:00000000: 00
> > ./sdc/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx3/state:
> > ./sdc/hctx2/cpu2/completed:16 37
> > ./sdc/hctx2/cpu2/merged:0
> > ./sdc/hctx2/cpu2/dispatched:16 37
> > ./sdc/hctx2/active:0
> > ./sdc/hctx2/run:73
> > ./sdc/hctx2/queued:53
> > ./sdc/hctx2/dispatched: 0 22
> > ./sdc/hctx2/dispatched: 1 52
> > ./sdc/hctx2/dispatched: 2 0
> > ./sdc/hctx2/dispatched: 4 0
> > ./sdc/hctx2/dispatched: 8 0
> > ./sdc/hctx2/dispatched: 16 0
> > ./sdc/hctx2/dispatched: 32+ 0
> > ./sdc/hctx2/io_poll:considered=0
> > ./sdc/hctx2/io_poll:invoked=0
> > ./sdc/hctx2/io_poll:success=0
> > ./sdc/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx2/tags:nr_tags=62
> > ./sdc/hctx2/tags:nr_reserved_tags=0
> > ./sdc/hctx2/tags:active_queues=0
> > ./sdc/hctx2/tags:
> > ./sdc/hctx2/tags:bitmap_tags:
> > ./sdc/hctx2/tags:depth=62
> > ./sdc/hctx2/tags:busy=0
> > ./sdc/hctx2/tags:bits_per_word=8
> > ./sdc/hctx2/tags:map_nr=8
> > ./sdc/hctx2/tags:alloc_hint={24, 37, 46, 30}
> > ./sdc/hctx2/tags:wake_batch=7
> > ./sdc/hctx2/tags:wake_index=0
> > ./sdc/hctx2/tags:ws={
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx2/tags:}
> > ./sdc/hctx2/tags:round_robin=0
> > ./sdc/hctx2/ctx_map:00000000: 00
> > ./sdc/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx2/state:
> > ./sdc/hctx1/cpu1/completed:32 8
> > ./sdc/hctx1/cpu1/merged:0
> > ./sdc/hctx1/cpu1/dispatched:32 8
> > ./sdc/hctx1/active:0
> > ./sdc/hctx1/run:10
> > ./sdc/hctx1/queued:40
> > ./sdc/hctx1/dispatched: 0 2
> > ./sdc/hctx1/dispatched: 1 8
> > ./sdc/hctx1/dispatched: 2 0
> > ./sdc/hctx1/dispatched: 4 0
> > ./sdc/hctx1/dispatched: 8 0
> > ./sdc/hctx1/dispatched: 16 0
> > ./sdc/hctx1/dispatched: 32+ 0
> > ./sdc/hctx1/io_poll:considered=0
> > ./sdc/hctx1/io_poll:invoked=0
> > ./sdc/hctx1/io_poll:success=0
> > ./sdc/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx1/tags:nr_tags=62
> > ./sdc/hctx1/tags:nr_reserved_tags=0
> > ./sdc/hctx1/tags:active_queues=0
> > ./sdc/hctx1/tags:
> > ./sdc/hctx1/tags:bitmap_tags:
> > ./sdc/hctx1/tags:depth=62
> > ./sdc/hctx1/tags:busy=0
> > ./sdc/hctx1/tags:bits_per_word=8
> > ./sdc/hctx1/tags:map_nr=8
> > ./sdc/hctx1/tags:alloc_hint={29, 0, 21, 37}
> > ./sdc/hctx1/tags:wake_batch=7
> > ./sdc/hctx1/tags:wake_index=0
> > ./sdc/hctx1/tags:ws={
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx1/tags:}
> > ./sdc/hctx1/tags:round_robin=0
> > ./sdc/hctx1/ctx_map:00000000: 00
> > ./sdc/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx1/state:
> > ./sdc/hctx0/cpu0/completed:39 12
> > ./sdc/hctx0/cpu0/merged:0
> > ./sdc/hctx0/cpu0/dispatched:39 12
> > ./sdc/hctx0/active:0
> > ./sdc/hctx0/run:48
> > ./sdc/hctx0/queued:51
> > ./sdc/hctx0/dispatched: 0 0
> > ./sdc/hctx0/dispatched: 1 48
> > ./sdc/hctx0/dispatched: 2 0
> > ./sdc/hctx0/dispatched: 4 0
> > ./sdc/hctx0/dispatched: 8 0
> > ./sdc/hctx0/dispatched: 16 0
> > ./sdc/hctx0/dispatched: 32+ 0
> > ./sdc/hctx0/io_poll:considered=0
> > ./sdc/hctx0/io_poll:invoked=0
> > ./sdc/hctx0/io_poll:success=0
> > ./sdc/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
> > ./sdc/hctx0/tags:nr_tags=62
> > ./sdc/hctx0/tags:nr_reserved_tags=0
> > ./sdc/hctx0/tags:active_queues=0
> > ./sdc/hctx0/tags:
> > ./sdc/hctx0/tags:bitmap_tags:
> > ./sdc/hctx0/tags:depth=62
> > ./sdc/hctx0/tags:busy=0
> > ./sdc/hctx0/tags:bits_per_word=8
> > ./sdc/hctx0/tags:map_nr=8
> > ./sdc/hctx0/tags:alloc_hint={27, 47, 46, 38}
> > ./sdc/hctx0/tags:wake_batch=7
> > ./sdc/hctx0/tags:wake_index=0
> > ./sdc/hctx0/tags:ws={
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
> > ./sdc/hctx0/tags:}
> > ./sdc/hctx0/tags:round_robin=0
> > ./sdc/hctx0/ctx_map:00000000: 00
> > ./sdc/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
> > ./sdc/hctx0/state:
> > ./sdc/write_hints:hint0: 0
> > ./sdc/write_hints:hint1: 0
> > ./sdc/write_hints:hint2: 0
> > ./sdc/write_hints:hint3: 0
> > ./sdc/write_hints:hint4: 0
> > ./sdc/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
> > ./sdc/poll_stat:read (512 Bytes): samples=0
> > ./sdc/poll_stat:write (512 Bytes): samples=0
> > ./sdc/poll_stat:read (1024 Bytes): samples=0
> > ./sdc/poll_stat:write (1024 Bytes): samples=0
> > ./sdc/poll_stat:read (2048 Bytes): samples=0
> > ./sdc/poll_stat:write (2048 Bytes): samples=0
> > ./sdc/poll_stat:read (4096 Bytes): samples=0
> > ./sdc/poll_stat:write (4096 Bytes): samples=0
> > ./sdc/poll_stat:read (8192 Bytes): samples=0
> > ./sdc/poll_stat:write (8192 Bytes): samples=0
> > ./sdc/poll_stat:read (16384 Bytes): samples=0
> > ./sdc/poll_stat:write (16384 Bytes): samples=0
> > ./sdc/poll_stat:read (32768 Bytes): samples=0
> > ./sdc/poll_stat:write (32768 Bytes): samples=0
> > ./sdc/poll_stat:read (65536 Bytes): samples=0
> > ./sdc/poll_stat:write (65536 Bytes): samples=0
> > ./sdb/hctx0/cpu3/completed:42 6
> > ./sdb/hctx0/cpu3/merged:0
> > ./sdb/hctx0/cpu3/dispatched:42 6
> > ./sdb/hctx0/cpu2/completed:110 13
> > ./sdb/hctx0/cpu2/merged:0
> > ./sdb/hctx0/cpu2/dispatched:110 13
> > ./sdb/hctx0/cpu1/completed:35 39
> > ./sdb/hctx0/cpu1/merged:0
> > ./sdb/hctx0/cpu1/dispatched:35 39
> > ./sdb/hctx0/cpu0/completed:35 26
> > ./sdb/hctx0/cpu0/merged:0
> > ./sdb/hctx0/cpu0/dispatched:35 26
> > ./sdb/hctx0/active:0
> > ./sdb/hctx0/run:300
> > ./sdb/hctx0/queued:306
> > ./sdb/hctx0/dispatched: 0 0
> > ./sdb/hctx0/dispatched: 1 306
> > ./sdb/hctx0/dispatched: 2 0
> > ./sdb/hctx0/dispatched: 4 0
> > ./sdb/hctx0/dispatched: 8 0
> > ./sdb/hctx0/dispatched: 16 0
> > ./sdb/hctx0/dispatched: 32+ 0
> > ./sdb/hctx0/io_poll:considered=0
> > ./sdb/hctx0/io_poll:invoked=0
> > ./sdb/hctx0/io_poll:success=0
> > ./sdb/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./sdb/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./sdb/hctx0/sched_tags:nr_tags=256
> > ./sdb/hctx0/sched_tags:nr_reserved_tags=0
> > ./sdb/hctx0/sched_tags:active_queues=0
> > ./sdb/hctx0/sched_tags:
> > ./sdb/hctx0/sched_tags:bitmap_tags:
> > ./sdb/hctx0/sched_tags:depth=256
> > ./sdb/hctx0/sched_tags:busy=0
> > ./sdb/hctx0/sched_tags:bits_per_word=64
> > ./sdb/hctx0/sched_tags:map_nr=4
> > ./sdb/hctx0/sched_tags:alloc_hint={68, 116, 75, 172}
> > ./sdb/hctx0/sched_tags:wake_batch=8
> > ./sdb/hctx0/sched_tags:wake_index=0
> > ./sdb/hctx0/sched_tags:ws={
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/sched_tags:}
> > ./sdb/hctx0/sched_tags:round_robin=0
> > ./sdb/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./sdb/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000
> > ./sdb/hctx0/tags:nr_tags=192
> > ./sdb/hctx0/tags:nr_reserved_tags=0
> > ./sdb/hctx0/tags:active_queues=0
> > ./sdb/hctx0/tags:
> > ./sdb/hctx0/tags:bitmap_tags:
> > ./sdb/hctx0/tags:depth=192
> > ./sdb/hctx0/tags:busy=0
> > ./sdb/hctx0/tags:bits_per_word=32
> > ./sdb/hctx0/tags:map_nr=6
> > ./sdb/hctx0/tags:alloc_hint={5, 24, 47, 126}
> > ./sdb/hctx0/tags:wake_batch=8
> > ./sdb/hctx0/tags:wake_index=0
> > ./sdb/hctx0/tags:ws={
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sdb/hctx0/tags:}
> > ./sdb/hctx0/tags:round_robin=0
> > ./sdb/hctx0/ctx_map:00000000: 00
> > ./sdb/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./sdb/hctx0/state:
> > ./sdb/sched/starved:0
> > ./sdb/sched/batching:1
> > ./sdb/write_hints:hint0: 0
> > ./sdb/write_hints:hint1: 0
> > ./sdb/write_hints:hint2: 0
> > ./sdb/write_hints:hint3: 0
> > ./sdb/write_hints:hint4: 0
> > ./sdb/state:BIDI|SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
> > ./sdb/poll_stat:read (512 Bytes): samples=0
> > ./sdb/poll_stat:write (512 Bytes): samples=0
> > ./sdb/poll_stat:read (1024 Bytes): samples=0
> > ./sdb/poll_stat:write (1024 Bytes): samples=0
> > ./sdb/poll_stat:read (2048 Bytes): samples=0
> > ./sdb/poll_stat:write (2048 Bytes): samples=0
> > ./sdb/poll_stat:read (4096 Bytes): samples=0
> > ./sdb/poll_stat:write (4096 Bytes): samples=0
> > ./sdb/poll_stat:read (8192 Bytes): samples=0
> > ./sdb/poll_stat:write (8192 Bytes): samples=0
> > ./sdb/poll_stat:read (16384 Bytes): samples=0
> > ./sdb/poll_stat:write (16384 Bytes): samples=0
> > ./sdb/poll_stat:read (32768 Bytes): samples=0
> > ./sdb/poll_stat:write (32768 Bytes): samples=0
> > ./sdb/poll_stat:read (65536 Bytes): samples=0
> > ./sdb/poll_stat:write (65536 Bytes): samples=0
> > ./sda/hctx0/cpu3/completed:46 3
> > ./sda/hctx0/cpu3/merged:0
> > ./sda/hctx0/cpu3/dispatched:46 3
> > ./sda/hctx0/cpu2/completed:41 27
> > ./sda/hctx0/cpu2/merged:0
> > ./sda/hctx0/cpu2/dispatched:41 27
> > ./sda/hctx0/cpu1/completed:2 8
> > ./sda/hctx0/cpu1/merged:0
> > ./sda/hctx0/cpu1/dispatched:2 8
> > ./sda/hctx0/cpu0/completed:1 43
> > ./sda/hctx0/cpu0/merged:0
> > ./sda/hctx0/cpu0/dispatched:1 43
> > ./sda/hctx0/active:0
> > ./sda/hctx0/run:144
> > ./sda/hctx0/queued:171
> > ./sda/hctx0/dispatched: 0 0
> > ./sda/hctx0/dispatched: 1 171
> > ./sda/hctx0/dispatched: 2 0
> > ./sda/hctx0/dispatched: 4 0
> > ./sda/hctx0/dispatched: 8 0
> > ./sda/hctx0/dispatched: 16 0
> > ./sda/hctx0/dispatched: 32+ 0
> > ./sda/hctx0/io_poll:considered=0
> > ./sda/hctx0/io_poll:invoked=0
> > ./sda/hctx0/io_poll:success=0
> > ./sda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./sda/hctx0/tags:nr_tags=128
> > ./sda/hctx0/tags:nr_reserved_tags=0
> > ./sda/hctx0/tags:active_queues=0
> > ./sda/hctx0/tags:
> > ./sda/hctx0/tags:bitmap_tags:
> > ./sda/hctx0/tags:depth=128
> > ./sda/hctx0/tags:busy=0
> > ./sda/hctx0/tags:bits_per_word=32
> > ./sda/hctx0/tags:map_nr=4
> > ./sda/hctx0/tags:alloc_hint={111, 80, 44, 68}
> > ./sda/hctx0/tags:wake_batch=8
> > ./sda/hctx0/tags:wake_index=0
> > ./sda/hctx0/tags:ws={
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./sda/hctx0/tags:}
> > ./sda/hctx0/tags:round_robin=0
> > ./sda/hctx0/ctx_map:00000000: 00
> > ./sda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
> > ./sda/hctx0/state:
> > ./sda/write_hints:hint0: 0
> > ./sda/write_hints:hint1: 0
> > ./sda/write_hints:hint2: 0
> > ./sda/write_hints:hint3: 0
> > ./sda/write_hints:hint4: 0
> > ./sda/state:SAME_COMP|IO_STAT|DISCARD|INIT_DONE|POLL|WC|REGISTERED|SCSI_PASSTHROUGH
> > ./sda/poll_stat:read (512 Bytes): samples=0
> > ./sda/poll_stat:write (512 Bytes): samples=0
> > ./sda/poll_stat:read (1024 Bytes): samples=0
> > ./sda/poll_stat:write (1024 Bytes): samples=0
> > ./sda/poll_stat:read (2048 Bytes): samples=0
> > ./sda/poll_stat:write (2048 Bytes): samples=0
> > ./sda/poll_stat:read (4096 Bytes): samples=0
> > ./sda/poll_stat:write (4096 Bytes): samples=0
> > ./sda/poll_stat:read (8192 Bytes): samples=0
> > ./sda/poll_stat:write (8192 Bytes): samples=0
> > ./sda/poll_stat:read (16384 Bytes): samples=0
> > ./sda/poll_stat:write (16384 Bytes): samples=0
> > ./sda/poll_stat:read (32768 Bytes): samples=0
> > ./sda/poll_stat:write (32768 Bytes): samples=0
> > ./sda/poll_stat:read (65536 Bytes): samples=0
> > ./sda/poll_stat:write (65536 Bytes): samples=0
> > ./vda/hctx0/cpu3/completed:1707 27
> > ./vda/hctx0/cpu3/merged:0
> > ./vda/hctx0/cpu3/dispatched:1707 27
> > ./vda/hctx0/cpu2/completed:1027 12
> > ./vda/hctx0/cpu2/merged:0
> > ./vda/hctx0/cpu2/dispatched:1027 12
> > ./vda/hctx0/cpu1/completed:1138 76
> > ./vda/hctx0/cpu1/merged:4
> > ./vda/hctx0/cpu1/dispatched:1138 76
> > ./vda/hctx0/cpu0/completed:2497 44
> > ./vda/hctx0/cpu0/merged:2
> > ./vda/hctx0/cpu0/dispatched:2497 44
> > ./vda/hctx0/active:0
> > ./vda/hctx0/run:6412
> > ./vda/hctx0/queued:6528
> > ./vda/hctx0/dispatched: 0 0
> > ./vda/hctx0/dispatched: 1 6361
> > ./vda/hctx0/dispatched: 2 20
> > ./vda/hctx0/dispatched: 4 13
> > ./vda/hctx0/dispatched: 8 8
> > ./vda/hctx0/dispatched: 16 8
> > ./vda/hctx0/dispatched: 32+ 0
> > ./vda/hctx0/io_poll:considered=0
> > ./vda/hctx0/io_poll:invoked=0
> > ./vda/hctx0/io_poll:success=0
> > ./vda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
> > ./vda/hctx0/tags:nr_tags=128
> > ./vda/hctx0/tags:nr_reserved_tags=0
> > ./vda/hctx0/tags:active_queues=0
> > ./vda/hctx0/tags:
> > ./vda/hctx0/tags:bitmap_tags:
> > ./vda/hctx0/tags:depth=128
> > ./vda/hctx0/tags:busy=0
> > ./vda/hctx0/tags:bits_per_word=32
> > ./vda/hctx0/tags:map_nr=4
> > ./vda/hctx0/tags:alloc_hint={28, 24, 114, 37}
> > ./vda/hctx0/tags:wake_batch=8
> > ./vda/hctx0/tags:wake_index=0
> > ./vda/hctx0/tags:ws={
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
> > ./vda/hctx0/tags:}
> > ./vda/hctx0/tags:round_robin=0
> > ./vda/hctx0/ctx_map:00000000: 00
> > ./vda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE
> > ./vda/hctx0/state:
> > ./vda/write_hints:hint0: 0
> > ./vda/write_hints:hint1: 0
> > ./vda/write_hints:hint2: 0
> > ./vda/write_hints:hint3: 0
> > ./vda/write_hints:hint4: 0
> > ./vda/state:SAME_COMP|IO_STAT|INIT_DONE|NO_SG_MERGE|POLL|WC|REGISTERED
> > ./vda/poll_stat:read (512 Bytes): samples=0
> > ./vda/poll_stat:write (512 Bytes): samples=0
> > ./vda/poll_stat:read (1024 Bytes): samples=0
> > ./vda/poll_stat:write (1024 Bytes): samples=0
> > ./vda/poll_stat:read (2048 Bytes): samples=0
> > ./vda/poll_stat:write (2048 Bytes): samples=0
> > ./vda/poll_stat:read (4096 Bytes): samples=0
> > ./vda/poll_stat:write (4096 Bytes): samples=0
> > ./vda/poll_stat:read (8192 Bytes): samples=0
> > ./vda/poll_stat:write (8192 Bytes): samples=0
> > ./vda/poll_stat:read (16384 Bytes): samples=0
> > ./vda/poll_stat:write (16384 Bytes): samples=0
> > ./vda/poll_stat:read (32768 Bytes): samples=0
> > ./vda/poll_stat:write (32768 Bytes): samples=0
> > ./vda/poll_stat:read (65536 Bytes): samples=0
> > ./vda/poll_stat:write (65536 Bytes): samples=0
> > ./sr0/hctx0/cpu3/completed:0 22
> > ./sr0/hctx0/cpu3/merged:0
> > ./sr0/hctx0/cpu3/dispatched:0 22
> > ./sr0/hctx0/cpu2/completed:46 107
> > ./sr0/hctx0/cpu2/merged:0
> > ./sr0/hctx0/cpu2/dispatched:46 107
> > ./sr0/hctx0/cpu1/completed:0 10
> > ./sr0/hctx0/cpu1/merged:0
> > ./sr0/hctx0/cpu1/dispatched:0 10
> > ./sr0/hctx0/cpu0/completed:0 18
> > ./sr0/hctx0/cpu0/merged:0
> > ./sr0/hctx0/cpu0/dispatched:0 18
> > ./sr0/hctx0/active:0
> > ./sr0/hctx0/run:203
> > ./sr0/hctx0/queued:203
> > ./sr0/hctx0/dispatched: 0 0
> > ./sr0/hctx0/dispatched: 1 203
> > ./sr0/hctx0/dispatched: 2 0
> > ./sr0/hctx0/dispatched: 4 0
> > ./sr0/hctx0/dispatched: 8 0
> > ./sr0/hctx0/dispatched: 16 0
> > ./sr0/hctx0/dispatched: 32+ 0
> > ./sr0/hctx0/io_poll:considered=0
> > ./sr0/hctx0/io_poll:invoked=0
> > ./sr0/hctx0/io_poll:success=0
> > ./sr0/hctx0/tags_bitmap:00000000: 00
> > ./sr0/hctx0/tags:nr_tags=1
> > ./sr0/hctx0/tags:nr_reserved_tags=0
> > ./sr0/hctx0/tags:active_queues=0
> > ./sr0/hctx0/tags:
> > ./sr0/hctx0/tags:bitmap_tags:
> > ./sr0/hctx0/tags:depth=1
> > ./sr0/hctx0/tags:busy=0
> > ./sr0/hctx0/tags:bits_per_word=64
> > ./sr0/hctx0/tags:map_nr=1
> > ./sr0/hctx0/tags:alloc_hint={0, 0, 0, 0}
> > ./sr0/hctx0/tags:wake_batch=1
> > ./sr0/hctx0/tags:wake_index=7
> > ./sr0/hctx0/tags:ws={
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
> > ./sr0/hctx0/tags:}
> > ./sr0/hctx0/tags:round_robin=1
> > ./sr0/hctx0/ctx_map:00000000: 00
> > ./sr0/hctx0/flags:alloc_policy=RR SHOULD_MERGE|SG_MERGE
> > ./sr0/hctx0/state:
> > ./sr0/write_hints:hint0: 0
> > ./sr0/write_hints:hint1: 0
> > ./sr0/write_hints:hint2: 0
> > ./sr0/write_hints:hint3: 0
> > ./sr0/write_hints:hint4: 0
> > ./sr0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|FLUSH_NQ|REGISTERED|SCSI_PASSTHROUGH
> > ./sr0/poll_stat:read (512 Bytes): samples=0
> > ./sr0/poll_stat:write (512 Bytes): samples=0
> > ./sr0/poll_stat:read (1024 Bytes): samples=0
> > ./sr0/poll_stat:write (1024 Bytes): samples=0
> > ./sr0/poll_stat:read (2048 Bytes): samples=0
> > ./sr0/poll_stat:write (2048 Bytes): samples=0
> > ./sr0/poll_stat:read (4096 Bytes): samples=0
> > ./sr0/poll_stat:write (4096 Bytes): samples=0
> > ./sr0/poll_stat:read (8192 Bytes): samples=0
> > ./sr0/poll_stat:write (8192 Bytes): samples=0
> > ./sr0/poll_stat:read (16384 Bytes): samples=0
> > ./sr0/poll_stat:write (16384 Bytes): samples=0
> > ./sr0/poll_stat:read (32768 Bytes): samples=0
> > ./sr0/poll_stat:write (32768 Bytes): samples=0
> > ./sr0/poll_stat:read (65536 Bytes): samples=0
> > ./sr0/poll_stat:write (65536 Bytes): samples=0
>
--
Ming
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 21:37 ` Mike Snitzer
@ 2018-01-17 21:49 ` Bart Van Assche
0 siblings, 0 replies; 12+ messages in thread
From: Bart Van Assche @ 2018-01-17 21:49 UTC (permalink / raw)
To: snitzer; +Cc: axboe, dm-devel, loberman, ming.lei
On Wed, 2018-01-17 at 16:37 -0500, Mike Snitzer wrote:
> How critical is it to have the latest SCSI changes that are queued for
> 4.16?
Hello Mike,
The IB tests I ran do not depend on any recent SCSI initiator (drivers/scsi)
or SRP driver changes. However, if you want to run the SRP drivers on top of
the rdma_rxe driver then you will need the latest version of the SRP initiator
and target drivers, including some patches that are not yet upstream (but
should go upstream soon).
The test I ran may depend on the SCSI target (drivers/target) patches in my
tree. The SCSI target patches in my tree significantly simplify the session
shutdown code. The patch for simplification of session shutdown may be a bug
fix. However, the original code is so hairy that I have not yet tried to figure
out whether or not my changes improve the behavior of the session shutdown code.
Bart.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 21:27 ` Bart Van Assche
@ 2018-01-17 21:37 ` Mike Snitzer
2018-01-17 21:49 ` Bart Van Assche
0 siblings, 1 reply; 12+ messages in thread
From: Mike Snitzer @ 2018-01-17 21:37 UTC (permalink / raw)
To: Bart Van Assche; +Cc: axboe, dm-devel, loberman, ming.lei
On Wed, Jan 17 2018 at 4:27pm -0500,
Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
> On Wed, 2018-01-17 at 15:14 -0500, Mike Snitzer wrote:
> > BUT my broader point stands: you aren't testing the dm-4.16 changes. By
> > just reverting that commit you're creating a self-fulfilling prophecy
> > (that you'll see hangs without it).
> >
> > Fact is you should pull all of dm-4.16 in, see:
> > https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.16
> >
> > But these dm-4.16 changes are particularly important:
> > 050af08ffb1b dm mpath: return DM_MAPIO_REQUEUE on blk-mq rq allocation failure
> > 459b54019cfe dm mpath: return DM_MAPIO_DELAY_REQUEUE if QUEUE_IO or PG_INIT_REQUIRED
> > ec3eaf9a6731 dm mpath: don't call blk_mq_delay_run_hw_queue() in case of BLK_STS_RESOURCE
> > 4dd6edd23e7e dm mpath: delay the retry of a request if the target responded as busy
> >
> > This last one is the commit that _should_ serve as a proper replacement
> > for the change you manually reverted in your branch.
> >
> > Please re-test after pulling in dm-4.16 and let us know how things fair.
>
> Hello Mike,
>
> If I replace the patch I referred to in my previous e-mail with your dm-4.16
> branch then I see the following:
> * Without I/O scheduler: dm path removal at the end of the test fails. This
> succeeded reliably in the past so I think this is a regression:
> # srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq
> [ ... ]
> Unmounting /root/mnt1 from /dev/mapper/mpathb
> SRP LUN /sys/class/scsi_device/4:0:0:0 / sdc: removing /dev/dm-1: done
> SRP LUN /sys/class/scsi_device/4:0:0:1 / sde: removing /dev/dm-2: done
> SRP LUN /sys/class/scsi_device/4:0:0:2 / sdd: removing /dev/dm-0: dm=$(dev_to_mpath "/dev/dm-0"): failed
> [ ... ]
So no IO hangs? Just removal of a dm device fails at the end? Anything
in the kernel log that might give a hint as to why? I'll need to
appreciate what the test is doing.
Like why is a single SRP scsi device being used to create a dm device?
What type of DM device?
> * With the Kyber I/O scheduler: I/O hangs.
> # srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq -e kyber
> [ ... ]
> Using /dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000 -> ../../dm-0
> (hangs)
Again this says little to me. But hopefully I'll find time to dig in
further and in parallel Laurence will be able to reproduce on his
testbed.
How critical is it to have the latest SCSI changes that are queued for
4.16?
Thanks,
Mike
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 20:14 ` Mike Snitzer
@ 2018-01-17 21:27 ` Bart Van Assche
2018-01-17 21:37 ` Mike Snitzer
0 siblings, 1 reply; 12+ messages in thread
From: Bart Van Assche @ 2018-01-17 21:27 UTC (permalink / raw)
To: snitzer; +Cc: axboe, dm-devel, loberman, ming.lei
[-- Attachment #1: Type: text/plain, Size: 1968 bytes --]
On Wed, 2018-01-17 at 15:14 -0500, Mike Snitzer wrote:
> BUT my broader point stands: you aren't testing the dm-4.16 changes. By
> just reverting that commit you're creating a self-fulfilling prophecy
> (that you'll see hangs without it).
>
> Fact is you should pull all of dm-4.16 in, see:
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.16
>
> But these dm-4.16 changes are particularly important:
> 050af08ffb1b dm mpath: return DM_MAPIO_REQUEUE on blk-mq rq allocation failure
> 459b54019cfe dm mpath: return DM_MAPIO_DELAY_REQUEUE if QUEUE_IO or PG_INIT_REQUIRED
> ec3eaf9a6731 dm mpath: don't call blk_mq_delay_run_hw_queue() in case of BLK_STS_RESOURCE
> 4dd6edd23e7e dm mpath: delay the retry of a request if the target responded as busy
>
> This last one is the commit that _should_ serve as a proper replacement
> for the change you manually reverted in your branch.
>
> Please re-test after pulling in dm-4.16 and let us know how things fair.
Hello Mike,
If I replace the patch I referred to in my previous e-mail with your dm-4.16
branch then I see the following:
* Without I/O scheduler: dm path removal at the end of the test fails. This
succeeded reliably in the past so I think this is a regression:
# srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq
[ ... ]
Unmounting /root/mnt1 from /dev/mapper/mpathb
SRP LUN /sys/class/scsi_device/4:0:0:0 / sdc: removing /dev/dm-1: done
SRP LUN /sys/class/scsi_device/4:0:0:1 / sde: removing /dev/dm-2: done
SRP LUN /sys/class/scsi_device/4:0:0:2 / sdd: removing /dev/dm-0: dm=$(dev_to_mpath "/dev/dm-0"): failed
[ ... ]
* With the Kyber I/O scheduler: I/O hangs.
# srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq -e kyber
[ ... ]
Using /dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000 -> ../../dm-0
(hangs)
I have attached a queue state dump for the last test case to this e-mail.
Bart.
[-- Attachment #2: blk-state --]
[-- Type: text/plain, Size: 67982 bytes --]
./dm-2/hctx0/cpu3/completed:2 0
./dm-2/hctx0/cpu3/merged:0
./dm-2/hctx0/cpu3/dispatched:2 0
./dm-2/hctx0/cpu2/completed:29 0
./dm-2/hctx0/cpu2/merged:0
./dm-2/hctx0/cpu2/dispatched:29 0
./dm-2/hctx0/cpu1/completed:0 0
./dm-2/hctx0/cpu1/merged:0
./dm-2/hctx0/cpu1/dispatched:0 0
./dm-2/hctx0/cpu0/completed:0 0
./dm-2/hctx0/cpu0/merged:0
./dm-2/hctx0/cpu0/dispatched:0 0
./dm-2/hctx0/active:0
./dm-2/hctx0/run:33
./dm-2/hctx0/queued:31
./dm-2/hctx0/dispatched: 0 6
./dm-2/hctx0/dispatched: 1 31
./dm-2/hctx0/dispatched: 2 0
./dm-2/hctx0/dispatched: 4 0
./dm-2/hctx0/dispatched: 8 0
./dm-2/hctx0/dispatched: 16 0
./dm-2/hctx0/dispatched: 32+ 0
./dm-2/hctx0/io_poll:considered=0
./dm-2/hctx0/io_poll:invoked=0
./dm-2/hctx0/io_poll:success=0
./dm-2/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/sched_tags:nr_tags=256
./dm-2/hctx0/sched_tags:nr_reserved_tags=0
./dm-2/hctx0/sched_tags:active_queues=0
./dm-2/hctx0/sched_tags:
./dm-2/hctx0/sched_tags:bitmap_tags:
./dm-2/hctx0/sched_tags:depth=256
./dm-2/hctx0/sched_tags:busy=0
./dm-2/hctx0/sched_tags:bits_per_word=64
./dm-2/hctx0/sched_tags:map_nr=4
./dm-2/hctx0/sched_tags:alloc_hint={33, 216, 163, 187}
./dm-2/hctx0/sched_tags:wake_batch=8
./dm-2/hctx0/sched_tags:wake_index=0
./dm-2/hctx0/sched_tags:ws={
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/sched_tags:}
./dm-2/hctx0/sched_tags:round_robin=0
./dm-2/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-2/hctx0/tags:nr_tags=2048
./dm-2/hctx0/tags:nr_reserved_tags=0
./dm-2/hctx0/tags:active_queues=0
./dm-2/hctx0/tags:
./dm-2/hctx0/tags:bitmap_tags:
./dm-2/hctx0/tags:depth=2048
./dm-2/hctx0/tags:busy=0
./dm-2/hctx0/tags:bits_per_word=64
./dm-2/hctx0/tags:map_nr=32
./dm-2/hctx0/tags:alloc_hint={760, 1496, 132, 171}
./dm-2/hctx0/tags:wake_batch=8
./dm-2/hctx0/tags:wake_index=0
./dm-2/hctx0/tags:ws={
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-2/hctx0/tags:}
./dm-2/hctx0/tags:round_robin=0
./dm-2/hctx0/ctx_map:00000000: 00
./dm-2/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./dm-2/hctx0/state:
./dm-2/sched/starved:0
./dm-2/sched/batching:1
./dm-2/write_hints:hint0: 0
./dm-2/write_hints:hint1: 0
./dm-2/write_hints:hint2: 0
./dm-2/write_hints:hint3: 0
./dm-2/write_hints:hint4: 0
./dm-2/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED
./dm-2/poll_stat:read (512 Bytes): samples=0
./dm-2/poll_stat:write (512 Bytes): samples=0
./dm-2/poll_stat:read (1024 Bytes): samples=0
./dm-2/poll_stat:write (1024 Bytes): samples=0
./dm-2/poll_stat:read (2048 Bytes): samples=0
./dm-2/poll_stat:write (2048 Bytes): samples=0
./dm-2/poll_stat:read (4096 Bytes): samples=0
./dm-2/poll_stat:write (4096 Bytes): samples=0
./dm-2/poll_stat:read (8192 Bytes): samples=0
./dm-2/poll_stat:write (8192 Bytes): samples=0
./dm-2/poll_stat:read (16384 Bytes): samples=0
./dm-2/poll_stat:write (16384 Bytes): samples=0
./dm-2/poll_stat:read (32768 Bytes): samples=0
./dm-2/poll_stat:write (32768 Bytes): samples=0
./dm-2/poll_stat:read (65536 Bytes): samples=0
./dm-2/poll_stat:write (65536 Bytes): samples=0
./sde/hctx3/cpu3/completed:41 10
./sde/hctx3/cpu3/merged:0
./sde/hctx3/cpu3/dispatched:41 10
./sde/hctx3/active:0
./sde/hctx3/run:19
./sde/hctx3/queued:51
./sde/hctx3/dispatched: 0 5
./sde/hctx3/dispatched: 1 14
./sde/hctx3/dispatched: 2 0
./sde/hctx3/dispatched: 4 0
./sde/hctx3/dispatched: 8 0
./sde/hctx3/dispatched: 16 0
./sde/hctx3/dispatched: 32+ 0
./sde/hctx3/io_poll:considered=0
./sde/hctx3/io_poll:invoked=0
./sde/hctx3/io_poll:success=0
./sde/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx3/tags:nr_tags=62
./sde/hctx3/tags:nr_reserved_tags=0
./sde/hctx3/tags:active_queues=0
./sde/hctx3/tags:
./sde/hctx3/tags:bitmap_tags:
./sde/hctx3/tags:depth=62
./sde/hctx3/tags:busy=0
./sde/hctx3/tags:bits_per_word=8
./sde/hctx3/tags:map_nr=8
./sde/hctx3/tags:alloc_hint={28, 33, 59, 34}
./sde/hctx3/tags:wake_batch=7
./sde/hctx3/tags:wake_index=0
./sde/hctx3/tags:ws={
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx3/tags:}
./sde/hctx3/tags:round_robin=0
./sde/hctx3/ctx_map:00000000: 00
./sde/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx3/state:
./sde/hctx2/cpu2/completed:97 46
./sde/hctx2/cpu2/merged:0
./sde/hctx2/cpu2/dispatched:97 46
./sde/hctx2/active:0
./sde/hctx2/run:80
./sde/hctx2/queued:143
./sde/hctx2/dispatched: 0 11
./sde/hctx2/dispatched: 1 70
./sde/hctx2/dispatched: 2 0
./sde/hctx2/dispatched: 4 0
./sde/hctx2/dispatched: 8 0
./sde/hctx2/dispatched: 16 0
./sde/hctx2/dispatched: 32+ 0
./sde/hctx2/io_poll:considered=0
./sde/hctx2/io_poll:invoked=0
./sde/hctx2/io_poll:success=0
./sde/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx2/tags:nr_tags=62
./sde/hctx2/tags:nr_reserved_tags=0
./sde/hctx2/tags:active_queues=0
./sde/hctx2/tags:
./sde/hctx2/tags:bitmap_tags:
./sde/hctx2/tags:depth=62
./sde/hctx2/tags:busy=0
./sde/hctx2/tags:bits_per_word=8
./sde/hctx2/tags:map_nr=8
./sde/hctx2/tags:alloc_hint={2, 0, 24, 0}
./sde/hctx2/tags:wake_batch=7
./sde/hctx2/tags:wake_index=0
./sde/hctx2/tags:ws={
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx2/tags:}
./sde/hctx2/tags:round_robin=0
./sde/hctx2/ctx_map:00000000: 00
./sde/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx2/state:
./sde/hctx1/cpu1/completed:75 17
./sde/hctx1/cpu1/merged:0
./sde/hctx1/cpu1/dispatched:75 17
./sde/hctx1/active:0
./sde/hctx1/run:45
./sde/hctx1/queued:92
./sde/hctx1/dispatched: 0 11
./sde/hctx1/dispatched: 1 36
./sde/hctx1/dispatched: 2 0
./sde/hctx1/dispatched: 4 0
./sde/hctx1/dispatched: 8 0
./sde/hctx1/dispatched: 16 0
./sde/hctx1/dispatched: 32+ 0
./sde/hctx1/io_poll:considered=0
./sde/hctx1/io_poll:invoked=0
./sde/hctx1/io_poll:success=0
./sde/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx1/tags:nr_tags=62
./sde/hctx1/tags:nr_reserved_tags=0
./sde/hctx1/tags:active_queues=0
./sde/hctx1/tags:
./sde/hctx1/tags:bitmap_tags:
./sde/hctx1/tags:depth=62
./sde/hctx1/tags:busy=0
./sde/hctx1/tags:bits_per_word=8
./sde/hctx1/tags:map_nr=8
./sde/hctx1/tags:alloc_hint={39, 24, 20, 46}
./sde/hctx1/tags:wake_batch=7
./sde/hctx1/tags:wake_index=0
./sde/hctx1/tags:ws={
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx1/tags:}
./sde/hctx1/tags:round_robin=0
./sde/hctx1/ctx_map:00000000: 00
./sde/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx1/state:
./sde/hctx0/cpu0/completed:11 20
./sde/hctx0/cpu0/merged:0
./sde/hctx0/cpu0/dispatched:11 20
./sde/hctx0/active:0
./sde/hctx0/run:29
./sde/hctx0/queued:31
./sde/hctx0/dispatched: 0 1
./sde/hctx0/dispatched: 1 28
./sde/hctx0/dispatched: 2 0
./sde/hctx0/dispatched: 4 0
./sde/hctx0/dispatched: 8 0
./sde/hctx0/dispatched: 16 0
./sde/hctx0/dispatched: 32+ 0
./sde/hctx0/io_poll:considered=0
./sde/hctx0/io_poll:invoked=0
./sde/hctx0/io_poll:success=0
./sde/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
./sde/hctx0/tags:nr_tags=62
./sde/hctx0/tags:nr_reserved_tags=0
./sde/hctx0/tags:active_queues=0
./sde/hctx0/tags:
./sde/hctx0/tags:bitmap_tags:
./sde/hctx0/tags:depth=62
./sde/hctx0/tags:busy=0
./sde/hctx0/tags:bits_per_word=8
./sde/hctx0/tags:map_nr=8
./sde/hctx0/tags:alloc_hint={7, 56, 59, 26}
./sde/hctx0/tags:wake_batch=7
./sde/hctx0/tags:wake_index=0
./sde/hctx0/tags:ws={
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sde/hctx0/tags:}
./sde/hctx0/tags:round_robin=0
./sde/hctx0/ctx_map:00000000: 00
./sde/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sde/hctx0/state:
./sde/write_hints:hint0: 0
./sde/write_hints:hint1: 0
./sde/write_hints:hint2: 0
./sde/write_hints:hint3: 0
./sde/write_hints:hint4: 0
./sde/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
./sde/poll_stat:read (512 Bytes): samples=0
./sde/poll_stat:write (512 Bytes): samples=0
./sde/poll_stat:read (1024 Bytes): samples=0
./sde/poll_stat:write (1024 Bytes): samples=0
./sde/poll_stat:read (2048 Bytes): samples=0
./sde/poll_stat:write (2048 Bytes): samples=0
./sde/poll_stat:read (4096 Bytes): samples=0
./sde/poll_stat:write (4096 Bytes): samples=0
./sde/poll_stat:read (8192 Bytes): samples=0
./sde/poll_stat:write (8192 Bytes): samples=0
./sde/poll_stat:read (16384 Bytes): samples=0
./sde/poll_stat:write (16384 Bytes): samples=0
./sde/poll_stat:read (32768 Bytes): samples=0
./sde/poll_stat:write (32768 Bytes): samples=0
./sde/poll_stat:read (65536 Bytes): samples=0
./sde/poll_stat:write (65536 Bytes): samples=0
./sdf/hctx3/cpu3/completed:7 40
./sdf/hctx3/cpu3/merged:0
./sdf/hctx3/cpu3/dispatched:7 40
./sdf/hctx3/active:0
./sdf/hctx3/run:48
./sdf/hctx3/queued:47
./sdf/hctx3/dispatched: 0 5
./sdf/hctx3/dispatched: 1 43
./sdf/hctx3/dispatched: 2 0
./sdf/hctx3/dispatched: 4 0
./sdf/hctx3/dispatched: 8 0
./sdf/hctx3/dispatched: 16 0
./sdf/hctx3/dispatched: 32+ 0
./sdf/hctx3/io_poll:considered=0
./sdf/hctx3/io_poll:invoked=0
./sdf/hctx3/io_poll:success=0
./sdf/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
./sdf/hctx3/tags:nr_tags=62
./sdf/hctx3/tags:nr_reserved_tags=0
./sdf/hctx3/tags:active_queues=0
./sdf/hctx3/tags:
./sdf/hctx3/tags:bitmap_tags:
./sdf/hctx3/tags:depth=62
./sdf/hctx3/tags:busy=0
./sdf/hctx3/tags:bits_per_word=8
./sdf/hctx3/tags:map_nr=8
./sdf/hctx3/tags:alloc_hint={28, 33, 59, 34}
./sdf/hctx3/tags:wake_batch=7
./sdf/hctx3/tags:wake_index=0
./sdf/hctx3/tags:ws={
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx3/tags:}
./sdf/hctx3/tags:round_robin=0
./sdf/hctx3/ctx_map:00000000: 00
./sdf/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdf/hctx3/state:
./sdf/hctx2/cpu2/completed:35 7
./sdf/hctx2/cpu2/merged:0
./sdf/hctx2/cpu2/dispatched:35 7
./sdf/hctx2/active:0
./sdf/hctx2/run:7
./sdf/hctx2/queued:42
./sdf/hctx2/dispatched: 0 0
./sdf/hctx2/dispatched: 1 7
./sdf/hctx2/dispatched: 2 0
./sdf/hctx2/dispatched: 4 0
./sdf/hctx2/dispatched: 8 0
./sdf/hctx2/dispatched: 16 0
./sdf/hctx2/dispatched: 32+ 0
./sdf/hctx2/io_poll:considered=0
./sdf/hctx2/io_poll:invoked=0
./sdf/hctx2/io_poll:success=0
./sdf/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
./sdf/hctx2/tags:nr_tags=62
./sdf/hctx2/tags:nr_reserved_tags=0
./sdf/hctx2/tags:active_queues=0
./sdf/hctx2/tags:
./sdf/hctx2/tags:bitmap_tags:
./sdf/hctx2/tags:depth=62
./sdf/hctx2/tags:busy=0
./sdf/hctx2/tags:bits_per_word=8
./sdf/hctx2/tags:map_nr=8
./sdf/hctx2/tags:alloc_hint={2, 0, 24, 0}
./sdf/hctx2/tags:wake_batch=7
./sdf/hctx2/tags:wake_index=0
./sdf/hctx2/tags:ws={
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx2/tags:}
./sdf/hctx2/tags:round_robin=0
./sdf/hctx2/ctx_map:00000000: 00
./sdf/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdf/hctx2/state:
./sdf/hctx1/cpu1/completed:52 11
./sdf/hctx1/cpu1/merged:0
./sdf/hctx1/cpu1/dispatched:52 11
./sdf/hctx1/active:0
./sdf/hctx1/run:63
./sdf/hctx1/queued:63
./sdf/hctx1/dispatched: 0 6
./sdf/hctx1/dispatched: 1 56
./sdf/hctx1/dispatched: 2 0
./sdf/hctx1/dispatched: 4 0
./sdf/hctx1/dispatched: 8 0
./sdf/hctx1/dispatched: 16 0
./sdf/hctx1/dispatched: 32+ 0
./sdf/hctx1/io_poll:considered=0
./sdf/hctx1/io_poll:invoked=0
./sdf/hctx1/io_poll:success=0
./sdf/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
./sdf/hctx1/tags:nr_tags=62
./sdf/hctx1/tags:nr_reserved_tags=0
./sdf/hctx1/tags:active_queues=0
./sdf/hctx1/tags:
./sdf/hctx1/tags:bitmap_tags:
./sdf/hctx1/tags:depth=62
./sdf/hctx1/tags:busy=0
./sdf/hctx1/tags:bits_per_word=8
./sdf/hctx1/tags:map_nr=8
./sdf/hctx1/tags:alloc_hint={39, 24, 20, 46}
./sdf/hctx1/tags:wake_batch=7
./sdf/hctx1/tags:wake_index=0
./sdf/hctx1/tags:ws={
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx1/tags:}
./sdf/hctx1/tags:round_robin=0
./sdf/hctx1/ctx_map:00000000: 00
./sdf/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdf/hctx1/state:
./sdf/hctx0/cpu0/completed:1 23
./sdf/hctx0/cpu0/merged:0
./sdf/hctx0/cpu0/dispatched:1 23
./sdf/hctx0/active:0
./sdf/hctx0/run:28
./sdf/hctx0/queued:24
./sdf/hctx0/dispatched: 0 5
./sdf/hctx0/dispatched: 1 23
./sdf/hctx0/dispatched: 2 0
./sdf/hctx0/dispatched: 4 0
./sdf/hctx0/dispatched: 8 0
./sdf/hctx0/dispatched: 16 0
./sdf/hctx0/dispatched: 32+ 0
./sdf/hctx0/io_poll:considered=0
./sdf/hctx0/io_poll:invoked=0
./sdf/hctx0/io_poll:success=0
./sdf/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
./sdf/hctx0/tags:nr_tags=62
./sdf/hctx0/tags:nr_reserved_tags=0
./sdf/hctx0/tags:active_queues=0
./sdf/hctx0/tags:
./sdf/hctx0/tags:bitmap_tags:
./sdf/hctx0/tags:depth=62
./sdf/hctx0/tags:busy=0
./sdf/hctx0/tags:bits_per_word=8
./sdf/hctx0/tags:map_nr=8
./sdf/hctx0/tags:alloc_hint={7, 56, 59, 26}
./sdf/hctx0/tags:wake_batch=7
./sdf/hctx0/tags:wake_index=0
./sdf/hctx0/tags:ws={
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdf/hctx0/tags:}
./sdf/hctx0/tags:round_robin=0
./sdf/hctx0/ctx_map:00000000: 00
./sdf/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdf/hctx0/state:
./sdf/write_hints:hint0: 0
./sdf/write_hints:hint1: 0
./sdf/write_hints:hint2: 0
./sdf/write_hints:hint3: 0
./sdf/write_hints:hint4: 0
./sdf/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
./sdf/poll_stat:read (512 Bytes): samples=0
./sdf/poll_stat:write (512 Bytes): samples=0
./sdf/poll_stat:read (1024 Bytes): samples=0
./sdf/poll_stat:write (1024 Bytes): samples=0
./sdf/poll_stat:read (2048 Bytes): samples=0
./sdf/poll_stat:write (2048 Bytes): samples=0
./sdf/poll_stat:read (4096 Bytes): samples=0
./sdf/poll_stat:write (4096 Bytes): samples=0
./sdf/poll_stat:read (8192 Bytes): samples=0
./sdf/poll_stat:write (8192 Bytes): samples=0
./sdf/poll_stat:read (16384 Bytes): samples=0
./sdf/poll_stat:write (16384 Bytes): samples=0
./sdf/poll_stat:read (32768 Bytes): samples=0
./sdf/poll_stat:write (32768 Bytes): samples=0
./sdf/poll_stat:read (65536 Bytes): samples=0
./sdf/poll_stat:write (65536 Bytes): samples=0
./sdc/hctx3/cpu3/completed:1 5
./sdc/hctx3/cpu3/merged:0
./sdc/hctx3/cpu3/dispatched:1 5
./sdc/hctx3/active:0
./sdc/hctx3/run:11
./sdc/hctx3/queued:6
./sdc/hctx3/dispatched: 0 5
./sdc/hctx3/dispatched: 1 6
./sdc/hctx3/dispatched: 2 0
./sdc/hctx3/dispatched: 4 0
./sdc/hctx3/dispatched: 8 0
./sdc/hctx3/dispatched: 16 0
./sdc/hctx3/dispatched: 32+ 0
./sdc/hctx3/io_poll:considered=0
./sdc/hctx3/io_poll:invoked=0
./sdc/hctx3/io_poll:success=0
./sdc/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx3/tags:nr_tags=62
./sdc/hctx3/tags:nr_reserved_tags=0
./sdc/hctx3/tags:active_queues=0
./sdc/hctx3/tags:
./sdc/hctx3/tags:bitmap_tags:
./sdc/hctx3/tags:depth=62
./sdc/hctx3/tags:busy=0
./sdc/hctx3/tags:bits_per_word=8
./sdc/hctx3/tags:map_nr=8
./sdc/hctx3/tags:alloc_hint={28, 33, 59, 34}
./sdc/hctx3/tags:wake_batch=7
./sdc/hctx3/tags:wake_index=0
./sdc/hctx3/tags:ws={
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx3/tags:}
./sdc/hctx3/tags:round_robin=0
./sdc/hctx3/ctx_map:00000000: 00
./sdc/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx3/state:
./sdc/hctx2/cpu2/completed:1 54
./sdc/hctx2/cpu2/merged:0
./sdc/hctx2/cpu2/dispatched:1 54
./sdc/hctx2/active:0
./sdc/hctx2/run:45
./sdc/hctx2/queued:55
./sdc/hctx2/dispatched: 0 6
./sdc/hctx2/dispatched: 1 39
./sdc/hctx2/dispatched: 2 0
./sdc/hctx2/dispatched: 4 0
./sdc/hctx2/dispatched: 8 0
./sdc/hctx2/dispatched: 16 0
./sdc/hctx2/dispatched: 32+ 0
./sdc/hctx2/io_poll:considered=0
./sdc/hctx2/io_poll:invoked=0
./sdc/hctx2/io_poll:success=0
./sdc/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx2/tags:nr_tags=62
./sdc/hctx2/tags:nr_reserved_tags=0
./sdc/hctx2/tags:active_queues=0
./sdc/hctx2/tags:
./sdc/hctx2/tags:bitmap_tags:
./sdc/hctx2/tags:depth=62
./sdc/hctx2/tags:busy=0
./sdc/hctx2/tags:bits_per_word=8
./sdc/hctx2/tags:map_nr=8
./sdc/hctx2/tags:alloc_hint={2, 0, 24, 0}
./sdc/hctx2/tags:wake_batch=7
./sdc/hctx2/tags:wake_index=0
./sdc/hctx2/tags:ws={
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx2/tags:}
./sdc/hctx2/tags:round_robin=0
./sdc/hctx2/ctx_map:00000000: 00
./sdc/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx2/state:
./sdc/hctx1/cpu1/completed:28 29
./sdc/hctx1/cpu1/merged:0
./sdc/hctx1/cpu1/dispatched:28 29
./sdc/hctx1/active:0
./sdc/hctx1/run:65
./sdc/hctx1/queued:57
./sdc/hctx1/dispatched: 0 13
./sdc/hctx1/dispatched: 1 55
./sdc/hctx1/dispatched: 2 0
./sdc/hctx1/dispatched: 4 0
./sdc/hctx1/dispatched: 8 0
./sdc/hctx1/dispatched: 16 0
./sdc/hctx1/dispatched: 32+ 0
./sdc/hctx1/io_poll:considered=0
./sdc/hctx1/io_poll:invoked=0
./sdc/hctx1/io_poll:success=0
./sdc/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx1/tags:nr_tags=62
./sdc/hctx1/tags:nr_reserved_tags=0
./sdc/hctx1/tags:active_queues=0
./sdc/hctx1/tags:
./sdc/hctx1/tags:bitmap_tags:
./sdc/hctx1/tags:depth=62
./sdc/hctx1/tags:busy=0
./sdc/hctx1/tags:bits_per_word=8
./sdc/hctx1/tags:map_nr=8
./sdc/hctx1/tags:alloc_hint={39, 24, 20, 46}
./sdc/hctx1/tags:wake_batch=7
./sdc/hctx1/tags:wake_index=0
./sdc/hctx1/tags:ws={
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx1/tags:}
./sdc/hctx1/tags:round_robin=0
./sdc/hctx1/ctx_map:00000000: 00
./sdc/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx1/state:
./sdc/hctx0/cpu0/completed:30 20
./sdc/hctx0/cpu0/merged:0
./sdc/hctx0/cpu0/dispatched:30 20
./sdc/hctx0/active:0
./sdc/hctx0/run:44
./sdc/hctx0/queued:50
./sdc/hctx0/dispatched: 0 1
./sdc/hctx0/dispatched: 1 43
./sdc/hctx0/dispatched: 2 0
./sdc/hctx0/dispatched: 4 0
./sdc/hctx0/dispatched: 8 0
./sdc/hctx0/dispatched: 16 0
./sdc/hctx0/dispatched: 32+ 0
./sdc/hctx0/io_poll:considered=0
./sdc/hctx0/io_poll:invoked=0
./sdc/hctx0/io_poll:success=0
./sdc/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000
./sdc/hctx0/tags:nr_tags=62
./sdc/hctx0/tags:nr_reserved_tags=0
./sdc/hctx0/tags:active_queues=0
./sdc/hctx0/tags:
./sdc/hctx0/tags:bitmap_tags:
./sdc/hctx0/tags:depth=62
./sdc/hctx0/tags:busy=0
./sdc/hctx0/tags:bits_per_word=8
./sdc/hctx0/tags:map_nr=8
./sdc/hctx0/tags:alloc_hint={7, 56, 59, 26}
./sdc/hctx0/tags:wake_batch=7
./sdc/hctx0/tags:wake_index=0
./sdc/hctx0/tags:ws={
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive},
./sdc/hctx0/tags:}
./sdc/hctx0/tags:round_robin=0
./sdc/hctx0/ctx_map:00000000: 00
./sdc/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE
./sdc/hctx0/state:
./sdc/write_hints:hint0: 0
./sdc/write_hints:hint1: 0
./sdc/write_hints:hint2: 0
./sdc/write_hints:hint3: 0
./sdc/write_hints:hint4: 0
./sdc/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH
./sdc/poll_stat:read (512 Bytes): samples=0
./sdc/poll_stat:write (512 Bytes): samples=0
./sdc/poll_stat:read (1024 Bytes): samples=0
./sdc/poll_stat:write (1024 Bytes): samples=0
./sdc/poll_stat:read (2048 Bytes): samples=0
./sdc/poll_stat:write (2048 Bytes): samples=0
./sdc/poll_stat:read (4096 Bytes): samples=0
./sdc/poll_stat:write (4096 Bytes): samples=0
./sdc/poll_stat:read (8192 Bytes): samples=0
./sdc/poll_stat:write (8192 Bytes): samples=0
./sdc/poll_stat:read (16384 Bytes): samples=0
./sdc/poll_stat:write (16384 Bytes): samples=0
./sdc/poll_stat:read (32768 Bytes): samples=0
./sdc/poll_stat:write (32768 Bytes): samples=0
./sdc/poll_stat:read (65536 Bytes): samples=0
./sdc/poll_stat:write (65536 Bytes): samples=0
./dm-1/hctx0/cpu3/completed:32 0
./dm-1/hctx0/cpu3/merged:0
./dm-1/hctx0/cpu3/dispatched:32 0
./dm-1/hctx0/cpu2/completed:56 0
./dm-1/hctx0/cpu2/merged:0
./dm-1/hctx0/cpu2/dispatched:56 0
./dm-1/hctx0/cpu1/completed:39 0
./dm-1/hctx0/cpu1/merged:0
./dm-1/hctx0/cpu1/dispatched:39 0
./dm-1/hctx0/cpu0/completed:2 0
./dm-1/hctx0/cpu0/merged:0
./dm-1/hctx0/cpu0/dispatched:2 0
./dm-1/hctx0/active:0
./dm-1/hctx0/run:140
./dm-1/hctx0/queued:129
./dm-1/hctx0/dispatched: 0 33
./dm-1/hctx0/dispatched: 1 130
./dm-1/hctx0/dispatched: 2 0
./dm-1/hctx0/dispatched: 4 0
./dm-1/hctx0/dispatched: 8 0
./dm-1/hctx0/dispatched: 16 0
./dm-1/hctx0/dispatched: 32+ 0
./dm-1/hctx0/io_poll:considered=0
./dm-1/hctx0/io_poll:invoked=0
./dm-1/hctx0/io_poll:success=0
./dm-1/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/sched_tags:nr_tags=256
./dm-1/hctx0/sched_tags:nr_reserved_tags=0
./dm-1/hctx0/sched_tags:active_queues=0
./dm-1/hctx0/sched_tags:
./dm-1/hctx0/sched_tags:bitmap_tags:
./dm-1/hctx0/sched_tags:depth=256
./dm-1/hctx0/sched_tags:busy=0
./dm-1/hctx0/sched_tags:bits_per_word=64
./dm-1/hctx0/sched_tags:map_nr=4
./dm-1/hctx0/sched_tags:alloc_hint={165, 112, 86, 40}
./dm-1/hctx0/sched_tags:wake_batch=8
./dm-1/hctx0/sched_tags:wake_index=0
./dm-1/hctx0/sched_tags:ws={
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/sched_tags:}
./dm-1/hctx0/sched_tags:round_robin=0
./dm-1/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-1/hctx0/tags:nr_tags=2048
./dm-1/hctx0/tags:nr_reserved_tags=0
./dm-1/hctx0/tags:active_queues=0
./dm-1/hctx0/tags:
./dm-1/hctx0/tags:bitmap_tags:
./dm-1/hctx0/tags:depth=2048
./dm-1/hctx0/tags:busy=0
./dm-1/hctx0/tags:bits_per_word=64
./dm-1/hctx0/tags:map_nr=32
./dm-1/hctx0/tags:alloc_hint={1468, 1663, 1467, 1182}
./dm-1/hctx0/tags:wake_batch=8
./dm-1/hctx0/tags:wake_index=0
./dm-1/hctx0/tags:ws={
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-1/hctx0/tags:}
./dm-1/hctx0/tags:round_robin=0
./dm-1/hctx0/ctx_map:00000000: 00
./dm-1/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./dm-1/hctx0/state:
./dm-1/sched/starved:0
./dm-1/sched/batching:1
./dm-1/write_hints:hint0: 0
./dm-1/write_hints:hint1: 0
./dm-1/write_hints:hint2: 0
./dm-1/write_hints:hint3: 0
./dm-1/write_hints:hint4: 0
./dm-1/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED
./dm-1/poll_stat:read (512 Bytes): samples=0
./dm-1/poll_stat:write (512 Bytes): samples=0
./dm-1/poll_stat:read (1024 Bytes): samples=0
./dm-1/poll_stat:write (1024 Bytes): samples=0
./dm-1/poll_stat:read (2048 Bytes): samples=0
./dm-1/poll_stat:write (2048 Bytes): samples=0
./dm-1/poll_stat:read (4096 Bytes): samples=0
./dm-1/poll_stat:write (4096 Bytes): samples=0
./dm-1/poll_stat:read (8192 Bytes): samples=0
./dm-1/poll_stat:write (8192 Bytes): samples=0
./dm-1/poll_stat:read (16384 Bytes): samples=0
./dm-1/poll_stat:write (16384 Bytes): samples=0
./dm-1/poll_stat:read (32768 Bytes): samples=0
./dm-1/poll_stat:write (32768 Bytes): samples=0
./dm-1/poll_stat:read (65536 Bytes): samples=0
./dm-1/poll_stat:write (65536 Bytes): samples=0
./dm-0/sched/async_depth:48
./dm-0/sched/other_tokens:depth=64
./dm-0/sched/other_tokens:busy=0
./dm-0/sched/other_tokens:bits_per_word=64
./dm-0/sched/other_tokens:map_nr=1
./dm-0/sched/other_tokens:alloc_hint={5, 1695, 36, 42}
./dm-0/sched/other_tokens:wake_batch=8
./dm-0/sched/other_tokens:wake_index=0
./dm-0/sched/other_tokens:ws={
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/other_tokens:}
./dm-0/sched/other_tokens:round_robin=0
./dm-0/sched/sync_write_tokens:depth=24
./dm-0/sched/sync_write_tokens:busy=24
./dm-0/sched/sync_write_tokens:bits_per_word=64
./dm-0/sched/sync_write_tokens:map_nr=1
./dm-0/sched/sync_write_tokens:alloc_hint={0, 0, 0, 11}
./dm-0/sched/sync_write_tokens:wake_batch=3
./dm-0/sched/sync_write_tokens:wake_index=1
./dm-0/sched/sync_write_tokens:ws={
./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive},
./dm-0/sched/sync_write_tokens: {.wait_cnt=1, .wait=active},
./dm-0/sched/sync_write_tokens: {.wait_cnt=1, .wait=inactive},
./dm-0/sched/sync_write_tokens: {.wait_cnt=1, .wait=inactive},
./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive},
./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive},
./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive},
./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive},
./dm-0/sched/sync_write_tokens:}
./dm-0/sched/sync_write_tokens:round_robin=0
./dm-0/sched/read_tokens:depth=256
./dm-0/sched/read_tokens:busy=256
./dm-0/sched/read_tokens:bits_per_word=64
./dm-0/sched/read_tokens:map_nr=4
./dm-0/sched/read_tokens:alloc_hint={0, 44, 0, 43}
./dm-0/sched/read_tokens:wake_batch=8
./dm-0/sched/read_tokens:wake_index=0
./dm-0/sched/read_tokens:ws={
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=active},
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive},
./dm-0/sched/read_tokens:}
./dm-0/sched/read_tokens:round_robin=0
./dm-0/hctx0/sched/batching:0
./dm-0/hctx0/sched/cur_domain:OTHER
./dm-0/hctx0/sched/other_waiting:0
./dm-0/hctx0/sched/sync_write_waiting:1
./dm-0/hctx0/sched/sync_write_rqs:000000001f573771 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=79}
./dm-0/hctx0/sched/sync_write_rqs:00000000e3b936ff {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=104}
./dm-0/hctx0/sched/sync_write_rqs:000000003cf95da1 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=105}
./dm-0/hctx0/sched/sync_write_rqs:00000000a533203e {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=102}
./dm-0/hctx0/sched/sync_write_rqs:0000000077ff59b5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=99}
./dm-0/hctx0/sched/sync_write_rqs:00000000d891eb85 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=59}
./dm-0/hctx0/sched/sync_write_rqs:00000000f8c029e7 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=123}
./dm-0/hctx0/sched/sync_write_rqs:0000000059a91beb {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=80}
./dm-0/hctx0/sched/sync_write_rqs:000000003f13f5bd {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=69}
./dm-0/hctx0/sched/sync_write_rqs:00000000aa3459ed {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=72}
./dm-0/hctx0/sched/sync_write_rqs:000000004f65153b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=107}
./dm-0/hctx0/sched/sync_write_rqs:00000000156ef8c1 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=40}
./dm-0/hctx0/sched/sync_write_rqs:000000006874301c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=103}
./dm-0/hctx0/sched/sync_write_rqs:000000006fd73113 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=70}
./dm-0/hctx0/sched/sync_write_rqs:00000000ce8cc3aa {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=241}
./dm-0/hctx0/sched/sync_write_rqs:000000009bfc32cd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=57}
./dm-0/hctx0/sched/sync_write_rqs:000000008ae86f6d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=62}
./dm-0/hctx0/sched/sync_write_rqs:000000002bea67f9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=2}
./dm-0/hctx0/sched/sync_write_rqs:000000004acf53b2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=5}
./dm-0/hctx0/sched/sync_write_rqs:00000000418f9bcb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=7}
./dm-0/hctx0/sched/sync_write_rqs:00000000487ad874 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=12}
./dm-0/hctx0/sched/sync_write_rqs:00000000d7cbf463 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=15}
./dm-0/hctx0/sched/sync_write_rqs:00000000f6bd8e3a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=20}
./dm-0/hctx0/sched/sync_write_rqs:0000000054db3535 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=106}
./dm-0/hctx0/sched/sync_write_rqs:00000000eb35cbdb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=110}
./dm-0/hctx0/sched/sync_write_rqs:000000007bdd531c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=113}
./dm-0/hctx0/sched/sync_write_rqs:00000000f4b1f373 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=133}
./dm-0/hctx0/sched/sync_write_rqs:000000009d0a7052 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=67}
./dm-0/hctx0/sched/sync_write_rqs:00000000cb5775b4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=64}
./dm-0/hctx0/sched/sync_write_rqs:00000000c49a47e2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=136}
./dm-0/hctx0/sched/sync_write_rqs:00000000ce2123f9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=98}
./dm-0/hctx0/sched/sync_write_rqs:00000000693d5140 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=87}
./dm-0/hctx0/sched/sync_write_rqs:000000004bf481dd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=138}
./dm-0/hctx0/sched/sync_write_rqs:0000000019b5d899 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=86}
./dm-0/hctx0/sched/sync_write_rqs:000000006815432d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=42}
./dm-0/hctx0/sched/sync_write_rqs:00000000751a985b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=46}
./dm-0/hctx0/sched/sync_write_rqs:00000000d14f2300 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=47}
./dm-0/hctx0/sched/sync_write_rqs:00000000c701ede2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=25}
./dm-0/hctx0/sched/sync_write_rqs:0000000078fbfaee {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=31}
./dm-0/hctx0/sched/sync_write_rqs:00000000c1654839 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=190}
./dm-0/hctx0/sched/sync_write_rqs:0000000045c6f16b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=131}
./dm-0/hctx0/sched/sync_write_rqs:00000000a6d12f31 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=78}
./dm-0/hctx0/sched/sync_write_rqs:00000000cdcc74c0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=100}
./dm-0/hctx0/sched/sync_write_rqs:00000000cd5796df {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=101}
./dm-0/hctx0/sched/sync_write_rqs:00000000bf164c9e {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=75}
./dm-0/hctx0/sched/sync_write_rqs:00000000df2e49ec {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=76}
./dm-0/hctx0/sched/sync_write_rqs:00000000ebeb68f9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=172}
./dm-0/hctx0/sched/sync_write_rqs:000000000526aa72 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=174}
./dm-0/hctx0/sched/sync_write_rqs:000000005abced20 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=177}
./dm-0/hctx0/sched/sync_write_rqs:000000009dc6fe76 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=179}
./dm-0/hctx0/sched/sync_write_rqs:000000001d5deaf7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=185}
./dm-0/hctx0/sched/sync_write_rqs:000000005539190f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=191}
./dm-0/hctx0/sched/sync_write_rqs:000000003b7659de {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=202}
./dm-0/hctx0/sched/sync_write_rqs:0000000013e8b2dd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=143}
./dm-0/hctx0/sched/sync_write_rqs:00000000407e2164 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=149}
./dm-0/hctx0/sched/sync_write_rqs:000000003e25a32f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=154}
./dm-0/hctx0/sched/sync_write_rqs:00000000ea72147a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=227}
./dm-0/hctx0/sched/sync_write_rqs:00000000d79b8d48 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=229}
./dm-0/hctx0/sched/sync_write_rqs:00000000a1889d65 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=212}
./dm-0/hctx0/sched/read_waiting:1
./dm-0/hctx0/sched/read_rqs:00000000f1d18e6a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=24}
./dm-0/hctx0/sched/read_rqs:00000000a199844e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=26}
./dm-0/hctx0/sched/read_rqs:000000007efcae62 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=28}
./dm-0/hctx0/sched/read_rqs:00000000af80be3b {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=29}
./dm-0/hctx0/sched/read_rqs:00000000aa25e9cc {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=34}
./dm-0/hctx0/sched/read_rqs:00000000449b58fa {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=35}
./dm-0/hctx0/sched/read_rqs:00000000083e07f2 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=38}
./dm-0/hctx0/sched/read_rqs:000000007815ced8 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=39}
./dm-0/hctx0/sched/read_rqs:00000000a2b71248 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=44}
./dm-0/hctx0/sched/read_rqs:00000000f5d2e05e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=45}
./dm-0/hctx0/sched/read_rqs:000000005f198b9e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=48}
./dm-0/hctx0/sched/read_rqs:000000008ee34144 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=49}
./dm-0/hctx0/sched/read_rqs:0000000076e8cbf9 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=71}
./dm-0/hctx0/sched/read_rqs:0000000069f53dae {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=73}
./dm-0/hctx0/sched/read_rqs:0000000029fa385c {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=222}
./dm-0/hctx0/sched/read_rqs:000000008a5edba7 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=247}
./dm-0/hctx0/sched/read_rqs:000000009ac21dec {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=200}
./dm-0/hctx0/sched/read_rqs:00000000e01ff63e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=74}
./dm-0/hctx0/sched/read_rqs:000000006c3f8b64 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=1}
./dm-0/hctx0/sched/read_rqs:0000000006f095fa {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=6}
./dm-0/hctx0/sched/read_rqs:000000006a0ec63c {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=10}
./dm-0/hctx0/sched/read_rqs:000000000a56cd0f {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=16}
./dm-0/hctx0/sched/read_rqs:00000000d2c3813a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=27}
./dm-0/hctx0/sched/read_rqs:000000003598c30a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=30}
./dm-0/hctx0/sched/read_rqs:000000001d9cbd54 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=209}
./dm-0/hctx0/sched/read_rqs:00000000bd9be489 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=228}
./dm-0/hctx0/sched/read_rqs:00000000e7f8caa3 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=236}
./dm-0/hctx0/sched/read_rqs:000000004c771941 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=254}
./dm-0/hctx0/sched/read_rqs:0000000006a08c24 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=219}
./dm-0/hctx0/sched/read_rqs:000000001e816561 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=220}
./dm-0/hctx0/sched/read_rqs:00000000370d6c62 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=225}
./dm-0/hctx0/sched/read_rqs:00000000f213e06d {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=237}
./dm-0/hctx0/sched/read_rqs:00000000df4bd4d4 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=238}
./dm-0/hctx0/sched/read_rqs:00000000181a5782 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=19}
./dm-0/hctx0/sched/read_rqs:00000000a50d648b {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=244}
./dm-0/hctx0/sched/read_rqs:0000000036dc390a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=246}
./dm-0/hctx0/sched/read_rqs:00000000e0cb7ef4 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=36}
./dm-0/hctx0/sched/read_rqs:00000000c09c951c {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=248}
./dm-0/hctx0/sched/read_rqs:000000006944c854 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=207}
./dm-0/hctx0/sched/read_rqs:000000008ce5e0ef {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=201}
./dm-0/hctx0/sched/read_rqs:00000000152a8d0d {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=203}
./dm-0/hctx0/sched/read_rqs:00000000db3bebb5 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=204}
./dm-0/hctx0/sched/read_rqs:00000000dd2875cf {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=206}
./dm-0/hctx0/sched/read_rqs:00000000a6c0229e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=215}
./dm-0/hctx0/sched/read_rqs:00000000803073e9 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=88}
./dm-0/hctx0/sched/read_rqs:00000000801ebbaa {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=52}
./dm-0/hctx0/sched/read_rqs:0000000031200a0d {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=4}
./dm-0/hctx0/sched/read_rqs:0000000095b25961 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=9}
./dm-0/hctx0/sched/read_rqs:00000000b3358e33 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=17}
./dm-0/hctx0/sched/read_rqs:0000000089c4cdb8 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=18}
./dm-0/hctx0/cpu3/completed:2658 0
./dm-0/hctx0/cpu3/merged:460
./dm-0/hctx0/cpu3/dispatched:2689 0
./dm-0/hctx0/cpu2/completed:2341 8
./dm-0/hctx0/cpu2/merged:371
./dm-0/hctx0/cpu2/dispatched:2358 8
./dm-0/hctx0/cpu1/completed:2015 0
./dm-0/hctx0/cpu1/merged:392
./dm-0/hctx0/cpu1/dispatched:2052 0
./dm-0/hctx0/cpu0/completed:1124 0
./dm-0/hctx0/cpu0/merged:193
./dm-0/hctx0/cpu0/dispatched:1148 0
./dm-0/hctx0/active:0
./dm-0/hctx0/run:15802
./dm-0/hctx0/queued:8255
./dm-0/hctx0/dispatched: 0 15202
./dm-0/hctx0/dispatched: 1 8354
./dm-0/hctx0/dispatched: 2 0
./dm-0/hctx0/dispatched: 4 0
./dm-0/hctx0/dispatched: 8 0
./dm-0/hctx0/dispatched: 16 0
./dm-0/hctx0/dispatched: 32+ 1
./dm-0/hctx0/io_poll:considered=0
./dm-0/hctx0/io_poll:invoked=0
./dm-0/hctx0/io_poll:success=0
./dm-0/hctx0/sched_tags_bitmap:00000000: f696 1fff dcf5 134a e9df c101 fc4f 0208
./dm-0/hctx0/sched_tags_bitmap:00000010: 2885 2004 0050 0ac2 00df 9258 3a70 d241
./dm-0/hctx0/sched_tags:nr_tags=256
./dm-0/hctx0/sched_tags:nr_reserved_tags=0
./dm-0/hctx0/sched_tags:active_queues=0
./dm-0/hctx0/sched_tags:
./dm-0/hctx0/sched_tags:bitmap_tags:
./dm-0/hctx0/sched_tags:depth=256
./dm-0/hctx0/sched_tags:busy=109
./dm-0/hctx0/sched_tags:bits_per_word=64
./dm-0/hctx0/sched_tags:map_nr=4
./dm-0/hctx0/sched_tags:alloc_hint={125, 75, 84, 53}
./dm-0/hctx0/sched_tags:wake_batch=8
./dm-0/hctx0/sched_tags:wake_index=1
./dm-0/hctx0/sched_tags:ws={
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/sched_tags:}
./dm-0/hctx0/sched_tags:round_robin=0
./dm-0/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000
./dm-0/hctx0/tags:nr_tags=2048
./dm-0/hctx0/tags:nr_reserved_tags=0
./dm-0/hctx0/tags:active_queues=0
./dm-0/hctx0/tags:
./dm-0/hctx0/tags:bitmap_tags:
./dm-0/hctx0/tags:depth=2048
./dm-0/hctx0/tags:busy=0
./dm-0/hctx0/tags:bits_per_word=64
./dm-0/hctx0/tags:map_nr=32
./dm-0/hctx0/tags:alloc_hint={1173, 1146, 1156, 1172}
./dm-0/hctx0/tags:wake_batch=8
./dm-0/hctx0/tags:wake_index=0
./dm-0/hctx0/tags:ws={
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./dm-0/hctx0/tags:}
./dm-0/hctx0/tags:round_robin=0
./dm-0/hctx0/ctx_map:00000000: 00
./dm-0/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./dm-0/hctx0/state:
./dm-0/write_hints:hint0: 0
./dm-0/write_hints:hint1: 0
./dm-0/write_hints:hint2: 0
./dm-0/write_hints:hint3: 0
./dm-0/write_hints:hint4: 0
./dm-0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|STATS|REGISTERED
./dm-0/poll_stat:read (512 Bytes): samples=0
./dm-0/poll_stat:write (512 Bytes): samples=0
./dm-0/poll_stat:read (1024 Bytes): samples=0
./dm-0/poll_stat:write (1024 Bytes): samples=0
./dm-0/poll_stat:read (2048 Bytes): samples=0
./dm-0/poll_stat:write (2048 Bytes): samples=0
./dm-0/poll_stat:read (4096 Bytes): samples=0
./dm-0/poll_stat:write (4096 Bytes): samples=0
./dm-0/poll_stat:read (8192 Bytes): samples=0
./dm-0/poll_stat:write (8192 Bytes): samples=0
./dm-0/poll_stat:read (16384 Bytes): samples=0
./dm-0/poll_stat:write (16384 Bytes): samples=0
./dm-0/poll_stat:read (32768 Bytes): samples=0
./dm-0/poll_stat:write (32768 Bytes): samples=0
./dm-0/poll_stat:read (65536 Bytes): samples=0
./dm-0/poll_stat:write (65536 Bytes): samples=0
./sdb/hctx0/cpu3/completed:118 11
./sdb/hctx0/cpu3/merged:0
./sdb/hctx0/cpu3/dispatched:118 11
./sdb/hctx0/cpu2/completed:150 47
./sdb/hctx0/cpu2/merged:0
./sdb/hctx0/cpu2/dispatched:150 47
./sdb/hctx0/cpu1/completed:119 9
./sdb/hctx0/cpu1/merged:0
./sdb/hctx0/cpu1/dispatched:119 9
./sdb/hctx0/cpu0/completed:46 17
./sdb/hctx0/cpu0/merged:0
./sdb/hctx0/cpu0/dispatched:46 17
./sdb/hctx0/active:0
./sdb/hctx0/run:511
./sdb/hctx0/queued:517
./sdb/hctx0/dispatched: 0 0
./sdb/hctx0/dispatched: 1 517
./sdb/hctx0/dispatched: 2 0
./sdb/hctx0/dispatched: 4 0
./sdb/hctx0/dispatched: 8 0
./sdb/hctx0/dispatched: 16 0
./sdb/hctx0/dispatched: 32+ 0
./sdb/hctx0/io_poll:considered=0
./sdb/hctx0/io_poll:invoked=0
./sdb/hctx0/io_poll:success=0
./sdb/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./sdb/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
./sdb/hctx0/sched_tags:nr_tags=256
./sdb/hctx0/sched_tags:nr_reserved_tags=0
./sdb/hctx0/sched_tags:active_queues=0
./sdb/hctx0/sched_tags:
./sdb/hctx0/sched_tags:bitmap_tags:
./sdb/hctx0/sched_tags:depth=256
./sdb/hctx0/sched_tags:busy=0
./sdb/hctx0/sched_tags:bits_per_word=64
./sdb/hctx0/sched_tags:map_nr=4
./sdb/hctx0/sched_tags:alloc_hint={47, 230, 128, 219}
./sdb/hctx0/sched_tags:wake_batch=8
./sdb/hctx0/sched_tags:wake_index=0
./sdb/hctx0/sched_tags:ws={
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/sched_tags:}
./sdb/hctx0/sched_tags:round_robin=0
./sdb/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./sdb/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000
./sdb/hctx0/tags:nr_tags=192
./sdb/hctx0/tags:nr_reserved_tags=0
./sdb/hctx0/tags:active_queues=0
./sdb/hctx0/tags:
./sdb/hctx0/tags:bitmap_tags:
./sdb/hctx0/tags:depth=192
./sdb/hctx0/tags:busy=0
./sdb/hctx0/tags:bits_per_word=32
./sdb/hctx0/tags:map_nr=6
./sdb/hctx0/tags:alloc_hint={112, 51, 6, 134}
./sdb/hctx0/tags:wake_batch=8
./sdb/hctx0/tags:wake_index=0
./sdb/hctx0/tags:ws={
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sdb/hctx0/tags:}
./sdb/hctx0/tags:round_robin=0
./sdb/hctx0/ctx_map:00000000: 00
./sdb/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./sdb/hctx0/state:
./sdb/sched/starved:0
./sdb/sched/batching:1
./sdb/write_hints:hint0: 0
./sdb/write_hints:hint1: 0
./sdb/write_hints:hint2: 0
./sdb/write_hints:hint3: 0
./sdb/write_hints:hint4: 0
./sdb/state:BIDI|SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH
./sdb/poll_stat:read (512 Bytes): samples=0
./sdb/poll_stat:write (512 Bytes): samples=0
./sdb/poll_stat:read (1024 Bytes): samples=0
./sdb/poll_stat:write (1024 Bytes): samples=0
./sdb/poll_stat:read (2048 Bytes): samples=0
./sdb/poll_stat:write (2048 Bytes): samples=0
./sdb/poll_stat:read (4096 Bytes): samples=0
./sdb/poll_stat:write (4096 Bytes): samples=0
./sdb/poll_stat:read (8192 Bytes): samples=0
./sdb/poll_stat:write (8192 Bytes): samples=0
./sdb/poll_stat:read (16384 Bytes): samples=0
./sdb/poll_stat:write (16384 Bytes): samples=0
./sdb/poll_stat:read (32768 Bytes): samples=0
./sdb/poll_stat:write (32768 Bytes): samples=0
./sdb/poll_stat:read (65536 Bytes): samples=0
./sdb/poll_stat:write (65536 Bytes): samples=0
./sda/hctx0/cpu3/completed:6 9
./sda/hctx0/cpu3/merged:0
./sda/hctx0/cpu3/dispatched:6 9
./sda/hctx0/cpu2/completed:41 46
./sda/hctx0/cpu2/merged:0
./sda/hctx0/cpu2/dispatched:41 46
./sda/hctx0/cpu1/completed:40 11
./sda/hctx0/cpu1/merged:0
./sda/hctx0/cpu1/dispatched:40 11
./sda/hctx0/cpu0/completed:3 32
./sda/hctx0/cpu0/merged:0
./sda/hctx0/cpu0/dispatched:3 32
./sda/hctx0/active:0
./sda/hctx0/run:161
./sda/hctx0/queued:188
./sda/hctx0/dispatched: 0 0
./sda/hctx0/dispatched: 1 188
./sda/hctx0/dispatched: 2 0
./sda/hctx0/dispatched: 4 0
./sda/hctx0/dispatched: 8 0
./sda/hctx0/dispatched: 16 0
./sda/hctx0/dispatched: 32+ 0
./sda/hctx0/io_poll:considered=0
./sda/hctx0/io_poll:invoked=0
./sda/hctx0/io_poll:success=0
./sda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./sda/hctx0/tags:nr_tags=128
./sda/hctx0/tags:nr_reserved_tags=0
./sda/hctx0/tags:active_queues=0
./sda/hctx0/tags:
./sda/hctx0/tags:bitmap_tags:
./sda/hctx0/tags:depth=128
./sda/hctx0/tags:busy=0
./sda/hctx0/tags:bits_per_word=32
./sda/hctx0/tags:map_nr=4
./sda/hctx0/tags:alloc_hint={112, 68, 4, 98}
./sda/hctx0/tags:wake_batch=8
./sda/hctx0/tags:wake_index=0
./sda/hctx0/tags:ws={
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./sda/hctx0/tags:}
./sda/hctx0/tags:round_robin=0
./sda/hctx0/ctx_map:00000000: 00
./sda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
./sda/hctx0/state:
./sda/write_hints:hint0: 0
./sda/write_hints:hint1: 0
./sda/write_hints:hint2: 0
./sda/write_hints:hint3: 0
./sda/write_hints:hint4: 0
./sda/state:SAME_COMP|IO_STAT|DISCARD|INIT_DONE|POLL|WC|REGISTERED|SCSI_PASSTHROUGH
./sda/poll_stat:read (512 Bytes): samples=0
./sda/poll_stat:write (512 Bytes): samples=0
./sda/poll_stat:read (1024 Bytes): samples=0
./sda/poll_stat:write (1024 Bytes): samples=0
./sda/poll_stat:read (2048 Bytes): samples=0
./sda/poll_stat:write (2048 Bytes): samples=0
./sda/poll_stat:read (4096 Bytes): samples=0
./sda/poll_stat:write (4096 Bytes): samples=0
./sda/poll_stat:read (8192 Bytes): samples=0
./sda/poll_stat:write (8192 Bytes): samples=0
./sda/poll_stat:read (16384 Bytes): samples=0
./sda/poll_stat:write (16384 Bytes): samples=0
./sda/poll_stat:read (32768 Bytes): samples=0
./sda/poll_stat:write (32768 Bytes): samples=0
./sda/poll_stat:read (65536 Bytes): samples=0
./sda/poll_stat:write (65536 Bytes): samples=0
./vda/hctx0/cpu3/completed:1348 162
./vda/hctx0/cpu3/merged:4
./vda/hctx0/cpu3/dispatched:1348 162
./vda/hctx0/cpu2/completed:1088 46
./vda/hctx0/cpu2/merged:7
./vda/hctx0/cpu2/dispatched:1088 46
./vda/hctx0/cpu1/completed:2092 20
./vda/hctx0/cpu1/merged:0
./vda/hctx0/cpu1/dispatched:2092 20
./vda/hctx0/cpu0/completed:2114 79
./vda/hctx0/cpu0/merged:16
./vda/hctx0/cpu0/dispatched:2114 79
./vda/hctx0/active:0
./vda/hctx0/run:6902
./vda/hctx0/queued:6949
./vda/hctx0/dispatched: 0 0
./vda/hctx0/dispatched: 1 6810
./vda/hctx0/dispatched: 2 42
./vda/hctx0/dispatched: 4 19
./vda/hctx0/dispatched: 8 16
./vda/hctx0/dispatched: 16 8
./vda/hctx0/dispatched: 32+ 0
./vda/hctx0/io_poll:considered=0
./vda/hctx0/io_poll:invoked=0
./vda/hctx0/io_poll:success=0
./vda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
./vda/hctx0/tags:nr_tags=128
./vda/hctx0/tags:nr_reserved_tags=0
./vda/hctx0/tags:active_queues=0
./vda/hctx0/tags:
./vda/hctx0/tags:bitmap_tags:
./vda/hctx0/tags:depth=128
./vda/hctx0/tags:busy=0
./vda/hctx0/tags:bits_per_word=32
./vda/hctx0/tags:map_nr=4
./vda/hctx0/tags:alloc_hint={97, 74, 111, 53}
./vda/hctx0/tags:wake_batch=8
./vda/hctx0/tags:wake_index=0
./vda/hctx0/tags:ws={
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive},
./vda/hctx0/tags:}
./vda/hctx0/tags:round_robin=0
./vda/hctx0/ctx_map:00000000: 00
./vda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE
./vda/hctx0/state:
./vda/write_hints:hint0: 0
./vda/write_hints:hint1: 0
./vda/write_hints:hint2: 0
./vda/write_hints:hint3: 0
./vda/write_hints:hint4: 0
./vda/state:SAME_COMP|IO_STAT|INIT_DONE|NO_SG_MERGE|POLL|WC|REGISTERED
./vda/poll_stat:read (512 Bytes): samples=0
./vda/poll_stat:write (512 Bytes): samples=0
./vda/poll_stat:read (1024 Bytes): samples=0
./vda/poll_stat:write (1024 Bytes): samples=0
./vda/poll_stat:read (2048 Bytes): samples=0
./vda/poll_stat:write (2048 Bytes): samples=0
./vda/poll_stat:read (4096 Bytes): samples=0
./vda/poll_stat:write (4096 Bytes): samples=0
./vda/poll_stat:read (8192 Bytes): samples=0
./vda/poll_stat:write (8192 Bytes): samples=0
./vda/poll_stat:read (16384 Bytes): samples=0
./vda/poll_stat:write (16384 Bytes): samples=0
./vda/poll_stat:read (32768 Bytes): samples=0
./vda/poll_stat:write (32768 Bytes): samples=0
./vda/poll_stat:read (65536 Bytes): samples=0
./vda/poll_stat:write (65536 Bytes): samples=0
./sr0/hctx0/cpu3/completed:7 10
./sr0/hctx0/cpu3/merged:0
./sr0/hctx0/cpu3/dispatched:7 10
./sr0/hctx0/cpu2/completed:39 299
./sr0/hctx0/cpu2/merged:0
./sr0/hctx0/cpu2/dispatched:39 299
./sr0/hctx0/cpu1/completed:0 37
./sr0/hctx0/cpu1/merged:0
./sr0/hctx0/cpu1/dispatched:0 37
./sr0/hctx0/cpu0/completed:0 26
./sr0/hctx0/cpu0/merged:0
./sr0/hctx0/cpu0/dispatched:0 26
./sr0/hctx0/active:0
./sr0/hctx0/run:418
./sr0/hctx0/queued:418
./sr0/hctx0/dispatched: 0 0
./sr0/hctx0/dispatched: 1 418
./sr0/hctx0/dispatched: 2 0
./sr0/hctx0/dispatched: 4 0
./sr0/hctx0/dispatched: 8 0
./sr0/hctx0/dispatched: 16 0
./sr0/hctx0/dispatched: 32+ 0
./sr0/hctx0/io_poll:considered=0
./sr0/hctx0/io_poll:invoked=0
./sr0/hctx0/io_poll:success=0
./sr0/hctx0/tags_bitmap:00000000: 00
./sr0/hctx0/tags:nr_tags=1
./sr0/hctx0/tags:nr_reserved_tags=0
./sr0/hctx0/tags:active_queues=0
./sr0/hctx0/tags:
./sr0/hctx0/tags:bitmap_tags:
./sr0/hctx0/tags:depth=1
./sr0/hctx0/tags:busy=0
./sr0/hctx0/tags:bits_per_word=64
./sr0/hctx0/tags:map_nr=1
./sr0/hctx0/tags:alloc_hint={0, 0, 0, 0}
./sr0/hctx0/tags:wake_batch=1
./sr0/hctx0/tags:wake_index=7
./sr0/hctx0/tags:ws={
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive},
./sr0/hctx0/tags:}
./sr0/hctx0/tags:round_robin=1
./sr0/hctx0/ctx_map:00000000: 00
./sr0/hctx0/flags:alloc_policy=RR SHOULD_MERGE|SG_MERGE
./sr0/hctx0/state:
./sr0/write_hints:hint0: 0
./sr0/write_hints:hint1: 0
./sr0/write_hints:hint2: 0
./sr0/write_hints:hint3: 0
./sr0/write_hints:hint4: 0
./sr0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|FLUSH_NQ|REGISTERED|SCSI_PASSTHROUGH
./sr0/poll_stat:read (512 Bytes): samples=0
./sr0/poll_stat:write (512 Bytes): samples=0
./sr0/poll_stat:read (1024 Bytes): samples=0
./sr0/poll_stat:write (1024 Bytes): samples=0
./sr0/poll_stat:read (2048 Bytes): samples=0
./sr0/poll_stat:write (2048 Bytes): samples=0
./sr0/poll_stat:read (4096 Bytes): samples=0
./sr0/poll_stat:write (4096 Bytes): samples=0
./sr0/poll_stat:read (8192 Bytes): samples=0
./sr0/poll_stat:write (8192 Bytes): samples=0
./sr0/poll_stat:read (16384 Bytes): samples=0
./sr0/poll_stat:write (16384 Bytes): samples=0
./sr0/poll_stat:read (32768 Bytes): samples=0
./sr0/poll_stat:write (32768 Bytes): samples=0
./sr0/poll_stat:read (65536 Bytes): samples=0
./sr0/poll_stat:write (65536 Bytes): samples=0
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
2018-01-17 19:45 ` Bart Van Assche
@ 2018-01-17 20:14 ` Mike Snitzer
2018-01-17 21:27 ` Bart Van Assche
0 siblings, 1 reply; 12+ messages in thread
From: Mike Snitzer @ 2018-01-17 20:14 UTC (permalink / raw)
To: Bart Van Assche; +Cc: axboe, dm-devel, loberman, ming.lei
On Wed, Jan 17 2018 at 2:45pm -0500,
Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
> On Wed, 2018-01-17 at 14:36 -0500, Mike Snitzer wrote:
> > That tree doesn't have the revert you think it does (because it doesn't have _any_ of the dm-4.16 changes).
> >
> > So there is some other reason for your hang.
>
> Hello Mike,
>
> Please have another look, namely at the following commit:
> https://github.com/bvanassche/linux/commit/cd0b6b9f8779a0b692b1ef1d1f5ba6cb2fd981c5
>
> That contents of that commit is as follows:
>
> Subject: [PATCH] Revert "dm rq: Avoid that request processing stalls sporadically"
>
> This reverts commit 6077c2d706097c00d8f2fed57d3f3c45cd228ee8.
> ---
> drivers/md/dm-rq.c | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
> index 85d80b8ccc33..a9e0f6ec62b6 100644
> --- a/drivers/md/dm-rq.c
> +++ b/drivers/md/dm-rq.c
> @@ -773,7 +773,6 @@ static blk_status_t dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx,
> /* Undo dm_start_request() before requeuing */
> rq_end_stats(md, rq);
> rq_completed(md, rq_data_dir(rq), false);
> - blk_mq_delay_run_hw_queue(hctx, 100/*ms*/);
> return BLK_STS_RESOURCE;
> }
>
Fine, the chrome browser on my phone displayed "master" rather than your
other branch.
BUT my broader point stands: you aren't testing the dm-4.16 changes. By
just reverting that commit you're creating a self-fulfilling prophecy
(that you'll see hangs without it).
Fact is you should pull all of dm-4.16 in, see:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.16
But these dm-4.16 changes are particularly important:
050af08ffb1b dm mpath: return DM_MAPIO_REQUEUE on blk-mq rq allocation failure
459b54019cfe dm mpath: return DM_MAPIO_DELAY_REQUEUE if QUEUE_IO or PG_INIT_REQUIRED
ec3eaf9a6731 dm mpath: don't call blk_mq_delay_run_hw_queue() in case of BLK_STS_RESOURCE
4dd6edd23e7e dm mpath: delay the retry of a request if the target responded as busy
This last one is the commit that _should_ serve as a proper replacement
for the change you manually reverted in your branch.
Please re-test after pulling in dm-4.16 and let us know how things fair.
Thanks!
Mike
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: dm-rq queue stalls
[not found] <a096db4d-1c32-4682-844d-4a7fc02d47da@email.android.com>
@ 2018-01-17 19:45 ` Bart Van Assche
2018-01-17 20:14 ` Mike Snitzer
0 siblings, 1 reply; 12+ messages in thread
From: Bart Van Assche @ 2018-01-17 19:45 UTC (permalink / raw)
To: snitzer; +Cc: axboe, dm-devel, loberman, ming.lei
On Wed, 2018-01-17 at 14:36 -0500, Mike Snitzer wrote:
> That tree doesn't have the revert you think it does (because it doesn't have _any_ of the dm-4.16 changes).
>
> So there is some other reason for your hang.
Hello Mike,
Please have another look, namely at the following commit:
https://github.com/bvanassche/linux/commit/cd0b6b9f8779a0b692b1ef1d1f5ba6cb2fd981c5
That contents of that commit is as follows:
Subject: [PATCH] Revert "dm rq: Avoid that request processing stalls sporadically"
This reverts commit 6077c2d706097c00d8f2fed57d3f3c45cd228ee8.
---
drivers/md/dm-rq.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
index 85d80b8ccc33..a9e0f6ec62b6 100644
--- a/drivers/md/dm-rq.c
+++ b/drivers/md/dm-rq.c
@@ -773,7 +773,6 @@ static blk_status_t dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx,
/* Undo dm_start_request() before requeuing */
rq_end_stats(md, rq);
rq_completed(md, rq_data_dir(rq), false);
- blk_mq_delay_run_hw_queue(hctx, 100/*ms*/);
return BLK_STS_RESOURCE;
}
Thanks,
Bart.
^ permalink raw reply related [flat|nested] 12+ messages in thread
end of thread, other threads:[~2018-01-18 1:53 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-17 18:05 dm-rq queue stalls Bart Van Assche
2018-01-17 18:18 ` Mike Snitzer
2018-01-17 18:27 ` Bart Van Assche
2018-01-17 18:29 ` Laurence Oberman
2018-01-17 19:29 ` Bart Van Assche
2018-01-17 19:46 ` Laurence Oberman
2018-01-18 1:53 ` Ming Lei
[not found] <a096db4d-1c32-4682-844d-4a7fc02d47da@email.android.com>
2018-01-17 19:45 ` Bart Van Assche
2018-01-17 20:14 ` Mike Snitzer
2018-01-17 21:27 ` Bart Van Assche
2018-01-17 21:37 ` Mike Snitzer
2018-01-17 21:49 ` Bart Van Assche
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.