[io_uring]: fio's ./t/io_uring on /dev/nullb0 causing CPU stalls

[io_uring]: fio's ./t/io_uring on /dev/nullb0 causing CPU stalls

[Stephen Bates]

Hi

I hit the following BUG when testing fio's t/io_uring test on a null block device. This was inside a 4 vCPU VM running Ubuntu 18.04 and a 5.1.11 kernel. I don't see the same issue if I run the same test on a fake NVMe device (I have not tested bare-metal or real NVMe devices). This is for the HEAD of fio as of yesterday. I also had the most recent version of liburing but I don't believe ./t/io_uring uses liburing. I will test 5.2-rc series to see if the issue also exists there.

Steps:

1. modprobe null_blk
2. sudo ./t/io_uring /dev/nullb0

batesste@io_uring-vm1:~/fio$ fio --version
fio-3.14-7-g7184a

batesste@io_uring-vm1:~/fio$ sudo ./t/io_uring /dev/nullb0
[sudo] password for batesste:
Added file /dev/nullb0
sq_ring ptr = 0x0x7f5ef3bec000
sqes ptr    = 0x0x7f5ef3bea000
cq_ring ptr = 0x0x7f5ef3be8000
polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
submitter=743
IOPS=259680, IOS/call=32/31, inflight=128 (128)

[  288.190467] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [io_uring:743]
[  288.191480] Modules linked in: crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd input_leds joydev cryptd serio_raw mac_hid glue_helper sch_fq_codel null_blk sunrpc ip_tables x_tables autofs4 8139too psmouse pata_acpi 8139cp i2c_piix4 mii floppy
[  288.191492] CPU: 1 PID: 743 Comm: io_uring Not tainted 5.1.11-cpacket-io-uring #1
[  288.191493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[  288.191499] RIP: 0010:blkdev_iopoll+0x6/0x30
[  288.191500] Code: 00 0f 1f 40 00 48 89 df e8 47 b5 fd ff 48 83 c4 08 4c 89 e0 5b 41 5c 5d c3 45 31 e4 eb ef 66 0f 1f 44 00 00 0f 1f 44 00 00 55 <48> 8b 07 89 f2 8b 77 28 0f b6 d2 48 89 e5 48 8b 80 f0 00 00 00 48
[  288.191501] RSP: 0018:ffffbc3e413e3de0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  288.191502] RAX: ffffffffbbee7650 RBX: 0000000000000001 RCX: ffff9d77f50afd18
[  288.191503] RDX: 00000000005040d0 RSI: 0000000000000001 RDI: ffff9d77f50afcc0
[  288.191504] RBP: ffffbc3e413e3e88 R08: ffff9d77f55d6401 R09: ffff9d77f55d6400
[  288.191504] R10: 0000000000000020 R11: ffff9d77ed5c3800 R12: ffff9d77f50afa80
[  288.191505] R13: ffff9d77ed5c3800 R14: ffffbc3e413e3ef4 R15: ffffbc3e413e3e08
[  288.191506] FS:  00007f5ef33b5700(0000) GS:ffff9d77f7a80000(0000) knlGS:0000000000000000
[  288.191507] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  288.191507] CR2: 00007efd77803000 CR3: 00000002335a6001 CR4: 00000000003606e0
[  288.191511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  288.191511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  288.191512] Call Trace:
[  288.191517]  ? io_iopoll_getevents+0xf7/0x2e0
[  288.191519]  io_iopoll_check+0x40/0x70
[  288.191521]  __x64_sys_io_uring_enter+0x156/0x300
[  288.191525]  do_syscall_64+0x5a/0x110
[  288.191528]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  288.191529] RIP: 0033:0x7f5ef34d1839
[  288.191531] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1f f6 2c 00 f7 d8 64 89 01 48
[  288.191531] RSP: 002b:00007f5ef33b4e78 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
[  288.191532] RAX: ffffffffffffffda RBX: 000056517d004260 RCX: 00007f5ef34d1839
[  288.191533] RDX: 0000000000000020 RSI: 0000000000000020 RDI: 0000000000000004
[  288.191533] RBP: 000056517d004270 R08: 0000000000000000 R09: 0000000000000000
[  288.191534] R10: 0000000000000001 R11: 0000000000000246 R12: 000000000000007f
[  288.191535] R13: 000000000003f680 R14: 0000000000000020 R15: 0000000000000020

Cheers
 
Stephen
 


Re: [io_uring]: fio's ./t/io_uring on /dev/nullb0 causing CPU stalls

[Stephen Bates]

Hi 

> I hit the following BUG when testing fio's t/io_uring test on a null block device

I confirmed the same BUG occurs on 5.2-rc5. I also managed to hit a similar bug if I use the io_uring engine in fio but only when the hipri engine option is enabled. 

Stephen    

Re: [io_uring]: fio's ./t/io_uring on /dev/nullb0 causing CPU stalls

[Jens Axboe]

On 6/21/19 6:06 AM, Stephen  Bates wrote:
> Hi
> 
>> I hit the following BUG when testing fio's t/io_uring test on a null
>> block device
> 
> I confirmed the same BUG occurs on 5.2-rc5. I also managed to hit a
> similar bug if I use the io_uring engine in fio but only when the
> hipri engine option is enabled.

Can't seem to reproduce this, but from your description, seems to be
specific to polled IO.

Do you have any options set for null_blk? Can you send your kernel
.config as well?

-- 
Jens Axboe