* io_uring on CentOS8?
@ 2021-02-14 15:17 Daniel Menzel
2021-02-14 16:49 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Menzel @ 2021-02-14 15:17 UTC (permalink / raw)
To: fio, Jens Axboe
[-- Attachment #1: Type: text/plain, Size: 979 bytes --]
/Somehow it seems as if I'm receiving messages from the mailing list -
but mine do not seem to get through. So I'm trying again./
Hi,
currently I started testing fairly fast NVMe-drives and I for the first
time I dealt with the question whether the io engine is sufficient (up
till now I just the default /libaio/). I found some information on
io_uring online and now tried to switch to that engine:
fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M
--filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1
--refill_buffers --ioengine=io_uring --group_reporting --fallocate=none
--runtime=60 --time_based --hipri
But it crashes with the following information: fio: pid=2490,
err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function
not implemented
My system:
1. fio-3.25
2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64)
3. I have installed the packes liburing and liburing-devel
Am I doing something wrong?
Kind regards
Daniel
[-- Attachment #2: Type: text/html, Size: 1512 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8?
2021-02-14 15:17 io_uring on CentOS8? Daniel Menzel
@ 2021-02-14 16:49 ` Jens Axboe
2021-02-16 19:00 ` Ober, Frank
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2021-02-14 16:49 UTC (permalink / raw)
To: Daniel Menzel, fio
On 2/14/21 8:17 AM, Daniel Menzel wrote:
> /Somehow it seems as if I'm receiving messages from the mailing list - but mine do not seem to get through. So I'm trying again./
>
> Hi,
>
> currently I started testing fairly fast NVMe-drives and I for the first time I dealt with the question whether the io engine is sufficient (up till now I just the default /libaio/). I found some information on io_uring online and now tried to switch to that engine:
>
> fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M --filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1 --refill_buffers --ioengine=io_uring --group_reporting --fallocate=none --runtime=60 --time_based --hipri
>
> But it crashes with the following information: fio: pid=2490, err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function not implemented
>
> My system:
>
> 1. fio-3.25
> 2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64)
> 3. I have installed the packes liburing and liburing-devel
>
> Am I doing something wrong?
You are not, but since you get -ENOSYS from the system call, that means that
the kernel doesn't support io_uring. It was officially added with the 5.1
kernel, so 4.18 predates that.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: io_uring on CentOS8?
2021-02-14 16:49 ` Jens Axboe
@ 2021-02-16 19:00 ` Ober, Frank
2021-02-16 19:46 ` Daniel Menzel
2021-02-16 19:51 ` Jens Axboe
0 siblings, 2 replies; 6+ messages in thread
From: Ober, Frank @ 2021-02-16 19:00 UTC (permalink / raw)
To: Jens Axboe, Daniel Menzel, fio
Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work.
Here's what I tried on PCIe Gen3 system with an Optane drive and your drive.
[root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \
> --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \
> --refill_buffers --ioengine=pvsync2 --group_reporting \
> --fallocate=none --runtime=60 --time_based --hipri
mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1
fio-3.24-5-g2ee2
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s]
I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2
Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s]
HTH,
Frank Ober
-----Original Message-----
From: Jens Axboe <axboe@kernel.dk>
Sent: Sunday, February 14, 2021 8:50 AM
To: Daniel Menzel <daniel.menzel@menzel-it.net>; fio@vger.kernel.org
Subject: Re: io_uring on CentOS8?
On 2/14/21 8:17 AM, Daniel Menzel wrote:
> /Somehow it seems as if I'm receiving messages from the mailing list -
> but mine do not seem to get through. So I'm trying again./
>
> Hi,
>
> currently I started testing fairly fast NVMe-drives and I for the first time I dealt with the question whether the io engine is sufficient (up till now I just the default /libaio/). I found some information on io_uring online and now tried to switch to that engine:
>
> fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M
> --filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1
> --refill_buffers --ioengine=io_uring --group_reporting
> --fallocate=none --runtime=60 --time_based --hipri
>
> But it crashes with the following information: fio: pid=2490,
> err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function
> not implemented
>
> My system:
>
> 1. fio-3.25
> 2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64) 3. I have installed
> the packes liburing and liburing-devel
>
> Am I doing something wrong?
You are not, but since you get -ENOSYS from the system call, that means that the kernel doesn't support io_uring. It was officially added with the 5.1 kernel, so 4.18 predates that.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8?
2021-02-16 19:00 ` Ober, Frank
@ 2021-02-16 19:46 ` Daniel Menzel
2021-02-16 19:51 ` Jens Axboe
1 sibling, 0 replies; 6+ messages in thread
From: Daniel Menzel @ 2021-02-16 19:46 UTC (permalink / raw)
To: Ober, Frank, Jens Axboe, fio
[-- Attachment #1: Type: text/plain, Size: 3106 bytes --]
Hi Frank,
thanks a lot for that input! Currently I'm switching to Ubuntu for my
tests - hoping that /io_uring/ will help. But I will definitely try out
pvsync2 as well.
Thanks!
Daniel
Am 16.02.21 um 20:00 schrieb Ober, Frank:
> Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work.
>
> Here's what I tried on PCIe Gen3 system with an Optane drive and your drive.
>
>
> [root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \
>> --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \
>> --refill_buffers --ioengine=pvsync2 --group_reporting \
>> --fallocate=none --runtime=60 --time_based --hipri
> mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1
> fio-3.24-5-g2ee2
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s]
>
> I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2
> Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s]
>
> HTH,
> Frank Ober
>
>
> -----Original Message-----
> From: Jens Axboe <axboe@kernel.dk>
> Sent: Sunday, February 14, 2021 8:50 AM
> To: Daniel Menzel <daniel.menzel@menzel-it.net>; fio@vger.kernel.org
> Subject: Re: io_uring on CentOS8?
>
> On 2/14/21 8:17 AM, Daniel Menzel wrote:
>> /Somehow it seems as if I'm receiving messages from the mailing list -
>> but mine do not seem to get through. So I'm trying again./
>>
>> Hi,
>>
>> currently I started testing fairly fast NVMe-drives and I for the first time I dealt with the question whether the io engine is sufficient (up till now I just the default /libaio/). I found some information on io_uring online and now tried to switch to that engine:
>>
>> fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M
>> --filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1
>> --refill_buffers --ioengine=io_uring --group_reporting
>> --fallocate=none --runtime=60 --time_based --hipri
>>
>> But it crashes with the following information: fio: pid=2490,
>> err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function
>> not implemented
>>
>> My system:
>>
>> 1. fio-3.25
>> 2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64) 3. I have installed
>> the packes liburing and liburing-devel
>>
>> Am I doing something wrong?
> You are not, but since you get -ENOSYS from the system call, that means that the kernel doesn't support io_uring. It was officially added with the 5.1 kernel, so 4.18 predates that.
>
>
> --
> Jens Axboe
>
--
Daniel Menzel
Geschäftsführer
Menzel IT GmbH
Charlottenburger Str. 33a
13086 Berlin
+49 (0) 30 / 5130 444 - 00
daniel.menzel@menzel-it.net
https://menzel-it.net
Geschäftsführer: Daniel Menzel, Josefin Menzel
Unternehmenssitz: Berlin
Handelsregister: Amtsgericht Charlottenburg
Handelsregister-Nummer: HRB 149835 B
USt-ID: DE 309 226 751
[-- Attachment #2: Type: text/html, Size: 4211 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8?
2021-02-16 19:00 ` Ober, Frank
2021-02-16 19:46 ` Daniel Menzel
@ 2021-02-16 19:51 ` Jens Axboe
2021-02-17 8:42 ` Daniel Menzel
1 sibling, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2021-02-16 19:51 UTC (permalink / raw)
To: Ober, Frank, Daniel Menzel, fio
On 2/16/21 12:00 PM, Ober, Frank wrote:
> Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work.
>
> Here's what I tried on PCIe Gen3 system with an Optane drive and your drive.
>
>
> [root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \
>> --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \
>> --refill_buffers --ioengine=pvsync2 --group_reporting \
>> --fallocate=none --runtime=60 --time_based --hipri
> mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1
> fio-3.24-5-g2ee2
> Starting 1 process
> Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s]
>
> I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2
> Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s]
That's not too surprising, if you're at queuedepth=1 or doing sync IO, then
I'd consider it a major win that the async API is just as fast or faster
than the sync one :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8?
2021-02-16 19:51 ` Jens Axboe
@ 2021-02-17 8:42 ` Daniel Menzel
0 siblings, 0 replies; 6+ messages in thread
From: Daniel Menzel @ 2021-02-17 8:42 UTC (permalink / raw)
To: Jens Axboe, Ober, Frank, fio
By the way: Does anyone do some system profiling (flame graphs or
something like that) while testing devices to make sure you reached the
devices' capabilities instead of the systems?
I just made some tests of nvme drives and a friend of mine (who is a
scientist) said that if that was a paper he would reject it because I do
not know whether I measure the devices or the system.
Am 16.02.21 um 20:51 schrieb Jens Axboe:
> On 2/16/21 12:00 PM, Ober, Frank wrote:
>> Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work.
>>
>> Here's what I tried on PCIe Gen3 system with an Optane drive and your drive.
>>
>>
>> [root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \
>>> --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \
>>> --refill_buffers --ioengine=pvsync2 --group_reporting \
>>> --fallocate=none --runtime=60 --time_based --hipri
>> mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1
>> fio-3.24-5-g2ee2
>> Starting 1 process
>> Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s]
>>
>> I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2
>> Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s]
> That's not too surprising, if you're at queuedepth=1 or doing sync IO, then
> I'd consider it a major win that the async API is just as fast or faster
> than the sync one :-)
>
>
--
Daniel Menzel
Geschäftsführer
Menzel IT GmbH
Charlottenburger Str. 33a
13086 Berlin
+49 (0) 30 / 5130 444 - 00
daniel.menzel@menzel-it.net
https://menzel-it.net
Geschäftsführer: Daniel Menzel, Josefin Menzel
Unternehmenssitz: Berlin
Handelsregister: Amtsgericht Charlottenburg
Handelsregister-Nummer: HRB 149835 B
USt-ID: DE 309 226 751
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-02-17 8:42 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-14 15:17 io_uring on CentOS8? Daniel Menzel
2021-02-14 16:49 ` Jens Axboe
2021-02-16 19:00 ` Ober, Frank
2021-02-16 19:46 ` Daniel Menzel
2021-02-16 19:51 ` Jens Axboe
2021-02-17 8:42 ` Daniel Menzel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.