* io_uring on CentOS8? @ 2021-02-14 15:17 Daniel Menzel 2021-02-14 16:49 ` Jens Axboe 0 siblings, 1 reply; 6+ messages in thread From: Daniel Menzel @ 2021-02-14 15:17 UTC (permalink / raw) To: fio, Jens Axboe [-- Attachment #1: Type: text/plain, Size: 979 bytes --] /Somehow it seems as if I'm receiving messages from the mailing list - but mine do not seem to get through. So I'm trying again./ Hi, currently I started testing fairly fast NVMe-drives and I for the first time I dealt with the question whether the io engine is sufficient (up till now I just the default /libaio/). I found some information on io_uring online and now tried to switch to that engine: fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M --filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1 --refill_buffers --ioengine=io_uring --group_reporting --fallocate=none --runtime=60 --time_based --hipri But it crashes with the following information: fio: pid=2490, err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function not implemented My system: 1. fio-3.25 2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64) 3. I have installed the packes liburing and liburing-devel Am I doing something wrong? Kind regards Daniel [-- Attachment #2: Type: text/html, Size: 1512 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8? 2021-02-14 15:17 io_uring on CentOS8? Daniel Menzel @ 2021-02-14 16:49 ` Jens Axboe 2021-02-16 19:00 ` Ober, Frank 0 siblings, 1 reply; 6+ messages in thread From: Jens Axboe @ 2021-02-14 16:49 UTC (permalink / raw) To: Daniel Menzel, fio On 2/14/21 8:17 AM, Daniel Menzel wrote: > /Somehow it seems as if I'm receiving messages from the mailing list - but mine do not seem to get through. So I'm trying again./ > > Hi, > > currently I started testing fairly fast NVMe-drives and I for the first time I dealt with the question whether the io engine is sufficient (up till now I just the default /libaio/). I found some information on io_uring online and now tried to switch to that engine: > > fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M --filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1 --refill_buffers --ioengine=io_uring --group_reporting --fallocate=none --runtime=60 --time_based --hipri > > But it crashes with the following information: fio: pid=2490, err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function not implemented > > My system: > > 1. fio-3.25 > 2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64) > 3. I have installed the packes liburing and liburing-devel > > Am I doing something wrong? You are not, but since you get -ENOSYS from the system call, that means that the kernel doesn't support io_uring. It was officially added with the 5.1 kernel, so 4.18 predates that. -- Jens Axboe ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: io_uring on CentOS8? 2021-02-14 16:49 ` Jens Axboe @ 2021-02-16 19:00 ` Ober, Frank 2021-02-16 19:46 ` Daniel Menzel 2021-02-16 19:51 ` Jens Axboe 0 siblings, 2 replies; 6+ messages in thread From: Ober, Frank @ 2021-02-16 19:00 UTC (permalink / raw) To: Jens Axboe, Daniel Menzel, fio Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work. Here's what I tried on PCIe Gen3 system with an Optane drive and your drive. [root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \ > --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \ > --refill_buffers --ioengine=pvsync2 --group_reporting \ > --fallocate=none --runtime=60 --time_based --hipri mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1 fio-3.24-5-g2ee2 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s] I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2 Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s] HTH, Frank Ober -----Original Message----- From: Jens Axboe <axboe@kernel.dk> Sent: Sunday, February 14, 2021 8:50 AM To: Daniel Menzel <daniel.menzel@menzel-it.net>; fio@vger.kernel.org Subject: Re: io_uring on CentOS8? On 2/14/21 8:17 AM, Daniel Menzel wrote: > /Somehow it seems as if I'm receiving messages from the mailing list - > but mine do not seem to get through. So I'm trying again./ > > Hi, > > currently I started testing fairly fast NVMe-drives and I for the first time I dealt with the question whether the io engine is sufficient (up till now I just the default /libaio/). I found some information on io_uring online and now tried to switch to that engine: > > fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M > --filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1 > --refill_buffers --ioengine=io_uring --group_reporting > --fallocate=none --runtime=60 --time_based --hipri > > But it crashes with the following information: fio: pid=2490, > err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function > not implemented > > My system: > > 1. fio-3.25 > 2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64) 3. I have installed > the packes liburing and liburing-devel > > Am I doing something wrong? You are not, but since you get -ENOSYS from the system call, that means that the kernel doesn't support io_uring. It was officially added with the 5.1 kernel, so 4.18 predates that. -- Jens Axboe ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8? 2021-02-16 19:00 ` Ober, Frank @ 2021-02-16 19:46 ` Daniel Menzel 2021-02-16 19:51 ` Jens Axboe 1 sibling, 0 replies; 6+ messages in thread From: Daniel Menzel @ 2021-02-16 19:46 UTC (permalink / raw) To: Ober, Frank, Jens Axboe, fio [-- Attachment #1: Type: text/plain, Size: 3106 bytes --] Hi Frank, thanks a lot for that input! Currently I'm switching to Ubuntu for my tests - hoping that /io_uring/ will help. But I will definitely try out pvsync2 as well. Thanks! Daniel Am 16.02.21 um 20:00 schrieb Ober, Frank: > Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work. > > Here's what I tried on PCIe Gen3 system with an Optane drive and your drive. > > > [root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \ >> --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \ >> --refill_buffers --ioengine=pvsync2 --group_reporting \ >> --fallocate=none --runtime=60 --time_based --hipri > mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1 > fio-3.24-5-g2ee2 > Starting 1 process > Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s] > > I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2 > Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s] > > HTH, > Frank Ober > > > -----Original Message----- > From: Jens Axboe <axboe@kernel.dk> > Sent: Sunday, February 14, 2021 8:50 AM > To: Daniel Menzel <daniel.menzel@menzel-it.net>; fio@vger.kernel.org > Subject: Re: io_uring on CentOS8? > > On 2/14/21 8:17 AM, Daniel Menzel wrote: >> /Somehow it seems as if I'm receiving messages from the mailing list - >> but mine do not seem to get through. So I'm trying again./ >> >> Hi, >> >> currently I started testing fairly fast NVMe-drives and I for the first time I dealt with the question whether the io engine is sufficient (up till now I just the default /libaio/). I found some information on io_uring online and now tried to switch to that engine: >> >> fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M >> --filename=/dev/nvme0n1 --name=mylittletest --direct=1 --fsync=1 >> --refill_buffers --ioengine=io_uring --group_reporting >> --fallocate=none --runtime=60 --time_based --hipri >> >> But it crashes with the following information: fio: pid=2490, >> err=38/file:engines/io_uring.c:699, func=io_queue_init, error=Function >> not implemented >> >> My system: >> >> 1. fio-3.25 >> 2. CentOS8 (Kernel 4.18.0-240.1.1.el8_3.x86_64) 3. I have installed >> the packes liburing and liburing-devel >> >> Am I doing something wrong? > You are not, but since you get -ENOSYS from the system call, that means that the kernel doesn't support io_uring. It was officially added with the 5.1 kernel, so 4.18 predates that. > > > -- > Jens Axboe > -- Daniel Menzel Geschäftsführer Menzel IT GmbH Charlottenburger Str. 33a 13086 Berlin +49 (0) 30 / 5130 444 - 00 daniel.menzel@menzel-it.net https://menzel-it.net Geschäftsführer: Daniel Menzel, Josefin Menzel Unternehmenssitz: Berlin Handelsregister: Amtsgericht Charlottenburg Handelsregister-Nummer: HRB 149835 B USt-ID: DE 309 226 751 [-- Attachment #2: Type: text/html, Size: 4211 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8? 2021-02-16 19:00 ` Ober, Frank 2021-02-16 19:46 ` Daniel Menzel @ 2021-02-16 19:51 ` Jens Axboe 2021-02-17 8:42 ` Daniel Menzel 1 sibling, 1 reply; 6+ messages in thread From: Jens Axboe @ 2021-02-16 19:51 UTC (permalink / raw) To: Ober, Frank, Daniel Menzel, fio On 2/16/21 12:00 PM, Ober, Frank wrote: > Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work. > > Here's what I tried on PCIe Gen3 system with an Optane drive and your drive. > > > [root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \ >> --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \ >> --refill_buffers --ioengine=pvsync2 --group_reporting \ >> --fallocate=none --runtime=60 --time_based --hipri > mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1 > fio-3.24-5-g2ee2 > Starting 1 process > Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s] > > I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2 > Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s] That's not too surprising, if you're at queuedepth=1 or doing sync IO, then I'd consider it a major win that the async API is just as fast or faster than the sync one :-) -- Jens Axboe ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: io_uring on CentOS8? 2021-02-16 19:51 ` Jens Axboe @ 2021-02-17 8:42 ` Daniel Menzel 0 siblings, 0 replies; 6+ messages in thread From: Daniel Menzel @ 2021-02-17 8:42 UTC (permalink / raw) To: Jens Axboe, Ober, Frank, fio By the way: Does anyone do some system profiling (flame graphs or something like that) while testing devices to make sure you reached the devices' capabilities instead of the systems? I just made some tests of nvme drives and a friend of mine (who is a scientist) said that if that was a paper he would reject it because I do not know whether I measure the devices or the system. Am 16.02.21 um 20:51 schrieb Jens Axboe: > On 2/16/21 12:00 PM, Ober, Frank wrote: >> Hi Daniel, Intel uses pvsync2 for QD1 testing so if you cannot use spdk or io_uring (these are the fastest ways to test a device), you can switch your engine to pvsync2 on this kernel and this will work. >> >> Here's what I tried on PCIe Gen3 system with an Optane drive and your drive. >> >> >> [root@fm42adsdemo001 block]# fio --bs=1M --rw=randwrite --numjobs=1 --iodepth=1 --size=10000M \ >>> --filename=/dev/nvme6n1 --name=mylittletest --direct=1 --fsync=1 \ >>> --refill_buffers --ioengine=pvsync2 --group_reporting \ >>> --fallocate=none --runtime=60 --time_based --hipri >> mylittletest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=pvsync2, iodepth=1 >> fio-3.24-5-g2ee2 >> Starting 1 process >> Jobs: 1 (f=1): [w(1)][100.0%][w=2048MiB/s][w=2048 IOPS][eta 00m:00s] >> >> I ran the same exact job on this mid-range Intel Xeon cpu and found io_uring only very slightly faster than pvsync2 >> Jobs: 1 (f=1): [w(1)][100.0%][w=2057MiB/s][w=2057 IOPS][eta 00m:00s] > That's not too surprising, if you're at queuedepth=1 or doing sync IO, then > I'd consider it a major win that the async API is just as fast or faster > than the sync one :-) > > -- Daniel Menzel Geschäftsführer Menzel IT GmbH Charlottenburger Str. 33a 13086 Berlin +49 (0) 30 / 5130 444 - 00 daniel.menzel@menzel-it.net https://menzel-it.net Geschäftsführer: Daniel Menzel, Josefin Menzel Unternehmenssitz: Berlin Handelsregister: Amtsgericht Charlottenburg Handelsregister-Nummer: HRB 149835 B USt-ID: DE 309 226 751 ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-02-17 8:42 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-02-14 15:17 io_uring on CentOS8? Daniel Menzel 2021-02-14 16:49 ` Jens Axboe 2021-02-16 19:00 ` Ober, Frank 2021-02-16 19:46 ` Daniel Menzel 2021-02-16 19:51 ` Jens Axboe 2021-02-17 8:42 ` Daniel Menzel
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.