* fio on vmware esxi @ 2018-10-10 18:24 eliezer 2018-10-10 19:05 ` Beierl, Mark 0 siblings, 1 reply; 20+ messages in thread From: eliezer @ 2018-10-10 18:24 UTC (permalink / raw) To: fio Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any helpful guides as to how to install it on vmware? I've searched google but could not find any. The only reference to fio+vmware is Disable libaio for ESXi build - bug#80 Thank you in advance. Regards, Eliezer ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-10 18:24 fio on vmware esxi eliezer @ 2018-10-10 19:05 ` Beierl, Mark 2018-10-10 19:56 ` Eliezer Nebab 0 siblings, 1 reply; 20+ messages in thread From: Beierl, Mark @ 2018-10-10 19:05 UTC (permalink / raw) To: eliezer, fio Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. Regards, Mark Mark Beierl SW System Sr Principal Developer Dell EMC | Cloud & Communication Service Provider Solution Mark.Beierl@Dell.com On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: [EXTERNAL EMAIL] Please report any suspicious attachments, links, or requests for sensitive information. Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any helpful guides as to how to install it on vmware? I've searched google but could not find any. The only reference to fio+vmware is Disable libaio for ESXi build - bug#80 Thank you in advance. Regards, Eliezer ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-10 19:05 ` Beierl, Mark @ 2018-10-10 19:56 ` Eliezer Nebab 2018-10-10 19:59 ` Beierl, Mark 0 siblings, 1 reply; 20+ messages in thread From: Eliezer Nebab @ 2018-10-10 19:56 UTC (permalink / raw) To: Beierl, Mark; +Cc: fio Thanks for the quick response Mark. Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. From your response it looks like (compile or precompiled) neither works. Thank you. Eliezer > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? > > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any > helpful guides as to how to install it on vmware? I've searched google but > could not find any. The only reference to fio+vmware is Disable libaio for > ESXi build - bug#80 > > Thank you in advance. > > Regards, > Eliezer > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-10 19:56 ` Eliezer Nebab @ 2018-10-10 19:59 ` Beierl, Mark 2018-10-11 5:34 ` Sitsofe Wheeler 0 siblings, 1 reply; 20+ messages in thread From: Beierl, Mark @ 2018-10-10 19:59 UTC (permalink / raw) To: Eliezer Nebab; +Cc: fio Ya, sorry I could not be of more help. I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? Regards, Mark Mark Beierl SW System Sr Principal Developer Dell EMC | Cloud & Communication Service Provider Solution Mark.Beierl@Dell.com On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: [EXTERNAL EMAIL] Please report any suspicious attachments, links, or requests for sensitive information. Thanks for the quick response Mark. Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. From your response it looks like (compile or precompiled) neither works. Thank you. Eliezer > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? > > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any > helpful guides as to how to install it on vmware? I've searched google but > could not find any. The only reference to fio+vmware is Disable libaio for > ESXi build - bug#80 > > Thank you in advance. > > Regards, > Eliezer > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-10 19:59 ` Beierl, Mark @ 2018-10-11 5:34 ` Sitsofe Wheeler 2018-10-11 13:42 ` Beierl, Mark 0 siblings, 1 reply; 20+ messages in thread From: Sitsofe Wheeler @ 2018-10-11 5:34 UTC (permalink / raw) To: eliezer; +Cc: Mark.Beierl, fio Hi, The question of how to compile fio for ESXi comes up every few years and as Mark suggested it IS doable by building a static fio binary from an appropriate real distro but there are a few gotchas. You can't take a "regular" distro's fio because the ESXi environment simply doesn't match that of the regular distro (for example things like a glibc are compiled with different options, libraries are missing etc). The last time it was asked about was over on Github (see https://github.com/axboe/fio/issues/464 ) where I mentioned the steps I used to get it going for ESXi 6.0 but note that I no longer have access to ESXi these days so I can't say what's needed for modern environments (but the general idea should be the same). On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Ya, sorry I could not be of more help. > > I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. > > I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Thanks for the quick response Mark. > > Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. > > From your response it looks like (compile or precompiled) neither works. > > Thank you. > > Eliezer > > > > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > > > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? > > > > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. > > > > Regards, > > Mark > > > > Mark Beierl > > SW System Sr Principal Developer > > Dell EMC | Cloud & Communication Service Provider Solution > > Mark.Beierl@Dell.com > > > > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: > > > > > > [EXTERNAL EMAIL] > > Please report any suspicious attachments, links, or requests for sensitive information. > > > > > > Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any > > helpful guides as to how to install it on vmware? I've searched google but > > could not find any. The only reference to fio+vmware is Disable libaio for > > ESXi build - bug#80 > > > > Thank you in advance. > > > > Regards, > > Eliezer -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-11 5:34 ` Sitsofe Wheeler @ 2018-10-11 13:42 ` Beierl, Mark 2018-10-11 19:28 ` Eliezer Nebab ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: Beierl, Mark @ 2018-10-11 13:42 UTC (permalink / raw) To: Sitsofe Wheeler, eliezer; +Cc: fio Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464. I have fio working on my ESXi 6.5 server. Here is the procedure I used. Install Docker and use this Dockerfile: ============================= FROM multiarch/alpine:x86_64-v3.6 RUN apk --no-cache add --update \ git \ alpine-sdk \ coreutils \ linux-headers \ libaio \ libaio-dev \ zlib-dev RUN git config --global http.sslVerify false RUN git clone http://git.kernel.dk/fio.git fio RUN cd fio && ./configure --esx RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install ============================= Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey) $ docker run -d fio sleep 600 Note the ID that it spits out. Use that ID in the next command: $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio . You can now scp that fio to your ESXi server and execute it there. [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 fio-3.11-6-gee6ce Starting 1 thread file1: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s] .... Hope this works for you, Eliezer! Regards, Mark Mark Beierl SW System Sr Principal Developer Dell EMC | Cloud & Communication Service Provider Solution Mark.Beierl@Dell.com On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: [EXTERNAL EMAIL] Please report any suspicious attachments, links, or requests for sensitive information. Hi, The question of how to compile fio for ESXi comes up every few years and as Mark suggested it IS doable by building a static fio binary from an appropriate real distro but there are a few gotchas. You can't take a "regular" distro's fio because the ESXi environment simply doesn't match that of the regular distro (for example things like a glibc are compiled with different options, libraries are missing etc). The last time it was asked about was over on Github (see https://github.com/axboe/fio/issues/464 ) where I mentioned the steps I used to get it going for ESXi 6.0 but note that I no longer have access to ESXi these days so I can't say what's needed for modern environments (but the general idea should be the same). On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Ya, sorry I could not be of more help. > > I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. > > I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Thanks for the quick response Mark. > > Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. > > From your response it looks like (compile or precompiled) neither works. > > Thank you. > > Eliezer > > > > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > > > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? > > > > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. > > > > Regards, > > Mark > > > > Mark Beierl > > SW System Sr Principal Developer > > Dell EMC | Cloud & Communication Service Provider Solution > > Mark.Beierl@Dell.com > > > > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: > > > > > > [EXTERNAL EMAIL] > > Please report any suspicious attachments, links, or requests for sensitive information. > > > > > > Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any > > helpful guides as to how to install it on vmware? I've searched google but > > could not find any. The only reference to fio+vmware is Disable libaio for > > ESXi build - bug#80 > > > > Thank you in advance. > > > > Regards, > > Eliezer -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-11 13:42 ` Beierl, Mark @ 2018-10-11 19:28 ` Eliezer Nebab 2018-10-11 19:33 ` Beierl, Mark 2018-10-12 6:13 ` Sitsofe Wheeler [not found] ` <555423228.215547.1540355274473@email.1and1.com> 2 siblings, 1 reply; 20+ messages in thread From: Eliezer Nebab @ 2018-10-11 19:28 UTC (permalink / raw) To: Beierl, Mark; +Cc: Sitsofe Wheeler, fio Excellent update to this thread. Thanks Sitsofe and Mark! I’ll try this and hope to be able to run fio the way you outlined. I see youre also using latest version 3.11. I hope to update this thread with some good news. If I could get this working, it will address a lot of performance questions from admins that are on ESXi environments. Regards, Eliezer > On Oct 11, 2018, at 8:42 AM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464. > > I have fio working on my ESXi 6.5 server. Here is the procedure I used. Install Docker and use this Dockerfile: > > ============================= > FROM multiarch/alpine:x86_64-v3.6 > > RUN apk --no-cache add --update \ > git \ > alpine-sdk \ > coreutils \ > linux-headers \ > libaio \ > libaio-dev \ > zlib-dev > > RUN git config --global http.sslVerify false > RUN git clone http://git.kernel.dk/fio.git fio > RUN cd fio && ./configure --esx > RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install > ============================= > > Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey) > $ docker run -d fio sleep 600 > > Note the ID that it spits out. Use that ID in the next command: > $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio . > > You can now scp that fio to your ESXi server and execute it there. > > [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio > file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 > fio-3.11-6-gee6ce > Starting 1 thread > file1: Laying out IO file (1 file / 1024MiB) > Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s] > .... > > Hope this works for you, Eliezer! > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Hi, > > The question of how to compile fio for ESXi comes up every few years > and as Mark suggested it IS doable by building a static fio binary > from an appropriate real distro but there are a few gotchas. You can't > take a "regular" distro's fio because the ESXi environment simply > doesn't match that of the regular distro (for example things like a > glibc are compiled with different options, libraries are missing etc). > > The last time it was asked about was over on Github (see > https://github.com/axboe/fio/issues/464 ) where I mentioned the steps > I used to get it going for ESXi 6.0 but note that I no longer have > access to ESXi these days so I can't say what's needed for modern > environments (but the general idea should be the same). > >> On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote: >> >> Ya, sorry I could not be of more help. >> >> I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. >> >> I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? >> >> Regards, >> Mark >> >> Mark Beierl >> SW System Sr Principal Developer >> Dell EMC | Cloud & Communication Service Provider Solution >> Mark.Beierl@Dell.com >> >> On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: >> >> >> [EXTERNAL EMAIL] >> Please report any suspicious attachments, links, or requests for sensitive information. >> >> >> Thanks for the quick response Mark. >> >> Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. >> >> From your response it looks like (compile or precompiled) neither works. >> >> Thank you. >> >> Eliezer >> >> >>> On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: >>> >>> Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? >>> >>> I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. >>> >>> Regards, >>> Mark >>> >>> Mark Beierl >>> SW System Sr Principal Developer >>> Dell EMC | Cloud & Communication Service Provider Solution >>> Mark.Beierl@Dell.com >>> >>> On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: >>> >>> >>> [EXTERNAL EMAIL] >>> Please report any suspicious attachments, links, or requests for sensitive information. >>> >>> >>> Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any >>> helpful guides as to how to install it on vmware? I've searched google but >>> could not find any. The only reference to fio+vmware is Disable libaio for >>> ESXi build - bug#80 >>> >>> Thank you in advance. >>> >>> Regards, >>> Eliezer > > -- > Sitsofe | http://sucs.org/~sits/ > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-11 19:28 ` Eliezer Nebab @ 2018-10-11 19:33 ` Beierl, Mark 0 siblings, 0 replies; 20+ messages in thread From: Beierl, Mark @ 2018-10-11 19:33 UTC (permalink / raw) To: Eliezer Nebab; +Cc: Sitsofe Wheeler, fio You are correct, in this Dockerfile, I am building the latest code from git. Add the following line after the git clone if you want to use a specific version: RUN cd fio && git checkout tags/fio-3.11 Or whatever tagged version you think is best :) Regards, Mark Mark Beierl SW System Sr Principal Developer Dell EMC | Cloud & Communication Service Provider Solution Mark.Beierl@Dell.com On 2018-10-11, 15:28, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: [EXTERNAL EMAIL] Please report any suspicious attachments, links, or requests for sensitive information. Excellent update to this thread. Thanks Sitsofe and Mark! I’ll try this and hope to be able to run fio the way you outlined. I see youre also using latest version 3.11. I hope to update this thread with some good news. If I could get this working, it will address a lot of performance questions from admins that are on ESXi environments. Regards, Eliezer > On Oct 11, 2018, at 8:42 AM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464. > > I have fio working on my ESXi 6.5 server. Here is the procedure I used. Install Docker and use this Dockerfile: > > ============================= > FROM multiarch/alpine:x86_64-v3.6 > > RUN apk --no-cache add --update \ > git \ > alpine-sdk \ > coreutils \ > linux-headers \ > libaio \ > libaio-dev \ > zlib-dev > > RUN git config --global http.sslVerify false > RUN git clone http://git.kernel.dk/fio.git fio > RUN cd fio && ./configure --esx > RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install > ============================= > > Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey) > $ docker run -d fio sleep 600 > > Note the ID that it spits out. Use that ID in the next command: > $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio . > > You can now scp that fio to your ESXi server and execute it there. > > [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio > file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 > fio-3.11-6-gee6ce > Starting 1 thread > file1: Laying out IO file (1 file / 1024MiB) > Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s] > .... > > Hope this works for you, Eliezer! > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Hi, > > The question of how to compile fio for ESXi comes up every few years > and as Mark suggested it IS doable by building a static fio binary > from an appropriate real distro but there are a few gotchas. You can't > take a "regular" distro's fio because the ESXi environment simply > doesn't match that of the regular distro (for example things like a > glibc are compiled with different options, libraries are missing etc). > > The last time it was asked about was over on Github (see > https://github.com/axboe/fio/issues/464 ) where I mentioned the steps > I used to get it going for ESXi 6.0 but note that I no longer have > access to ESXi these days so I can't say what's needed for modern > environments (but the general idea should be the same). > >> On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote: >> >> Ya, sorry I could not be of more help. >> >> I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. >> >> I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? >> >> Regards, >> Mark >> >> Mark Beierl >> SW System Sr Principal Developer >> Dell EMC | Cloud & Communication Service Provider Solution >> Mark.Beierl@Dell.com >> >> On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: >> >> >> [EXTERNAL EMAIL] >> Please report any suspicious attachments, links, or requests for sensitive information. >> >> >> Thanks for the quick response Mark. >> >> Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. >> >> From your response it looks like (compile or precompiled) neither works. >> >> Thank you. >> >> Eliezer >> >> >>> On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: >>> >>> Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? >>> >>> I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. >>> >>> Regards, >>> Mark >>> >>> Mark Beierl >>> SW System Sr Principal Developer >>> Dell EMC | Cloud & Communication Service Provider Solution >>> Mark.Beierl@Dell.com >>> >>> On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: >>> >>> >>> [EXTERNAL EMAIL] >>> Please report any suspicious attachments, links, or requests for sensitive information. >>> >>> >>> Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any >>> helpful guides as to how to install it on vmware? I've searched google but >>> could not find any. The only reference to fio+vmware is Disable libaio for >>> ESXi build - bug#80 >>> >>> Thank you in advance. >>> >>> Regards, >>> Eliezer > > -- > Sitsofe | http://sucs.org/~sits/ > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-11 13:42 ` Beierl, Mark 2018-10-11 19:28 ` Eliezer Nebab @ 2018-10-12 6:13 ` Sitsofe Wheeler 2018-10-12 18:18 ` Beierl, Mark [not found] ` <555423228.215547.1540355274473@email.1and1.com> 2 siblings, 1 reply; 20+ messages in thread From: Sitsofe Wheeler @ 2018-10-12 6:13 UTC (permalink / raw) To: Mark.Beierl; +Cc: eliezer, fio Thank you for following up on this Mark - I'm sure others will find that information useful! Glancing at your steps: > RUN git config --global http.sslVerify false I'm not sure you need this as the next line isn't using https: > RUN git clone http://git.kernel.dk/fio.git fio I don't know if alpine builds a static version of libaio (e.g. there isn't a prepackaged *static* libaio on CentOS/RHEL) but if it does you may find you can nobble fio's configure script to allow its detection even though you're using the --esx option... On Thu, 11 Oct 2018 at 14:42, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464. > > I have fio working on my ESXi 6.5 server. Here is the procedure I used. Install Docker and use this Dockerfile: > > ============================= > FROM multiarch/alpine:x86_64-v3.6 > > RUN apk --no-cache add --update \ > git \ > alpine-sdk \ > coreutils \ > linux-headers \ > libaio \ > libaio-dev \ > zlib-dev > > RUN git config --global http.sslVerify false > RUN git clone http://git.kernel.dk/fio.git fio > RUN cd fio && ./configure --esx > RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install > ============================= > > Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey) > $ docker run -d fio sleep 600 > > Note the ID that it spits out. Use that ID in the next command: > $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio . > > You can now scp that fio to your ESXi server and execute it there. > > [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio > file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 > fio-3.11-6-gee6ce > Starting 1 thread > file1: Laying out IO file (1 file / 1024MiB) > Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s] > .... > > Hope this works for you, Eliezer! > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Hi, > > The question of how to compile fio for ESXi comes up every few years > and as Mark suggested it IS doable by building a static fio binary > from an appropriate real distro but there are a few gotchas. You can't > take a "regular" distro's fio because the ESXi environment simply > doesn't match that of the regular distro (for example things like a > glibc are compiled with different options, libraries are missing etc). > > The last time it was asked about was over on Github (see > https://github.com/axboe/fio/issues/464 ) where I mentioned the steps > I used to get it going for ESXi 6.0 but note that I no longer have > access to ESXi these days so I can't say what's needed for modern > environments (but the general idea should be the same). > > On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > > > Ya, sorry I could not be of more help. > > > > I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. > > > > I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? > > > > Regards, > > Mark > > > > Mark Beierl > > SW System Sr Principal Developer > > Dell EMC | Cloud & Communication Service Provider Solution > > Mark.Beierl@Dell.com > > > > On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: > > > > > > [EXTERNAL EMAIL] > > Please report any suspicious attachments, links, or requests for sensitive information. > > > > > > Thanks for the quick response Mark. > > > > Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. > > > > From your response it looks like (compile or precompiled) neither works. > > > > Thank you. > > > > Eliezer > > > > > > > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > > > > > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? > > > > > > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. > > > > > > Regards, > > > Mark > > > > > > Mark Beierl > > > SW System Sr Principal Developer > > > Dell EMC | Cloud & Communication Service Provider Solution > > > Mark.Beierl@Dell.com > > > > > > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: > > > > > > > > > [EXTERNAL EMAIL] > > > Please report any suspicious attachments, links, or requests for sensitive information. > > > > > > > > > Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any > > > helpful guides as to how to install it on vmware? I've searched google but > > > could not find any. The only reference to fio+vmware is Disable libaio for > > > ESXi build - bug#80 > > > > > > Thank you in advance. > > > > > > Regards, > > > Eliezer > > -- > Sitsofe | http://sucs.org/~sits/ > > -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-12 6:13 ` Sitsofe Wheeler @ 2018-10-12 18:18 ` Beierl, Mark 2018-10-12 18:50 ` Eliezer Nebab 0 siblings, 1 reply; 20+ messages in thread From: Beierl, Mark @ 2018-10-12 18:18 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: eliezer, fio Oops, thanks Sitsofe. I over-enthusiatically copied that from another docker file where I was also doing https requests. You are right, it has no purpose in this case. Hopefully I can look at the libaio with --esx option next week. Regards, Mark Mark Beierl SW System Sr Principal Developer Dell EMC | Cloud & Communication Service Provider Solution Mark.Beierl@Dell.com On 2018-10-12, 02:13, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: [EXTERNAL EMAIL] Please report any suspicious attachments, links, or requests for sensitive information. Thank you for following up on this Mark - I'm sure others will find that information useful! Glancing at your steps: > RUN git config --global http.sslVerify false I'm not sure you need this as the next line isn't using https: > RUN git clone http://git.kernel.dk/fio.git fio I don't know if alpine builds a static version of libaio (e.g. there isn't a prepackaged *static* libaio on CentOS/RHEL) but if it does you may find you can nobble fio's configure script to allow its detection even though you're using the --esx option... On Thu, 11 Oct 2018 at 14:42, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464. > > I have fio working on my ESXi 6.5 server. Here is the procedure I used. Install Docker and use this Dockerfile: > > ============================= > FROM multiarch/alpine:x86_64-v3.6 > > RUN apk --no-cache add --update \ > git \ > alpine-sdk \ > coreutils \ > linux-headers \ > libaio \ > libaio-dev \ > zlib-dev > > RUN git config --global http.sslVerify false > RUN git clone http://git.kernel.dk/fio.git fio > RUN cd fio && ./configure --esx > RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install > ============================= > > Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey) > $ docker run -d fio sleep 600 > > Note the ID that it spits out. Use that ID in the next command: > $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio . > > You can now scp that fio to your ESXi server and execute it there. > > [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio > file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 > fio-3.11-6-gee6ce > Starting 1 thread > file1: Laying out IO file (1 file / 1024MiB) > Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s] > .... > > Hope this works for you, Eliezer! > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Hi, > > The question of how to compile fio for ESXi comes up every few years > and as Mark suggested it IS doable by building a static fio binary > from an appropriate real distro but there are a few gotchas. You can't > take a "regular" distro's fio because the ESXi environment simply > doesn't match that of the regular distro (for example things like a > glibc are compiled with different options, libraries are missing etc). > > The last time it was asked about was over on Github (see > https://github.com/axboe/fio/issues/464 ) where I mentioned the steps > I used to get it going for ESXi 6.0 but note that I no longer have > access to ESXi these days so I can't say what's needed for modern > environments (but the general idea should be the same). > > On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > > > Ya, sorry I could not be of more help. > > > > I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. > > > > I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? > > > > Regards, > > Mark > > > > Mark Beierl > > SW System Sr Principal Developer > > Dell EMC | Cloud & Communication Service Provider Solution > > Mark.Beierl@Dell.com > > > > On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: > > > > > > [EXTERNAL EMAIL] > > Please report any suspicious attachments, links, or requests for sensitive information. > > > > > > Thanks for the quick response Mark. > > > > Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. > > > > From your response it looks like (compile or precompiled) neither works. > > > > Thank you. > > > > Eliezer > > > > > > > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > > > > > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? > > > > > > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. > > > > > > Regards, > > > Mark > > > > > > Mark Beierl > > > SW System Sr Principal Developer > > > Dell EMC | Cloud & Communication Service Provider Solution > > > Mark.Beierl@Dell.com > > > > > > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: > > > > > > > > > [EXTERNAL EMAIL] > > > Please report any suspicious attachments, links, or requests for sensitive information. > > > > > > > > > Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any > > > helpful guides as to how to install it on vmware? I've searched google but > > > could not find any. The only reference to fio+vmware is Disable libaio for > > > ESXi build - bug#80 > > > > > > Thank you in advance. > > > > > > Regards, > > > Eliezer > > -- > Sitsofe | http://sucs.org/~sits/ > > -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-12 18:18 ` Beierl, Mark @ 2018-10-12 18:50 ` Eliezer Nebab 0 siblings, 0 replies; 20+ messages in thread From: Eliezer Nebab @ 2018-10-12 18:50 UTC (permalink / raw) To: Beierl, Mark; +Cc: Sitsofe Wheeler, fio I dont blame ya for the over enthusiasm Mark. I’m just as excited and enthused when I got your reply thay you got fio to work on vmware! Ive yet to try this out, hoping this weekend to spend some quality time on this and will report back. Eliezer > On Oct 12, 2018, at 1:18 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Oops, thanks Sitsofe. > > I over-enthusiatically copied that from another docker file where I was also doing https requests. You are right, it has no purpose in this case. > > Hopefully I can look at the libaio with --esx option next week. > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-12, 02:13, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > Thank you for following up on this Mark - I'm sure others will find > that information useful! > > Glancing at your steps: > >> RUN git config --global http.sslVerify false > > I'm not sure you need this as the next line isn't using https: > >> RUN git clone http://git.kernel.dk/fio.git fio > > I don't know if alpine builds a static version of libaio (e.g. there > isn't a prepackaged *static* libaio on CentOS/RHEL) but if it does you > may find you can nobble fio's configure script to allow its detection > even though you're using the --esx option... > >> On Thu, 11 Oct 2018 at 14:42, Beierl, Mark <Mark.Beierl@dell.com> wrote: >> >> Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464. >> >> I have fio working on my ESXi 6.5 server. Here is the procedure I used. Install Docker and use this Dockerfile: >> >> ============================= >> FROM multiarch/alpine:x86_64-v3.6 >> >> RUN apk --no-cache add --update \ >> git \ >> alpine-sdk \ >> coreutils \ >> linux-headers \ >> libaio \ >> libaio-dev \ >> zlib-dev >> >> RUN git config --global http.sslVerify false >> RUN git clone http://git.kernel.dk/fio.git fio >> RUN cd fio && ./configure --esx >> RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install >> ============================= >> >> Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey) >> $ docker run -d fio sleep 600 >> >> Note the ID that it spits out. Use that ID in the next command: >> $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio . >> >> You can now scp that fio to your ESXi server and execute it there. >> >> [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio >> file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 >> fio-3.11-6-gee6ce >> Starting 1 thread >> file1: Laying out IO file (1 file / 1024MiB) >> Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s] >> .... >> >> Hope this works for you, Eliezer! >> >> Regards, >> Mark >> >> Mark Beierl >> SW System Sr Principal Developer >> Dell EMC | Cloud & Communication Service Provider Solution >> Mark.Beierl@Dell.com >> >> On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote: >> >> >> [EXTERNAL EMAIL] >> Please report any suspicious attachments, links, or requests for sensitive information. >> >> >> Hi, >> >> The question of how to compile fio for ESXi comes up every few years >> and as Mark suggested it IS doable by building a static fio binary >> from an appropriate real distro but there are a few gotchas. You can't >> take a "regular" distro's fio because the ESXi environment simply >> doesn't match that of the regular distro (for example things like a >> glibc are compiled with different options, libraries are missing etc). >> >> The last time it was asked about was over on Github (see >> https://github.com/axboe/fio/issues/464 ) where I mentioned the steps >> I used to get it going for ESXi 6.0 but note that I no longer have >> access to ESXi these days so I can't say what's needed for modern >> environments (but the general idea should be the same). >> >>> On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote: >>> >>> Ya, sorry I could not be of more help. >>> >>> I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell. A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job. >>> >>> I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it. So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful. Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage? >>> >>> Regards, >>> Mark >>> >>> Mark Beierl >>> SW System Sr Principal Developer >>> Dell EMC | Cloud & Communication Service Provider Solution >>> Mark.Beierl@Dell.com >>> >>> On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote: >>> >>> >>> [EXTERNAL EMAIL] >>> Please report any suspicious attachments, links, or requests for sensitive information. >>> >>> >>> Thanks for the quick response Mark. >>> >>> Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers. >>> >>> From your response it looks like (compile or precompiled) neither works. >>> >>> Thank you. >>> >>> Eliezer >>> >>> >>>> On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote: >>>> >>>> Are you asking if FIO can be built or run on the hypervisor itself, like via ssh? >>>> >>>> I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start. Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes. So, I’m not sure it can be done. ESXi is not a general purpose OS, so it lacks some features. >>>> >>>> Regards, >>>> Mark >>>> >>>> Mark Beierl >>>> SW System Sr Principal Developer >>>> Dell EMC | Cloud & Communication Service Provider Solution >>>> Mark.Beierl@Dell.com >>>> >>>> On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: >>>> >>>> >>>> [EXTERNAL EMAIL] >>>> Please report any suspicious attachments, links, or requests for sensitive information. >>>> >>>> >>>> Is fio supported on vmware esxi 5 and 6 servers? Does anyone have any >>>> helpful guides as to how to install it on vmware? I've searched google but >>>> could not find any. The only reference to fio+vmware is Disable libaio for >>>> ESXi build - bug#80 >>>> >>>> Thank you in advance. >>>> >>>> Regards, >>>> Eliezer >> >> -- >> Sitsofe | http://sucs.org/~sits/ >> >> > > > -- > Sitsofe | http://sucs.org/~sits/ > > ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <555423228.215547.1540355274473@email.1and1.com>]
* Re: fio on vmware esxi [not found] ` <555423228.215547.1540355274473@email.1and1.com> @ 2018-10-24 5:05 ` Sitsofe Wheeler 2018-10-26 14:53 ` Eliezer Nebab 0 siblings, 1 reply; 20+ messages in thread From: Sitsofe Wheeler @ 2018-10-24 5:05 UTC (permalink / raw) To: eliezer; +Cc: Mark Beierl, fio lHi, If you don't need asynchrony pvsync2 tends to be good. If you do I think the only choice from the engines listed is posixaio... Regarding libaio, see my earlier reply to Mark... PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about . On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab <eliezer@integritech.solutions> wrote: > > Mark, > > > Finally had a chance to try this out this weekend. > > > VMWare version: > > # vmware -v > VMware ESXi 6.5.0 build-8294253 > > > I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described. > > I then copied the ID: > > # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio . > # ls -l > total 5832 > -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile > -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio > > > I scp'd the "fio" file to my esxi server. > > To my excitement, tried to run "fio" with the libaio which did not work. The only engines available out of that fio build are as follows: > > /tmp/fio --enghelp > > > Available IO engines: > cpuio > mmap > sync > psync > vsync > pvsync > pvsync2 > null > net > netsplice > ftruncate > filecreate > posixaio > falloc > e4defrag > splice > mtd > sg > > Otherwise, fio is working just as you described. Thank you very much Mark!!! > > So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing? -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-24 5:05 ` Sitsofe Wheeler @ 2018-10-26 14:53 ` Eliezer Nebab 2018-10-26 14:54 ` Beierl, Mark 0 siblings, 1 reply; 20+ messages in thread From: Eliezer Nebab @ 2018-10-26 14:53 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: Mark Beierl, fio posixaio engine worked! Thank you very much Sitsofe and Mark im good to go! > On Oct 24, 2018, at 12:05 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote: > > lHi, > > If you don't need asynchrony pvsync2 tends to be good. If you do I > think the only choice from the engines listed is posixaio... > > Regarding libaio, see my earlier reply to Mark... > > PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about . > > On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab > <eliezer@integritech.solutions> wrote: >> >> Mark, >> >> >> Finally had a chance to try this out this weekend. >> >> >> VMWare version: >> >> # vmware -v >> VMware ESXi 6.5.0 build-8294253 >> >> >> I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described. >> >> I then copied the ID: >> >> # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio . >> # ls -l >> total 5832 >> -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile >> -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio >> >> >> I scp'd the "fio" file to my esxi server. >> >> To my excitement, tried to run "fio" with the libaio which did not work. The only engines available out of that fio build are as follows: >> >> /tmp/fio --enghelp >> >> >> Available IO engines: >> cpuio >> mmap >> sync >> psync >> vsync >> pvsync >> pvsync2 >> null >> net >> netsplice >> ftruncate >> filecreate >> posixaio >> falloc >> e4defrag >> splice >> mtd >> sg >> >> Otherwise, fio is working just as you described. Thank you very much Mark!!! >> >> So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing? > > -- > Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-26 14:53 ` Eliezer Nebab @ 2018-10-26 14:54 ` Beierl, Mark 2018-10-26 17:10 ` Eliezer Nebab 0 siblings, 1 reply; 20+ messages in thread From: Beierl, Mark @ 2018-10-26 14:54 UTC (permalink / raw) To: Eliezer Nebab, Sitsofe Wheeler; +Cc: fio Can you clarify – did you need to rebuild FIO or were you just able to run it with posixaio from the Dockerfile example? Regards, Mark Mark Beierl SW System Sr Principal Developer Dell EMC | Cloud & Communication Service Provider Solution Mark.Beierl@Dell.com On 2018-10-26, 10:53, "fio-owner@vger.kernel.org on behalf of Eliezer Nebab" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: [EXTERNAL EMAIL] Please report any suspicious attachments, links, or requests for sensitive information. posixaio engine worked! Thank you very much Sitsofe and Mark im good to go! > On Oct 24, 2018, at 12:05 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote: > > lHi, > > If you don't need asynchrony pvsync2 tends to be good. If you do I > think the only choice from the engines listed is posixaio... > > Regarding libaio, see my earlier reply to Mark... > > PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about . > > On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab > <eliezer@integritech.solutions> wrote: >> >> Mark, >> >> >> Finally had a chance to try this out this weekend. >> >> >> VMWare version: >> >> # vmware -v >> VMware ESXi 6.5.0 build-8294253 >> >> >> I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described. >> >> I then copied the ID: >> >> # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio . >> # ls -l >> total 5832 >> -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile >> -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio >> >> >> I scp'd the "fio" file to my esxi server. >> >> To my excitement, tried to run "fio" with the libaio which did not work. The only engines available out of that fio build are as follows: >> >> /tmp/fio --enghelp >> >> >> Available IO engines: >> cpuio >> mmap >> sync >> psync >> vsync >> pvsync >> pvsync2 >> null >> net >> netsplice >> ftruncate >> filecreate >> posixaio >> falloc >> e4defrag >> splice >> mtd >> sg >> >> Otherwise, fio is working just as you described. Thank you very much Mark!!! >> >> So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing? > > -- > Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-26 14:54 ` Beierl, Mark @ 2018-10-26 17:10 ` Eliezer Nebab 2018-10-26 21:59 ` eliezer 0 siblings, 1 reply; 20+ messages in thread From: Eliezer Nebab @ 2018-10-26 17:10 UTC (permalink / raw) To: Beierl, Mark; +Cc: Sitsofe Wheeler, fio Sorry, let me clarify. I was able to run the posixaio engine from your Docker file example. Thank you. Eliezer > On Oct 26, 2018, at 9:54 AM, Beierl, Mark <Mark.Beierl@dell.com> wrote: > > Can you clarify – did you need to rebuild FIO or were you just able to run it with posixaio from the Dockerfile example? > > Regards, > Mark > > Mark Beierl > SW System Sr Principal Developer > Dell EMC | Cloud & Communication Service Provider Solution > Mark.Beierl@Dell.com > > On 2018-10-26, 10:53, "fio-owner@vger.kernel.org on behalf of Eliezer Nebab" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > posixaio engine worked! > > Thank you very much Sitsofe and Mark im good to go! > > > >> On Oct 24, 2018, at 12:05 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote: >> >> lHi, >> >> If you don't need asynchrony pvsync2 tends to be good. If you do I >> think the only choice from the engines listed is posixaio... >> >> Regarding libaio, see my earlier reply to Mark... >> >> PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about . >> >> On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab >> <eliezer@integritech.solutions> wrote: >>> >>> Mark, >>> >>> >>> Finally had a chance to try this out this weekend. >>> >>> >>> VMWare version: >>> >>> # vmware -v >>> VMware ESXi 6.5.0 build-8294253 >>> >>> >>> I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described. >>> >>> I then copied the ID: >>> >>> # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio . >>> # ls -l >>> total 5832 >>> -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile >>> -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio >>> >>> >>> I scp'd the "fio" file to my esxi server. >>> >>> To my excitement, tried to run "fio" with the libaio which did not work. The only engines available out of that fio build are as follows: >>> >>> /tmp/fio --enghelp >>> >>> >>> Available IO engines: >>> cpuio >>> mmap >>> sync >>> psync >>> vsync >>> pvsync >>> pvsync2 >>> null >>> net >>> netsplice >>> ftruncate >>> filecreate >>> posixaio >>> falloc >>> e4defrag >>> splice >>> mtd >>> sg >>> >>> Otherwise, fio is working just as you described. Thank you very much Mark!!! >>> >>> So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing? >> >> -- >> Sitsofe | http://sucs.org/~sits/ > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* RE: fio on vmware esxi 2018-10-26 17:10 ` Eliezer Nebab @ 2018-10-26 21:59 ` eliezer 2018-10-26 22:30 ` Elliott, Robert (Persistent Memory) 0 siblings, 1 reply; 20+ messages in thread From: eliezer @ 2018-10-26 21:59 UTC (permalink / raw) To: 'Beierl, Mark'; +Cc: 'Sitsofe Wheeler', 'fio' Mark, Sitsofe, So I've been testing FIO on VMware using the different ioengines below and found something interesting. -psync -pvsync -sync -posixaio The OS is a VMware ESXi: VMware ESXi 6.5.0 build-8294253 FIO version compiled via Docker (thanks to Mark and Sitsofe): fio-3.11-22-g425d I mounted 3 storage devices (2 are different block storage types) on the same baremetal ESXi server: -500 IOPS NFS -1000 IOPS Reliant Tier iSCSI Block Storage -1000 IOPS Perf Tier iSCSI Block Storage I run FIO with the following parameters and used the different engines I listed above: --randrepeat=1 --ioengine=posixaio --direct=1 --sync=1 --name=mytest --filename=mytestfile.fio --overwrite=1 --iodepth=64 --size=100MB --readwrite=randrw --rwmixread=50 --rwmixwrite=50 --bs=16k Here are the results: NFS::: -ioengine=psync on a 500 IOPS NFS storage::: read=245 write=256 IOPS* -ioengine=pvsync on a 500 IOPS NFS Storage:::: read=252 write=263 IOPS* -ioengine=sync on a 500 IOPS NFS Storage:::: read=258 write=270 IOPS* -ioengine=posixaio on a 500 IOPS NFS Storage:::: read=172 write=180 IOPS BLOCK STORAGE::: 1000 IOPS Reliant Tier -ioengine=psync on a 1000 IOPS Reliant storage::: read=249 write=260 IOPS -ioengine=pvsync on a 1000 IOPS Reliant storage::: read=321 write=335 IOPS -ioengine=sync on a 1000 IOPS Reliant storage::: read=236 write=246 IOPS -ioengine=posixaio on a 1000 IOPS Reliant storage::: read=535 write=559 IOPS* 1000 IOPS Perf Tier -ioengine=psync on a 1000 IOPS Perf storage::: read=177 write=185 IOPS -ioengine=pvsync on a 1000 IOPS Perf storage::: read=176 write=184 IOPS -ioengine=sync on a 1000 IOPS Perf storage::: read=145 write=152 IOPS -ioengine=posixaio on a 1000 IOPS Perf storage::: read=528 write=551 IOPS* Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage is "posixaio"; and for NFS it can be psync or pvsync or sync. If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS. However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS. I'd have to use either psync, or pvsync, or sync to see the IOPS I'm expecting. These tests are on the same baremetal server running the version of VMWare I posted above. Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test an NFS storage? The same goes with ioengines psync/pvsync/sync skews IOPS results on a block storage device? Thanks in advance. Eliezer ^ permalink raw reply [flat|nested] 20+ messages in thread
* RE: fio on vmware esxi 2018-10-26 21:59 ` eliezer @ 2018-10-26 22:30 ` Elliott, Robert (Persistent Memory) 2018-10-26 22:44 ` eliezer ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: Elliott, Robert (Persistent Memory) @ 2018-10-26 22:30 UTC (permalink / raw) To: 'eliezer@integritech.solutions', 'Beierl, Mark' Cc: 'Sitsofe Wheeler', 'fio' > I run FIO with the following parameters and used the different engines I listed above: > --randrepeat=1 > --ioengine=posixaio > --direct=1 > --sync=1 > --name=mytest > --filename=mytestfile.fio > --overwrite=1 > --iodepth=64 > --size=100MB > --readwrite=randrw > --rwmixread=50 > --rwmixwrite=50 > --bs=16k > ... > -ioengine=psync on a 1000 IOPS Perf storage::: read=177 write=185 IOPS > -ioengine=pvsync on a 1000 IOPS Perf storage::: read=176 write=184 IOPS > -ioengine=sync on a 1000 IOPS Perf storage::: read=145 write=152 IOPS > -ioengine=posixaio on a 1000 IOPS Perf storage::: read=528 write=551 IOPS* > > Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage > is "posixaio"; and for NFS it can be psync or pvsync or sync. > If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS. > However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS. I'd > have to use either psync, or pvsync, or sync to see the IOPS I'm expecting. > These tests are on the same baremetal server running the version of VMWare I posted above. > > Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test > an NFS storage? The same goes with ioengines psync/pvsync/sync skews IOPS results on a block > storage device? NFS might introduce all sorts of problems. For the block device cases... Synchronous engines like sync do not honor the iodepth; you need to use jobs= to get multiple concurrent IOs. posixaio is an asynchronous engine, so it is honoring your iodepth (it spawns threads for you). Also, if your storage devices do any sort of buffering and caching, the tiny 100 MiB size is likely to result in lots of cache hits, distorting the results. --- Robert Elliott, HPE Persistent Memory ^ permalink raw reply [flat|nested] 20+ messages in thread
* RE: fio on vmware esxi 2018-10-26 22:30 ` Elliott, Robert (Persistent Memory) @ 2018-10-26 22:44 ` eliezer 2018-10-26 22:52 ` Beierl, Mark 2018-10-27 20:34 ` Sitsofe Wheeler 2 siblings, 0 replies; 20+ messages in thread From: eliezer @ 2018-10-26 22:44 UTC (permalink / raw) To: 'Elliott, Robert (Persistent Memory)', 'Beierl, Mark' Cc: 'Sitsofe Wheeler', 'fio' Thanks for the quick feedback Robert. I appreciate it. I forgot to add, when I created a new NFS datastore in VMware ESXi 6.5 I have to NFS versions to choose from: -NFS 3 allows the datastore to be accessed by ESX/ESXi hosts of version earlier than 6.0 -NFS 4.1 provides multipathing for servers and supports the Kerberos authentication protocol The results FIO results I posted were from NFS v.3. so I continued testing... I unmounted that NFS volume and remounted it as a new datastore using NFS v4.1 and I'm now showing the expected IOPS using the "posixaio" engine on NFS! Great! So, use "posixaio" FIO engine on NFS v 4.1; don’t use "posixaio" engine on NFS v 3. Eliezer -----Original Message----- From: Elliott, Robert (Persistent Memory) [mailto:elliott@hpe.com] Sent: Friday, October 26, 2018 5:30 PM To: 'eliezer@integritech.solutions'; 'Beierl, Mark' Cc: 'Sitsofe Wheeler'; 'fio' Subject: RE: fio on vmware esxi > I run FIO with the following parameters and used the different engines I listed above: > --randrepeat=1 > --ioengine=posixaio > --direct=1 > --sync=1 > --name=mytest > --filename=mytestfile.fio > --overwrite=1 > --iodepth=64 > --size=100MB > --readwrite=randrw > --rwmixread=50 > --rwmixwrite=50 > --bs=16k > ... > -ioengine=psync on a 1000 IOPS Perf storage::: read=177 write=185 IOPS > -ioengine=pvsync on a 1000 IOPS Perf storage::: read=176 write=184 IOPS > -ioengine=sync on a 1000 IOPS Perf storage::: read=145 write=152 IOPS > -ioengine=posixaio on a 1000 IOPS Perf storage::: read=528 write=551 IOPS* > > Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage > is "posixaio"; and for NFS it can be psync or pvsync or sync. > If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS. > However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS. I'd > have to use either psync, or pvsync, or sync to see the IOPS I'm expecting. > These tests are on the same baremetal server running the version of VMWare I posted above. > > Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test > an NFS storage? The same goes with ioengines psync/pvsync/sync skews IOPS results on a block > storage device? NFS might introduce all sorts of problems. For the block device cases... Synchronous engines like sync do not honor the iodepth; you need to use jobs= to get multiple concurrent IOs. posixaio is an asynchronous engine, so it is honoring your iodepth (it spawns threads for you). Also, if your storage devices do any sort of buffering and caching, the tiny 100 MiB size is likely to result in lots of cache hits, distorting the results. --- Robert Elliott, HPE Persistent Memory ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-26 22:30 ` Elliott, Robert (Persistent Memory) 2018-10-26 22:44 ` eliezer @ 2018-10-26 22:52 ` Beierl, Mark 2018-10-27 20:34 ` Sitsofe Wheeler 2 siblings, 0 replies; 20+ messages in thread From: Beierl, Mark @ 2018-10-26 22:52 UTC (permalink / raw) To: Elliott, Robert (Persistent Memory); +Cc: eliezer, Sitsofe Wheeler, fio > On Oct 26, 2018, at 18:30, Elliott, Robert (Persistent Memory) <elliott@hpe.com> wrote: > > > [EXTERNAL EMAIL] > Please report any suspicious attachments, links, or requests for sensitive information. > > > >> I run FIO with the following parameters and used the different engines I listed above: >> --randrepeat=1 >> --ioengine=posixaio >> --direct=1 >> --sync=1 >> --name=mytest >> --filename=mytestfile.fio >> --overwrite=1 >> --iodepth=64 >> --size=100MB >> --readwrite=randrw >> --rwmixread=50 >> --rwmixwrite=50 >> --bs=16k >> > ... >> -ioengine=psync on a 1000 IOPS Perf storage::: read=177 write=185 IOPS >> -ioengine=pvsync on a 1000 IOPS Perf storage::: read=176 write=184 IOPS >> -ioengine=sync on a 1000 IOPS Perf storage::: read=145 write=152 IOPS >> -ioengine=posixaio on a 1000 IOPS Perf storage::: read=528 write=551 IOPS* >> >> Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage >> is "posixaio"; and for NFS it can be psync or pvsync or sync. >> If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS. >> However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS. I'd >> have to use either psync, or pvsync, or sync to see the IOPS I'm expecting. >> These tests are on the same baremetal server running the version of VMWare I posted above. >> >> Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test >> an NFS storage? The same goes with ioengines psync/pvsync/sync skews IOPS results on a block >> storage device? > > NFS might introduce all sorts of problems. For the block device cases... > > Synchronous engines like sync do not honor the iodepth; you need to use > jobs= to get multiple concurrent IOs. posixaio is an asynchronous engine, > so it is honoring your iodepth (it spawns threads for you). > > Also, if your storage devices do any sort of buffering and caching, the > tiny 100 MiB size is likely to result in lots of cache hits, distorting > the results. > > --- > Robert Elliott, HPE Persistent Memory > My typical rule of thumb is to make your test data size be around 5x the total size of all cache available. This can become very large, btw, once you factor in SDS like vSAN. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: fio on vmware esxi 2018-10-26 22:30 ` Elliott, Robert (Persistent Memory) 2018-10-26 22:44 ` eliezer 2018-10-26 22:52 ` Beierl, Mark @ 2018-10-27 20:34 ` Sitsofe Wheeler 2 siblings, 0 replies; 20+ messages in thread From: Sitsofe Wheeler @ 2018-10-27 20:34 UTC (permalink / raw) To: Elliott, Robert (Persistent Memory); +Cc: eliezer, Mark Beierl, fio On Fri, 26 Oct 2018 at 23:30, Elliott, Robert (Persistent Memory) <elliott@hpe.com> wrote: > > > > I run FIO with the following parameters and used the different engines I listed above: > > --randrepeat=1 > > --ioengine=posixaio > > --direct=1 > > --sync=1 > > --name=mytest > > --filename=mytestfile.fio > > --overwrite=1 > > --iodepth=64 > > --size=100MB > > --readwrite=randrw > > --rwmixread=50 > > --rwmixwrite=50 > > --bs=16k > > > ... > > -ioengine=psync on a 1000 IOPS Perf storage::: read=177 write=185 IOPS > > -ioengine=pvsync on a 1000 IOPS Perf storage::: read=176 write=184 IOPS > > -ioengine=sync on a 1000 IOPS Perf storage::: read=145 write=152 IOPS > > -ioengine=posixaio on a 1000 IOPS Perf storage::: read=528 write=551 IOPS* > > > > Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage > > is "posixaio"; and for NFS it can be psync or pvsync or sync. > > If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS. > > However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS. I'd > > have to use either psync, or pvsync, or sync to see the IOPS I'm expecting. > > These tests are on the same baremetal server running the version of VMWare I posted above. > > > > Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test > > an NFS storage? The same goes with ioengines psync/pvsync/sync skews IOPS results on a block > > storage device? > > NFS might introduce all sorts of problems. For the block device cases... > > Synchronous engines like sync do not honor the iodepth; you need to use > jobs= to get multiple concurrent IOs. posixaio is an asynchronous engine, > so it is honoring your iodepth (it spawns threads for you). > > Also, if your storage devices do any sort of buffering and caching, the > tiny 100 MiB size is likely to result in lots of cache hits, distorting > the results. On the NFS front, perhaps you requesting direct=1 the kernel is actually using synchronous buffered I/O behind the scenes and all those syncs are slowing it down? Does using sync=0 show less of a difference between engines? -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2018-10-28 5:16 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-10 18:24 fio on vmware esxi eliezer
2018-10-10 19:05 ` Beierl, Mark
2018-10-10 19:56 ` Eliezer Nebab
2018-10-10 19:59 ` Beierl, Mark
2018-10-11 5:34 ` Sitsofe Wheeler
2018-10-11 13:42 ` Beierl, Mark
2018-10-11 19:28 ` Eliezer Nebab
2018-10-11 19:33 ` Beierl, Mark
2018-10-12 6:13 ` Sitsofe Wheeler
2018-10-12 18:18 ` Beierl, Mark
2018-10-12 18:50 ` Eliezer Nebab
[not found] ` <555423228.215547.1540355274473@email.1and1.com>
2018-10-24 5:05 ` Sitsofe Wheeler
2018-10-26 14:53 ` Eliezer Nebab
2018-10-26 14:54 ` Beierl, Mark
2018-10-26 17:10 ` Eliezer Nebab
2018-10-26 21:59 ` eliezer
2018-10-26 22:30 ` Elliott, Robert (Persistent Memory)
2018-10-26 22:44 ` eliezer
2018-10-26 22:52 ` Beierl, Mark
2018-10-27 20:34 ` Sitsofe Wheeler
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.