fio on vmware esxi

fio on vmware esxi

[eliezer]

Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
helpful guides as to how to install it on vmware?  I've searched google but
could not find any.  The only reference to fio+vmware is Disable libaio for
ESXi build - bug#80

Thank you in advance.

Regards,
Eliezer

Re: fio on vmware esxi

[Beierl, Mark]

Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?

I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.

Regards,
Mark
 
Mark Beierl
SW System Sr Principal Developer
Dell EMC | Cloud & Communication Service Provider Solution
Mark.Beierl@Dell.com

On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:

    
    [EXTERNAL EMAIL] 
    Please report any suspicious attachments, links, or requests for sensitive information.
    
    
    Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
    helpful guides as to how to install it on vmware?  I've searched google but
    could not find any.  The only reference to fio+vmware is Disable libaio for
    ESXi build - bug#80
    
    Thank you in advance.
    
    Regards,
    Eliezer
    
    

Re: fio on vmware esxi

[Eliezer Nebab]

Thanks for the quick response Mark.

Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.  
 
From your response it looks like (compile or precompiled) neither works.

Thank you.

Eliezer


> On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
> 
> Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
> 
> I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
> 
> Regards,
> Mark
> 
> Mark Beierl
> SW System Sr Principal Developer
> Dell EMC | Cloud & Communication Service Provider Solution
> Mark.Beierl@Dell.com
> 
> On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
> 
> 
>    [EXTERNAL EMAIL] 
>    Please report any suspicious attachments, links, or requests for sensitive information.
> 
> 
>    Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
>    helpful guides as to how to install it on vmware?  I've searched google but
>    could not find any.  The only reference to fio+vmware is Disable libaio for
>    ESXi build - bug#80
> 
>    Thank you in advance.
> 
>    Regards,
>    Eliezer
> 
> 
> 

Re: fio on vmware esxi

[Beierl, Mark]

Ya, sorry I could not be of more help.

I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.

I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?

Regards,
Mark
 
Mark Beierl
SW System Sr Principal Developer
Dell EMC | Cloud & Communication Service Provider Solution
Mark.Beierl@Dell.com

On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:

    
    [EXTERNAL EMAIL] 
    Please report any suspicious attachments, links, or requests for sensitive information.
    
    
    Thanks for the quick response Mark.
    
    Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.  
     
    From your response it looks like (compile or precompiled) neither works.
    
    Thank you.
    
    Eliezer
    
    
    > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    > 
    > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
    > 
    > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
    > 
    > Regards,
    > Mark
    > 
    > Mark Beierl
    > SW System Sr Principal Developer
    > Dell EMC | Cloud & Communication Service Provider Solution
    > Mark.Beierl@Dell.com
    > 
    > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
    > 
    > 
    >    [EXTERNAL EMAIL] 
    >    Please report any suspicious attachments, links, or requests for sensitive information.
    > 
    > 
    >    Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
    >    helpful guides as to how to install it on vmware?  I've searched google but
    >    could not find any.  The only reference to fio+vmware is Disable libaio for
    >    ESXi build - bug#80
    > 
    >    Thank you in advance.
    > 
    >    Regards,
    >    Eliezer
    > 
    > 
    > 
    
    

Re: fio on vmware esxi

[Sitsofe Wheeler]

Hi,

The question of how to compile fio for ESXi comes up every few years
and as Mark suggested it IS doable by building a static fio binary
from an appropriate real distro but there are a few gotchas. You can't
take a "regular" distro's fio because the ESXi environment simply
doesn't match that of the regular distro (for example things like a
glibc are compiled with different options, libraries are missing etc).

The last time it was asked about was over on Github (see
https://github.com/axboe/fio/issues/464 ) where I mentioned the steps
I used to get it going for ESXi 6.0 but note that I no longer have
access to ESXi these days so I can't say what's needed for modern
environments (but the general idea should be the same).

On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>
> Ya, sorry I could not be of more help.
>
> I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.
>
> I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?
>
> Regards,
> Mark
>
> Mark Beierl
> SW System Sr Principal Developer
> Dell EMC | Cloud & Communication Service Provider Solution
> Mark.Beierl@Dell.com
>
> On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:
>
>
>     [EXTERNAL EMAIL]
>     Please report any suspicious attachments, links, or requests for sensitive information.
>
>
>     Thanks for the quick response Mark.
>
>     Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.
>
>     From your response it looks like (compile or precompiled) neither works.
>
>     Thank you.
>
>     Eliezer
>
>
>     > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>     >
>     > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
>     >
>     > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
>     >
>     > Regards,
>     > Mark
>     >
>     > Mark Beierl
>     > SW System Sr Principal Developer
>     > Dell EMC | Cloud & Communication Service Provider Solution
>     > Mark.Beierl@Dell.com
>     >
>     > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
>     >
>     >
>     >    [EXTERNAL EMAIL]
>     >    Please report any suspicious attachments, links, or requests for sensitive information.
>     >
>     >
>     >    Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
>     >    helpful guides as to how to install it on vmware?  I've searched google but
>     >    could not find any.  The only reference to fio+vmware is Disable libaio for
>     >    ESXi build - bug#80
>     >
>     >    Thank you in advance.
>     >
>     >    Regards,
>     >    Eliezer

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio on vmware esxi

[Beierl, Mark]

Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464.

I have fio working on my ESXi 6.5 server.  Here is the procedure I used.  Install Docker and use this Dockerfile:

=============================
FROM multiarch/alpine:x86_64-v3.6

RUN apk --no-cache add --update \
    git \
    alpine-sdk \
    coreutils \
    linux-headers \
    libaio \
    libaio-dev \
    zlib-dev

RUN git config --global http.sslVerify false
RUN git clone http://git.kernel.dk/fio.git fio
RUN cd fio && ./configure --esx
RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install
=============================

Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey)
$ docker run -d fio sleep 600

Note the ID that it spits out.  Use that ID in the next command:
$ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio .

You can now scp that fio to your ESXi server and execute it there.

[root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio 
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
fio-3.11-6-gee6ce
Starting 1 thread
file1: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s]
....

Hope this works for you, Eliezer!

Regards,
Mark
 
Mark Beierl
SW System Sr Principal Developer
Dell EMC | Cloud & Communication Service Provider Solution
Mark.Beierl@Dell.com

On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    
    [EXTERNAL EMAIL] 
    Please report any suspicious attachments, links, or requests for sensitive information.
    
    
    Hi,
    
    The question of how to compile fio for ESXi comes up every few years
    and as Mark suggested it IS doable by building a static fio binary
    from an appropriate real distro but there are a few gotchas. You can't
    take a "regular" distro's fio because the ESXi environment simply
    doesn't match that of the regular distro (for example things like a
    glibc are compiled with different options, libraries are missing etc).
    
    The last time it was asked about was over on Github (see
    https://github.com/axboe/fio/issues/464 ) where I mentioned the steps
    I used to get it going for ESXi 6.0 but note that I no longer have
    access to ESXi these days so I can't say what's needed for modern
    environments (but the general idea should be the same).
    
    On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    >
    > Ya, sorry I could not be of more help.
    >
    > I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.
    >
    > I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?
    >
    > Regards,
    > Mark
    >
    > Mark Beierl
    > SW System Sr Principal Developer
    > Dell EMC | Cloud & Communication Service Provider Solution
    > Mark.Beierl@Dell.com
    >
    > On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:
    >
    >
    >     [EXTERNAL EMAIL]
    >     Please report any suspicious attachments, links, or requests for sensitive information.
    >
    >
    >     Thanks for the quick response Mark.
    >
    >     Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.
    >
    >     From your response it looks like (compile or precompiled) neither works.
    >
    >     Thank you.
    >
    >     Eliezer
    >
    >
    >     > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    >     >
    >     > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
    >     >
    >     > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
    >     >
    >     > Regards,
    >     > Mark
    >     >
    >     > Mark Beierl
    >     > SW System Sr Principal Developer
    >     > Dell EMC | Cloud & Communication Service Provider Solution
    >     > Mark.Beierl@Dell.com
    >     >
    >     > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
    >     >
    >     >
    >     >    [EXTERNAL EMAIL]
    >     >    Please report any suspicious attachments, links, or requests for sensitive information.
    >     >
    >     >
    >     >    Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
    >     >    helpful guides as to how to install it on vmware?  I've searched google but
    >     >    could not find any.  The only reference to fio+vmware is Disable libaio for
    >     >    ESXi build - bug#80
    >     >
    >     >    Thank you in advance.
    >     >
    >     >    Regards,
    >     >    Eliezer
    
    -- 
    Sitsofe | http://sucs.org/~sits/
    

Re: fio on vmware esxi

[Eliezer Nebab]

Excellent update to this thread.

Thanks Sitsofe and Mark!

I’ll try this and hope to be able to run fio the way you outlined.   I see youre also using latest version 3.11.

I hope to update this thread with some good news.  If I could get this working, it will address a lot of performance questions from admins that are on ESXi environments.

Regards,
Eliezer


> On Oct 11, 2018, at 8:42 AM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
> 
> Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464.
> 
> I have fio working on my ESXi 6.5 server.  Here is the procedure I used.  Install Docker and use this Dockerfile:
> 
> =============================
> FROM multiarch/alpine:x86_64-v3.6
> 
> RUN apk --no-cache add --update \
>    git \
>    alpine-sdk \
>    coreutils \
>    linux-headers \
>    libaio \
>    libaio-dev \
>    zlib-dev
> 
> RUN git config --global http.sslVerify false
> RUN git clone http://git.kernel.dk/fio.git fio
> RUN cd fio && ./configure --esx
> RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install
> =============================
> 
> Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey)
> $ docker run -d fio sleep 600
> 
> Note the ID that it spits out.  Use that ID in the next command:
> $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio .
> 
> You can now scp that fio to your ESXi server and execute it there.
> 
> [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio 
> file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
> fio-3.11-6-gee6ce
> Starting 1 thread
> file1: Laying out IO file (1 file / 1024MiB)
> Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s]
> ....
> 
> Hope this works for you, Eliezer!
> 
> Regards,
> Mark
> 
> Mark Beierl
> SW System Sr Principal Developer
> Dell EMC | Cloud & Communication Service Provider Solution
> Mark.Beierl@Dell.com
> 
> On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
> 
> 
>    [EXTERNAL EMAIL] 
>    Please report any suspicious attachments, links, or requests for sensitive information.
> 
> 
>    Hi,
> 
>    The question of how to compile fio for ESXi comes up every few years
>    and as Mark suggested it IS doable by building a static fio binary
>    from an appropriate real distro but there are a few gotchas. You can't
>    take a "regular" distro's fio because the ESXi environment simply
>    doesn't match that of the regular distro (for example things like a
>    glibc are compiled with different options, libraries are missing etc).
> 
>    The last time it was asked about was over on Github (see
>    https://github.com/axboe/fio/issues/464 ) where I mentioned the steps
>    I used to get it going for ESXi 6.0 but note that I no longer have
>    access to ESXi these days so I can't say what's needed for modern
>    environments (but the general idea should be the same).
> 
>>    On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>> 
>> Ya, sorry I could not be of more help.
>> 
>> I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.
>> 
>> I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?
>> 
>> Regards,
>> Mark
>> 
>> Mark Beierl
>> SW System Sr Principal Developer
>> Dell EMC | Cloud & Communication Service Provider Solution
>> Mark.Beierl@Dell.com
>> 
>> On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:
>> 
>> 
>>    [EXTERNAL EMAIL]
>>    Please report any suspicious attachments, links, or requests for sensitive information.
>> 
>> 
>>    Thanks for the quick response Mark.
>> 
>>    Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.
>> 
>>    From your response it looks like (compile or precompiled) neither works.
>> 
>>    Thank you.
>> 
>>    Eliezer
>> 
>> 
>>> On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>>> 
>>> Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
>>> 
>>> I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
>>> 
>>> Regards,
>>> Mark
>>> 
>>> Mark Beierl
>>> SW System Sr Principal Developer
>>> Dell EMC | Cloud & Communication Service Provider Solution
>>> Mark.Beierl@Dell.com
>>> 
>>> On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
>>> 
>>> 
>>>   [EXTERNAL EMAIL]
>>>   Please report any suspicious attachments, links, or requests for sensitive information.
>>> 
>>> 
>>>   Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
>>>   helpful guides as to how to install it on vmware?  I've searched google but
>>>   could not find any.  The only reference to fio+vmware is Disable libaio for
>>>   ESXi build - bug#80
>>> 
>>>   Thank you in advance.
>>> 
>>>   Regards,
>>>   Eliezer
> 
>    -- 
>    Sitsofe | http://sucs.org/~sits/
> 
> 

Re: fio on vmware esxi

[Beierl, Mark]

You are correct, in this Dockerfile, I am building the latest code from git.  Add the following line after the git clone if you want to use a specific version:

RUN cd fio && git checkout tags/fio-3.11

Or whatever tagged version you think is best :)

Regards,
Mark
 
Mark Beierl
SW System Sr Principal Developer
Dell EMC | Cloud & Communication Service Provider Solution
Mark.Beierl@Dell.com

On 2018-10-11, 15:28, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:

    
    [EXTERNAL EMAIL] 
    Please report any suspicious attachments, links, or requests for sensitive information.
    
    
    Excellent update to this thread.
    
    Thanks Sitsofe and Mark!
    
    I’ll try this and hope to be able to run fio the way you outlined.   I see youre also using latest version 3.11.
    
    I hope to update this thread with some good news.  If I could get this working, it will address a lot of performance questions from admins that are on ESXi environments.
    
    Regards,
    Eliezer
    
    
    > On Oct 11, 2018, at 8:42 AM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    > 
    > Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464.
    > 
    > I have fio working on my ESXi 6.5 server.  Here is the procedure I used.  Install Docker and use this Dockerfile:
    > 
    > =============================
    > FROM multiarch/alpine:x86_64-v3.6
    > 
    > RUN apk --no-cache add --update \
    >    git \
    >    alpine-sdk \
    >    coreutils \
    >    linux-headers \
    >    libaio \
    >    libaio-dev \
    >    zlib-dev
    > 
    > RUN git config --global http.sslVerify false
    > RUN git clone http://git.kernel.dk/fio.git fio
    > RUN cd fio && ./configure --esx
    > RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install
    > =============================
    > 
    > Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey)
    > $ docker run -d fio sleep 600
    > 
    > Note the ID that it spits out.  Use that ID in the next command:
    > $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio .
    > 
    > You can now scp that fio to your ESXi server and execute it there.
    > 
    > [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio 
    > file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
    > fio-3.11-6-gee6ce
    > Starting 1 thread
    > file1: Laying out IO file (1 file / 1024MiB)
    > Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s]
    > ....
    > 
    > Hope this works for you, Eliezer!
    > 
    > Regards,
    > Mark
    > 
    > Mark Beierl
    > SW System Sr Principal Developer
    > Dell EMC | Cloud & Communication Service Provider Solution
    > Mark.Beierl@Dell.com
    > 
    > On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
    > 
    > 
    >    [EXTERNAL EMAIL] 
    >    Please report any suspicious attachments, links, or requests for sensitive information.
    > 
    > 
    >    Hi,
    > 
    >    The question of how to compile fio for ESXi comes up every few years
    >    and as Mark suggested it IS doable by building a static fio binary
    >    from an appropriate real distro but there are a few gotchas. You can't
    >    take a "regular" distro's fio because the ESXi environment simply
    >    doesn't match that of the regular distro (for example things like a
    >    glibc are compiled with different options, libraries are missing etc).
    > 
    >    The last time it was asked about was over on Github (see
    >    https://github.com/axboe/fio/issues/464 ) where I mentioned the steps
    >    I used to get it going for ESXi 6.0 but note that I no longer have
    >    access to ESXi these days so I can't say what's needed for modern
    >    environments (but the general idea should be the same).
    > 
    >>    On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    >> 
    >> Ya, sorry I could not be of more help.
    >> 
    >> I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.
    >> 
    >> I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?
    >> 
    >> Regards,
    >> Mark
    >> 
    >> Mark Beierl
    >> SW System Sr Principal Developer
    >> Dell EMC | Cloud & Communication Service Provider Solution
    >> Mark.Beierl@Dell.com
    >> 
    >> On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:
    >> 
    >> 
    >>    [EXTERNAL EMAIL]
    >>    Please report any suspicious attachments, links, or requests for sensitive information.
    >> 
    >> 
    >>    Thanks for the quick response Mark.
    >> 
    >>    Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.
    >> 
    >>    From your response it looks like (compile or precompiled) neither works.
    >> 
    >>    Thank you.
    >> 
    >>    Eliezer
    >> 
    >> 
    >>> On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    >>> 
    >>> Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
    >>> 
    >>> I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
    >>> 
    >>> Regards,
    >>> Mark
    >>> 
    >>> Mark Beierl
    >>> SW System Sr Principal Developer
    >>> Dell EMC | Cloud & Communication Service Provider Solution
    >>> Mark.Beierl@Dell.com
    >>> 
    >>> On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
    >>> 
    >>> 
    >>>   [EXTERNAL EMAIL]
    >>>   Please report any suspicious attachments, links, or requests for sensitive information.
    >>> 
    >>> 
    >>>   Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
    >>>   helpful guides as to how to install it on vmware?  I've searched google but
    >>>   could not find any.  The only reference to fio+vmware is Disable libaio for
    >>>   ESXi build - bug#80
    >>> 
    >>>   Thank you in advance.
    >>> 
    >>>   Regards,
    >>>   Eliezer
    > 
    >    -- 
    >    Sitsofe | http://sucs.org/~sits/
    > 
    > 
    
    

Re: fio on vmware esxi

[Sitsofe Wheeler]

Thank you for following up on this Mark - I'm sure others will find
that information useful!

Glancing at your steps:

> RUN git config --global http.sslVerify false

I'm not sure you need this as the next line isn't using https:

> RUN git clone http://git.kernel.dk/fio.git fio

I don't know if alpine builds a static version of libaio (e.g. there
isn't a prepackaged *static* libaio on CentOS/RHEL) but if it does you
may find you can nobble fio's configure script to allow its detection
even though you're using the --esx option...

On Thu, 11 Oct 2018 at 14:42, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>
> Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464.
>
> I have fio working on my ESXi 6.5 server.  Here is the procedure I used.  Install Docker and use this Dockerfile:
>
> =============================
> FROM multiarch/alpine:x86_64-v3.6
>
> RUN apk --no-cache add --update \
>     git \
>     alpine-sdk \
>     coreutils \
>     linux-headers \
>     libaio \
>     libaio-dev \
>     zlib-dev
>
> RUN git config --global http.sslVerify false
> RUN git clone http://git.kernel.dk/fio.git fio
> RUN cd fio && ./configure --esx
> RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install
> =============================
>
> Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey)
> $ docker run -d fio sleep 600
>
> Note the ID that it spits out.  Use that ID in the next command:
> $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio .
>
> You can now scp that fio to your ESXi server and execute it there.
>
> [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio
> file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
> fio-3.11-6-gee6ce
> Starting 1 thread
> file1: Laying out IO file (1 file / 1024MiB)
> Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s]
> ....
>
> Hope this works for you, Eliezer!
>
> Regards,
> Mark
>
> Mark Beierl
> SW System Sr Principal Developer
> Dell EMC | Cloud & Communication Service Provider Solution
> Mark.Beierl@Dell.com
>
> On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
>
>
>     [EXTERNAL EMAIL]
>     Please report any suspicious attachments, links, or requests for sensitive information.
>
>
>     Hi,
>
>     The question of how to compile fio for ESXi comes up every few years
>     and as Mark suggested it IS doable by building a static fio binary
>     from an appropriate real distro but there are a few gotchas. You can't
>     take a "regular" distro's fio because the ESXi environment simply
>     doesn't match that of the regular distro (for example things like a
>     glibc are compiled with different options, libraries are missing etc).
>
>     The last time it was asked about was over on Github (see
>     https://github.com/axboe/fio/issues/464 ) where I mentioned the steps
>     I used to get it going for ESXi 6.0 but note that I no longer have
>     access to ESXi these days so I can't say what's needed for modern
>     environments (but the general idea should be the same).
>
>     On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>     >
>     > Ya, sorry I could not be of more help.
>     >
>     > I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.
>     >
>     > I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?
>     >
>     > Regards,
>     > Mark
>     >
>     > Mark Beierl
>     > SW System Sr Principal Developer
>     > Dell EMC | Cloud & Communication Service Provider Solution
>     > Mark.Beierl@Dell.com
>     >
>     > On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:
>     >
>     >
>     >     [EXTERNAL EMAIL]
>     >     Please report any suspicious attachments, links, or requests for sensitive information.
>     >
>     >
>     >     Thanks for the quick response Mark.
>     >
>     >     Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.
>     >
>     >     From your response it looks like (compile or precompiled) neither works.
>     >
>     >     Thank you.
>     >
>     >     Eliezer
>     >
>     >
>     >     > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>     >     >
>     >     > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
>     >     >
>     >     > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
>     >     >
>     >     > Regards,
>     >     > Mark
>     >     >
>     >     > Mark Beierl
>     >     > SW System Sr Principal Developer
>     >     > Dell EMC | Cloud & Communication Service Provider Solution
>     >     > Mark.Beierl@Dell.com
>     >     >
>     >     > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
>     >     >
>     >     >
>     >     >    [EXTERNAL EMAIL]
>     >     >    Please report any suspicious attachments, links, or requests for sensitive information.
>     >     >
>     >     >
>     >     >    Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
>     >     >    helpful guides as to how to install it on vmware?  I've searched google but
>     >     >    could not find any.  The only reference to fio+vmware is Disable libaio for
>     >     >    ESXi build - bug#80
>     >     >
>     >     >    Thank you in advance.
>     >     >
>     >     >    Regards,
>     >     >    Eliezer
>
>     --
>     Sitsofe | http://sucs.org/~sits/
>
>


-- 
Sitsofe | http://sucs.org/~sits/

Re: fio on vmware esxi

[Beierl, Mark]

Oops, thanks Sitsofe.

I over-enthusiatically copied that from another docker file where I was also doing https requests.  You are right, it has no purpose in this case.

Hopefully I can look at the libaio with --esx option next week.

Regards,
Mark
 
Mark Beierl
SW System Sr Principal Developer
Dell EMC | Cloud & Communication Service Provider Solution
Mark.Beierl@Dell.com

On 2018-10-12, 02:13, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    
    [EXTERNAL EMAIL] 
    Please report any suspicious attachments, links, or requests for sensitive information.
    
    
    Thank you for following up on this Mark - I'm sure others will find
    that information useful!
    
    Glancing at your steps:
    
    > RUN git config --global http.sslVerify false
    
    I'm not sure you need this as the next line isn't using https:
    
    > RUN git clone http://git.kernel.dk/fio.git fio
    
    I don't know if alpine builds a static version of libaio (e.g. there
    isn't a prepackaged *static* libaio on CentOS/RHEL) but if it does you
    may find you can nobble fio's configure script to allow its detection
    even though you're using the --esx option...
    
    On Thu, 11 Oct 2018 at 14:42, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    >
    > Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464.
    >
    > I have fio working on my ESXi 6.5 server.  Here is the procedure I used.  Install Docker and use this Dockerfile:
    >
    > =============================
    > FROM multiarch/alpine:x86_64-v3.6
    >
    > RUN apk --no-cache add --update \
    >     git \
    >     alpine-sdk \
    >     coreutils \
    >     linux-headers \
    >     libaio \
    >     libaio-dev \
    >     zlib-dev
    >
    > RUN git config --global http.sslVerify false
    > RUN git clone http://git.kernel.dk/fio.git fio
    > RUN cd fio && ./configure --esx
    > RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install
    > =============================
    >
    > Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey)
    > $ docker run -d fio sleep 600
    >
    > Note the ID that it spits out.  Use that ID in the next command:
    > $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio .
    >
    > You can now scp that fio to your ESXi server and execute it there.
    >
    > [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio
    > file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
    > fio-3.11-6-gee6ce
    > Starting 1 thread
    > file1: Laying out IO file (1 file / 1024MiB)
    > Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s]
    > ....
    >
    > Hope this works for you, Eliezer!
    >
    > Regards,
    > Mark
    >
    > Mark Beierl
    > SW System Sr Principal Developer
    > Dell EMC | Cloud & Communication Service Provider Solution
    > Mark.Beierl@Dell.com
    >
    > On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
    >
    >
    >     [EXTERNAL EMAIL]
    >     Please report any suspicious attachments, links, or requests for sensitive information.
    >
    >
    >     Hi,
    >
    >     The question of how to compile fio for ESXi comes up every few years
    >     and as Mark suggested it IS doable by building a static fio binary
    >     from an appropriate real distro but there are a few gotchas. You can't
    >     take a "regular" distro's fio because the ESXi environment simply
    >     doesn't match that of the regular distro (for example things like a
    >     glibc are compiled with different options, libraries are missing etc).
    >
    >     The last time it was asked about was over on Github (see
    >     https://github.com/axboe/fio/issues/464 ) where I mentioned the steps
    >     I used to get it going for ESXi 6.0 but note that I no longer have
    >     access to ESXi these days so I can't say what's needed for modern
    >     environments (but the general idea should be the same).
    >
    >     On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    >     >
    >     > Ya, sorry I could not be of more help.
    >     >
    >     > I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.
    >     >
    >     > I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?
    >     >
    >     > Regards,
    >     > Mark
    >     >
    >     > Mark Beierl
    >     > SW System Sr Principal Developer
    >     > Dell EMC | Cloud & Communication Service Provider Solution
    >     > Mark.Beierl@Dell.com
    >     >
    >     > On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:
    >     >
    >     >
    >     >     [EXTERNAL EMAIL]
    >     >     Please report any suspicious attachments, links, or requests for sensitive information.
    >     >
    >     >
    >     >     Thanks for the quick response Mark.
    >     >
    >     >     Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.
    >     >
    >     >     From your response it looks like (compile or precompiled) neither works.
    >     >
    >     >     Thank you.
    >     >
    >     >     Eliezer
    >     >
    >     >
    >     >     > On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
    >     >     >
    >     >     > Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
    >     >     >
    >     >     > I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
    >     >     >
    >     >     > Regards,
    >     >     > Mark
    >     >     >
    >     >     > Mark Beierl
    >     >     > SW System Sr Principal Developer
    >     >     > Dell EMC | Cloud & Communication Service Provider Solution
    >     >     > Mark.Beierl@Dell.com
    >     >     >
    >     >     > On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
    >     >     >
    >     >     >
    >     >     >    [EXTERNAL EMAIL]
    >     >     >    Please report any suspicious attachments, links, or requests for sensitive information.
    >     >     >
    >     >     >
    >     >     >    Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
    >     >     >    helpful guides as to how to install it on vmware?  I've searched google but
    >     >     >    could not find any.  The only reference to fio+vmware is Disable libaio for
    >     >     >    ESXi build - bug#80
    >     >     >
    >     >     >    Thank you in advance.
    >     >     >
    >     >     >    Regards,
    >     >     >    Eliezer
    >
    >     --
    >     Sitsofe | http://sucs.org/~sits/
    >
    >
    
    
    -- 
    Sitsofe | http://sucs.org/~sits/
    

Re: fio on vmware esxi

[Eliezer Nebab]

I dont blame ya for the over enthusiasm Mark.  I’m just as excited and enthused when I got your reply thay you got fio to work on vmware!

Ive yet to try this out, hoping this weekend to spend some quality time on this and will report back.

Eliezer

> On Oct 12, 2018, at 1:18 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
> 
> Oops, thanks Sitsofe.
> 
> I over-enthusiatically copied that from another docker file where I was also doing https requests.  You are right, it has no purpose in this case.
> 
> Hopefully I can look at the libaio with --esx option next week.
> 
> Regards,
> Mark
> 
> Mark Beierl
> SW System Sr Principal Developer
> Dell EMC | Cloud & Communication Service Provider Solution
> Mark.Beierl@Dell.com
> 
> On 2018-10-12, 02:13, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
> 
> 
>    [EXTERNAL EMAIL] 
>    Please report any suspicious attachments, links, or requests for sensitive information.
> 
> 
>    Thank you for following up on this Mark - I'm sure others will find
>    that information useful!
> 
>    Glancing at your steps:
> 
>> RUN git config --global http.sslVerify false
> 
>    I'm not sure you need this as the next line isn't using https:
> 
>> RUN git clone http://git.kernel.dk/fio.git fio
> 
>    I don't know if alpine builds a static version of libaio (e.g. there
>    isn't a prepackaged *static* libaio on CentOS/RHEL) but if it does you
>    may find you can nobble fio's configure script to allow its detection
>    even though you're using the --esx option...
> 
>>    On Thu, 11 Oct 2018 at 14:42, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>> 
>> Hey all, and special thanks to Sitsofe for pointing me at the github fio issue 464.
>> 
>> I have fio working on my ESXi 6.5 server.  Here is the procedure I used.  Install Docker and use this Dockerfile:
>> 
>> =============================
>> FROM multiarch/alpine:x86_64-v3.6
>> 
>> RUN apk --no-cache add --update \
>>    git \
>>    alpine-sdk \
>>    coreutils \
>>    linux-headers \
>>    libaio \
>>    libaio-dev \
>>    zlib-dev
>> 
>> RUN git config --global http.sslVerify false
>> RUN git clone http://git.kernel.dk/fio.git fio
>> RUN cd fio && ./configure --esx
>> RUN cd fio && EXTFLAGS="-static" make -j $(grep -c ^processor /proc/cpuinfo) install
>> =============================
>> 
>> Then, start the docker container with a dummy command (I’m sure there are better ways to do this, but it works, so hey)
>> $ docker run -d fio sleep 600
>> 
>> Note the ID that it spits out.  Use that ID in the next command:
>> $ docker cp 69e64de38659d350fb07e45b5d7d2d4098c642ec1ca81a6eb18a9b34e622082f:/fio/fio .
>> 
>> You can now scp that fio to your ESXi server and execute it there.
>> 
>> [root@R156U37ESXI:/vmfs/volumes/5a611782-2d14bf65-c453-246e96146f88/testdir] ~/fio test.fio
>> file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
>> fio-3.11-6-gee6ce
>> Starting 1 thread
>> file1: Laying out IO file (1 file / 1024MiB)
>> Jobs: 1 (f=1): [f(1)][100.0%][r=1346KiB/s,w=899KiB/s][r=336,w=224 IOPS][eta 00m:00s]
>> ....
>> 
>> Hope this works for you, Eliezer!
>> 
>> Regards,
>> Mark
>> 
>> Mark Beierl
>> SW System Sr Principal Developer
>> Dell EMC | Cloud & Communication Service Provider Solution
>> Mark.Beierl@Dell.com
>> 
>> On 2018-10-11, 01:34, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
>> 
>> 
>>    [EXTERNAL EMAIL]
>>    Please report any suspicious attachments, links, or requests for sensitive information.
>> 
>> 
>>    Hi,
>> 
>>    The question of how to compile fio for ESXi comes up every few years
>>    and as Mark suggested it IS doable by building a static fio binary
>>    from an appropriate real distro but there are a few gotchas. You can't
>>    take a "regular" distro's fio because the ESXi environment simply
>>    doesn't match that of the regular distro (for example things like a
>>    glibc are compiled with different options, libraries are missing etc).
>> 
>>    The last time it was asked about was over on Github (see
>>    https://github.com/axboe/fio/issues/464 ) where I mentioned the steps
>>    I used to get it going for ESXi 6.0 but note that I no longer have
>>    access to ESXi these days so I can't say what's needed for modern
>>    environments (but the general idea should be the same).
>> 
>>>    On Wed, 10 Oct 2018 at 21:00, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>>> 
>>> Ya, sorry I could not be of more help.
>>> 
>>> I doubt that FIO could be compiled on raw ESXi as there is no compiler available on that from what I can tell.  A statically linked one should work, but I really don’t know enough to be able to say why it could not start a job.
>>> 
>>> I do know that ESXi can pass a full device (raw disk) to a VM and that VM takes full ownership of it.  So you could load a VM with CentOS or something and have FIO profile the raw device, but that is probably not going to be useful.  Chances are you are looking to something a little more advanced like profile vSAN or some other form of software defined storage?
>>> 
>>> Regards,
>>> Mark
>>> 
>>> Mark Beierl
>>> SW System Sr Principal Developer
>>> Dell EMC | Cloud & Communication Service Provider Solution
>>> Mark.Beierl@Dell.com
>>> 
>>> On 2018-10-10, 15:56, "Eliezer Nebab" <eliezer@integritech.solutions> wrote:
>>> 
>>> 
>>>    [EXTERNAL EMAIL]
>>>    Please report any suspicious attachments, links, or requests for sensitive information.
>>> 
>>> 
>>>    Thanks for the quick response Mark.
>>> 
>>>    Yes, I was looking for documentation or guide on if fio can be compiled or installed on esxi servers.
>>> 
>>>    From your response it looks like (compile or precompiled) neither works.
>>> 
>>>    Thank you.
>>> 
>>>    Eliezer
>>> 
>>> 
>>>> On Oct 10, 2018, at 2:05 PM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
>>>> 
>>>> Are you asking if FIO can be built or run on the hypervisor itself, like via ssh?
>>>> 
>>>> I have tried a test before where I statically compiled FIO and copied it to an ESXi server, but I was never able to get any jobs to start.  Not sure if the kernel supports the fork() or whatever call FIO uses to spawn the job processes.  So, I’m not sure it can be done.  ESXi is not a general purpose OS, so it lacks some features.
>>>> 
>>>> Regards,
>>>> Mark
>>>> 
>>>> Mark Beierl
>>>> SW System Sr Principal Developer
>>>> Dell EMC | Cloud & Communication Service Provider Solution
>>>> Mark.Beierl@Dell.com
>>>> 
>>>> On 2018-10-10, 14:24, "fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
>>>> 
>>>> 
>>>>   [EXTERNAL EMAIL]
>>>>   Please report any suspicious attachments, links, or requests for sensitive information.
>>>> 
>>>> 
>>>>   Is fio supported on vmware esxi 5 and 6 servers?  Does anyone have any
>>>>   helpful guides as to how to install it on vmware?  I've searched google but
>>>>   could not find any.  The only reference to fio+vmware is Disable libaio for
>>>>   ESXi build - bug#80
>>>> 
>>>>   Thank you in advance.
>>>> 
>>>>   Regards,
>>>>   Eliezer
>> 
>>    --
>>    Sitsofe | http://sucs.org/~sits/
>> 
>> 
> 
> 
>    -- 
>    Sitsofe | http://sucs.org/~sits/
> 
> 

Re: fio on vmware esxi

[Sitsofe Wheeler]

lHi,

If you don't need asynchrony pvsync2 tends to be good. If you do I
think the only choice from the engines listed is posixaio...

Regarding libaio, see my earlier reply to Mark...

PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about .

On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab
<eliezer@integritech.solutions> wrote:
>
> Mark,
>
>
> Finally had a chance to try this out this weekend.
>
>
> VMWare version:
>
> # vmware -v
> VMware ESXi 6.5.0 build-8294253
>
>
> I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described.
>
> I then copied the ID:
>
> # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio .
> # ls -l
> total 5832
> -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile
> -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio
>
>
> I scp'd the "fio" file to my esxi server.
>
> To my excitement, tried to run "fio" with the libaio which did not work.  The only engines available out of that fio build are as follows:
>
> /tmp/fio --enghelp
>
>
> Available IO engines:
> cpuio
> mmap
> sync
> psync
> vsync
> pvsync
> pvsync2
> null
> net
> netsplice
> ftruncate
> filecreate
> posixaio
> falloc
> e4defrag
> splice
> mtd
> sg
>
> Otherwise, fio is working just as you described.   Thank you very much Mark!!!
>
> So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing?

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio on vmware esxi

[Eliezer Nebab]

posixaio engine worked!

Thank you very much Sitsofe and Mark im good to go!



> On Oct 24, 2018, at 12:05 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> 
> lHi,
> 
> If you don't need asynchrony pvsync2 tends to be good. If you do I
> think the only choice from the engines listed is posixaio...
> 
> Regarding libaio, see my earlier reply to Mark...
> 
> PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about .
> 
> On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab
> <eliezer@integritech.solutions> wrote:
>> 
>> Mark,
>> 
>> 
>> Finally had a chance to try this out this weekend.
>> 
>> 
>> VMWare version:
>> 
>> # vmware -v
>> VMware ESXi 6.5.0 build-8294253
>> 
>> 
>> I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described.
>> 
>> I then copied the ID:
>> 
>> # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio .
>> # ls -l
>> total 5832
>> -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile
>> -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio
>> 
>> 
>> I scp'd the "fio" file to my esxi server.
>> 
>> To my excitement, tried to run "fio" with the libaio which did not work.  The only engines available out of that fio build are as follows:
>> 
>> /tmp/fio --enghelp
>> 
>> 
>> Available IO engines:
>> cpuio
>> mmap
>> sync
>> psync
>> vsync
>> pvsync
>> pvsync2
>> null
>> net
>> netsplice
>> ftruncate
>> filecreate
>> posixaio
>> falloc
>> e4defrag
>> splice
>> mtd
>> sg
>> 
>> Otherwise, fio is working just as you described.   Thank you very much Mark!!!
>> 
>> So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing?
> 
> -- 
> Sitsofe | http://sucs.org/~sits/

Re: fio on vmware esxi

[Beierl, Mark]

Can you clarify – did you need to rebuild FIO or were you just able to run it with posixaio from the Dockerfile example?

Regards,
Mark
 
Mark Beierl
SW System Sr Principal Developer
Dell EMC | Cloud & Communication Service Provider Solution
Mark.Beierl@Dell.com

On 2018-10-26, 10:53, "fio-owner@vger.kernel.org on behalf of Eliezer Nebab" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:

    
    [EXTERNAL EMAIL] 
    Please report any suspicious attachments, links, or requests for sensitive information.
    
    
    posixaio engine worked!
    
    Thank you very much Sitsofe and Mark im good to go!
    
    
    
    > On Oct 24, 2018, at 12:05 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
    > 
    > lHi,
    > 
    > If you don't need asynchrony pvsync2 tends to be good. If you do I
    > think the only choice from the engines listed is posixaio...
    > 
    > Regarding libaio, see my earlier reply to Mark...
    > 
    > PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about .
    > 
    > On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab
    > <eliezer@integritech.solutions> wrote:
    >> 
    >> Mark,
    >> 
    >> 
    >> Finally had a chance to try this out this weekend.
    >> 
    >> 
    >> VMWare version:
    >> 
    >> # vmware -v
    >> VMware ESXi 6.5.0 build-8294253
    >> 
    >> 
    >> I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described.
    >> 
    >> I then copied the ID:
    >> 
    >> # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio .
    >> # ls -l
    >> total 5832
    >> -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile
    >> -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio
    >> 
    >> 
    >> I scp'd the "fio" file to my esxi server.
    >> 
    >> To my excitement, tried to run "fio" with the libaio which did not work.  The only engines available out of that fio build are as follows:
    >> 
    >> /tmp/fio --enghelp
    >> 
    >> 
    >> Available IO engines:
    >> cpuio
    >> mmap
    >> sync
    >> psync
    >> vsync
    >> pvsync
    >> pvsync2
    >> null
    >> net
    >> netsplice
    >> ftruncate
    >> filecreate
    >> posixaio
    >> falloc
    >> e4defrag
    >> splice
    >> mtd
    >> sg
    >> 
    >> Otherwise, fio is working just as you described.   Thank you very much Mark!!!
    >> 
    >> So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing?
    > 
    > -- 
    > Sitsofe | http://sucs.org/~sits/
    
    

Re: fio on vmware esxi

[Eliezer Nebab]

Sorry, let me clarify.

I was able to run the posixaio engine from your Docker file example.

Thank you.

Eliezer

> On Oct 26, 2018, at 9:54 AM, Beierl, Mark <Mark.Beierl@dell.com> wrote:
> 
> Can you clarify – did you need to rebuild FIO or were you just able to run it with posixaio from the Dockerfile example?
> 
> Regards,
> Mark
> 
> Mark Beierl
> SW System Sr Principal Developer
> Dell EMC | Cloud & Communication Service Provider Solution
> Mark.Beierl@Dell.com
> 
> On 2018-10-26, 10:53, "fio-owner@vger.kernel.org on behalf of Eliezer Nebab" <fio-owner@vger.kernel.org on behalf of eliezer@integritech.solutions> wrote:
> 
> 
>    [EXTERNAL EMAIL] 
>    Please report any suspicious attachments, links, or requests for sensitive information.
> 
> 
>    posixaio engine worked!
> 
>    Thank you very much Sitsofe and Mark im good to go!
> 
> 
> 
>> On Oct 24, 2018, at 12:05 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> 
>> lHi,
>> 
>> If you don't need asynchrony pvsync2 tends to be good. If you do I
>> think the only choice from the engines listed is posixaio...
>> 
>> Regarding libaio, see my earlier reply to Mark...
>> 
>> PS: VMware is spelt with a lowercase w - http://thelowercasew.com/about .
>> 
>> On Wed, 24 Oct 2018 at 05:27, Eliezer Nebab
>> <eliezer@integritech.solutions> wrote:
>>> 
>>> Mark,
>>> 
>>> 
>>> Finally had a chance to try this out this weekend.
>>> 
>>> 
>>> VMWare version:
>>> 
>>> # vmware -v
>>> VMware ESXi 6.5.0 build-8294253
>>> 
>>> 
>>> I built the docker file per your procedure, but I excluded the line that Sitsofe mentioned ("RUN git config --global http.sslVerify false") and started docker exactly the way you described.
>>> 
>>> I then copied the ID:
>>> 
>>> # docker cp 997cd757e1f1d350e353bd6b4a252c5b16497e49d0b4ed46e32273dc 6a82aa56:/fio/fio .
>>> # ls -l
>>> total 5832
>>> -rw-r--r--. 1 root root 341 Oct 23 22:39 Dockerfile
>>> -rwxr-xr-x. 1 root root 5951600 Oct 23 22:46 fio
>>> 
>>> 
>>> I scp'd the "fio" file to my esxi server.
>>> 
>>> To my excitement, tried to run "fio" with the libaio which did not work.  The only engines available out of that fio build are as follows:
>>> 
>>> /tmp/fio --enghelp
>>> 
>>> 
>>> Available IO engines:
>>> cpuio
>>> mmap
>>> sync
>>> psync
>>> vsync
>>> pvsync
>>> pvsync2
>>> null
>>> net
>>> netsplice
>>> ftruncate
>>> filecreate
>>> posixaio
>>> falloc
>>> e4defrag
>>> splice
>>> mtd
>>> sg
>>> 
>>> Otherwise, fio is working just as you described.   Thank you very much Mark!!!
>>> 
>>> So, out of the engines listed above, which ioengine would be suitable to use for VMWare testing?
>> 
>> -- 
>> Sitsofe | http://sucs.org/~sits/
> 
> 
> 

RE: fio on vmware esxi

[eliezer]

Mark, Sitsofe,

So I've been testing FIO on VMware using the different ioengines below and found something interesting.
-psync
-pvsync
-sync
-posixaio

The OS is a VMware ESXi:
VMware ESXi 6.5.0 build-8294253

FIO version compiled via Docker (thanks to Mark and Sitsofe):
fio-3.11-22-g425d

I mounted 3 storage devices (2 are different block storage types) on the same baremetal ESXi server:
-500 IOPS NFS
-1000 IOPS Reliant Tier iSCSI Block Storage
-1000 IOPS Perf Tier iSCSI Block Storage

I run FIO with the following parameters and used the different engines I listed above:
--randrepeat=1 
--ioengine=posixaio 
--direct=1 
--sync=1  
--name=mytest
--filename=mytestfile.fio
 --overwrite=1 
--iodepth=64 
--size=100MB 
--readwrite=randrw 
--rwmixread=50 
--rwmixwrite=50
 --bs=16k

Here are the results:

NFS:::
-ioengine=psync on a 500 IOPS NFS storage:::      read=245 write=256 IOPS*
-ioengine=pvsync on a 500 IOPS NFS Storage::::   read=252 write=263 IOPS*
-ioengine=sync on a 500 IOPS NFS Storage::::        read=258 write=270 IOPS*
-ioengine=posixaio on a 500 IOPS NFS Storage:::: read=172 write=180 IOPS

BLOCK STORAGE:::
1000 IOPS Reliant Tier
-ioengine=psync on a 1000 IOPS Reliant storage:::     read=249 write=260 IOPS
-ioengine=pvsync on a 1000 IOPS Reliant storage:::    read=321 write=335 IOPS
-ioengine=sync on a 1000 IOPS Reliant storage:::        read=236 write=246 IOPS
-ioengine=posixaio on a 1000 IOPS Reliant storage:::  read=535 write=559 IOPS*

1000 IOPS Perf Tier
-ioengine=psync on a 1000 IOPS Perf storage:::     read=177 write=185 IOPS
-ioengine=pvsync on a 1000 IOPS Perf storage:::    read=176 write=184 IOPS
-ioengine=sync on a 1000 IOPS Perf storage:::      read=145 write=152 IOPS
-ioengine=posixaio on a 1000 IOPS Perf storage:::  read=528 write=551 IOPS*

Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage is "posixaio"; and for NFS it can be psync or pvsync or sync.
If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS.  
However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS.  I'd have to use either psync, or pvsync, or sync to see the IOPS I'm expecting.
  These tests are on the same baremetal server running the version of VMWare I posted above.

Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test an NFS storage?   The same goes with ioengines psync/pvsync/sync skews IOPS  results on a block storage device?

Thanks in advance.

Eliezer

RE: fio on vmware esxi

[Elliott, Robert (Persistent Memory)]

> I run FIO with the following parameters and used the different engines I listed above:
> --randrepeat=1
> --ioengine=posixaio
> --direct=1
> --sync=1
> --name=mytest
> --filename=mytestfile.fio
>  --overwrite=1
> --iodepth=64
> --size=100MB
> --readwrite=randrw
> --rwmixread=50
> --rwmixwrite=50
>  --bs=16k
> 
...
> -ioengine=psync on a 1000 IOPS Perf storage:::     read=177 write=185 IOPS
> -ioengine=pvsync on a 1000 IOPS Perf storage:::    read=176 write=184 IOPS
> -ioengine=sync on a 1000 IOPS Perf storage:::      read=145 write=152 IOPS
> -ioengine=posixaio on a 1000 IOPS Perf storage:::  read=528 write=551 IOPS*
> 
> Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage
> is "posixaio"; and for NFS it can be psync or pvsync or sync.
> If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS.
> However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS.  I'd
> have to use either psync, or pvsync, or sync to see the IOPS I'm expecting.
>   These tests are on the same baremetal server running the version of VMWare I posted above.
> 
> Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test
> an NFS storage?   The same goes with ioengines psync/pvsync/sync skews IOPS  results on a block
> storage device?

NFS might introduce all sorts of problems.  For the block device cases...

Synchronous engines like sync do not honor the iodepth; you need to use
jobs= to get multiple concurrent IOs.  posixaio is an  asynchronous engine, 
so it is honoring your iodepth (it spawns threads for you).

Also, if your storage devices do any sort of buffering and caching, the
tiny 100 MiB size is likely to result in lots of cache hits, distorting
the results.

---
Robert Elliott, HPE Persistent Memory

RE: fio on vmware esxi

[eliezer]

Thanks for the quick feedback Robert.  I appreciate it.

I forgot to add,  when I created a new NFS datastore in VMware ESXi 6.5 I have to NFS versions to choose from:

-NFS 3 allows the datastore to be accessed by ESX/ESXi hosts of version earlier than 6.0
-NFS 4.1 provides multipathing for servers and supports the Kerberos authentication protocol

The results FIO results I posted were from NFS v.3. so I continued testing...

I unmounted that NFS volume and remounted it as a new datastore using NFS v4.1 and I'm now showing the expected IOPS using the "posixaio" engine on NFS!  Great!

So, use "posixaio" FIO engine on NFS v 4.1;  don’t use "posixaio" engine on NFS v 3.

Eliezer 

-----Original Message-----
From: Elliott, Robert (Persistent Memory) [mailto:elliott@hpe.com] 
Sent: Friday, October 26, 2018 5:30 PM
To: 'eliezer@integritech.solutions'; 'Beierl, Mark'
Cc: 'Sitsofe Wheeler'; 'fio'
Subject: RE: fio on vmware esxi


> I run FIO with the following parameters and used the different engines I listed above:
> --randrepeat=1
> --ioengine=posixaio
> --direct=1
> --sync=1
> --name=mytest
> --filename=mytestfile.fio
>  --overwrite=1
> --iodepth=64
> --size=100MB
> --readwrite=randrw
> --rwmixread=50
> --rwmixwrite=50
>  --bs=16k
> 
...
> -ioengine=psync on a 1000 IOPS Perf storage:::     read=177 write=185 IOPS
> -ioengine=pvsync on a 1000 IOPS Perf storage:::    read=176 write=184 IOPS
> -ioengine=sync on a 1000 IOPS Perf storage:::      read=145 write=152 IOPS
> -ioengine=posixaio on a 1000 IOPS Perf storage:::  read=528 write=551 IOPS*
> 
> Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage
> is "posixaio"; and for NFS it can be psync or pvsync or sync.
> If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS.
> However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS.  I'd
> have to use either psync, or pvsync, or sync to see the IOPS I'm expecting.
>   These tests are on the same baremetal server running the version of VMWare I posted above.
> 
> Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test
> an NFS storage?   The same goes with ioengines psync/pvsync/sync skews IOPS  results on a block
> storage device?

NFS might introduce all sorts of problems.  For the block device cases...

Synchronous engines like sync do not honor the iodepth; you need to use
jobs= to get multiple concurrent IOs.  posixaio is an  asynchronous engine, 
so it is honoring your iodepth (it spawns threads for you).

Also, if your storage devices do any sort of buffering and caching, the
tiny 100 MiB size is likely to result in lots of cache hits, distorting
the results.

---
Robert Elliott, HPE Persistent Memory

Re: fio on vmware esxi

[Beierl, Mark]

> On Oct 26, 2018, at 18:30, Elliott, Robert (Persistent Memory) <elliott@hpe.com> wrote:
> 
> 
> [EXTERNAL EMAIL] 
> Please report any suspicious attachments, links, or requests for sensitive information.
> 
> 
> 
>> I run FIO with the following parameters and used the different engines I listed above:
>> --randrepeat=1
>> --ioengine=posixaio
>> --direct=1
>> --sync=1
>> --name=mytest
>> --filename=mytestfile.fio
>> --overwrite=1
>> --iodepth=64
>> --size=100MB
>> --readwrite=randrw
>> --rwmixread=50
>> --rwmixwrite=50
>> --bs=16k
>> 
> ...
>> -ioengine=psync on a 1000 IOPS Perf storage:::     read=177 write=185 IOPS
>> -ioengine=pvsync on a 1000 IOPS Perf storage:::    read=176 write=184 IOPS
>> -ioengine=sync on a 1000 IOPS Perf storage:::      read=145 write=152 IOPS
>> -ioengine=posixaio on a 1000 IOPS Perf storage:::  read=528 write=551 IOPS*
>> 
>> Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage
>> is "posixaio"; and for NFS it can be psync or pvsync or sync.
>> If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS.
>> However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS.  I'd
>> have to use either psync, or pvsync, or sync to see the IOPS I'm expecting.
>>  These tests are on the same baremetal server running the version of VMWare I posted above.
>> 
>> Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test
>> an NFS storage?   The same goes with ioengines psync/pvsync/sync skews IOPS  results on a block
>> storage device?
> 
> NFS might introduce all sorts of problems.  For the block device cases...
> 
> Synchronous engines like sync do not honor the iodepth; you need to use
> jobs= to get multiple concurrent IOs.  posixaio is an  asynchronous engine, 
> so it is honoring your iodepth (it spawns threads for you).
> 
> Also, if your storage devices do any sort of buffering and caching, the
> tiny 100 MiB size is likely to result in lots of cache hits, distorting
> the results.
> 
> ---
> Robert Elliott, HPE Persistent Memory
> 

My typical rule of thumb is to make your test data size be around 5x the total size of all cache available. This can become very large, btw, once you factor in SDS like vSAN. 

Re: fio on vmware esxi

[Sitsofe Wheeler]

On Fri, 26 Oct 2018 at 23:30, Elliott, Robert (Persistent Memory)
<elliott@hpe.com> wrote:
>
>
> > I run FIO with the following parameters and used the different engines I listed above:
> > --randrepeat=1
> > --ioengine=posixaio
> > --direct=1
> > --sync=1
> > --name=mytest
> > --filename=mytestfile.fio
> >  --overwrite=1
> > --iodepth=64
> > --size=100MB
> > --readwrite=randrw
> > --rwmixread=50
> > --rwmixwrite=50
> >  --bs=16k
> >
> ...
> > -ioengine=psync on a 1000 IOPS Perf storage:::     read=177 write=185 IOPS
> > -ioengine=pvsync on a 1000 IOPS Perf storage:::    read=176 write=184 IOPS
> > -ioengine=sync on a 1000 IOPS Perf storage:::      read=145 write=152 IOPS
> > -ioengine=posixaio on a 1000 IOPS Perf storage:::  read=528 write=551 IOPS*
> >
> > Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage
> > is "posixaio"; and for NFS it can be psync or pvsync or sync.
> > If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS.
> > However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS.  I'd
> > have to use either psync, or pvsync, or sync to see the IOPS I'm expecting.
> >   These tests are on the same baremetal server running the version of VMWare I posted above.
> >
> > Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test
> > an NFS storage?   The same goes with ioengines psync/pvsync/sync skews IOPS  results on a block
> > storage device?
>
> NFS might introduce all sorts of problems.  For the block device cases...
>
> Synchronous engines like sync do not honor the iodepth; you need to use
> jobs= to get multiple concurrent IOs.  posixaio is an  asynchronous engine,
> so it is honoring your iodepth (it spawns threads for you).
>
> Also, if your storage devices do any sort of buffering and caching, the
> tiny 100 MiB size is likely to result in lots of cache hits, distorting
> the results.

On the NFS front, perhaps you requesting direct=1 the kernel is
actually using synchronous buffered I/O behind the scenes and all
those syncs are slowing it down? Does using sync=0 show less of a
difference between engines?

-- 
Sitsofe | http://sucs.org/~sits/