FIO and replay_redirect

FIO and replay_redirect

[neto from Brazil]

Hi All,

This is neto from Brazil

How are you?

I have captured a blktrace from an Oracle workload. 18 dm devices.

Steps to capture:

blktrace <device 0> <device 1> ... -w 40
blkparse -i <device 0> -i <device 1> ... -o /dev/null -d oracle_trace.out

Trying to replay with FIO (here I have 2 devices - /dev/sdh and /dev/sdi)

Testing with one device:

[workload]
ioengine=libaio
iodepth=1
size=100g
numjobs=1
direct=1
thread
read_iolog=/root/traces/oracle-s144/oracle_trace.out
replay_no_stall=1
replay_redirect=/dev/sdh

[root@oraperf1 fio]# ./fio oracle-replay.cfg
workload: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio-2.2.9-26-g669e
Starting 1 thread
fio: ioengines.c:422: td_io_open_file: Assertion `!fio_file_open(f)'
failed.
Aborted (core dumped)





Questions:

1) The workload that I capture, all DM devices have 200GB size. My devices
where I want to replay have 100GB. I believe I will need to match the size
of all disks (where I captured the workload) with the destination where I
am going to replay. For example: If I capture a workload with 18 LUNs
200GB, to replay I need to have 18 LUNs with 200GB minimum.

2) I need to replay to more than one device. How can I do that with fio?
Seems to me replay_redirect is for a single specified device.

3) Any help why I am getting that error? fio: ioengines.c:422:
td_io_open_file: Assertion `!fio_file_open(f)' failed.

Thank you very much for all help.

All the best

neto

Re: FIO and replay_redirect

[Alireza Haghdoost]

On Thu, Feb 4, 2016 at 4:32 PM, neto from Brazil
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
> Hi All,
>
> This is neto from Brazil
>
> How are you?
>
> I have captured a blktrace from an Oracle workload. 18 dm devices.
>
> Steps to capture:
>
> blktrace <device 0> <device 1> ... -w 40
> blkparse -i <device 0> -i <device 1> ... -o /dev/null -d oracle_trace.out
>
> Trying to replay with FIO (here I have 2 devices - /dev/sdh and /dev/sdi)
>
> Testing with one device:
>
> [workload]
> ioengine=libaio
> iodepth=1
> size=100g
> numjobs=1
> direct=1
> thread
> read_iolog=/root/traces/oracle-s144/oracle_trace.out
> replay_no_stall=1
> replay_redirect=/dev/sdh
>
> [root@oraperf1 fio]# ./fio oracle-replay.cfg
> workload: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
> fio-2.2.9-26-g669e
> Starting 1 thread
> fio: ioengines.c:422: td_io_open_file: Assertion `!fio_file_open(f)'
> failed.
> Aborted (core dumped)
>
>
>
>
>
> Questions:
>
> 1) The workload that I capture, all DM devices have 200GB size. My devices
> where I want to replay have 100GB. I believe I will need to match the size
> of all disks (where I captured the workload) with the destination where I
> am going to replay. For example: If I capture a workload with 18 LUNs
> 200GB, to replay I need to have 18 LUNs with 200GB minimum.

That is right. Your best bet to create one 100GB partition on each one
of those 200GB DMs. Therefore, physically you have 200GB DM. However,
the workload only goes to the first 100GB. Then capture new trace and
try to replay your new trace.

>
> 2) I need to replay to more than one device. How can I do that with fio?
> Seems to me replay_redirect is for a single specified device.

replay with btreplay or blkreplay instead of fio.

> 3) Any help why I am getting that error? fio: ioengines.c:422:
> td_io_open_file: Assertion `!fio_file_open(f)' failed.
>
> Thank you very much for all help.
>
> All the best
>
> neto
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FIO and replay_redirect

[neto from Brazil]

On 2/4/16, 6:11 PM, "Alireza Haghdoost" <alireza@cs.umn.edu> wrote:

>On Thu, Feb 4, 2016 at 4:32 PM, neto from Brazil
><Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>> Hi All,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> I have captured a blktrace from an Oracle workload. 18 dm devices.
>>
>> Steps to capture:
>>
>> blktrace <device 0> <device 1> ... -w 40
>> blkparse -i <device 0> -i <device 1> ... -o /dev/null -d
>>oracle_trace.out
>>
>> Trying to replay with FIO (here I have 2 devices - /dev/sdh and
>>/dev/sdi)
>>
>> Testing with one device:
>>
>> [workload]
>> ioengine=libaio
>> iodepth=1
>> size=100g
>> numjobs=1
>> direct=1
>> thread
>> read_iolog=/root/traces/oracle-s144/oracle_trace.out
>> replay_no_stall=1
>> replay_redirect=/dev/sdh
>>
>> [root@oraperf1 fio]# ./fio oracle-replay.cfg
>> workload: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
>>iodepth=1
>> fio-2.2.9-26-g669e
>> Starting 1 thread
>> fio: ioengines.c:422: td_io_open_file: Assertion `!fio_file_open(f)'
>> failed.
>> Aborted (core dumped)
>>
>>
>>
>>
>>
>> Questions:
>>
>> 1) The workload that I capture, all DM devices have 200GB size. My
>>devices
>> where I want to replay have 100GB. I believe I will need to match the
>>size
>> of all disks (where I captured the workload) with the destination where
>>I
>> am going to replay. For example: If I capture a workload with 18 LUNs
>> 200GB, to replay I need to have 18 LUNs with 200GB minimum.
>
>That is right. Your best bet to create one 100GB partition on each one
>of those 200GB DMs. Therefore, physically you have 200GB DM. However,
>the workload only goes to the first 100GB. Then capture new trace and
>try to replay your new trace.
>
>>
>> 2) I need to replay to more than one device. How can I do that with fio?
>> Seems to me replay_redirect is for a single specified device.
>
>replay with btreplay or blkreplay instead of fio.
>
>> 3) Any help why I am getting that error? fio: ioengines.c:422:
>> td_io_open_file: Assertion `!fio_file_open(f)' failed.
>>
>> Thank you very much for all help.
>>
>> All the best
>>
>> neto
>>


Hi Alireza,

This is neto from Brazil

How are you?

Actually I have to reply with FIO and I will explain why... :-)

The workload that I have been generating is HIGHLY compressible and I
would like to use FIO to control the % of the compressible for the
workload. Unfortunately, I won't be able to do that with blkreplay.

Thanks for your reply.

Thanks

neto

Re: FIO and replay_redirect

[neto from Brazil]

>Hi All,
>
>This is neto from Brazil
>
>How are you?
>
>I have captured a blktrace from an Oracle workload. 18 dm devices.
>
>Steps to capture:
>
>blktrace <device 0> <device 1> ... -w 40
>blkparse -i <device 0> -i <device 1> ... -o /dev/null -d oracle_trace.out
>
>Trying to replay with FIO (here I have 2 devices - /dev/sdh and /dev/sdi)
>
>Testing with one device:
>
>[workload]
>ioengine=libaio
>iodepth=1
>size=100g
>numjobs=1
>direct=1
>thread
>read_iolog=/root/traces/oracle-s144/oracle_trace.out
>replay_no_stall=1
>replay_redirect=/dev/sdh
>
>[root@oraperf1 fio]# ./fio oracle-replay.cfg
>workload: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
>fio-2.2.9-26-g669e
>Starting 1 thread
>fio: ioengines.c:422: td_io_open_file: Assertion `!fio_file_open(f)'
>failed.
>Aborted (core dumped)
>
>
>
>
>
>Questions:
>
>1) The workload that I capture, all DM devices have 200GB size. My devices
>where I want to replay have 100GB. I believe I will need to match the size
>of all disks (where I captured the workload) with the destination where I
>am going to replay. For example: If I capture a workload with 18 LUNs
>200GB, to replay I need to have 18 LUNs with 200GB minimum.
>
>2) I need to replay to more than one device. How can I do that with fio?
>Seems to me replay_redirect is for a single specified device.
>
>3) Any help why I am getting that error? fio: ioengines.c:422:
>td_io_open_file: Assertion `!fio_file_open(f)' failed.
>
>Thank you very much for all help.
>
>All the best
>
>Neto


Hi All,

This is neto from Brazil

How are you?

Any suggestions here?

Thank you,

neto

Re: FIO and replay_redirect

[Alireza Haghdoost]

Neto,

Once again, I would recommend you to use blkreplay instead of fio. You
can control the compressibility ratio in blkreplay using fill_random
parameter.

--Alireza

On Tue, Feb 9, 2016 at 2:29 PM, neto from Brazil
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>
>>Hi All,
>>
>>This is neto from Brazil
>>
>>How are you?
>>
>>I have captured a blktrace from an Oracle workload. 18 dm devices.
>>
>>Steps to capture:
>>
>>blktrace <device 0> <device 1> ... -w 40
>>blkparse -i <device 0> -i <device 1> ... -o /dev/null -d oracle_trace.out
>>
>>Trying to replay with FIO (here I have 2 devices - /dev/sdh and /dev/sdi)
>>
>>Testing with one device:
>>
>>[workload]
>>ioengine=libaio
>>iodepth=1
>>size=100g
>>numjobs=1
>>direct=1
>>thread
>>read_iolog=/root/traces/oracle-s144/oracle_trace.out
>>replay_no_stall=1
>>replay_redirect=/dev/sdh
>>
>>[root@oraperf1 fio]# ./fio oracle-replay.cfg
>>workload: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
>>fio-2.2.9-26-g669e
>>Starting 1 thread
>>fio: ioengines.c:422: td_io_open_file: Assertion `!fio_file_open(f)'
>>failed.
>>Aborted (core dumped)
>>
>>
>>
>>
>>
>>Questions:
>>
>>1) The workload that I capture, all DM devices have 200GB size. My devices
>>where I want to replay have 100GB. I believe I will need to match the size
>>of all disks (where I captured the workload) with the destination where I
>>am going to replay. For example: If I capture a workload with 18 LUNs
>>200GB, to replay I need to have 18 LUNs with 200GB minimum.
>>
>>2) I need to replay to more than one device. How can I do that with fio?
>>Seems to me replay_redirect is for a single specified device.
>>
>>3) Any help why I am getting that error? fio: ioengines.c:422:
>>td_io_open_file: Assertion `!fio_file_open(f)' failed.
>>
>>Thank you very much for all help.
>>
>>All the best
>>
>>Neto
>
>
> Hi All,
>
> This is neto from Brazil
>
> How are you?
>
> Any suggestions here?
>
> Thank you,
>
> neto
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FIO and replay_redirect

[Sitsofe Wheeler]

(This is one I meant to reply to but didn't get around to for some reason)

On 4 February 2016 at 22:32, neto from Brazil
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>
> 1) The workload that I capture, all DM devices have 200GB size. My devices
> where I want to replay have 100GB. I believe I will need to match the size
> of all disks (where I captured the workload) with the destination where I
> am going to replay. For example: If I capture a workload with 18 LUNs
> 200GB, to replay I need to have 18 LUNs with 200GB minimum.
>
> 2) I need to replay to more than one device. How can I do that with fio?
> Seems to me replay_redirect is for a single specified device.
>
> 3) Any help why I am getting that error? fio: ioengines.c:422:
> td_io_open_file: Assertion `!fio_file_open(f)' failed.

1) You should be able to use replay_scale
(http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-replay-scale
)..
2) This still can't be done with fio.
3) Should be fixed since
https://github.com/axboe/fio/commit/96d663ad34f1def5c41a8fe656c15d5c312dffe2
(e.g. 2.16 and later).

-- 
Sitsofe | http://sucs.org/~sits/