Compressible and Dedupable ratios using fio

Compressible and Dedupable ratios using fio

[Neto, Antonio Jose Rodrigues]

Hi All,

This is neto from Brazil

How are you?

I need to know how to control the % of compression and deduplication using
fio?

My idea is to generate some workload based on:

Workload Random
0% compressible and 0% dedupable

Workload Mixed
20% compressible and 20% dedupable

Workload Easier
100% compressible and 100% dedupable

Could you please help me?

Thank you very much,

All the best

neto

Re: Compressible and Dedupable ratios using fio

[Matthew Eaton]

On Mon, Mar 10, 2014 at 2:18 PM, Neto, Antonio Jose Rodrigues
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
> Hi All,
>
> This is neto from Brazil
>
> How are you?
>
> I need to know how to control the % of compression and deduplication using
> fio?
>
> My idea is to generate some workload based on:
>
> Workload Random
> 0% compressible and 0% dedupable
>
> Workload Mixed
> 20% compressible and 20% dedupable
>
> Workload Easier
> 100% compressible and 100% dedupable
>
> Could you please help me?
>
> Thank you very much,
>
> All the best
>
> neto
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Neto,

Have you tried using buffer_compress_percentage?  Setting this to 100
would get you 100% compressible data while 0 would be 0% compressible.
 Note that if you use fio prior to version 2.1.6 in git, you will also
want to set scramble_buffers=0 to get the desired compression.

I don't know how you would control how dedupable the data is, since to
my knowledge there is no way to specify how repeatable the data
pattern is.  But I think fully incompressible data is naturally not
dedupable, while the opposite is true for fully compressible data.

Re: Compressible and Dedupable ratios using fio

[Neto, Antonio Jose Rodrigues]

On 3/16/14, 7:53 PM, "Matthew Eaton" <m.eaton82@gmail.com> wrote:

>On Mon, Mar 10, 2014 at 2:18 PM, Neto, Antonio Jose Rodrigues
><Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>> Hi All,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> I need to know how to control the % of compression and deduplication
>>using
>> fio?
>>
>> My idea is to generate some workload based on:
>>
>> Workload Random
>> 0% compressible and 0% dedupable
>>
>> Workload Mixed
>> 20% compressible and 20% dedupable
>>
>> Workload Easier
>> 100% compressible and 100% dedupable
>>
>> Could you please help me?
>>
>> Thank you very much,
>>
>> All the best
>>
>> neto
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>Hi Neto,
>
>Have you tried using buffer_compress_percentage?  Setting this to 100
>would get you 100% compressible data while 0 would be 0% compressible.
> Note that if you use fio prior to version 2.1.6 in git, you will also
>want to set scramble_buffers=0 to get the desired compression.
>
>I don't know how you would control how dedupable the data is, since to
>my knowledge there is no way to specify how repeatable the data
>pattern is.  But I think fully incompressible data is naturally not
>dedupable, while the opposite is true for fully compressible data.

Hi Matthew,

This is neto from Brazil

How are you?

Thank you very much for your reply.

I will try to do some tests this week and I will share the results.

Thanks again,

All the best,

neto

Re: Compressible and Dedupable ratios using fio

[Neto, Antonio Jose Rodrigues]

>><Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>> Hi All,
>>>
>>> This is neto from Brazil
>>>
>>> How are you?
>>>
>>> I need to know how to control the % of compression and deduplication
>>>using
>>> fio?
>>>
>>> My idea is to generate some workload based on:
>>>
>>> Workload Random
>>> 0% compressible and 0% dedupable
>>>
>>> Workload Mixed
>>> 20% compressible and 20% dedupable
>>>
>>> Workload Easier
>>> 100% compressible and 100% dedupable
>>>
>>> Could you please help me?
>>>
>>> Thank you very much,
>>>
>>> All the best
>>>
>>> neto
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>Hi Neto,
>>
>>Have you tried using buffer_compress_percentage?  Setting this to 100
>>would get you 100% compressible data while 0 would be 0% compressible.
>> Note that if you use fio prior to version 2.1.6 in git, you will also
>>want to set scramble_buffers=0 to get the desired compression.
>>
>>I don't know how you would control how dedupable the data is, since to
>>my knowledge there is no way to specify how repeatable the data
>>pattern is.  But I think fully incompressible data is naturally not
>>dedupable, while the opposite is true for fully compressible data.

Hi All,

This is neto from Brazil

How are you?

I am doing some tests here and I would like to share the results.

Go fio :-)

[oracle@s1][flash1][~/fio]$ ./fio -version
fio-2.1.6.1-18-g190b

Config file:
[workload]
bs=8k
ioengine=libaio
iodepth=1
numjobs=1
direct=1
runtime=10
size=1024m
filename=/home/oracle/fio/bigfile
rw=read
thread
unified_rw_reporting=1
group_reporting=1
buffer_compress_percentage=100

Assuming 100% compressible - it's working



[oracle@s1][flash1][~/fio]$ gzip -v bigfile
bigfile:	 99.9% -- replaced with bigfile.gz



Now I will try to generate something with 0 compressible:

[workload]
bs=8k
ioengine=libaio
iodepth=1
numjobs=1
direct=1
runtime=10
size=1024m
filename=/home/oracle/fio/bigfile
rw=read
thread
unified_rw_reporting=1
group_reporting=1
buffer_compress_percentage=0


[oracle@s1][flash1][~/fio]$ gzip -v bigfile
bigfile:	  0.3% -- replaced with bigfile.gz

One question:

1) Are these parameters (randrepeat=0, norandommap=1,
file_service_type=random) going to help in the randomness and
compressibility of the data?

2) Do you have any practical example where I have to use those parameters?

In my tests, I haven't notice any difference.


Thank you very much,

All the best

neto

Re: Compressible and Dedupable ratios using fio

[Matthew Eaton]

> One question:
>
> 1) Are these parameters (randrepeat=0, norandommap=1,
> file_service_type=random) going to help in the randomness and
> compressibility of the data?
>
> 2) Do you have any practical example where I have to use those parameters?
>
> In my tests, I haven't notice any difference.

Data compressibility is only affected by the data pattern.  Randrepeat
and norandommap will only affect the randomness of the io for random
workloads.  For file_service_type, I think this is only used if fio
has multiple files open at the same time, e.g. if multiple filenames
are specified or nrfiles is used.