* FIO - dedup checksums or specified blocksize does not match.
@ 2015-04-27 13:18 Srinivasa Chamarthy
2015-04-27 14:39 ` Jens Axboe
0 siblings, 1 reply; 5+ messages in thread
From: Srinivasa Chamarthy @ 2015-04-27 13:18 UTC (permalink / raw)
To: fio, Jens Axboe
I was just verifying if i could generate 100% duplicable data with
FIO. I have configured small workload with bs of 256k and writing 2MB
of file. I tried to get the checksum of each of 256k blocks of data
from the file and the checksums do not match. If i am not wrong, when
i specify data as 100% deduppable, my checksums should match isn't it?
# cat ddp_file.fio
[dedupe]
filename=test.tmp
bs=256k
rw=write
size=2m
dedupe_percentage=100
write_iolog=test.tmp.log
# fio ddp_file.fio
dedupe: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K,
ioengine=sync, iodepth=1
fio-2.2.7-24-g7c30
Starting 1 process
dedupe: Laying out IO file(s) (1 file(s) / 2MB)
dedupe: (groupid=0, jobs=1): err= 0: pid=31497: Mon Apr 27 09:13:35 2015
write: io=2048.0KB, bw=2000.0MB/s, iops=8000, runt= 1msec
clat (usec): min=123, max=183, avg=150.50, stdev=22.35
lat (usec): min=125, max=184, avg=152.38, stdev=22.08
clat percentiles (usec):
| 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 124],
| 30.00th=[ 139], 40.00th=[ 145], 50.00th=[ 145], 60.00th=[ 155],
| 70.00th=[ 159], 80.00th=[ 177], 90.00th=[ 183], 95.00th=[ 183],
| 99.00th=[ 183], 99.50th=[ 183], 99.90th=[ 183], 99.95th=[ 183],
| 99.99th=[ 183]
lat (usec) : 250=100.00%
cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=8/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=2048KB, aggrb=2000.0MB/s, minb=2000.0MB/s,
maxb=2000.0MB/s, mint=1msec, maxt=1msec
Disk stats (read/write):
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
# ls -lh test.tmp
-rw-r--r-- 1 root root 2.0M Apr 27 09:13 test.tmp
# cat test.tmp.log
fio version 2 iolog
test.tmp add
test.tmp open
test.tmp write 0 262144
test.tmp write 262144 262144
test.tmp write 524288 262144
test.tmp write 786432 262144
test.tmp write 1048576 262144
test.tmp write 1310720 262144
test.tmp write 1572864 262144
test.tmp write 1835008 262144
test.tmp close
# for each in {0..7}; do dd if=test.tmp bs=262144 count=1 skip=$each
2>/dev/null | hexdump -C | md5sum; done
71a1660503bcff7c4e20a763d569d069 -
9c9bb7ec1020b4d4249028aecc896e6b -
68b9685812d47c822532854201c9b352 -
e5c8ef471a27ba92b86893ee5ded654b -
14e0e798a8af3f4e6abdaf022ddf91c3 -
85528ae970bd25dde8c39ecaaffa4cf3 -
60b8ccf0e0793094b9356544fb541f3a -
ef736cc9cbf7588cb7b84467cb37c44e -
# fio -v
fio-2.2.7-24-g7c30
--
Srinivasa R Chamarthy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: FIO - dedup checksums or specified blocksize does not match.
2015-04-27 13:18 FIO - dedup checksums or specified blocksize does not match Srinivasa Chamarthy
@ 2015-04-27 14:39 ` Jens Axboe
2015-04-28 5:15 ` Srinivasa Chamarthy
0 siblings, 1 reply; 5+ messages in thread
From: Jens Axboe @ 2015-04-27 14:39 UTC (permalink / raw)
To: Srinivasa Chamarthy, fio
On 04/27/2015 07:18 AM, Srinivasa Chamarthy wrote:
> I was just verifying if i could generate 100% duplicable data with
> FIO. I have configured small workload with bs of 256k and writing 2MB
> of file. I tried to get the checksum of each of 256k blocks of data
> from the file and the checksums do not match. If i am not wrong, when
> i specify data as 100% deduppable, my checksums should match isn't it?
>
> # cat ddp_file.fio
> [dedupe]
> filename=test.tmp
> bs=256k
> rw=write
> size=2m
> dedupe_percentage=100
> write_iolog=test.tmp.log
>
> # fio ddp_file.fio
> dedupe: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K,
> ioengine=sync, iodepth=1
> fio-2.2.7-24-g7c30
> Starting 1 process
> dedupe: Laying out IO file(s) (1 file(s) / 2MB)
>
> dedupe: (groupid=0, jobs=1): err= 0: pid=31497: Mon Apr 27 09:13:35 2015
> write: io=2048.0KB, bw=2000.0MB/s, iops=8000, runt= 1msec
> clat (usec): min=123, max=183, avg=150.50, stdev=22.35
> lat (usec): min=125, max=184, avg=152.38, stdev=22.08
> clat percentiles (usec):
> | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 124],
> | 30.00th=[ 139], 40.00th=[ 145], 50.00th=[ 145], 60.00th=[ 155],
> | 70.00th=[ 159], 80.00th=[ 177], 90.00th=[ 183], 95.00th=[ 183],
> | 99.00th=[ 183], 99.50th=[ 183], 99.90th=[ 183], 99.95th=[ 183],
> | 99.99th=[ 183]
> lat (usec) : 250=100.00%
> cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=28
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued : total=r=0/w=8/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: io=2048KB, aggrb=2000.0MB/s, minb=2000.0MB/s,
> maxb=2000.0MB/s, mint=1msec, maxt=1msec
>
> Disk stats (read/write):
> sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>
> # ls -lh test.tmp
> -rw-r--r-- 1 root root 2.0M Apr 27 09:13 test.tmp
>
> # cat test.tmp.log
> fio version 2 iolog
> test.tmp add
> test.tmp open
> test.tmp write 0 262144
> test.tmp write 262144 262144
> test.tmp write 524288 262144
> test.tmp write 786432 262144
> test.tmp write 1048576 262144
> test.tmp write 1310720 262144
> test.tmp write 1572864 262144
> test.tmp write 1835008 262144
> test.tmp close
>
> # for each in {0..7}; do dd if=test.tmp bs=262144 count=1 skip=$each
> 2>/dev/null | hexdump -C | md5sum; done
> 71a1660503bcff7c4e20a763d569d069 -
> 9c9bb7ec1020b4d4249028aecc896e6b -
> 68b9685812d47c822532854201c9b352 -
> e5c8ef471a27ba92b86893ee5ded654b -
> 14e0e798a8af3f4e6abdaf022ddf91c3 -
> 85528ae970bd25dde8c39ecaaffa4cf3 -
> 60b8ccf0e0793094b9356544fb541f3a -
> ef736cc9cbf7588cb7b84467cb37c44e -
>
> # fio -v
> fio-2.2.7-24-g7c30
Can you try with current -git? The corner cases of being 100% dedupable
was broken.
--
Jens Axboe
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: FIO - dedup checksums or specified blocksize does not match.
2015-04-27 14:39 ` Jens Axboe
@ 2015-04-28 5:15 ` Srinivasa Chamarthy
2015-04-28 7:03 ` Srinivasa Chamarthy
0 siblings, 1 reply; 5+ messages in thread
From: Srinivasa Chamarthy @ 2015-04-28 5:15 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Seems working now. Thanks for the great support.
for each in {0..7}; do dd if=test.tmp.comp bs=262144 count=1
skip=$each 2>/dev/null | hexdump -C | md5sum; done
e1d3c034e3fc15481e5c8610333ad9cd -
e1d3c034e3fc15481e5c8610333ad9cd -
e1d3c034e3fc15481e5c8610333ad9cd -
e1d3c034e3fc15481e5c8610333ad9cd -
e1d3c034e3fc15481e5c8610333ad9cd -
e1d3c034e3fc15481e5c8610333ad9cd -
e1d3c034e3fc15481e5c8610333ad9cd -
e1d3c034e3fc15481e5c8610333ad9cd -
Srinivasa R Chamarthy
On Mon, Apr 27, 2015 at 10:39 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 04/27/2015 07:18 AM, Srinivasa Chamarthy wrote:
>>
>> I was just verifying if i could generate 100% duplicable data with
>> FIO. I have configured small workload with bs of 256k and writing 2MB
>> of file. I tried to get the checksum of each of 256k blocks of data
>> from the file and the checksums do not match. If i am not wrong, when
>> i specify data as 100% deduppable, my checksums should match isn't it?
>>
>> # cat ddp_file.fio
>> [dedupe]
>> filename=test.tmp
>> bs=256k
>> rw=write
>> size=2m
>> dedupe_percentage=100
>> write_iolog=test.tmp.log
>>
>> # fio ddp_file.fio
>> dedupe: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K,
>> ioengine=sync, iodepth=1
>> fio-2.2.7-24-g7c30
>> Starting 1 process
>> dedupe: Laying out IO file(s) (1 file(s) / 2MB)
>>
>> dedupe: (groupid=0, jobs=1): err= 0: pid=31497: Mon Apr 27 09:13:35 2015
>> write: io=2048.0KB, bw=2000.0MB/s, iops=8000, runt= 1msec
>> clat (usec): min=123, max=183, avg=150.50, stdev=22.35
>> lat (usec): min=125, max=184, avg=152.38, stdev=22.08
>> clat percentiles (usec):
>> | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[
>> 124],
>> | 30.00th=[ 139], 40.00th=[ 145], 50.00th=[ 145], 60.00th=[
>> 155],
>> | 70.00th=[ 159], 80.00th=[ 177], 90.00th=[ 183], 95.00th=[
>> 183],
>> | 99.00th=[ 183], 99.50th=[ 183], 99.90th=[ 183], 99.95th=[
>> 183],
>> | 99.99th=[ 183]
>> lat (usec) : 250=100.00%
>> cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=28
>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=8/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
>> latency : target=0, window=0, percentile=100.00%, depth=1
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2048KB, aggrb=2000.0MB/s, minb=2000.0MB/s,
>> maxb=2000.0MB/s, mint=1msec, maxt=1msec
>>
>> Disk stats (read/write):
>> sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>>
>> # ls -lh test.tmp
>> -rw-r--r-- 1 root root 2.0M Apr 27 09:13 test.tmp
>>
>> # cat test.tmp.log
>> fio version 2 iolog
>> test.tmp add
>> test.tmp open
>> test.tmp write 0 262144
>> test.tmp write 262144 262144
>> test.tmp write 524288 262144
>> test.tmp write 786432 262144
>> test.tmp write 1048576 262144
>> test.tmp write 1310720 262144
>> test.tmp write 1572864 262144
>> test.tmp write 1835008 262144
>> test.tmp close
>>
>> # for each in {0..7}; do dd if=test.tmp bs=262144 count=1 skip=$each
>> 2>/dev/null | hexdump -C | md5sum; done
>> 71a1660503bcff7c4e20a763d569d069 -
>> 9c9bb7ec1020b4d4249028aecc896e6b -
>> 68b9685812d47c822532854201c9b352 -
>> e5c8ef471a27ba92b86893ee5ded654b -
>> 14e0e798a8af3f4e6abdaf022ddf91c3 -
>> 85528ae970bd25dde8c39ecaaffa4cf3 -
>> 60b8ccf0e0793094b9356544fb541f3a -
>> ef736cc9cbf7588cb7b84467cb37c44e -
>>
>> # fio -v
>> fio-2.2.7-24-g7c30
>
>
> Can you try with current -git? The corner cases of being 100% dedupable was
> broken.
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: FIO - dedup checksums or specified blocksize does not match.
2015-04-28 5:15 ` Srinivasa Chamarthy
@ 2015-04-28 7:03 ` Srinivasa Chamarthy
2015-04-28 10:54 ` Srinivasa Chamarthy
0 siblings, 1 reply; 5+ messages in thread
From: Srinivasa Chamarthy @ 2015-04-28 7:03 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
There seems to be one more issue with dedup for data that is not 100%
dedupable. I tried with 50% and 80% and it give only 35 for 50 and 60
for 80.
# cat ddp_file.fio
[dedupe]
filename=test.tmp.comp
bs=256k
rw=write
size=10m
dedupe_percentage=80
write_iolog=test.tmp.log.comp
# fio ddp_file.fio
dedupe: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K,
ioengine=sync, iodepth=1
fio-2.2.7-26-g9451b
Starting 1 process
dedupe: Laying out IO file(s) (1 file(s) / 10MB)
dedupe: (groupid=0, jobs=1): err= 0: pid=13376: Tue Apr 28 02:54:02 2015
write: io=10240KB, bw=731429KB/s, iops=2857, runt= 14msec
clat (usec): min=170, max=374, avg=235.80, stdev=41.11
lat (usec): min=173, max=378, avg=239.10, stdev=41.75
clat percentiles (usec):
| 1.00th=[ 171], 5.00th=[ 175], 10.00th=[ 197], 20.00th=[ 213],
| 30.00th=[ 217], 40.00th=[ 221], 50.00th=[ 231], 60.00th=[ 235],
| 70.00th=[ 239], 80.00th=[ 253], 90.00th=[ 262], 95.00th=[ 318],
| 99.00th=[ 374], 99.50th=[ 374], 99.90th=[ 374], 99.95th=[ 374],
| 99.99th=[ 374]
lat (usec) : 250=77.50%, 500=22.50%
cpu : usr=57.14%, sys=28.57%, ctx=1, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=40/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=10240KB, aggrb=731428KB/s, minb=731428KB/s,
maxb=731428KB/s, mint=14msec, maxt=14msec
Disk stats (read/write):
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
# fio/t/fio-dedupe -b 262144 test.tmp.comp
Will check <test.tmp.comp>, size <10485760>, using 8 threads
Threads(8): 40 items processed
Extents=40, Unique extents=15
De-dupe ratio: 1:1.67
Fio setting: dedupe_percentage=63
I also confirmed the same by taking checksum of the data file by
individual blocks of size bs.
# for each in {0..39}; do dd if=test.tmp.comp bs=262144 count=1
skip=$each 2>/dev/null | hexdump -C | md5sum; done | wc -l
40 <<< have 40 blocks as expected.
# for each in {0..39}; do dd if=test.tmp.comp bs=262144 count=1
skip=$each 2>/dev/null | hexdump -C | md5sum; done | sort | uniq | wc
-l
16 <<< returns 16 unique blocks
In a 80% dedupable size, i would expect around 8 unique blocks. Is that true.?
Also, from the fio/t/fio-dedupe output, it shows that there are only
15 unique extents. Checking manually returns 16.
Thanks,
Srinivasa Chamarthy
Srinivasa R Chamarthy
On Tue, Apr 28, 2015 at 1:15 PM, Srinivasa Chamarthy
<chamarthy.raju@gmail.com> wrote:
> Seems working now. Thanks for the great support.
>
> for each in {0..7}; do dd if=test.tmp.comp bs=262144 count=1
> skip=$each 2>/dev/null | hexdump -C | md5sum; done
> e1d3c034e3fc15481e5c8610333ad9cd -
> e1d3c034e3fc15481e5c8610333ad9cd -
> e1d3c034e3fc15481e5c8610333ad9cd -
> e1d3c034e3fc15481e5c8610333ad9cd -
> e1d3c034e3fc15481e5c8610333ad9cd -
> e1d3c034e3fc15481e5c8610333ad9cd -
> e1d3c034e3fc15481e5c8610333ad9cd -
> e1d3c034e3fc15481e5c8610333ad9cd -
> Srinivasa R Chamarthy
>
>
> On Mon, Apr 27, 2015 at 10:39 PM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 04/27/2015 07:18 AM, Srinivasa Chamarthy wrote:
>>>
>>> I was just verifying if i could generate 100% duplicable data with
>>> FIO. I have configured small workload with bs of 256k and writing 2MB
>>> of file. I tried to get the checksum of each of 256k blocks of data
>>> from the file and the checksums do not match. If i am not wrong, when
>>> i specify data as 100% deduppable, my checksums should match isn't it?
>>>
>>> # cat ddp_file.fio
>>> [dedupe]
>>> filename=test.tmp
>>> bs=256k
>>> rw=write
>>> size=2m
>>> dedupe_percentage=100
>>> write_iolog=test.tmp.log
>>>
>>> # fio ddp_file.fio
>>> dedupe: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K,
>>> ioengine=sync, iodepth=1
>>> fio-2.2.7-24-g7c30
>>> Starting 1 process
>>> dedupe: Laying out IO file(s) (1 file(s) / 2MB)
>>>
>>> dedupe: (groupid=0, jobs=1): err= 0: pid=31497: Mon Apr 27 09:13:35 2015
>>> write: io=2048.0KB, bw=2000.0MB/s, iops=8000, runt= 1msec
>>> clat (usec): min=123, max=183, avg=150.50, stdev=22.35
>>> lat (usec): min=125, max=184, avg=152.38, stdev=22.08
>>> clat percentiles (usec):
>>> | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[
>>> 124],
>>> | 30.00th=[ 139], 40.00th=[ 145], 50.00th=[ 145], 60.00th=[
>>> 155],
>>> | 70.00th=[ 159], 80.00th=[ 177], 90.00th=[ 183], 95.00th=[
>>> 183],
>>> | 99.00th=[ 183], 99.50th=[ 183], 99.90th=[ 183], 99.95th=[
>>> 183],
>>> | 99.99th=[ 183]
>>> lat (usec) : 250=100.00%
>>> cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=28
>>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> issued : total=r=0/w=8/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
>>> latency : target=0, window=0, percentile=100.00%, depth=1
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2048KB, aggrb=2000.0MB/s, minb=2000.0MB/s,
>>> maxb=2000.0MB/s, mint=1msec, maxt=1msec
>>>
>>> Disk stats (read/write):
>>> sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>>>
>>> # ls -lh test.tmp
>>> -rw-r--r-- 1 root root 2.0M Apr 27 09:13 test.tmp
>>>
>>> # cat test.tmp.log
>>> fio version 2 iolog
>>> test.tmp add
>>> test.tmp open
>>> test.tmp write 0 262144
>>> test.tmp write 262144 262144
>>> test.tmp write 524288 262144
>>> test.tmp write 786432 262144
>>> test.tmp write 1048576 262144
>>> test.tmp write 1310720 262144
>>> test.tmp write 1572864 262144
>>> test.tmp write 1835008 262144
>>> test.tmp close
>>>
>>> # for each in {0..7}; do dd if=test.tmp bs=262144 count=1 skip=$each
>>> 2>/dev/null | hexdump -C | md5sum; done
>>> 71a1660503bcff7c4e20a763d569d069 -
>>> 9c9bb7ec1020b4d4249028aecc896e6b -
>>> 68b9685812d47c822532854201c9b352 -
>>> e5c8ef471a27ba92b86893ee5ded654b -
>>> 14e0e798a8af3f4e6abdaf022ddf91c3 -
>>> 85528ae970bd25dde8c39ecaaffa4cf3 -
>>> 60b8ccf0e0793094b9356544fb541f3a -
>>> ef736cc9cbf7588cb7b84467cb37c44e -
>>>
>>> # fio -v
>>> fio-2.2.7-24-g7c30
>>
>>
>> Can you try with current -git? The corner cases of being 100% dedupable was
>> broken.
>>
>> --
>> Jens Axboe
>>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: FIO - dedup checksums or specified blocksize does not match.
2015-04-28 7:03 ` Srinivasa Chamarthy
@ 2015-04-28 10:54 ` Srinivasa Chamarthy
0 siblings, 0 replies; 5+ messages in thread
From: Srinivasa Chamarthy @ 2015-04-28 10:54 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
It may be a problem with using smaller io size. If we increase the
size of the io, i think its getting near to what is expected.
# cat ddp_file.fio
[dedupe]
filename=test.tmp.comp
bs=32k
rw=write
size=1g
dedupe_percentage=80
write_iolog=test.tmp.log.comp
# fio/t/fio-dedupe -b 32768 test.tmp.comp
Will check <test.tmp.comp>, size <1073741824>, using 8 threads
Threads(8): 32768 items processed
Extents=32768, Unique extents=6623
De-dupe ratio: 1:3.95
Fio setting: dedupe_percentage=80
Srinivasa R Chamarthy
On Tue, Apr 28, 2015 at 3:03 PM, Srinivasa Chamarthy
<chamarthy.raju@gmail.com> wrote:
> There seems to be one more issue with dedup for data that is not 100%
> dedupable. I tried with 50% and 80% and it give only 35 for 50 and 60
> for 80.
>
> # cat ddp_file.fio
> [dedupe]
> filename=test.tmp.comp
> bs=256k
> rw=write
> size=10m
> dedupe_percentage=80
> write_iolog=test.tmp.log.comp
>
> # fio ddp_file.fio
> dedupe: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K,
> ioengine=sync, iodepth=1
> fio-2.2.7-26-g9451b
> Starting 1 process
> dedupe: Laying out IO file(s) (1 file(s) / 10MB)
>
> dedupe: (groupid=0, jobs=1): err= 0: pid=13376: Tue Apr 28 02:54:02 2015
> write: io=10240KB, bw=731429KB/s, iops=2857, runt= 14msec
> clat (usec): min=170, max=374, avg=235.80, stdev=41.11
> lat (usec): min=173, max=378, avg=239.10, stdev=41.75
> clat percentiles (usec):
> | 1.00th=[ 171], 5.00th=[ 175], 10.00th=[ 197], 20.00th=[ 213],
> | 30.00th=[ 217], 40.00th=[ 221], 50.00th=[ 231], 60.00th=[ 235],
> | 70.00th=[ 239], 80.00th=[ 253], 90.00th=[ 262], 95.00th=[ 318],
> | 99.00th=[ 374], 99.50th=[ 374], 99.90th=[ 374], 99.95th=[ 374],
> | 99.99th=[ 374]
> lat (usec) : 250=77.50%, 500=22.50%
> cpu : usr=57.14%, sys=28.57%, ctx=1, majf=0, minf=27
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued : total=r=0/w=40/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs):
> WRITE: io=10240KB, aggrb=731428KB/s, minb=731428KB/s,
> maxb=731428KB/s, mint=14msec, maxt=14msec
>
> Disk stats (read/write):
> sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>
> # fio/t/fio-dedupe -b 262144 test.tmp.comp
> Will check <test.tmp.comp>, size <10485760>, using 8 threads
> Threads(8): 40 items processed
> Extents=40, Unique extents=15
> De-dupe ratio: 1:1.67
> Fio setting: dedupe_percentage=63
>
> I also confirmed the same by taking checksum of the data file by
> individual blocks of size bs.
>
> # for each in {0..39}; do dd if=test.tmp.comp bs=262144 count=1
> skip=$each 2>/dev/null | hexdump -C | md5sum; done | wc -l
> 40 <<< have 40 blocks as expected.
>
> # for each in {0..39}; do dd if=test.tmp.comp bs=262144 count=1
> skip=$each 2>/dev/null | hexdump -C | md5sum; done | sort | uniq | wc
> -l
> 16 <<< returns 16 unique blocks
>
> In a 80% dedupable size, i would expect around 8 unique blocks. Is that true.?
> Also, from the fio/t/fio-dedupe output, it shows that there are only
> 15 unique extents. Checking manually returns 16.
>
> Thanks,
> Srinivasa Chamarthy
> Srinivasa R Chamarthy
>
>
> On Tue, Apr 28, 2015 at 1:15 PM, Srinivasa Chamarthy
> <chamarthy.raju@gmail.com> wrote:
>> Seems working now. Thanks for the great support.
>>
>> for each in {0..7}; do dd if=test.tmp.comp bs=262144 count=1
>> skip=$each 2>/dev/null | hexdump -C | md5sum; done
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> e1d3c034e3fc15481e5c8610333ad9cd -
>> Srinivasa R Chamarthy
>>
>>
>> On Mon, Apr 27, 2015 at 10:39 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>> On 04/27/2015 07:18 AM, Srinivasa Chamarthy wrote:
>>>>
>>>> I was just verifying if i could generate 100% duplicable data with
>>>> FIO. I have configured small workload with bs of 256k and writing 2MB
>>>> of file. I tried to get the checksum of each of 256k blocks of data
>>>> from the file and the checksums do not match. If i am not wrong, when
>>>> i specify data as 100% deduppable, my checksums should match isn't it?
>>>>
>>>> # cat ddp_file.fio
>>>> [dedupe]
>>>> filename=test.tmp
>>>> bs=256k
>>>> rw=write
>>>> size=2m
>>>> dedupe_percentage=100
>>>> write_iolog=test.tmp.log
>>>>
>>>> # fio ddp_file.fio
>>>> dedupe: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K,
>>>> ioengine=sync, iodepth=1
>>>> fio-2.2.7-24-g7c30
>>>> Starting 1 process
>>>> dedupe: Laying out IO file(s) (1 file(s) / 2MB)
>>>>
>>>> dedupe: (groupid=0, jobs=1): err= 0: pid=31497: Mon Apr 27 09:13:35 2015
>>>> write: io=2048.0KB, bw=2000.0MB/s, iops=8000, runt= 1msec
>>>> clat (usec): min=123, max=183, avg=150.50, stdev=22.35
>>>> lat (usec): min=125, max=184, avg=152.38, stdev=22.08
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[
>>>> 124],
>>>> | 30.00th=[ 139], 40.00th=[ 145], 50.00th=[ 145], 60.00th=[
>>>> 155],
>>>> | 70.00th=[ 159], 80.00th=[ 177], 90.00th=[ 183], 95.00th=[
>>>> 183],
>>>> | 99.00th=[ 183], 99.50th=[ 183], 99.90th=[ 183], 99.95th=[
>>>> 183],
>>>> | 99.99th=[ 183]
>>>> lat (usec) : 250=100.00%
>>>> cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=28
>>>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>>> >=64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> issued : total=r=0/w=8/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
>>>> latency : target=0, window=0, percentile=100.00%, depth=1
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2048KB, aggrb=2000.0MB/s, minb=2000.0MB/s,
>>>> maxb=2000.0MB/s, mint=1msec, maxt=1msec
>>>>
>>>> Disk stats (read/write):
>>>> sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>>>>
>>>> # ls -lh test.tmp
>>>> -rw-r--r-- 1 root root 2.0M Apr 27 09:13 test.tmp
>>>>
>>>> # cat test.tmp.log
>>>> fio version 2 iolog
>>>> test.tmp add
>>>> test.tmp open
>>>> test.tmp write 0 262144
>>>> test.tmp write 262144 262144
>>>> test.tmp write 524288 262144
>>>> test.tmp write 786432 262144
>>>> test.tmp write 1048576 262144
>>>> test.tmp write 1310720 262144
>>>> test.tmp write 1572864 262144
>>>> test.tmp write 1835008 262144
>>>> test.tmp close
>>>>
>>>> # for each in {0..7}; do dd if=test.tmp bs=262144 count=1 skip=$each
>>>> 2>/dev/null | hexdump -C | md5sum; done
>>>> 71a1660503bcff7c4e20a763d569d069 -
>>>> 9c9bb7ec1020b4d4249028aecc896e6b -
>>>> 68b9685812d47c822532854201c9b352 -
>>>> e5c8ef471a27ba92b86893ee5ded654b -
>>>> 14e0e798a8af3f4e6abdaf022ddf91c3 -
>>>> 85528ae970bd25dde8c39ecaaffa4cf3 -
>>>> 60b8ccf0e0793094b9356544fb541f3a -
>>>> ef736cc9cbf7588cb7b84467cb37c44e -
>>>>
>>>> # fio -v
>>>> fio-2.2.7-24-g7c30
>>>
>>>
>>> Can you try with current -git? The corner cases of being 100% dedupable was
>>> broken.
>>>
>>> --
>>> Jens Axboe
>>>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-04-28 10:54 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-27 13:18 FIO - dedup checksums or specified blocksize does not match Srinivasa Chamarthy
2015-04-27 14:39 ` Jens Axboe
2015-04-28 5:15 ` Srinivasa Chamarthy
2015-04-28 7:03 ` Srinivasa Chamarthy
2015-04-28 10:54 ` Srinivasa Chamarthy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.