Large no.of extents for a single small file with a fresh RAID5 setup

Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

Hi,

With RAID5, using 3 loop-back devices each has 4GB. On a fresh setup, created a 2mb file.
While checking no.of extents,it shows like:

# ./btrfs-debugfs -f /mnt/file2m.txt 
(257 0): ram 110592 disk 145227776 disk_size 114688
(257 110592): ram 4096 disk 145461248 disk_size 4096
(257 114688): ram 114688 disk 145342464 disk_size 118784
(257 229376): ram 712704 disk 145620992 disk_size 716800
(257 942080): ram 4096 disk 145096704 disk_size 4096
(257 946176): ram 737280 disk 146407424 disk_size 741376
(257 1683456): ram 413696 disk 147193856 disk_size 413696
file: /mnt/file2m.txt extents 7 disk size 2113536 logical size 2097152 ratio 0.99

Is this expected behaviour? Is there any mount option to minimize no.of extents for
a smaller file?

Cheers.
Lakshmipathi.G

Re: Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

On Mon, Feb 20, 2017 at 03:53:15PM +0800, Qu Wenruo wrote:
> 
> 
> At 02/18/2017 10:07 AM, Lakshmipathi.G wrote:
> >Hi,
> >
> >With RAID5, using 3 loop-back devices each has 4GB. On a fresh setup, created a 2mb file.
> >While checking no.of extents,it shows like:
> >
> ># ./btrfs-debugfs -f /mnt/file2m.txt
> >(257 0): ram 110592 disk 145227776 disk_size 114688
> >(257 110592): ram 4096 disk 145461248 disk_size 4096
> >(257 114688): ram 114688 disk 145342464 disk_size 118784
> >(257 229376): ram 712704 disk 145620992 disk_size 716800
> >(257 942080): ram 4096 disk 145096704 disk_size 4096
> >(257 946176): ram 737280 disk 146407424 disk_size 741376
> >(257 1683456): ram 413696 disk 147193856 disk_size 413696
> >file: /mnt/file2m.txt extents 7 disk size 2113536 logical size 2097152 ratio 0.99
> >
> >Is this expected behaviour? Is there any mount option to minimize no.of extents for
> >a smaller file?
> 
> What mount option did you use?
> Are you using compression?

Used the default mount options (mount /dev/loop0 /mnt) and compression not enabled.

> How did you create the 2m file?

Yes,this is the problem. script invokes 'dd' with 'notrunc' for each
byte. If the file is 1kb, 'dd' invoked 1024 times with notrunc.
This seems to be creating the issue.

Tried using open-fd (exec FD<>/path/f, write finally close FD) method
even this runs into possible sync and creates multiple extents.

so finally found a simple trick, created the file outside btrfs then copy it
into btrfs-mnt-pt :) this workaround helps to solve the script file-creation
issue.

> 
> This seems quite strange, if you go through normal buffered write without
> compression, 2m can easily be embedded into one file extent.
> 
> And according to extent at offset 0, 112K, 224K, its data extent is larger
> than its file extent, which means it goes through CoW.
> 
> Either you did fsync/sync between writes or you goes through Direct IO.
> 
> Anyway, the way you create the 2M file is quite important here now.
> 
> Thanks,
> Qu
> >

thanks for the help.

Cheers.
Lakshmipathi.G

Re: Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

> >>How did you create the 2m file?
> >
> >Yes,this is the problem. script invokes 'dd' with 'notrunc' for each
> >byte. If the file is 1kb, 'dd' invoked 1024 times with notrunc.
> >This seems to be creating the issue.
> 
> IIRC that's not the direct cause though.
> 
> Since kernel is using delayed allocation, each time we trigger buffered
> write, kernel just info fs to do accounting and copy the data into page
> cache, no real write is triggered.
> 
> Only sync/fsync and memory pressure will make us to write pages into disc,
> and until then we allocate space for them.
> 
> So unless you're trying such operation on a busy server which triggered
> several sync/fsync/memory pressure during the several seconds of dd, it
> won't cause so many fragments.
> 
> I also tried your dd bs=1 method, no problem and still one single extent.
> 
> Thanks,
> Qu
I'm using Ubuntu(16.04) desktop version (not server) running Xorg and others. May be its possible its
flushing data to disk at constant time. If you like to give it a try again with servers, below is the
exact script along with its timing & multiextent output.

one more curious thing, is it fine to have extents with size 4096 on RAID5 setup?

Cheers.
Lakshmipathi.G

#time /root/check_extents.sh f2MB 32
real	1m30.284s
user	1m22.796s
sys	0m39.324s

laks/btrfs-progs# ./btrfs-debugfs -f tests/mnt/f2MB
(263 0): ram 110592 disk 151015424 disk_size 114688
(263 110592): ram 4096 disk 147103744 disk_size 4096
(263 114688): ram 110592 disk 151130112 disk_size 114688
(263 225280): ram 696320 disk 151257088 disk_size 700416
(263 921600): ram 4096 disk 147107840 disk_size 4096
(263 925696): ram 700416 disk 152043520 disk_size 704512
(263 1626112): ram 4096 disk 147111936 disk_size 4096
(263 1630208): ram 344064 disk 153223168 disk_size 348160
(263 1974272): ram 4096 disk 147116032 disk_size 4096
(263 1978368): ram 118784 disk 152748032 disk_size 118784
file: tests/mnt/f2MB extents 10 disk size 2117632 logical size 2097152 ratio 0.99

--
$cat /root/check_extents.sh 
#!/bin/bash
#$1 Filename
#$2 Expected no.of data stripes for the file.

create_layout(){
        fname=$1
        size=$(( $2 * 65536 ))
        n=0
        bs_value=1
        stripe=0
        while (( $n < $size ))
        do
                if [ $(( $n % 65536 )) -eq 0 ]; then
                        val='D'$stripe
                        echo -n $val
                        stripe=$(( $stripe+1 ))
                        # ensure proper value           
                        bs_value=`echo "${#val}"` 
                else
                        echo -n 'x'
                        bs_value=1
                fi
        n=$(( $n+$bs_value ))
        done | dd of="/home/laks/btrfs-progs/tests/mnt/$fname" bs=$bs_value conv=notrunc &> /dev/null
	##EDIT above hard-coded path
}
create_layout $1 $2
--

Re: Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

> 
> >I'm using Ubuntu(16.04) desktop version (not server) running Xorg and others. May be its possible its
> >flushing data to disk at constant time. If you like to give it a try again with servers, below is the
> >exact script along with its timing & multiextent output.
> 
> Archlinux here, with my btrfs vm and browser and music player, still one
> extent.
> 
looks like I have better setup to produce single-extent files than yours :-)
> Are you under heavy memory usage?
> 
no, not as per top and free. 'top' shows mem.usage as:
KiB Mem :  7092516 total,  5961796 free,   794836 used,   335884 buff/cache

where 'compiz' process uses around 2.5% of memory and 'free' command reports
almost 5gb as free. May be some 'sysctl' or others needs to be adjusted on my setup?
> >
> >one more curious thing, is it fine to have extents with size 4096 on RAID5 setup?
> >
> 
> Sure, why not?
> 
> Any size aligned to sector size is valid, no matter the profile.
> 
> For 4K extent, RAID5/6 will do read-modify-write, which is very common.
> 
> Thanks,
> Qu
> 
While looking into mapping from logical->physical chunks on RAID5. I was wondering 
how these 4KB extents will fit into 64KB data-stripe?

I didn't have idea about read-modify-write, just found few wiki links on it, will 
check. thanks

Cheers.
Lakshmipathi.G