Re: Problem in EROFS: Not able to read the files after mount

[Gao Xiang via Linux-erofs <linux-erofs@lists.ozlabs.org>]

Hi Saumya,

On Wed, Jan 29, 2020 at 09:43:37AM +0530, Saumya Panda wrote:
> Hi Gao,
>   How you got the read amplification? I ran FIO on enwik9 (both Erofs and
> SquashFs) and got the below output. Is there anyway to calculate the read
> amplification from the below logs.

No. FIO doesn't provide such number as far as I know, you'd get all statistic
number by some block device information.

BTW, I'd suggest you umount, drop_caches, mount the filesystem every time
in order to get rid of filesystem itself internal cache and bdev buffer
cache. It would not be useful in real scenarios except for benchmark use
only.

https://github.com/erofs/erofs-openbenchmark/

Thanks,
Gao Xiang

> 
>      Here filename (/mnt/enwik9_erofs/enwik9, /mnt/enwiki_sqfs/enwik9)
> points to the mounted readonly file system(squasfs, erofs). But if I give
> directory as a parameter instead of filename I am getting error(see the
> logs at the end).
> 
> *FIO on Erofs:*
> 
> localhost:~> fio --name=randread --ioengine=libaio --iodepth=16
> --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4 --runtime=240
> --group_reporting --filename=/mnt/enwik9_erofs/enwik9
> 
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=16
> 
> ...
> 
> fio-3.17-90-gd9b7
> 
> Starting 4 processes
> 
> Jobs: 4 (f=4): [r(4)][100.0%][r=381MiB/s][r=97.6k IOPS][eta 00m:00s]
> 
> randread: (groupid=0, jobs=4): err= 0: pid=34282: Mon Jan 27 01:04:55 2020
> 
>   read: IOPS=36.7k, BW=144MiB/s (150MB/s)(2048MiB/14271msec)
> 
>     slat (nsec): min=1305, max=135688k, avg=106650.48, stdev=493480.73
> 
>     clat (nsec): min=1970, max=136593k, avg=1629459.90, stdev=2639786.83
> 
>      lat (usec): min=3, max=136625, avg=1736.29, stdev=2772.32
> 
>     clat percentiles (usec):
> 
>      |  1.00th=[    48],  5.00th=[    69], 10.00th=[   251], 20.00th=[
> 437],
> 
>      | 30.00th=[   570], 40.00th=[   701], 50.00th=[   848], 60.00th=[
> 1029],
> 
>      | 70.00th=[  1336], 80.00th=[  2147], 90.00th=[  4015], 95.00th=[
> 5932],
> 
>      | 99.00th=[ 11600], 99.50th=[ 13304], 99.90th=[ 17171], 99.95th=[
> 20579],
> 
>      | 99.99th=[135267]
> 
>    bw (  KiB/s): min=16510, max=295435, per=76.91%, avg=113025.79,
> stdev=23830.42, samples=112
> 
>    iops        : min= 4126, max=73857, avg=28254.82, stdev=5957.62,
> samples=112
> 
>   lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.37%
> 
>   lat (usec)   : 100=5.45%, 250=3.15%, 500=14.74%, 750=18.99%, 1000=14.99%
> 
>   lat (msec)   : 2=20.14%, 4=11.09%, 10=8.42%, 20=1.62%, 50=0.04%
> 
>   lat (msec)   : 250=0.01%
> 
>   cpu          : usr=1.87%, sys=8.28%, ctx=144023, majf=1, minf=114
> 
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
> >=64=0.0%
> 
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> 
>      latency   : target=0, window=0, percentile=100.00%, depth=16
> 
> 
> 
> Run status group 0 (all jobs):
> 
>    READ: bw=144MiB/s (150MB/s), 144MiB/s-144MiB/s (150MB/s-150MB/s),
> io=2048MiB (2147MB), run=14271-14271msec
> 
> 
> 
> Disk stats (read/write):
> 
>   loop0: ios=137357/0, merge=0/0, ticks=23020/0, in_queue=460, util=97.70%
> 
> 
> *FIO on SquashFs:*
> 
> 
> localhost:~/Downloads/erofs-utils> fio --name=randread --ioengine=libaio
> --iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4
> --runtime=240 --group_reporting --filename=/mnt/enwik9_sqsh/enwik9
> 
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=16
> 
> ...
> 
> fio-3.17-90-gd9b7
> 
> Starting 4 processes
> 
> Jobs: 4 (f=4): [r(4)][66.7%][r=1175MiB/s][r=301k IOPS][eta 00m:05s]
> 
> randread: (groupid=0, jobs=4): err= 0: pid=34389: Mon Jan 27 01:07:56 2020
> 
>   read: IOPS=55.4k, BW=216MiB/s (227MB/s)(2048MiB/9467msec)
> 
>     slat (nsec): min=1194, max=61065k, avg=67581.76, stdev=754174.73
> 
>     clat (usec): min=2, max=222014, avg=1075.25, stdev=5969.94
> 
>      lat (usec): min=3, max=235437, avg=1143.13, stdev=6341.32
> 
>     clat percentiles (usec):
> 
>      |  1.00th=[    39],  5.00th=[    40], 10.00th=[    40], 20.00th=[
> 41],
> 
>      | 30.00th=[    42], 40.00th=[    43], 50.00th=[    43], 60.00th=[
> 44],
> 
>      | 70.00th=[    45], 80.00th=[    48], 90.00th=[    63], 95.00th=[
> 3163],
> 
>      | 99.00th=[ 28443], 99.50th=[ 41157], 99.90th=[ 78119], 99.95th=[
> 89654],
> 
>      | 99.99th=[125305]
> 
>    bw (  KiB/s): min= 1985, max=991826, per=63.49%, avg=140649.83,
> stdev=78204.76, samples=72
> 
>    iops        : min=  495, max=247955, avg=35161.00, stdev=19551.19,
> samples=72
> 
>   lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=84.82%, 100=8.18%
> 
>   lat (usec)   : 250=0.37%, 500=0.09%, 750=0.24%, 1000=0.54%
> 
>   lat (msec)   : 2=0.43%, 4=0.46%, 10=1.29%, 20=1.93%, 50=1.30%
> 
>   lat (msec)   : 100=0.33%, 250=0.02%
> 
>   cpu          : usr=1.76%, sys=16.29%, ctx=14519, majf=0, minf=104
> 
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
> >=64=0.0%
> 
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> 
>      latency   : target=0, window=0, percentile=100.00%, depth=16
> 
> 
> 
> Run status group 0 (all jobs):
> 
>    READ: bw=216MiB/s (227MB/s), 216MiB/s-216MiB/s (227MB/s-227MB/s),
> io=2048MiB (2147MB), run=9467-9467msec
> 
> 
> 
> Disk stats (read/write):
> 
>   loop1: ios=177240/0, merge=0/0, ticks=199386/0, in_queue=75984,
> util=73.95%
> 
> 
> 
>  Fio Test on SquashFs dir:
> 
> 
> localhost:~/Downloads/erofs-utils> fio --name=randread --ioengine=libaio
> --iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4
> --runtime=240 --group_reporting --directory=/mnt/enwik9_sqsh/
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=16
> ...
> fio-3.17-90-gd9b7
> Starting 4 processes
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> 
> 
> Run status group 0 (all jobs):
> 
> 
> On Wed, Jan 22, 2020 at 10:07 AM Gao Xiang <hsiangkao@aol.com> wrote:
> 
> > On Wed, Jan 22, 2020 at 09:27:45AM +0530, Saumya Panda wrote:
> > > Hi Gao,
> > >   Thanks for the info. After I enabled the said configuration, I am now
> > > able to read the files after mount. But I am seeing Squashfs has better
> > > compression ratio compared to Erofs (more than 60% than that of Erofs).
> > Am
> > > I missing something? I used lz4hc while making the Erofs image.
> > >
> > > ls -l enwik*
> > > -rw-r--r-- 1 saumya users  61280256 Jan 21 03:22 enwik8.erofs.img
> > > -rw-r--r-- 1 saumya users  37355520 Jan 21 03:34 enwik8.sqsh
> > > -rw-r--r-- 1 saumya users 558133248 Jan 21 03:25 enwik9.erofs.img
> > > -rw-r--r-- 1 saumya users 331481088 Jan 21 03:35 enwik9.sqsh
> >
> > Yes, it's working as expect. Currently EROFS is compressed in 4k
> > fixed-sized output compression granularity as mentioned in many
> > available materials. That is the use case for our smartphones.
> > You should compare with similar block configuration of squashfs.
> > and there are some 3rd data by other folks as well [1].
> >
> > In the future, we will support other compression algorithms and
> > larger compressed size (> 4k).
> >
> > [1] In chinese,
> >     https://blog.csdn.net/scnutiger/article/details/102507596
> >
> > Thanks,
> > Gao Xiang
> >
> >
> 
> -- 
> Thanks,
> Saumya Prakash Panda