linux-erofs.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
* Problem in EROFS: Not able to read the files after mount
@ 2020-01-20  6:55 Saumya Panda
  2020-01-20  7:41 ` Gao Xiang via Linux-erofs
  0 siblings, 1 reply; 9+ messages in thread
From: Saumya Panda @ 2020-01-20  6:55 UTC (permalink / raw)
  To: linux-erofs

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

Hi Experts,
   I am testing EROFS in openSuse(Kernel: 5.4.7-1-default). I used the
enwik8 and enwik9 as source file (
https://cs.fit.edu/~mmahoney/compression/textdata.html) for compression.
But after I mount the erofs image, I am not able to read it (it is saying
operation not permitted). Simple "ls" command is not working. But if I
create EROFS image without compression flag, then after mount I am able to
read the files. Seems there is some problem during compression.

I will appreciate if someone can help me out why this is happening.

Steps followed:
*Erofs image creation & mount: *
mkfs.erofs -zlz4hc enwik8.erofs.img enwik8/
mkfs.erofs 1.0
        c_version:           [     1.0]
        c_dbg_lvl:           [       0]
        c_dry_run:           [       0]
mount enwik8.erofs.img /mnt/enwik8/ -t erofs -o loop

ls -l /mnt/enwik8/
ls: cannot access '/mnt/enwik8/enwik8': Operation not supported
total 0
-????????? ? ? ? ?            ? enwik8

The problem seen for both lz4 & lz4hc.

*Erofs image creation & mount without compression: *
mkfs.erofs  enwik8_nocomp.erofs.img enwik8/
mkfs.erofs 1.0
        c_version:           [     1.0]
        c_dbg_lvl:           [       0]
        c_dry_run:           [       0]

mount enwik8_nocomp.erofs.img /mnt/enwik8_nocomp/ -t erofs -o loop

ls -l /mnt/enwik8_nocomp/
total 97660
-rw-r--r-- 1 root root 100000000 Jan 20 01:27 enwik8

*Original enwik8 file:*
ls -l enwik8
total 97660
-rw-r--r-- 1 root root 100000000 Jan 20 01:14 enwik8

*Source code used for Lz4 and Erofs utils:*
https://github.com/hsiangkao/erofs-utils
https://github.com/lz4/lz4

-- 
Thanks,
Saumya Prakash Panda

[-- Attachment #2: Type: text/html, Size: 2475 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-01-20  6:55 Problem in EROFS: Not able to read the files after mount Saumya Panda
@ 2020-01-20  7:41 ` Gao Xiang via Linux-erofs
  2020-01-22  3:57   ` Saumya Panda
  0 siblings, 1 reply; 9+ messages in thread
From: Gao Xiang via Linux-erofs @ 2020-01-20  7:41 UTC (permalink / raw)
  To: Saumya Panda; +Cc: linux-erofs

Hi Saumya,

On Mon, Jan 20, 2020 at 12:25:15PM +0530, Saumya Panda wrote:
> Hi Experts,
>    I am testing EROFS in openSuse(Kernel: 5.4.7-1-default). I used the
> enwik8 and enwik9 as source file (
> https://cs.fit.edu/~mmahoney/compression/textdata.html) for compression.
> But after I mount the erofs image, I am not able to read it (it is saying
> operation not permitted). Simple "ls" command is not working. But if I
> create EROFS image without compression flag, then after mount I am able to
> read the files. Seems there is some problem during compression.
> 
> I will appreciate if someone can help me out why this is happening.

Could you please check if your opensuse kernel has been enabled
the following configuration?

CONFIG_EROFS_FS_ZIP=y
CONFIG_EROFS_FS_CLUSTER_PAGE_LIMIT=1

By default, they should be enabled, but it seems not according to
the following information you mentioned.

Thanks,
Gao Xiang

> 
> Steps followed:
> *Erofs image creation & mount: *
> mkfs.erofs -zlz4hc enwik8.erofs.img enwik8/
> mkfs.erofs 1.0
>         c_version:           [     1.0]
>         c_dbg_lvl:           [       0]
>         c_dry_run:           [       0]
> mount enwik8.erofs.img /mnt/enwik8/ -t erofs -o loop
> 
> ls -l /mnt/enwik8/
> ls: cannot access '/mnt/enwik8/enwik8': Operation not supported
> total 0
> -????????? ? ? ? ?            ? enwik8
> 
> The problem seen for both lz4 & lz4hc.
> 
> *Erofs image creation & mount without compression: *
> mkfs.erofs  enwik8_nocomp.erofs.img enwik8/
> mkfs.erofs 1.0
>         c_version:           [     1.0]
>         c_dbg_lvl:           [       0]
>         c_dry_run:           [       0]
> 
> mount enwik8_nocomp.erofs.img /mnt/enwik8_nocomp/ -t erofs -o loop
> 
> ls -l /mnt/enwik8_nocomp/
> total 97660
> -rw-r--r-- 1 root root 100000000 Jan 20 01:27 enwik8
> 
> *Original enwik8 file:*
> ls -l enwik8
> total 97660
> -rw-r--r-- 1 root root 100000000 Jan 20 01:14 enwik8
> 
> *Source code used for Lz4 and Erofs utils:*
> https://github.com/hsiangkao/erofs-utils
> https://github.com/lz4/lz4
> 
> -- 
> Thanks,
> Saumya Prakash Panda

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-01-20  7:41 ` Gao Xiang via Linux-erofs
@ 2020-01-22  3:57   ` Saumya Panda
  2020-01-22  4:37     ` Gao Xiang via Linux-erofs
  0 siblings, 1 reply; 9+ messages in thread
From: Saumya Panda @ 2020-01-22  3:57 UTC (permalink / raw)
  To: Gao Xiang; +Cc: linux-erofs

[-- Attachment #1: Type: text/plain, Size: 2888 bytes --]

Hi Gao,
  Thanks for the info. After I enabled the said configuration, I am now
able to read the files after mount. But I am seeing Squashfs has better
compression ratio compared to Erofs (more than 60% than that of Erofs). Am
I missing something? I used lz4hc while making the Erofs image.

ls -l enwik*
-rw-r--r-- 1 saumya users  61280256 Jan 21 03:22 enwik8.erofs.img
-rw-r--r-- 1 saumya users  37355520 Jan 21 03:34 enwik8.sqsh
-rw-r--r-- 1 saumya users 558133248 Jan 21 03:25 enwik9.erofs.img
-rw-r--r-- 1 saumya users 331481088 Jan 21 03:35 enwik9.sqsh

On Mon, Jan 20, 2020 at 1:11 PM Gao Xiang <hsiangkao@aol.com> wrote:

> Hi Saumya,
>
> On Mon, Jan 20, 2020 at 12:25:15PM +0530, Saumya Panda wrote:
> > Hi Experts,
> >    I am testing EROFS in openSuse(Kernel: 5.4.7-1-default). I used the
> > enwik8 and enwik9 as source file (
> > https://cs.fit.edu/~mmahoney/compression/textdata.html) for compression.
> > But after I mount the erofs image, I am not able to read it (it is saying
> > operation not permitted). Simple "ls" command is not working. But if I
> > create EROFS image without compression flag, then after mount I am able
> to
> > read the files. Seems there is some problem during compression.
> >
> > I will appreciate if someone can help me out why this is happening.
>
> Could you please check if your opensuse kernel has been enabled
> the following configuration?
>
> CONFIG_EROFS_FS_ZIP=y
> CONFIG_EROFS_FS_CLUSTER_PAGE_LIMIT=1
>
> By default, they should be enabled, but it seems not according to
> the following information you mentioned.
>
> Thanks,
> Gao Xiang
>
> >
> > Steps followed:
> > *Erofs image creation & mount: *
> > mkfs.erofs -zlz4hc enwik8.erofs.img enwik8/
> > mkfs.erofs 1.0
> >         c_version:           [     1.0]
> >         c_dbg_lvl:           [       0]
> >         c_dry_run:           [       0]
> > mount enwik8.erofs.img /mnt/enwik8/ -t erofs -o loop
> >
> > ls -l /mnt/enwik8/
> > ls: cannot access '/mnt/enwik8/enwik8': Operation not supported
> > total 0
> > -????????? ? ? ? ?            ? enwik8
> >
> > The problem seen for both lz4 & lz4hc.
> >
> > *Erofs image creation & mount without compression: *
> > mkfs.erofs  enwik8_nocomp.erofs.img enwik8/
> > mkfs.erofs 1.0
> >         c_version:           [     1.0]
> >         c_dbg_lvl:           [       0]
> >         c_dry_run:           [       0]
> >
> > mount enwik8_nocomp.erofs.img /mnt/enwik8_nocomp/ -t erofs -o loop
> >
> > ls -l /mnt/enwik8_nocomp/
> > total 97660
> > -rw-r--r-- 1 root root 100000000 Jan 20 01:27 enwik8
> >
> > *Original enwik8 file:*
> > ls -l enwik8
> > total 97660
> > -rw-r--r-- 1 root root 100000000 Jan 20 01:14 enwik8
> >
> > *Source code used for Lz4 and Erofs utils:*
> > https://github.com/hsiangkao/erofs-utils
> > https://github.com/lz4/lz4
> >
> > --
> > Thanks,
> > Saumya Prakash Panda
>


-- 
Thanks,
Saumya Prakash Panda

[-- Attachment #2: Type: text/html, Size: 4074 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-01-22  3:57   ` Saumya Panda
@ 2020-01-22  4:37     ` Gao Xiang via Linux-erofs
  2020-01-29  4:13       ` Saumya Panda
  0 siblings, 1 reply; 9+ messages in thread
From: Gao Xiang via Linux-erofs @ 2020-01-22  4:37 UTC (permalink / raw)
  To: Saumya Panda; +Cc: linux-erofs

On Wed, Jan 22, 2020 at 09:27:45AM +0530, Saumya Panda wrote:
> Hi Gao,
>   Thanks for the info. After I enabled the said configuration, I am now
> able to read the files after mount. But I am seeing Squashfs has better
> compression ratio compared to Erofs (more than 60% than that of Erofs). Am
> I missing something? I used lz4hc while making the Erofs image.
> 
> ls -l enwik*
> -rw-r--r-- 1 saumya users  61280256 Jan 21 03:22 enwik8.erofs.img
> -rw-r--r-- 1 saumya users  37355520 Jan 21 03:34 enwik8.sqsh
> -rw-r--r-- 1 saumya users 558133248 Jan 21 03:25 enwik9.erofs.img
> -rw-r--r-- 1 saumya users 331481088 Jan 21 03:35 enwik9.sqsh

Yes, it's working as expect. Currently EROFS is compressed in 4k
fixed-sized output compression granularity as mentioned in many
available materials. That is the use case for our smartphones.
You should compare with similar block configuration of squashfs.
and there are some 3rd data by other folks as well [1].

In the future, we will support other compression algorithms and
larger compressed size (> 4k).

[1] In chinese,
    https://blog.csdn.net/scnutiger/article/details/102507596

Thanks,
Gao Xiang


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-01-22  4:37     ` Gao Xiang via Linux-erofs
@ 2020-01-29  4:13       ` Saumya Panda
  2020-01-29  4:50         ` Gao Xiang via Linux-erofs
  2020-01-29  4:59         ` Gao Xiang via Linux-erofs
  0 siblings, 2 replies; 9+ messages in thread
From: Saumya Panda @ 2020-01-29  4:13 UTC (permalink / raw)
  To: Gao Xiang; +Cc: linux-erofs

[-- Attachment #1: Type: text/plain, Size: 7215 bytes --]

Hi Gao,
  How you got the read amplification? I ran FIO on enwik9 (both Erofs and
SquashFs) and got the below output. Is there anyway to calculate the read
amplification from the below logs.

     Here filename (/mnt/enwik9_erofs/enwik9, /mnt/enwiki_sqfs/enwik9)
points to the mounted readonly file system(squasfs, erofs). But if I give
directory as a parameter instead of filename I am getting error(see the
logs at the end).

*FIO on Erofs:*

localhost:~> fio --name=randread --ioengine=libaio --iodepth=16
--rw=randread --bs=4k --direct=0 --size=512M --numjobs=4 --runtime=240
--group_reporting --filename=/mnt/enwik9_erofs/enwik9

randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=16

...

fio-3.17-90-gd9b7

Starting 4 processes

Jobs: 4 (f=4): [r(4)][100.0%][r=381MiB/s][r=97.6k IOPS][eta 00m:00s]

randread: (groupid=0, jobs=4): err= 0: pid=34282: Mon Jan 27 01:04:55 2020

  read: IOPS=36.7k, BW=144MiB/s (150MB/s)(2048MiB/14271msec)

    slat (nsec): min=1305, max=135688k, avg=106650.48, stdev=493480.73

    clat (nsec): min=1970, max=136593k, avg=1629459.90, stdev=2639786.83

     lat (usec): min=3, max=136625, avg=1736.29, stdev=2772.32

    clat percentiles (usec):

     |  1.00th=[    48],  5.00th=[    69], 10.00th=[   251], 20.00th=[
437],

     | 30.00th=[   570], 40.00th=[   701], 50.00th=[   848], 60.00th=[
1029],

     | 70.00th=[  1336], 80.00th=[  2147], 90.00th=[  4015], 95.00th=[
5932],

     | 99.00th=[ 11600], 99.50th=[ 13304], 99.90th=[ 17171], 99.95th=[
20579],

     | 99.99th=[135267]

   bw (  KiB/s): min=16510, max=295435, per=76.91%, avg=113025.79,
stdev=23830.42, samples=112

   iops        : min= 4126, max=73857, avg=28254.82, stdev=5957.62,
samples=112

  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.37%

  lat (usec)   : 100=5.45%, 250=3.15%, 500=14.74%, 750=18.99%, 1000=14.99%

  lat (msec)   : 2=20.14%, 4=11.09%, 10=8.42%, 20=1.62%, 50=0.04%

  lat (msec)   : 250=0.01%

  cpu          : usr=1.87%, sys=8.28%, ctx=144023, majf=1, minf=114

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
>=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
>=64=0.0%

     issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=16



Run status group 0 (all jobs):

   READ: bw=144MiB/s (150MB/s), 144MiB/s-144MiB/s (150MB/s-150MB/s),
io=2048MiB (2147MB), run=14271-14271msec



Disk stats (read/write):

  loop0: ios=137357/0, merge=0/0, ticks=23020/0, in_queue=460, util=97.70%


*FIO on SquashFs:*


localhost:~/Downloads/erofs-utils> fio --name=randread --ioengine=libaio
--iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4
--runtime=240 --group_reporting --filename=/mnt/enwik9_sqsh/enwik9

randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=16

...

fio-3.17-90-gd9b7

Starting 4 processes

Jobs: 4 (f=4): [r(4)][66.7%][r=1175MiB/s][r=301k IOPS][eta 00m:05s]

randread: (groupid=0, jobs=4): err= 0: pid=34389: Mon Jan 27 01:07:56 2020

  read: IOPS=55.4k, BW=216MiB/s (227MB/s)(2048MiB/9467msec)

    slat (nsec): min=1194, max=61065k, avg=67581.76, stdev=754174.73

    clat (usec): min=2, max=222014, avg=1075.25, stdev=5969.94

     lat (usec): min=3, max=235437, avg=1143.13, stdev=6341.32

    clat percentiles (usec):

     |  1.00th=[    39],  5.00th=[    40], 10.00th=[    40], 20.00th=[
41],

     | 30.00th=[    42], 40.00th=[    43], 50.00th=[    43], 60.00th=[
44],

     | 70.00th=[    45], 80.00th=[    48], 90.00th=[    63], 95.00th=[
3163],

     | 99.00th=[ 28443], 99.50th=[ 41157], 99.90th=[ 78119], 99.95th=[
89654],

     | 99.99th=[125305]

   bw (  KiB/s): min= 1985, max=991826, per=63.49%, avg=140649.83,
stdev=78204.76, samples=72

   iops        : min=  495, max=247955, avg=35161.00, stdev=19551.19,
samples=72

  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=84.82%, 100=8.18%

  lat (usec)   : 250=0.37%, 500=0.09%, 750=0.24%, 1000=0.54%

  lat (msec)   : 2=0.43%, 4=0.46%, 10=1.29%, 20=1.93%, 50=1.30%

  lat (msec)   : 100=0.33%, 250=0.02%

  cpu          : usr=1.76%, sys=16.29%, ctx=14519, majf=0, minf=104

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
>=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
>=64=0.0%

     issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=16



Run status group 0 (all jobs):

   READ: bw=216MiB/s (227MB/s), 216MiB/s-216MiB/s (227MB/s-227MB/s),
io=2048MiB (2147MB), run=9467-9467msec



Disk stats (read/write):

  loop1: ios=177240/0, merge=0/0, ticks=199386/0, in_queue=75984,
util=73.95%



 Fio Test on SquashFs dir:


localhost:~/Downloads/erofs-utils> fio --name=randread --ioengine=libaio
--iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4
--runtime=240 --group_reporting --directory=/mnt/enwik9_sqsh/
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.17-90-gd9b7
Starting 4 processes
randread: Laying out IO file (1 file / 512MiB)
fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
system
randread: Laying out IO file (1 file / 512MiB)
fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
system
randread: Laying out IO file (1 file / 512MiB)
fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
system
randread: Laying out IO file (1 file / 512MiB)
fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
system


Run status group 0 (all jobs):


On Wed, Jan 22, 2020 at 10:07 AM Gao Xiang <hsiangkao@aol.com> wrote:

> On Wed, Jan 22, 2020 at 09:27:45AM +0530, Saumya Panda wrote:
> > Hi Gao,
> >   Thanks for the info. After I enabled the said configuration, I am now
> > able to read the files after mount. But I am seeing Squashfs has better
> > compression ratio compared to Erofs (more than 60% than that of Erofs).
> Am
> > I missing something? I used lz4hc while making the Erofs image.
> >
> > ls -l enwik*
> > -rw-r--r-- 1 saumya users  61280256 Jan 21 03:22 enwik8.erofs.img
> > -rw-r--r-- 1 saumya users  37355520 Jan 21 03:34 enwik8.sqsh
> > -rw-r--r-- 1 saumya users 558133248 Jan 21 03:25 enwik9.erofs.img
> > -rw-r--r-- 1 saumya users 331481088 Jan 21 03:35 enwik9.sqsh
>
> Yes, it's working as expect. Currently EROFS is compressed in 4k
> fixed-sized output compression granularity as mentioned in many
> available materials. That is the use case for our smartphones.
> You should compare with similar block configuration of squashfs.
> and there are some 3rd data by other folks as well [1].
>
> In the future, we will support other compression algorithms and
> larger compressed size (> 4k).
>
> [1] In chinese,
>     https://blog.csdn.net/scnutiger/article/details/102507596
>
> Thanks,
> Gao Xiang
>
>

-- 
Thanks,
Saumya Prakash Panda

[-- Attachment #2: Type: text/html, Size: 34706 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-01-29  4:13       ` Saumya Panda
@ 2020-01-29  4:50         ` Gao Xiang via Linux-erofs
  2020-01-29  4:59         ` Gao Xiang via Linux-erofs
  1 sibling, 0 replies; 9+ messages in thread
From: Gao Xiang via Linux-erofs @ 2020-01-29  4:50 UTC (permalink / raw)
  To: Saumya Panda; +Cc: linux-erofs

Hi Saumya,

On Wed, Jan 29, 2020 at 09:43:37AM +0530, Saumya Panda wrote:
> Hi Gao,
>   How you got the read amplification? I ran FIO on enwik9 (both Erofs and
> SquashFs) and got the below output. Is there anyway to calculate the read
> amplification from the below logs.

No. FIO doesn't provide such number as far as I know, you'd get all statistic
number by some block device information.

BTW, I'd suggest you umount, drop_caches, mount the filesystem every time
in order to get rid of filesystem itself internal cache and bdev buffer
cache. It would not be useful in real scenarios except for benchmark use
only.

https://github.com/erofs/erofs-openbenchmark/

Thanks,
Gao Xiang

> 
>      Here filename (/mnt/enwik9_erofs/enwik9, /mnt/enwiki_sqfs/enwik9)
> points to the mounted readonly file system(squasfs, erofs). But if I give
> directory as a parameter instead of filename I am getting error(see the
> logs at the end).
> 
> *FIO on Erofs:*
> 
> localhost:~> fio --name=randread --ioengine=libaio --iodepth=16
> --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4 --runtime=240
> --group_reporting --filename=/mnt/enwik9_erofs/enwik9
> 
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=16
> 
> ...
> 
> fio-3.17-90-gd9b7
> 
> Starting 4 processes
> 
> Jobs: 4 (f=4): [r(4)][100.0%][r=381MiB/s][r=97.6k IOPS][eta 00m:00s]
> 
> randread: (groupid=0, jobs=4): err= 0: pid=34282: Mon Jan 27 01:04:55 2020
> 
>   read: IOPS=36.7k, BW=144MiB/s (150MB/s)(2048MiB/14271msec)
> 
>     slat (nsec): min=1305, max=135688k, avg=106650.48, stdev=493480.73
> 
>     clat (nsec): min=1970, max=136593k, avg=1629459.90, stdev=2639786.83
> 
>      lat (usec): min=3, max=136625, avg=1736.29, stdev=2772.32
> 
>     clat percentiles (usec):
> 
>      |  1.00th=[    48],  5.00th=[    69], 10.00th=[   251], 20.00th=[
> 437],
> 
>      | 30.00th=[   570], 40.00th=[   701], 50.00th=[   848], 60.00th=[
> 1029],
> 
>      | 70.00th=[  1336], 80.00th=[  2147], 90.00th=[  4015], 95.00th=[
> 5932],
> 
>      | 99.00th=[ 11600], 99.50th=[ 13304], 99.90th=[ 17171], 99.95th=[
> 20579],
> 
>      | 99.99th=[135267]
> 
>    bw (  KiB/s): min=16510, max=295435, per=76.91%, avg=113025.79,
> stdev=23830.42, samples=112
> 
>    iops        : min= 4126, max=73857, avg=28254.82, stdev=5957.62,
> samples=112
> 
>   lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.37%
> 
>   lat (usec)   : 100=5.45%, 250=3.15%, 500=14.74%, 750=18.99%, 1000=14.99%
> 
>   lat (msec)   : 2=20.14%, 4=11.09%, 10=8.42%, 20=1.62%, 50=0.04%
> 
>   lat (msec)   : 250=0.01%
> 
>   cpu          : usr=1.87%, sys=8.28%, ctx=144023, majf=1, minf=114
> 
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
> >=64=0.0%
> 
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> 
>      latency   : target=0, window=0, percentile=100.00%, depth=16
> 
> 
> 
> Run status group 0 (all jobs):
> 
>    READ: bw=144MiB/s (150MB/s), 144MiB/s-144MiB/s (150MB/s-150MB/s),
> io=2048MiB (2147MB), run=14271-14271msec
> 
> 
> 
> Disk stats (read/write):
> 
>   loop0: ios=137357/0, merge=0/0, ticks=23020/0, in_queue=460, util=97.70%
> 
> 
> *FIO on SquashFs:*
> 
> 
> localhost:~/Downloads/erofs-utils> fio --name=randread --ioengine=libaio
> --iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4
> --runtime=240 --group_reporting --filename=/mnt/enwik9_sqsh/enwik9
> 
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=16
> 
> ...
> 
> fio-3.17-90-gd9b7
> 
> Starting 4 processes
> 
> Jobs: 4 (f=4): [r(4)][66.7%][r=1175MiB/s][r=301k IOPS][eta 00m:05s]
> 
> randread: (groupid=0, jobs=4): err= 0: pid=34389: Mon Jan 27 01:07:56 2020
> 
>   read: IOPS=55.4k, BW=216MiB/s (227MB/s)(2048MiB/9467msec)
> 
>     slat (nsec): min=1194, max=61065k, avg=67581.76, stdev=754174.73
> 
>     clat (usec): min=2, max=222014, avg=1075.25, stdev=5969.94
> 
>      lat (usec): min=3, max=235437, avg=1143.13, stdev=6341.32
> 
>     clat percentiles (usec):
> 
>      |  1.00th=[    39],  5.00th=[    40], 10.00th=[    40], 20.00th=[
> 41],
> 
>      | 30.00th=[    42], 40.00th=[    43], 50.00th=[    43], 60.00th=[
> 44],
> 
>      | 70.00th=[    45], 80.00th=[    48], 90.00th=[    63], 95.00th=[
> 3163],
> 
>      | 99.00th=[ 28443], 99.50th=[ 41157], 99.90th=[ 78119], 99.95th=[
> 89654],
> 
>      | 99.99th=[125305]
> 
>    bw (  KiB/s): min= 1985, max=991826, per=63.49%, avg=140649.83,
> stdev=78204.76, samples=72
> 
>    iops        : min=  495, max=247955, avg=35161.00, stdev=19551.19,
> samples=72
> 
>   lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=84.82%, 100=8.18%
> 
>   lat (usec)   : 250=0.37%, 500=0.09%, 750=0.24%, 1000=0.54%
> 
>   lat (msec)   : 2=0.43%, 4=0.46%, 10=1.29%, 20=1.93%, 50=1.30%
> 
>   lat (msec)   : 100=0.33%, 250=0.02%
> 
>   cpu          : usr=1.76%, sys=16.29%, ctx=14519, majf=0, minf=104
> 
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
> >=64=0.0%
> 
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> 
>      issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> 
>      latency   : target=0, window=0, percentile=100.00%, depth=16
> 
> 
> 
> Run status group 0 (all jobs):
> 
>    READ: bw=216MiB/s (227MB/s), 216MiB/s-216MiB/s (227MB/s-227MB/s),
> io=2048MiB (2147MB), run=9467-9467msec
> 
> 
> 
> Disk stats (read/write):
> 
>   loop1: ios=177240/0, merge=0/0, ticks=199386/0, in_queue=75984,
> util=73.95%
> 
> 
> 
>  Fio Test on SquashFs dir:
> 
> 
> localhost:~/Downloads/erofs-utils> fio --name=randread --ioengine=libaio
> --iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4
> --runtime=240 --group_reporting --directory=/mnt/enwik9_sqsh/
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=16
> ...
> fio-3.17-90-gd9b7
> Starting 4 processes
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> randread: Laying out IO file (1 file / 512MiB)
> fio: pid=0, err=30/file:filesetup.c:150, func=unlink, error=Read-only file
> system
> 
> 
> Run status group 0 (all jobs):
> 
> 
> On Wed, Jan 22, 2020 at 10:07 AM Gao Xiang <hsiangkao@aol.com> wrote:
> 
> > On Wed, Jan 22, 2020 at 09:27:45AM +0530, Saumya Panda wrote:
> > > Hi Gao,
> > >   Thanks for the info. After I enabled the said configuration, I am now
> > > able to read the files after mount. But I am seeing Squashfs has better
> > > compression ratio compared to Erofs (more than 60% than that of Erofs).
> > Am
> > > I missing something? I used lz4hc while making the Erofs image.
> > >
> > > ls -l enwik*
> > > -rw-r--r-- 1 saumya users  61280256 Jan 21 03:22 enwik8.erofs.img
> > > -rw-r--r-- 1 saumya users  37355520 Jan 21 03:34 enwik8.sqsh
> > > -rw-r--r-- 1 saumya users 558133248 Jan 21 03:25 enwik9.erofs.img
> > > -rw-r--r-- 1 saumya users 331481088 Jan 21 03:35 enwik9.sqsh
> >
> > Yes, it's working as expect. Currently EROFS is compressed in 4k
> > fixed-sized output compression granularity as mentioned in many
> > available materials. That is the use case for our smartphones.
> > You should compare with similar block configuration of squashfs.
> > and there are some 3rd data by other folks as well [1].
> >
> > In the future, we will support other compression algorithms and
> > larger compressed size (> 4k).
> >
> > [1] In chinese,
> >     https://blog.csdn.net/scnutiger/article/details/102507596
> >
> > Thanks,
> > Gao Xiang
> >
> >
> 
> -- 
> Thanks,
> Saumya Prakash Panda

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-01-29  4:13       ` Saumya Panda
  2020-01-29  4:50         ` Gao Xiang via Linux-erofs
@ 2020-01-29  4:59         ` Gao Xiang via Linux-erofs
  2020-03-20  8:00           ` Saumya Panda
  1 sibling, 1 reply; 9+ messages in thread
From: Gao Xiang via Linux-erofs @ 2020-01-29  4:59 UTC (permalink / raw)
  To: Saumya Panda; +Cc: linux-erofs

On Wed, Jan 29, 2020 at 09:43:37AM +0530, Saumya Panda wrote:
> 
> localhost:~> fio --name=randread --ioengine=libaio --iodepth=16
> --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4 --runtime=240
> --group_reporting --filename=/mnt/enwik9_erofs/enwik9
> 
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=16

And I don't think such configuration is useful to calculate read ampfication
since you read 100% finally, use multi-thread without memory limitation (all
compressed data will be cached, so the total read is compressed size).

I have no idea what you want to get via doing comparsion between EROFS and
Squashfs. Larger block size much like readahead in bulk. If you benchmark
uncompressed file systems, you will notice such filesystems cannot get such
high 100% randread number.

Thank,
Gao Xiang


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-01-29  4:59         ` Gao Xiang via Linux-erofs
@ 2020-03-20  8:00           ` Saumya Panda
  2020-03-20 11:16             ` Gao Xiang via Linux-erofs
  0 siblings, 1 reply; 9+ messages in thread
From: Saumya Panda @ 2020-03-20  8:00 UTC (permalink / raw)
  To: Gao Xiang; +Cc: linux-erofs

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

Hi Gao,
  I am trying to evaluate Erofs on my device. Right now SquashFS is used
for system files. Hence I am trying to compare Erofs with SquashFs. On my
device with the below environment I am seeing Erofs is 3 times faster than
SquashFS 128k (I used enwik8 (100MB) as testing file)) while doing Seq
Read. Your test result shows it is near to SquasFs 128k. How Erofs is so
fast for Seq Read?  I also tested  it on Suse VM with low memory(free
memory 425MB) and I am seeing Erofs is pretty fast.

Also Can you tell me how to run FIO on directory instead of files ?
 fio -filename=$i -rw=read -bs=4k -name=seqbench

             Test on Embedded Device:

Total Memory 5.5 GB:

 Free Memory 1515

 No Swap


$: /fio/erofs_test]$ free -m

              total        used        free      shared  buff/cache
available

Mem:           5384        2315        1515        1378        1553
1592

Swap:             0           0           0





Seq Read



Rand Read





squashFS 4k



51.8MB/s

1931msec

45.7MB/s

2187msec



SquashFS 128k



116MB/s

861msec

14MB/s

877msec



SquashFS 1M



124MB/s-124MB/s

805msec

119MB/s

837msec





Erofs 4k



658MB/s-658MB/s

152msec



103MB

974msec







 Test on Suse VM:


Total Memory 1.5 GB:

 Free Memory 425

 No Swap

localhost:/home/saumya/Documents/erofs_test # free -m
              total        used        free      shared  buff/cache
available
Mem:           1436         817         425           5         192
444
Swap:             0           0           0






Seq Read



Rand Read





squashFS 4k



30.7MB/s

3216msec

9333kB/s

10715msec



SquashFS 128k



318MB/s

314msec

5946kB/s

16819msec











Erofs 4k



469MB/s

213msec



11.9MB/s

8414msec











On Wed, Jan 29, 2020 at 10:30 AM Gao Xiang <hsiangkao@aol.com> wrote:

> On Wed, Jan 29, 2020 at 09:43:37AM +0530, Saumya Panda wrote:
> >
> > localhost:~> fio --name=randread --ioengine=libaio --iodepth=16
> > --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4 --runtime=240
> > --group_reporting --filename=/mnt/enwik9_erofs/enwik9
> >
> > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > 4096B-4096B, ioengine=libaio, iodepth=16
>
> And I don't think such configuration is useful to calculate read
> ampfication
> since you read 100% finally, use multi-thread without memory limitation
> (all
> compressed data will be cached, so the total read is compressed size).
>
> I have no idea what you want to get via doing comparsion between EROFS and
> Squashfs. Larger block size much like readahead in bulk. If you benchmark
> uncompressed file systems, you will notice such filesystems cannot get such
> high 100% randread number.
>
> Thank,
> Gao Xiang
>
>

-- 
Thanks,
Saumya Prakash Panda

[-- Attachment #2: Type: text/html, Size: 30834 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Problem in EROFS: Not able to read the files after mount
  2020-03-20  8:00           ` Saumya Panda
@ 2020-03-20 11:16             ` Gao Xiang via Linux-erofs
  0 siblings, 0 replies; 9+ messages in thread
From: Gao Xiang via Linux-erofs @ 2020-03-20 11:16 UTC (permalink / raw)
  To: Saumya Panda; +Cc: linux-erofs

Hi Saumya,

On Fri, Mar 20, 2020 at 01:30:39PM +0530, Saumya Panda wrote:
> Hi Gao,
>   I am trying to evaluate Erofs on my device. Right now SquashFS is used
> for system files. Hence I am trying to compare Erofs with SquashFs. On my
> device with the below environment I am seeing Erofs is 3 times faster than
> SquashFS 128k (I used enwik8 (100MB) as testing file)) while doing Seq
> Read. Your test result shows it is near to SquasFs 128k. How Erofs is so
> fast for Seq Read?  I also tested  it on Suse VM with low memory(free
> memory 425MB) and I am seeing Erofs is pretty fast.
> 
> Also Can you tell me how to run FIO on directory instead of files ?
>  fio -filename=$i -rw=read -bs=4k -name=seqbench

Thanks for your detailed words.

Firstly, I cannot think out some way to run FIO on directory directly.
And maybe some numbers below are still strange in my opinion.

OK, Actually, I don't want to leave a lot of (maybe aggressive) comments
publicly to compare one filesystem with other filesystems, such as EROFS
vs squashfs (or ext4 vs f2fs). But there are actually some exist materials
which did this before, if you have some extra time, you could read through
the following reference materials about EROFS (although some parts are outdated):

[1] https://static.sched.com/hosted_files/kccncosschn19chi/ce/EROFS%20file%20system_OSS2019_Final.pdf
[2] https://www.usenix.org/system/files/atc19-gao.pdf

The reason why I think in this way is that (Objectively, I think) people
have their own judgement / insistance on every stuffs. But okay, there are
some hints why EROFS behaves well in this email (compared with Squashfs, but
I really want to avoid such aggressive topics):

 o EROFS has carefully designed critial paths, such as async decompression
   path. that partly answers your question about sequential read behavior;

 o EROFS has well-designed compression metadata (called EROFS compacted
   index). Each logic compressed block only takes 2-byte metadata on average
   (high information entropy, so no need to compress compacted indexes again)
   and it supports random read without pervious meta dependence. In contrast,
   the on-disk metadata of Squashfs doesn't support random read (and even
   metadata itself could be compressed), which means you have to cached more
   metadata in memory for random read, or you'll stand with its bad metadata
   random access performance. some hint: see ondisk blocklist, index cache
   and read_blocklist();

 o EROFS firstly uses fixed-sized output compression in filesystem field.
   By using fixed-sized output compression, EROFS can easily implement
   in-place decompression (or at least in-place I/O), which means that it
   doesn't allocate physical pages for most cases, therefore fewer memory
   reclaim/compaction possibility and keeps useful file-backed page cache
   as much as possible;

 o EROFS has designed on-disk directory format, it supports directory
   random access compared with current Squashfs;

In a word, I don't think the current on-disk squashfs is a well-designed
stuff in the long term. In other words, EROFS is a completely different
stuff either from its principle, the on-disk format  and runtime
implementation...)

By the way, the pervious link
https://blog.csdn.net/scnutiger/article/details/102507596
was _not_ written by me. I just noticed it by chance, I think
it was written by some Chinese kernel developer from some other
Android vendor.

And FIO cannot benchmark all cases, heavy memory workload
doesn't completely equal to low memory as well.

However, there is my FIO test script to benchmark different fses:

https://github.com/erofs/erofs-openbenchmark/blob/master/fio-benchmark.sh

for reference. Personally, I think it's reasonable.

It makes more sense to use designed dynamic model. Huawei interally uses
several well-designed light/heavy workloads to benchmark the whole system.

In addition, I noticed many complaints about Squashfs, e.g:

https://forum.snapcraft.io/t/squashfs-is-a-terrible-storage-format/9466

I don't want to comment the whole content itself. But for such runtime
workloads, I'd suggest using EROFS instead and see if it performs better
(compared with any configuration of squashfs+lz4).

There are many ongoing stuffs to do, but I'm really busy recently. After
implementing LZMA and larger compress cluster, I think EROFS will be more
useful, but it needs to be carefully designed first in order to avoid
further complexity of the whole solution. 

Sorry about my English, hope it of some help..

Thanks,
Gao Xiang


> 
>              Test on Embedded Device:
> 
> Total Memory 5.5 GB:
> 
>  Free Memory 1515
> 
>  No Swap
> 
> 
> $: /fio/erofs_test]$ free -m
> 
>               total        used        free      shared  buff/cache
> available
> 
> Mem:           5384        2315        1515        1378        1553
> 1592
> 
> Swap:             0           0           0
> 
> 
> 
> 
> 
> Seq Read
> 
> 
> 
> Rand Read
> 
> 
> 
> 
> 
> squashFS 4k
> 
> 
> 
> 51.8MB/s
> 
> 1931msec
> 
> 45.7MB/s
> 
> 2187msec
> 
> 
> 
> SquashFS 128k
> 
> 
> 
> 116MB/s
> 
> 861msec
> 
> 14MB/s
> 
> 877msec
> 
> 
> 
> SquashFS 1M
> 
> 
> 
> 124MB/s-124MB/s
> 
> 805msec
> 
> 119MB/s
> 
> 837msec
> 
> 
> 
> 
> 
> Erofs 4k
> 
> 
> 
> 658MB/s-658MB/s
> 
> 152msec
> 
> 
> 
> 103MB
> 
> 974msec
> 
> 
> 
> 
> 
> 
> 
>  Test on Suse VM:
> 
> 
> Total Memory 1.5 GB:
> 
>  Free Memory 425
> 
>  No Swap
> 
> localhost:/home/saumya/Documents/erofs_test # free -m
>               total        used        free      shared  buff/cache
> available
> Mem:           1436         817         425           5         192
> 444
> Swap:             0           0           0
> 
> 
> 
> 
> 
> 
> Seq Read
> 
> 
> 
> Rand Read
> 
> 
> 
> 
> 
> squashFS 4k
> 
> 
> 
> 30.7MB/s
> 
> 3216msec
> 
> 9333kB/s
> 
> 10715msec
> 
> 
> 
> SquashFS 128k
> 
> 
> 
> 318MB/s
> 
> 314msec
> 
> 5946kB/s
> 
> 16819msec
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Erofs 4k
> 
> 
> 
> 469MB/s
> 
> 213msec
> 
> 
> 
> 11.9MB/s
> 
> 8414msec
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Wed, Jan 29, 2020 at 10:30 AM Gao Xiang <hsiangkao@aol.com> wrote:
> 
> > On Wed, Jan 29, 2020 at 09:43:37AM +0530, Saumya Panda wrote:
> > >
> > > localhost:~> fio --name=randread --ioengine=libaio --iodepth=16
> > > --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4 --runtime=240
> > > --group_reporting --filename=/mnt/enwik9_erofs/enwik9
> > >
> > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > > 4096B-4096B, ioengine=libaio, iodepth=16
> >
> > And I don't think such configuration is useful to calculate read
> > ampfication
> > since you read 100% finally, use multi-thread without memory limitation
> > (all
> > compressed data will be cached, so the total read is compressed size).
> >
> > I have no idea what you want to get via doing comparsion between EROFS and
> > Squashfs. Larger block size much like readahead in bulk. If you benchmark
> > uncompressed file systems, you will notice such filesystems cannot get such
> > high 100% randread number.
> >
> > Thank,
> > Gao Xiang
> >
> >
> 
> -- 
> Thanks,
> Saumya Prakash Panda

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-03-20 11:17 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-20  6:55 Problem in EROFS: Not able to read the files after mount Saumya Panda
2020-01-20  7:41 ` Gao Xiang via Linux-erofs
2020-01-22  3:57   ` Saumya Panda
2020-01-22  4:37     ` Gao Xiang via Linux-erofs
2020-01-29  4:13       ` Saumya Panda
2020-01-29  4:50         ` Gao Xiang via Linux-erofs
2020-01-29  4:59         ` Gao Xiang via Linux-erofs
2020-03-20  8:00           ` Saumya Panda
2020-03-20 11:16             ` Gao Xiang via Linux-erofs

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).