jffs2 Erase Block Summary overhead

jffs2 Erase Block Summary overhead

[Matthieu CASTET]

Hi,

after doing some test on a 64MB nand flash, we see that only 50 MB can 
be stored on the filesystem [1]. The extra 10 MB are taken by jffs2 summary.
If we disable jffs2 sumary in the kernel, the 10MB become available.

Is there a way to reduce this overhead ?
20 % seems really big.

May be Centralized Summary [3] give better result ?
What is its status ?

Matthieu

[1]
# time mount /dev/mtdblock3 -t jffs2 /tmp/mnt/ ; time cat 
/tmp/mnt/root/file.1 > /dev/null ; du -sh /tmp/mnt; df
real    0m 3.05s
user    0m 0.00s
sys     0m 3.05s
real    0m 0.34s
user    0m 0.01s
sys     0m 0.33s
50.6M   /tmp/mnt
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock3           61.5M     60.7M    832.0k  99% /tmp/mnt

[2]
# time mount /dev/mtdblock3 -t jffs2 /tmp/mnt/ ; time cat 
/tmp/mnt/root/file.1 > /dev/null ; du -sh /tmp/mnt; df
real    0m 11.30s
user    0m 0.01s
sys     0m 11.28s
real    0m 0.35s
user    0m 0.00s
sys     0m 0.33s
50.6M   /tmp/mnt
/dev/mtdblock3           61.5M     51.2M     10.3M  83% /tmp/mnt

[3]
http://www.inf.u-szeged.hu/jffs2/mount.php

Re: jffs2 Erase Block Summary overhead

[David Woodhouse]

On Thu, 2007-02-15 at 16:30 +0100, Matthieu CASTET wrote:
> after doing some test on a 64MB nand flash, we see that only 50 MB can 
> be stored on the filesystem [1]. The extra 10 MB are taken by jffs2 summary.
> If we disable jffs2 sumary in the kernel, the 10MB become available.
> 
> Is there a way to reduce this overhead ?
> 20 % seems really big.

There's a lot of redundant information in the summary; I think we could
halve it fairly easily. I don't believe your 20% figure though -- that's
too high. You really can't rely on the output of df.

-- 
dwmw2

Re: jffs2 Erase Block Summary overhead

[Matthieu CASTET]

Hi,

David Woodhouse a écrit :
> 
> There's a lot of redundant information in the summary; I think we could
> halve it fairly easily.
That would be great :)

> I don't believe your 20% figure though -- that's
> too high. You really can't rely on the output of df.
> 
But the jffs2 layer say "no space left on device", so the it is really full.

For more information on the test procedure, I fill the jffs2 filesystem 
with someting like :
"i=0; while true; do dd if=/dev/urandom of=file.$i count=1024; 
i=$(($i+1));done"
And I can create only 93 complete files, a incomplete file of 430 KB 
(and there should be 4 MB taken file others files).


May be I am on a worse test case ?

If I remove "Erase Block Summary", I can really fill the jffs2 up to 
60MB [1] instead of 51MB.

Thanks,

Matthieu


[1]

# time mount /dev/mtdblock3 -t jffs2 /tmp/mnt/ ; time cat 
/tmp/mnt/root/file.1 > /dev/null ; du -sh /tmp/mnt; df
real    0m 11.91s
user    0m 0.00s
sys     0m 11.91s
real    0m 0.36s
user    0m 0.01s
sys     0m 0.35s
59.8M   /tmp/mnt
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0                 4.4M      4.4M         0 100% /
none                      7.0M     16.0k      7.0M   0% /tmp
none                      7.0M     16.0k      7.0M   0% /tmp
tmpfs                     7.0M     16.0k      7.0M   0% /tmp
/dev/mtdblock3           61.5M     60.9M    640.0k  99% /tmp/mnt

Re: jffs2 Erase Block Summary overhead

[Indrek Kruusa]

Ühel kenal päeval (neljapäev 15 veebruar 2007 5:30 pm) kirjutas Matthieu 
CASTET:
> Hi,
>
> after doing some test on a 64MB nand flash, we see that only 50 MB can
> be stored on the filesystem [1]. The extra 10 MB are taken by jffs2
> summary. If we disable jffs2 sumary in the kernel, the 10MB become
> available.

This is interesting. I have a Puppy Linux build within 52MB (ps 512B/ebs 16KB) 
and sumtool grows it to 57MB.
JFFS2/device with ps of 2048B/ebs 128KB has much much better compression level 
and sum adds ca. 3-4% for this case.
You can play with "dry" mkfs.jffs2 and different page;block sizes/file 
counts/file types to get the whole picture. Or read the code ;)
Somebody could put a nice summary table into FAQ with estimated results.

cheers,
Indrek



>
> Is there a way to reduce this overhead ?
> 20 % seems really big.
>
> May be Centralized Summary [3] give better result ?
> What is its status ?
>
> Matthieu
>
> [1]
> # time mount /dev/mtdblock3 -t jffs2 /tmp/mnt/ ; time cat
> /tmp/mnt/root/file.1 > /dev/null ; du -sh /tmp/mnt; df
> real    0m 3.05s
> user    0m 0.00s
> sys     0m 3.05s
> real    0m 0.34s
> user    0m 0.01s
> sys     0m 0.33s
> 50.6M   /tmp/mnt
> Filesystem                Size      Used Available Use% Mounted on
> /dev/mtdblock3           61.5M     60.7M    832.0k  99% /tmp/mnt
>
> [2]
> # time mount /dev/mtdblock3 -t jffs2 /tmp/mnt/ ; time cat
> /tmp/mnt/root/file.1 > /dev/null ; du -sh /tmp/mnt; df
> real    0m 11.30s
> user    0m 0.01s
> sys     0m 11.28s
> real    0m 0.35s
> user    0m 0.00s
> sys     0m 0.33s
> 50.6M   /tmp/mnt
> /dev/mtdblock3           61.5M     51.2M     10.3M  83% /tmp/mnt
>
> [3]
> http://www.inf.u-szeged.hu/jffs2/mount.php
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: jffs2 Erase Block Summary overhead

[Matthieu CASTET]

Matthieu CASTET a écrit :
> Hi,
> 
> David Woodhouse a écrit :
>> There's a lot of redundant information in the summary; I think we could
>> halve it fairly easily.
> That would be great :)
> 
>> I don't believe your 20% figure though -- that's
>> too high. You really can't rely on the output of df.
>>
After an accurate benchmark it is more 15 % than 20%, but it still big.

> But the jffs2 layer say "no space left on device", so the it is really full.
> 
> For more information on the test procedure, I fill the jffs2 filesystem 
> with someting like :
> "i=0; while true; do dd if=/dev/urandom of=file.$i count=1024; 
> i=$(($i+1));done"
> And I can create only 93 complete files, a incomplete file of 430 KB 
> (and there should be 4 MB taken file others files).
> 
> 
> May be I am on a worse test case ?
> 
> If I remove "Erase Block Summary", I can really fill the jffs2 up to 
> 60MB [1] instead of 51MB.
> 
> Thanks,
> 
> Matthieu
> 
> 
> [1]
> 
> # time mount /dev/mtdblock3 -t jffs2 /tmp/mnt/ ; time cat 
> /tmp/mnt/root/file.1 > /dev/null ; du -sh /tmp/mnt; df
> real    0m 11.91s
> user    0m 0.00s
> sys     0m 11.91s
> real    0m 0.36s
> user    0m 0.01s
> sys     0m 0.35s
> 59.8M   /tmp/mnt
> Filesystem                Size      Used Available Use% Mounted on
> /dev/ram0                 4.4M      4.4M         0 100% /
> none                      7.0M     16.0k      7.0M   0% /tmp
> none                      7.0M     16.0k      7.0M   0% /tmp
> tmpfs                     7.0M     16.0k      7.0M   0% /tmp
> /dev/mtdblock3           61.5M     60.9M    640.0k  99% /tmp/mnt
> 

Re: jffs2 Erase Block Summary overhead

[Matthieu CASTET]

Indrek Kruusa a écrit :
> Ühel kenal päeval (neljapäev 15 veebruar 2007 5:30 pm) kirjutas Matthieu 
> CASTET:
>> Hi,
>>
>> after doing some test on a 64MB nand flash, we see that only 50 MB can
>> be stored on the filesystem [1]. The extra 10 MB are taken by jffs2
>> summary. If we disable jffs2 sumary in the kernel, the 10MB become
>> available.
> 
> This is interesting. I have a Puppy Linux build within 52MB (ps 512B/ebs 16KB) 
> and sumtool grows it to 57MB.
> JFFS2/device with ps of 2048B/ebs 128KB has much much better compression level 
> and sum adds ca. 3-4% for this case.
Mine is a ps 512B/ebs 4KB.

> You can play with "dry" mkfs.jffs2 and different page;block sizes/file 
> counts/file types to get the whole picture.
I don't get the same result with image generated with mkfs.jffs2 and on
the fly written file :
- with mkfs.jffs2 + sumtool give me only 7 % overhead
- on the fly write give me 15 % overhead


Matthieu