ext4 settings in an embedded system

ext4 settings in an embedded system

[Ohlsson, Fredrik (GE Healthcare, consultant)]

Hi,
I would like to have advice regarding features and options.
I am working with an embedded system equipped with an IDE Flash Disk and the ext4 filesystem. I have identified 3 problems that I would like to solve in our product. The power is abruptly turned off from time to time, this has sometimes resulted in broken Superblock (inode8) and empty files with size 0 bytes. It also happens that file changes is not committed to disk even if minutes pass before a power loss. This is very undesirable and expensive in our case, we are searching for a solution or a workaround to the problems.
List with my problems I like to solve:
1. Broken Superblock (inode8).
2. Empty files, size 0.
3. Very long auto commit times, several minutes with default settings.


Is ext4 a bad choice for an embedded system with I 1Gb IDE Flash Disk and Debian 2.6.32-5-686?  Should we change filesystem?

I am planning to set the mount flags: barrier=1, commit=1 and data=journal, is this the way to go with an embedded system and ext4? 

Is there more ext4 options/settings I can use to get an embedded system reliable when a power loss occurs?

Best Regards

Fredrik Ohlsson
Software Engineer



--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext4 settings in an embedded system

[Theodore Ts'o]

On Wed, Nov 14, 2012 at 11:41:59AM +0100, Ohlsson, Fredrik (GE Healthcare, consultant) wrote:
> I am working with an embedded system equipped with an IDE Flash Disk
> and the ext4 filesystem. I have identified 3 problems that I would
> like to solve in our product. The power is abruptly turned off from
> time to time, this has sometimes resulted in broken Superblock
> (inode8) and empty files with size 0 bytes. It also happens that
> file changes is not committed to disk even if minutes pass before a
> power loss. This is very undesirable and expensive in our case, we
> are searching for a solution or a workaround to the problems.

I'm not sure what you mean by "broken Superblock (inode 8)".  Inode #8
is the journal superblock.  I'm guessing you're seeing some kind of
corrupted journal superblock?  It would be useful if you could send
kernel logs or e2fsck output so we can see exactly what is going on.

> List with my problems I like to solve:
> 1. Broken Superblock (inode8).
> 2. Empty files, size 0.
> 3. Very long auto commit times, several minutes with default settings.

The default auto commit time is 5 seconds.  *However*, with delayed
allocation, writeback takes place after a 30 second timer, and
depending on how many dirty pages are outstanding, it might take a
while for all of the writeback to be completed.  If you want to
simulate the behaviour you are used to with ext3, where at a journal
commit we force all writeback to complete before the commit is allowed
to proceed, you could use the nodelalloc mount option, but you will
see a corresponding hit in performance as a result.  The better thing
to do is to make sure programs that care about data hitting stable
store use fsync(2) as appropriate, but unfortunately there are many
applications out there which don't do this, and I do understand that
fixing them all might be problematic.  (On the other hand, for an
embedded system, it should be easier since you do control all of your
userspace applicaitons.)

The other thing which may be going on is that there is crappy flash
devices out there which do not handle unexpected power failures
correctly.  Hence, even if you have pushed data out to disk using a
CACHE FLUSH request (which is what barrier=1 does, and which is the
default BTW), there are flash devices which essentially lie and which
do not guarantee that data written before the CACHE FLUSH is stable by
the time the CACHE FLUSH command returns.

If you are seeing a corrupted journal superblock (which is what I
assume you meant by Broken Superblock inode 8), that's an indication
that the hardware is lying to us, and unfortuantely, there's not much
any file system can do in that case.  If the hardware is lying, you're
pretty much out of luck, and the only solution is to replace the
hardware with something which is competently engineered....

I would suggest trying to tackle these two problems separately.  If
you want to make sure fsync is handled correctly, so that files are
flushed out when you need them to be, try doing a reset of the device
--- without dropping power, and see if you can get rid of the zero
length files.   That should be relatively easy to handle.

Then you can try to see what happens with a power drop.
Unfortunately, if it's what I suspect is going on, you have faulty
hardware, and there really is not anything we can do at the OS layer.
If I am correct that your IDE Flash Disk is some cheap piece of cr*p,
you can try using any file system you want, but you're probably going
to end up losing big time.

Regards,

					- Ted

Re: ext4 settings in an embedded system

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 2193 bytes --]

Hi, Fredrik,

On Wed, 2012-11-14 at 11:41 +0100, Ohlsson, Fredrik (GE Healthcare,
consultant) wrote:
> 2. Empty files, size 0.

Well, this is expected in some cases. If create a file, start appending
data, and have a power cut, you may end up with zero-sized files.

This is a Linux feature - the data is cached in RAM until write-back
happens or something like fsync() happens.

Ext4 has a feature that if you write to a file, then close it, ext4 will
initiate write-back for you right away. It was added a couple of years
ago to make it more user-friendly.

You really should investigate what are those files and what was
happening to them just before the power cut. Zero-length files may be
normal, in general.

However, strictly _all_ files you care about should be explicitly
synced. This is just safer.

If you write serious medical software, you should take data integrity
seriously in your apps.

I wrote this section for UBIFS users long time ago, and it is the same
(modulo UBIFS-specific details) for other Linux file-systems, including
ext4:

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_exceptions

> Is ext4 a bad choice for an embedded system with I 1Gb IDE Flash Disk and Debian 2.6.32-5-686?  Should we change filesystem?

I do not know for sure, but I doubt there is serious power-cut testing
regularly conducted for ext4, but people may correct me.

So if power-cut tolerance is important for you, you should conduct good
power-cut tests.

And remember, the disk quality is very important for
power-cut-tolerance. If you uses something like bad managed flash (bad
SSD, eMMC), it may lose recently written data on power-cut. So testing
is important.

Of course you should have barriers on as well.

We conducted some 3 years ago. Results were quite good for ext4 - in
many cases it could recover without a need to run ckfs.ext4, sometimes
it was not mountable, but ckfs.ext4 helped.

On the opposite, ext3 constantly required ckfs.ext3, and sometimes died
so badly that even ckfs.ext3 could not recover it.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ext4 settings in an embedded system

[Andrey Sidorov]

On Thu, Nov 15, 2012 at 12:42 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:

> We conducted some 3 years ago. Results were quite good for ext4 - in
> many cases it could recover without a need to run ckfs.ext4, sometimes
> it was not mountable, but ckfs.ext4 helped.
>
> On the opposite, ext3 constantly required ckfs.ext3, and sometimes died
> so badly that even ckfs.ext3 could not recover it.

We ran about 6000 cycles of power resets with linux 2.6.37. The test
was to run 3 tar processes unpacking linux kernel archive and power
off after about 15 seconds. There were only 3 failures when file
system couldn't be mounted, but that was due to HDD failure
(unreadable sector in journal area). e2fsck successfully recovered
those corruptions. As for software itself, there was no single issue
and we never needed to run fsck after power loss. So I'd say that ext4
is very tolerant to power losses at least in 2.6.37 assuming barriers
and ordered data mode. I however understand this test is quite basic
and any way results can be different for different kernels.

Re: ext4 settings in an embedded system

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Thu, 2012-11-15 at 13:01 +0300, Andrey Sidorov wrote:
> On Thu, Nov 15, 2012 at 12:42 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> 
> > We conducted some 3 years ago. Results were quite good for ext4 - in
> > many cases it could recover without a need to run ckfs.ext4, sometimes
> > it was not mountable, but ckfs.ext4 helped.
> >
> > On the opposite, ext3 constantly required ckfs.ext3, and sometimes died
> > so badly that even ckfs.ext3 could not recover it.
> 
> We ran about 6000 cycles of power resets with linux 2.6.37. The test
> was to run 3 tar processes unpacking linux kernel archive and power
> off after about 15 seconds. There were only 3 failures when file
> system couldn't be mounted, but that was due to HDD failure
> (unreadable sector in journal area). e2fsck successfully recovered
> those corruptions. As for software itself, there was no single issue
> and we never needed to run fsck after power loss. So I'd say that ext4
> is very tolerant to power losses at least in 2.6.37 assuming barriers
> and ordered data mode. I however understand this test is quite basic
> and any way results can be different for different kernels.

Very different experience indeed, shoes that everyone has to conduct own
power-cut tests in own system. I did not say that we were running on
eMMC.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ext4 settings in an embedded system

[Eric Sandeen]

On 11/15/12 4:01 AM, Andrey Sidorov wrote:
> On Thu, Nov 15, 2012 at 12:42 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> 
>> We conducted some 3 years ago. Results were quite good for ext4 - in
>> many cases it could recover without a need to run ckfs.ext4, sometimes
>> it was not mountable, but ckfs.ext4 helped.
>>
>> On the opposite, ext3 constantly required ckfs.ext3, and sometimes died
>> so badly that even ckfs.ext3 could not recover it.

Depending on your storage, if barriers were not enabled on your storage,
this is expected.  Evaporating write caches on power cut do not play well
with journaling guarantees.

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/writebarr.html

Barriers were not made default on ext3 until 2011, in kernel v3.1,
astonishingly.  So it makes sense that ext3 fared worse than ext4.

Unplayable journals are not surprising with non-battery-backed
writeback caches, no explicit cache flushing, and power loss.

> We ran about 6000 cycles of power resets with linux 2.6.37. The test
> was to run 3 tar processes unpacking linux kernel archive and power
> off after about 15 seconds. There were only 3 failures when file
> system couldn't be mounted, but that was due to HDD failure
> (unreadable sector in journal area). e2fsck successfully recovered
> those corruptions. As for software itself, there was no single issue
> and we never needed to run fsck after power loss. So I'd say that ext4
> is very tolerant to power losses at least in 2.6.37 assuming barriers
> and ordered data mode. I however understand this test is quite basic
> and any way results can be different for different kernels.

Right - barriers.

Of course you probably did lose *file* data even if the fs metadata
was correct.

Remember that journaling ensures a consistent metadata structure, but
does not guarantee data integrity.

-Eric

Re: ext4 settings in an embedded system

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

On Fri, 2012-11-16 at 09:40 -0600, Eric Sandeen wrote:
> Barriers were not made default on ext3 until 2011, in kernel v3.1,
> astonishingly.  So it makes sense that ext3 fared worse than ext4.

Very probable, I do not remember if we had them. But we were testing on
top of eMMC with not write-cache.

Anyway, that was long time ago.

While on it, one problem I remember in an unrelated testing of ext3 on
top of eMMC was related to _read_ errors. Sometimes after an power cut
eMMC returned an ECC error, so a sector could be unreadable. But after
writing to this sector it became fine.

ext3 and the tools treated a read error as fatal. However, in case of
eMMC that was something "normal".

I do not know how the situation changes since then, this was probably in
2009.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ext4 settings in an embedded system

[Eric Sandeen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/15/12 2:42 AM, Artem Bityutskiy wrote:
> Hi, Fredrik,
> 
> On Wed, 2012-11-14 at 11:41 +0100, Ohlsson, Fredrik (GE Healthcare,
> consultant) wrote:
>> 2. Empty files, size 0.
> 
> Well, this is expected in some cases. If create a file, start appending
> data, and have a power cut, you may end up with zero-sized files.
> 
> This is a Linux feature - the data is cached in RAM until write-back
> happens or something like fsync() happens.
> 
> Ext4 has a feature that if you write to a file, then close it, ext4 will
> initiate write-back for you right away. It was added a couple of years
> ago to make it more user-friendly.
> 
> You really should investigate what are those files and what was
> happening to them just before the power cut. Zero-length files may be
> normal, in general.
> 
> However, strictly _all_ files you care about should be explicitly
> synced. This is just safer.
> 
> If you write serious medical software, you should take data integrity
> seriously in your apps.
> 
> I wrote this section for UBIFS users long time ago, and it is the same
> (modulo UBIFS-specific details) for other Linux file-systems, including
> ext4:
> 
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_exceptions

Jeff Moyer also has a very good article on this:

http://lwn.net/Articles/457667/
 
>> Is ext4 a bad choice for an embedded system with I 1Gb IDE Flash Disk and Debian 2.6.32-5-686?  Should we change filesystem?
> 
> I do not know for sure, but I doubt there is serious power-cut testing
> regularly conducted for ext4, but people may correct me.

We do it here, though maybe not as regularly as we should.
I also periodically test journal replay, but not in a way that simulates
lost write caches or misbehaving hardware.  OTOH, those things are out
of our control in the real world (if users disable barriers, or if the
hardware lies to us).

But you absolutely should test YOUR system, audit YOUR software, and
YOUR configuration to be sure that it is behaving as you require.

It's absolutely possible to build & configure a system (software+hardware)
which correctly persists data even in the face of a power loss.
(or, in the case of a crash before data integrity syscalls are complete,
your software _knows_ that it was unable to persist the data).

- -Eric

> So if power-cut tolerance is important for you, you should conduct good
> power-cut tests.
> 
> And remember, the disk quality is very important for
> power-cut-tolerance. If you uses something like bad managed flash (bad
> SSD, eMMC), it may lose recently written data on power-cut. So testing
> is important.
> 
> Of course you should have barriers on as well.
> 
> We conducted some 3 years ago. Results were quite good for ext4 - in
> many cases it could recover without a need to run ckfs.ext4, sometimes
> it was not mountable, but ckfs.ext4 helped.
> 
> On the opposite, ext3 constantly required ckfs.ext3, and sometimes died
> so badly that even ckfs.ext3 could not recover it.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQpl9jAAoJECCuFpLhPd7go70P/AiAnzxaR7BhpBY+hMxEyA2O
YFmfvUsY8N51NdReE4NqO3ebu7BWIflTNHmzGaKWDQgXf2T6b1p6Cy2YVphBLsH3
Tfd5B1Aem8ZKVWq2pyuCjuQpNYOmlxibQPT4SWfzKukVO5sxgwiqL/e73awN6luF
U8zTCNnh1d8rOEEq/mSH1eeQSPdlY1h0lXegdBZ0CLkeI5/sTIFaEy8JiPBmKbrU
GsxsAyVEeko7aDL7Mh62RWFN953L/ZPkeVXWOoId8ANcZ+/Lrt/JEjkKIblkyHiK
FJTMSHeHTX8gg88HzFgWqkuB7aBlUB16Ppf5W2FFTUJOHtOcvIVlC5VGyYhYSe9L
DZZh/rxqjDpB8hFpvD6JPWLaTZ/UcIEBdYVlb+va5gfPjE+TBCRmUxRISLCg8opv
tsAQQo6kF818zGYQMtF3pwGkIHqP3DA+SRch/mh77ChTy0kGssgeZeKIoiz15cDJ
bgXR2hLHiuQ3/F3rqV5lCi8ioSPYvH2BmG1dhQdEAQAq48o25WjHOl7Rzpd38gEZ
BaAn0brYOlmW+oL09IKvBnSorZsmlPz5snwmZdxI7afp0ZbGJR1jx+7iqnl5sioG
4GtSwschP+tSnTSczQMlE6siGh3Alyq8vr7maIzsUODRyzdfsrx8koJs05hQEAIp
VzzJPsk6mv4J7E9VKhmH
=S0cP
-----END PGP SIGNATURE-----

Re: ext4 settings in an embedded system

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

On Fri, 2012-11-16 at 09:44 -0600, Eric Sandeen wrote:
> > I do not know for sure, but I doubt there is serious power-cut testing
> > regularly conducted for ext4, but people may correct me.
> 
> We do it here, though maybe not as regularly as we should.
> I also periodically test journal replay, but not in a way that simulates
> lost write caches or misbehaving hardware.  OTOH, those things are out
> of our control in the real world (if users disable barriers, or if the
> hardware lies to us).

Good to know, thanks. I actually now recall Josef mentioned that RedHad
does some power-cut testing.

> But you absolutely should test YOUR system, audit YOUR software, and
> YOUR configuration to be sure that it is behaving as you require.

Sure.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ext4 settings in an embedded system

[Eric Sandeen]

On 11/14/12 4:41 AM, Ohlsson, Fredrik (GE Healthcare, consultant) wrote:
> Hi,
> I would like to have advice regarding features and options.
> I am working with an embedded system equipped with an IDE Flash Disk
> and the ext4 filesystem. I have identified 3 problems that I would
> like to solve in our product. The power is abruptly turned off from
> time to time, this has sometimes resulted in broken Superblock
> (inode8) and empty files with size 0 bytes. It also happens that file
> changes is not committed to disk even if minutes pass before a power
> loss. This is very undesirable and expensive in our case, we are
> searching for a solution or a workaround to the problems.

> List with my problems I like to solve:
> 1. Broken Superblock (inode8).
> 2. Empty files, size 0.
> 3. Very long auto commit times, several minutes with default settings.
> 
> 
> Is ext4 a bad choice for an embedded system with I 1Gb IDE Flash Disk
> and Debian 2.6.32-5-686? Should we change filesystem?
> 
> I am planning to set the mount flags: barrier=1, commit=1 and
> data=journal, is this the way to go with an embedded system and ext4?

When you say "with barrier=1", were you using barrier=0 (or "nobarrier")
before?

If you have disabled barriers explicitly, just start with re-enabling
them.  If your software is doing all the right data integrity syscalls,
then that is probably enough.

-Eric

> Is there more ext4 options/settings I can use to get an embedded
> system reliable when a power loss occurs?
> 
> Best Regards
> 
> Fredrik Ohlsson
> Software Engineer
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: ext4 settings in an embedded system

[Ohlsson, Fredrik (GE Healthcare, consultant)]

[-- Attachment #1: Type: text/plain, Size: 2371 bytes --]

Thank you very much for your helpful response and answers.

I would like to describe some background for our embedded system. The
system has recently been upgraded, before the upgrade we did not see
"filesystem" related problems. Before the upgrade we used  another
filesystem, kernel and IDE Flash Disk(kernel 2.6.12 kernel, reiserfs and
a smaller 256 MB IDE Flash Disk).  Today we have kernel 2.6.32,
ext4(default options) and a 1GB Transend TS1GDOM44H-S IDE Flash Disk.
We have a very low IO intensity towards the flash disk and low
requirements on the filesystem performance. 

In our case with the file of size 0 bytes. We use a bash shell script to
upgrade our application. The shell script calls the program "tar" and
tar overwrites/recreates  our application bin file. After some minutes
the power is cut and our bin file had the new size 0 bytes when the
system came up again . This particular case was solved by adding a sync
in the end of our upgrade shell script. I still don't understand why the
data is not committed to the disk after several minutes? Even if tar
leaves the file truncated tar must have closed the file and ext4 would
have done an implicit write-back?
We are worried that this will happen again where we use programs not
written by ourselves. Will the nodelalloc option solve this behavior?

If I understand you right the corrupted journal superblock (inode #8) is
most probably a result of a problem in the IDE Flash Disk.  I have
attached dumpe2fs.output from this problem. I booted the system from a
usb-drive and the filesystem could be repaired by e2fsck, but this is
not something the customer can do, we have to replace units like this.
The Transend TS1GDOM44H-S IDE Flash Disk is intended for demanding
embedded systems that require reliability, I guess it could still be the
part creating this problem. 

I think you are saying that ext4 should work fine in our setup were we
have regular power-cuts if we use sync/fsync and applies the following
settings:
-barrier, which we already use by default (Can't see barrier in the
attached dumpe2fs.output thou).
-nodelalloc
You also advise us to implement or own power-cut tests.

Are there more settings that could be to our favor, like
journal_checksum? Tune2fs data=journal?

Best Regards

Fredrik Ohlsson
Software Engineer







[-- Attachment #2: dumpe2fs.output --]
[-- Type: application/octet-stream, Size: 24139 bytes --]

Filesystem volume name:   ideflash
Last mounted on:          /
Filesystem UUID:          3dc25bd2-e6b2-4970-a8f4-a1c6195c133d
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              61056
Block count:              244224
Reserved block count:     12211
Free blocks:              111899
Free inodes:              36510
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      59
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         7632
Inode blocks per group:   477
Flex block group size:    16
Filesystem created:       Tue May 11 02:06:07 2010
Last mount time:          Fri Sep 16 03:04:33 2011
Last write time:          Fri Sep 16 02:51:41 2011
Mount count:              1
Maximum mount count:      -1
Last checked:             Fri Sep 16 02:51:41 2011
Check interval:           0 (<none>)
Lifetime writes:          1332 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      cde378c2-7b1c-40d0-b21a-d3f44dca32d8
Journal backup:           inode blocks
Journal features:         (none)
Journal size:             16M
Journal length:           4096
Journal sequence:         0x00000002
Journal start:            1


Group 0: (Blocks 0-32767) [ITABLE_ZEROED]
  Checksum 0x6dd1, unused inodes 0
  Primary superblock at 0, Group descriptors at 1-1
  Reserved GDT blocks at 2-60
  Block bitmap at 61 (+61), Inode bitmap at 77 (+77)
  Inode table at 93-569 (+93)
  26088 free blocks, 0 free inodes, 760 directories
  Free blocks: 6284-6294, 6296-6299, 6303-6314, 6316-6323, 6357-6373, 6389-6396, 6434-6469, 6677-6816, 6916-32767
  Free inodes: 
Group 1: (Blocks 32768-65535) [ITABLE_ZEROED]
  Checksum 0xd8f0, unused inodes 0
  Backup superblock at 32768, Group descriptors at 32769-32769
  Reserved GDT blocks at 32770-32828
  Block bitmap at 62, Inode bitmap at 78
  Inode table at 570-1046
  12395 free blocks, 0 free inodes, 855 directories
  Free blocks: 32929, 32944-32945, 32952-32959, 33066-33067, 33124-33135, 33198-33199, 33230-33233, 33236-33237, 33240-33246, 33276-33277, 33279, 33304, 33307, 33512-33513, 33653, 33792-33795, 33797-33801, 33815-33817, 33819-33823, 33845-33855, 33916-33918, 33920-33923, 33925-33941, 33943-33966, 33984-33986, 33988-34009, 34011-34018, 34020-34032, 34034-34078, 34080-34089, 34092-34094, 34098-34109, 34111-34130, 34132-34137, 34139-34141, 34143-34170, 34172-34175, 34181-34207, 34272-34296, 34298-34308, 34310-34313, 34346, 34348-34367, 34428-34450, 34452-34495, 34688-34851, 34854-34943, 34964-34967, 34970, 35005-35071, 35328-35337, 35339-35343, 35360-35455, 35560-35845, 35887-35888, 35901-36361, 36364, 36366, 36388-36389, 36415-37119, 37148-37156, 37158-37376, 37378-37891, 37894-38404, 38408-38911, 38971-39423, 39995-40453, 40455-40959, 41018-41020, 41105-41108, 41110, 41195, 41258, 41904-41919, 44032-44415, 45556-45567, 45632-45695, 46384-46389, 46396, 46453, 48206-48207, 48574, 48597-48598, 48601, 48603, 48637, 48665, 48700-48703, 48724-48727, 48763-48831, 48869, 48880-48895, 49095, 49104-49119, 49147-49157, 49160, 49162, 49168-49671, 49673-50422, 50424-50702, 50708-50714, 50721, 50726, 50757-50758, 50761, 50765, 50774-50775, 50789-50790, 50794, 50797, 50807, 50844-50858, 50863-51205, 51208-51209, 51211-51711, 52224-52231, 52233-52741, 52744-52745, 52748-53297, 53299-53765, 53768-54297, 54300-54303, 54333, 54373, 54376-54399, 54425, 54428-54455, 54459-54463, 54509, 54512-54527, 54585, 54588-54591, 54648-54655, 54676-54687, 54716-54719, 54757, 54760-54783, 54801, 54804-54844, 54887-54911, 54935-54943, 54973, 55004-55007, 55025, 55028-55039, 55096-55103, 55151-55167, 55237, 55240-55263, 55284-55295, 55492-55519, 55540-55551, 55577, 55580-55583, 55609, 55612-55615, 55644-55647, 55676-55679, 55720-55743, 55769, 55772-55775, 56131-56159, 56185, 56188-56191, 56247-56255, 56293, 56296-56319, 56360-56383, 56404-56415, 56439-56447, 56500-56511, 56563-56575, 56611-56639, 56696-56703, 56735, 56761, 56764-56767, 56789, 56792-56799, 56817, 56820-56831, 56852-56863, 56892-56895, 56920-56927, 56948-56959, 56984-57023, 57060-57087, 57111-57119, 57144-57151, 57191-57215, 57251-57279, 57317, 57320-57343, 57704-57727, 57752-57759, 57780-57791, 57812-57823, 57844-57855, 57892, 57895-57899, 57902, 58672-58687, 58727-58751, 58788-58815, 58852-58879, 58904-58911, 58937, 58940-58943, 58972-58975, 59000-59039, 59064-59071, 59128-59135, 59209, 59212-59231, 59256-59263, 59340-59359, 59382-59391, 59431-59455, 59511-59519, 59583, 59604-59647, 59692-59711, 59735-59743, 59770-59775, 59795-59807, 59825-59903, 60006-60031, 60106-60127, 60152-60159, 60202-60223, 60248-60255, 60285-60287, 60402-60415, 61165, 61168-61169, 61172, 61175, 61183-61184, 61187, 61190, 61204-61205, 61208, 61212, 61219-61220, 61224, 61231, 61242-61243, 61248-61249, 61516-61535, 61555-61567, 61623-61631, 61666-61695, 61797-61823, 61898-61919, 61940-61951, 62024-62047, 62076-62079, 62151-62175, 62206-62207, 62292-62303, 62331-62335, 62404-62431, 62455-62463, 62757, 62761, 62774-62775, 62781, 62784-62790, 62976, 62980-62981, 62994-62997, 63504-63505, 63508, 63511, 63555-63556, 63559, 63562, 63564-63565, 63568, 63572, 63592-63593, 63595-63596, 63750-63751, 63754-63755, 63831-63832, 63834-63835, 63845-63846, 63860, 63864, 64082-64083, 64086, 64088, 64311-64312, 64314-64315, 64447-64448, 64450-64451, 64509-64510, 64512-64513, 64525-64527, 64530-64532, 64553, 64555-64556, 64559, 64561, 64599, 64611-64612, 64617, 64620-64627, 65058-65059, 65062, 65066, 65076-65077, 65079-65080, 65151-65152, 65154-65155, 65208-65209, 65219, 65223, 65263-65264, 65282, 65287
  Free inodes: 
Group 2: (Blocks 65536-98303) [ITABLE_ZEROED]
  Checksum 0x1b27, unused inodes 0
  Block bitmap at 63, Inode bitmap at 79
  Inode table at 1047-1523
  7705 free blocks, 0 free inodes, 481 directories
  Free blocks: 65545-69631, 70864-70865, 70875, 70879, 70892-70895, 71203-71204, 71206, 71208, 71256-71257, 71260, 71264-71265, 71634-71679, 71758-71807, 71882-71903, 71936-71999, 72019-72031, 72050-72159, 72177-72191, 72262-72287, 72306-72319, 72338-72447, 72522-72543, 72562-72632, 72640, 72644, 72852-72853, 72864, 72868, 72912-72913, 72916, 72919, 72961-72962, 72965, 72967, 72987-72988, 72991, 72999, 73019-73020, 73023, 73026, 73051-73052, 73055, 73058, 73073-73074, 73076-73077, 73092-73093, 73097, 73099, 73550-73567, 73587-73599, 73674-73695, 74086-74111, 74182-74207, 74585-74591, 74610-74623, 74677-74687, 74723-74751, 75092-75103, 75122-75135, 75212-75231, 75263, 75404-75423, 75443-75455, 75491-75519, 75740-75743, 75767-75775, 75859-75860, 75862, 75864, 75874-75875, 75883, 75887, 76324-76351, 76385-76415, 76449-76479, 76514-76543, 76562-76575, 76603-76607, 76643-76671, 76705-76735, 76769-76799, 76837-76863, 76882-76895, 76927, 76968-76991, 77009-77023, 77048-77055, 77091-77119, 77138-77151, 77169-77183, 77226-77247, 77267-77279, 77305-77311, 77937-77938, 77941, 77944, 77987-77988, 77991, 77994, 78006-78007, 78010, 78013, 78029-78030, 78033, 78036, 78040-78041, 78044, 78047, 78055-78056, 78059, 78062, 78069-78070, 78076, 78080, 78590-78591, 78783, 78838-78847, 79324-79327, 79357-79359, 79836-79839, 79870-79871, 79954-79967, 79999, 80082-80095, 80121-80127, 80220-80223, 80241-80255, 80359-80383, 81235-81247, 81270-81279, 81377-81407, 81511-81535, 81593-81599, 81642-81663, 81702-81727, 81766-81791, 81837-81855, 81889-81919, 81954-81983, 82020-82047, 82082-82111, 82143, 82166-82175, 82214-82239, 82289-82303, 82329-82335, 82366-82367, 82431, 82467-82495, 82538-82559, 82588-82591, 82622-82623, 82655, 82684-82687, 82726-82751, 82788-82815, 82839-82847, 82879, 82916-82943, 82982-83007, 83039, 83069-83071, 83127-83135, 83164-83167, 83187-83199, 83317-83327, 83351-83359, 83378-83391, 83428-83455, 83473-83474, 83478, 83482, 83535-83536, 83538, 83541, 83555-83556, 83559, 83563, 83725-83726, 83728, 83731, 83869-83870, 83872-83873, 83908-83909, 83912, 83916, 83935-83936, 83942, 83945, 84003-84031, 84062-84063, 84083-84095, 84122-84127, 84157-84159, 84214-84223, 84281-84287, 84308-84319, 84340-84351, 84440-84447, 84467-84479, 84872-84873, 84875-84876, 85052-85053, 85057, 85061, 85093-85094, 85096, 85098, 85212-85213, 85216, 85219, 85229-85230, 85234, 85237, 85499-85500, 85502-85503, 85528-85529, 85531-85532, 85580-85581, 85585, 85588, 85614-85615, 85618, 85621, 85672-85673, 85677, 85679, 85700, 85703-85704, 85717, 85723, 86006-86008, 86012-86013, 86124-86143, 86178-86207, 86232-86239, 86268-86271, 86321-86335, 86357-86367, 86392-86399, 86445-86463, 86498-86527, 86758-86759, 86763, 86766, 86829-86830, 86835, 86839, 87154-87155, 87157, 87160, 87207-87208, 87212, 87215, 87291-87292, 87296, 87301, 87419-87420, 87424, 87428, 87491, 87493-87494, 87496, 87499, 87530-87531, 87534, 87537, 87594-87615, 87635-87647, 87667-87679, 87701-87711, 87733-87743, 87781-87807, 87881-87903, 87925-88063, 88075-88076, 88079, 88083, 88122-88123, 88125-88126, 88144-88145, 88160, 88162, 88247-88248, 88251-88252, 88380-88381, 88383, 88386, 88417-88418, 88438, 88443-88444, 88612-88639, 88675-88703, 88785-88831, 88987-89087, 89244-89245, 89249, 89252, 89293-89294, 89296, 89300, 89308-89309, 89311, 89313, 89716-90111, 90624-90669, 90690, 90697, 90756-90757, 91210-91223, 91225-91234, 91239-91240, 91432-91433, 91452, 91459, 92429-92430, 92433, 92436, 92448-92449, 92455, 92463, 92691-92692, 92733-92734, 92739, 92743-92746, 92869, 93032-93051, 93053-93095, 93097-93113, 93115-93116, 93121-93122, 93130-93143, 93145-93146, 93173-93174, 93176, 93178, 93277-93278, 93282, 93286, 93288-93289, 93357-93390, 93392-93393, 93398-93399, 93405-93406, 93420, 93432, 93434, 93438-93439, 93455, 93465
  Free inodes: 
Group 3: (Blocks 98304-131071) [ITABLE_ZEROED]
  Checksum 0x63ab, unused inodes 5967
  Backup superblock at 98304, Group descriptors at 98305-98305
  Reserved GDT blocks at 98306-98364
  Block bitmap at 64, Inode bitmap at 80
  Inode table at 1524-2000
  6230 free blocks, 5980 free inodes, 211 directories, 5967 unused inodes
  Free blocks: 98366-98367, 98389-98399, 98427-98431, 98476-98495, 98516-98527, 98556-98559, 98664-98687, 98707-98719, 98739-98751, 98789-98975, 99044-99071, 99097-99103, 99136-99167, 99188-99199, 99229-99231, 99263, 99324-99327, 99415-99423, 99445-99455, 99476-99487, 99510-99519, 99575-99583, 99736-99743, 99767-99775, 99794-99807, 99835-99839, 100112-100127, 100157-100159, 100191, 100210-100223, 100295-100319, 100339-100351, 100452-100479, 100500-100511, 100529-100543, 100577-100607, 100641-100671, 100705-100735, 100765-100767, 100789-100799, 100833-100863, 100898-100927, 100958-100959, 100986-100991, 101019-101023, 101091-101119, 101163-101183, 101220-101247, 101268-101279, 101301-101311, 101352-101375, 101508-101535, 101555-101567, 101630-101631, 101718-101727, 101750-101759, 101797-101823, 101862-101887, 102186-102207, 102260-102463, 102521-102527, 102563-102591, 102630-102655, 102694-102719, 102744-102751, 102783, 102809-102815, 102840-102847, 102876-102879, 102905-102911, 103079-103103, 103142-103167, 103213-103231, 103250-103263, 103292-103295, 103330-103359, 103394-103423, 103464-103487, 103545-103551, 103586-103615, 103661-103679, 103716-103743, 103807, 103856-103871, 103893-103903, 103928-104127, 104166-104191, 104225-104255, 104285-104287, 104308-104319, 104356-104383, 104408-104415, 104440-104447, 104466-104479, 104508-104511, 104531-104543, 104564-104575, 104599-104607, 104628-104639, 104693-104703, 104732-104735, 104757-104767, 104812-104831, 104854-104863, 104892-104895, 104916-104927, 104958-104959, 104981-104991, 105010-105023, 105076-105087, 105108-105119, 105139-105151, 105208-105215, 105241-105247, 105273-105279, 105313-105343, 105373-105375, 105395-105407, 105431-105439, 105460-105471, 105641-105663, 105705-105727, 105830-105855, 105938-105951, 105971-105983, 106054-106079, 106102-106111, 106158-106175, 106219-106239, 106268-106271, 106289-106303, 106325-106335, 106356-106367, 106403-106431, 106491-106495, 106599-106623, 106653-106655, 106674-106687, 106731-106751, 106821-106847, 106865-106879, 106913-106943, 106982-107007, 107090-107103, 107126-107135, 107154-107167, 107190-107199, 107221-107231, 107253-107263, 107384-107391, 107464-107487, 107511-107519, 107540-107551, 107573-107583, 107607-107615, 107639-107647, 107687-107711, 107732-107743, 107770-107775, 107815-107839, 107876-107903, 107935, 107966-107967, 108003-108031, 108592-108607, 108626-108639, 108661-108671, 108689-108703, 108726-108735, 108761-108767, 108796-108799, 108860-108863, 108890-108895, 108913-108927, 108947-108959, 108980-108991, 109050-109055, 110611-110623, 110649-110655, 110673-110687, 110709-110719, 110821-110847, 110913-110943, 110962-110975, 111006-111007, 111031-111039, 111076-111103, 112170-112191, 112246-112255, 112277-112287, 112310-112319, 112366-112383, 112444-112447, 112497-112511, 112611-112639, 113329-113343, 113391-113407, 113486-113503, 113522-113535, 113553-113567, 113585-113599, 113635-113663, 114294-114303, 114327-114335, 114357-114367, 114412-114431, 114451-114463, 114481-114495, 114520-114527, 114554-114559, 114577-114591, 114613-114623, 114642-114655, 114677-114687, 115775, 115796-115807, 115829-115839, 115870-115871, 115899-115903, 115921-115935, 115955-115967, 116006-116031, 116053-116063, 116084-116095, 116221-116223, 116827-116831, 116856-116863, 116902-116927, 116948-116959, 116981-116991, 117020-117023, 117053-117055, 117089-117119, 117177-117183, 117203-117215, 117241-117247, 117537-117538, 117540-117541, 117547-117570, 117573, 117576, 117597-117598, 117602, 117604, 117627-117628, 117648-117666, 117668-117669, 117679-117680, 117685-117698, 117700-117701, 117706-117707, 117716-117734, 117738, 117742, 117747-117748, 117965-117983, 118003-118015, 118070-118079, 118124-118143, 118260-118271, 118346-118367, 118456-118463, 118502-118527, 118604-118623, 118645-118655, 118683-118687, 118713-118719, 118755-118783, 118851-118879, 118908-118911, 118979-119007, 119242-119295, 119612-119615, 119664-119807, 120370-120831, 120835, 120837-120848, 120850, 120852, 120854-120856, 121205-121343, 121359-121377, 121379, 121382, 121393-121400, 121403, 121406, 121513-121540, 121542-121543, 121548-121549, 121552-121571, 121574, 121577, 121583-121584, 121586-121587, 121629-121630, 121634, 121638, 121640-121641, 121673-121674, 121680, 121683, 121694, 121709-121738, 121742, 121744-121746, 121759-121855, 122180-122410, 122413-122414, 122419-122420, 122426, 122430, 123581-123582, 123588-123589, 123591, 123595, 123611, 123625-123626, 123637, 123640, 124567, 124586-124587, 124589, 124592, 124610, 124628-124629, 124633, 124636, 124650, 124743-124744, 124747, 124751, 124758, 124780-124781, 124784-124785, 124919-124920, 124923, 124926, 124929, 124977, 125098-125099, 125102, 125105, 125141-125142, 125145, 125148, 125163-125164, 125167, 125170, 125199-125200, 125203, 125206, 125312-125313, 125316, 125319, 125333, 125356-125357, 125364, 125368-125369, 125397-125398, 125400, 125404-125405, 125421-125422, 125425, 125427, 125437, 125440, 125468-125469, 125471, 125474, 125661-125662, 125664-125665, 125673-125674, 125676, 125678, 125693, 125706-125708, 125711, 125804-125805, 125815-125816, 127616-127999, 128093-128094, 128106, 128110, 128131, 128150-128151, 128168, 128173, 128210, 128227-128228, 128231, 128235, 128246, 128266-128267, 128271, 128275, 128288, 128515-128518, 128521, 128523-128525, 128530, 128550-128551, 128554, 128557, 128566-128567, 128570, 128573, 128597-128598, 128601, 128604, 128610-128611, 128614, 128618, 128639-128640, 128643, 128648, 128662-128663, 128666, 128677, 128697-128698, 128701, 128706, 128720-128721, 128724, 128727, 128748-128749, 128752, 128755, 128763-128764, 128767, 128770, 128778-128779, 128782, 128785, 128801-128802, 128805, 128808, 128816-128817, 128820, 128823, 128843-128844, 128847, 128850, 128858-128859, 128861-128863, 128872-128873, 128876, 128879, 128886, 128892-128893, 128897, 128909, 128918, 129003-129006, 129021, 129545-129546, 129553, 129556, 129559-129560, 129563, 129567, 129578, 129586-129587, 129589-129590, 129604, 129614-129615, 129617, 129620-129621, 129631-129632, 129635, 129638, 129762, 129773-129774, 129777, 129780, 129798, 129808-129809, 129811, 129814, 129819-129820, 129822, 129825, 129830-129831, 129834, 129838, 129860-129861, 129864, 129867, 129871-129872, 129875, 129878, 129884-129885, 129888, 129891, 129909-129910, 129913, 129916, 129930-129931, 129934, 129938, 129953-129954, 129957, 129962, 129976-129977, 129980, 129983, 130015-130016, 130019, 130024, 130038-130039, 130042, 130047, 130076-130077, 130080, 130085, 130099-130100, 130103, 130106, 130135-130136, 130138, 130142, 130173, 130177-130178, 130181, 130184, 130205-130206, 130209, 130212, 130230-130231, 130234, 130242, 130248-130249, 130251-130252, 130286, 130297-130298, 130300-130301, 130377, 130379, 130386-130387, 130390, 130394, 130413, 130478-130479, 130482, 130487, 130524-130525, 130528, 130536, 130554-130555, 130560, 130564, 130793, 130836-130837, 130839-130840
  Free inodes: 24514, 24521, 24523, 24525, 24527, 24535-24539, 24541, 24543, 24556, 24562-30528
Group 4: (Blocks 131072-163839) [INODE_UNINIT, ITABLE_ZEROED]
  Checksum 0x75a0, unused inodes 7632
  Block bitmap at 65, Inode bitmap at 81
  Inode table at 2001-2477
  3755 free blocks, 7632 free inodes, 0 directories, 7632 unused inodes
  Free blocks: 132624, 141398-141407, 141426-141439, 141457-141471, 141501-141503, 141523-141535, 141562-141567, 141598-141599, 141629-141631, 141667-141695, 141744-141759, 141783-141791, 141815-141823, 142271, 142290-142303, 142327-142335, 142642-142655, 142680-142687, 142705-142719, 142787-142815, 142841-142847, 143192-143199, 143223-143231, 143254-143263, 143290-143359, 143507-143519, 143550-143551, 143594-143615, 143662-143679, 143726-143743, 143790-143807, 143825-143839, 143864-143871, 144068-144095, 144121-144127, 144147-144159, 144183-144191, 144215-144223, 144250-144255, 144279-144287, 144305-144319, 144343-144351, 144370-144383, 144407-144415, 144441-144447, 144471-144479, 144504-144511, 144535-144543, 144567-144575, 144594-144607, 144637-144639, 144658-144671, 144689-144703, 144729-144735, 144756-144767, 144852-144863, 144891-144895, 145073-145087, 145126-145151, 145219-145247, 145277-145279, 145300-145311, 145375, 145397-145407, 146745-146746, 146749, 146753, 146774-146775, 146783, 146789, 146805, 146810-146811, 146814, 146818, 146881, 146922-146923, 146926, 146931, 146981, 147020-147021, 147023-147024, 147042, 147048-147049, 147063, 147065, 147504-147519, 147566-147583, 147611-147615, 147640-147647, 147686-147711, 147800-147807, 147832-147839, 147864-147871, 147900-147903, 147927-147935, 147957-147967, 148046-148063, 148085-148095, 148113-148127, 148155-148159, 148186-148191, 148211-148223, 148411-148415, 148453-148479, 148710-148735, 148910-148927, 148961-148991, 149265-149279, 149299-149311, 149329-149343, 149663, 149681-149695, 149713-149727, 149745-149759, 149810-149823, 149886-149887, 149914-149919, 149941-149951, 150001-150015, 151351-151359, 151395-151423, 151469-151487, 151525-151551, 151810-151811, 151822, 151825, 152612-152639, 152676-152703, 152723-152735, 152811-152831, 152868-152895, 152938-152959, 153014-153023, 153067-153087, 153155-153183, 153203-153215, 153286-153311, 153336-153343, 153594-153599, 154419-154431, 154473-154495, 154520-154527, 154552-154559, 154580-154591, 154613-154623, 154905-154911, 154931-154943, 155000-155007, 155061-155071, 155093-155103, 155125-155135, 155386-155391, 155412-155423, 155447-155455, 155476-155487, 155508-155519, 155541-155551, 155572-155583, 155606-155615, 155635-155647, 155651, 155708-155709, 155711-155712, 155756, 155771-155772, 155776, 155779, 155781, 155813-155814, 155816, 155819, 155910-155911, 155914, 155917-155918, 155956-155957, 155959-155960, 155976, 155986-155987, 155990, 155993, 156003, 156009-156010, 156014, 156017, 156200, 156223-156224, 156227, 156230, 156232, 156235, 156257-156258, 156262, 156266, 156269, 156416-156417, 156421, 156424, 156432, 156436-156437, 156439-156440, 156495, 156499-156500, 156509-156510, 156742-156767, 156786-156799, 156876-156895, 156919-156927, 157156-157183, 157476-157503, 157560-157567, 157634-157663, 157687-157695, 157839-157855, 157879-157887, 157932-157951, 158031-158047, 158074-158079, 158186-158207, 158768-158783, 158840-158847, 158898-158911, 158948-158975, 159085-159103, 159150-159167, 159217-159231, 159845-159871, 159986-159999, 160080-160095, 160121-160127, 160208-160223, 160252-160255, 160634-160635, 160638, 160641, 160649-160650, 160653, 160656, 160667-160668, 160671, 160678, 160682-160683, 160688-160690, 160817-160831, 160856-160863, 160895, 161021-161023, 161045-161055, 161074-161279, 161467-161468, 161471, 161482, 161501-161502, 161505, 161509, 161532-161533, 161536, 161539, 161548-161549, 161552, 161554, 161569, 161665-161666, 161670, 161673, 161684, 161692-161693, 161695-161696, 161702, 161714-161715, 161719, 161722, 161730-161731, 161755-161791, 162076-162444, 162446-162624, 162626-162627, 162629-162630, 162697-162698, 162701, 162705, 162710, 162719-162720, 162723, 162725-162727, 162757-162758, 162761, 162767, 162802-162803, 162806, 162808, 162942-162943, 162956, 163038-163093, 163097-163129, 163131-163154, 163161, 163171, 163197-163198, 163328-163368, 163424-163431, 163456-163458, 163467-163472, 163474-163480, 163519-163839
  Free inodes: 30529-38160
Group 5: (Blocks 163840-196607) [INODE_UNINIT, ITABLE_ZEROED]
  Checksum 0x54a9, unused inodes 7632
  Backup superblock at 163840, Group descriptors at 163841-163841
  Reserved GDT blocks at 163842-163900
  Block bitmap at 66, Inode bitmap at 82
  Inode table at 2478-2954
  13589 free blocks, 7632 free inodes, 0 directories, 7632 unused inodes
  Free blocks: 163901, 163960-163967, 164087-164369, 164469-164476, 164537-164543, 164603-164863, 164912-164920, 164987-164991, 165096-165114, 165318-165375, 165888-165929, 165950-166008, 166144-166462, 166739-166965, 167811-167871, 167913-167941, 167958-168447, 168492-168511, 168545-169023, 169084-169472, 169475-170111, 170161-170175, 170236-170495, 170610-171007, 171566-172031, 174361-174591, 174608-175103, 175105-175615, 175629-176127, 176129-177151, 177664-177685, 177687-178175, 182272-182278, 182284-183343, 183345-183816, 183818-184319, 187641-187916, 187925-187926, 187933-187934, 187941-187942, 187950, 187952-187953, 187967-187968, 187972, 187976, 187987, 187996-187997, 188001, 188004, 188007, 188029-188031, 188039-188424, 188426-188936, 188940-189455, 189457-189458, 189462-189951, 189962-190463, 194560-194570, 194572-195081, 195083-195583
  Free inodes: 38161-45792
Group 6: (Blocks 196608-229375) [INODE_UNINIT, ITABLE_ZEROED]
  Checksum 0xf64a, unused inodes 7632
  Block bitmap at 67, Inode bitmap at 83
  Inode table at 2955-3431
  27303 free blocks, 7632 free inodes, 0 directories, 7632 unused inodes
  Free blocks: 198166-198527, 198535, 198551, 198566-202253, 202259, 202699-205320, 205322-205839, 205928-206847, 210169-214534, 214536-215051, 215056-216070, 216072-216582, 216585-217145, 217147-217608, 217611-219144, 219146-219667, 219671-220681, 220683-222727, 222729-229375
  Free inodes: 45793-53424
Group 7: (Blocks 229376-244223) [INODE_UNINIT, ITABLE_ZEROED]
  Checksum 0x6fc9, unused inodes 7632
  Backup superblock at 229376, Group descriptors at 229377-229377
  Reserved GDT blocks at 229378-229436
  Block bitmap at 68, Inode bitmap at 84
  Inode table at 3432-3908
  14787 free blocks, 7632 free inodes, 0 directories, 7632 unused inodes
  Free blocks: 229437-244223
  Free inodes: 53425-61056

[-- Attachment #3: e2fsck.output --]
[-- Type: application/octet-stream, Size: 1895 bytes --]

Superblock has an invalid journal (inode 8).
Clear? yes

*** ext3 journal has been deleted - filesystem is now ext2 only ***

Superblock last write time (Wed Nov  9 14:57:58 2011,
	now = Fri Sep 16 00:51:41 2011) is in the future.
Fix? yes

ideflash was not cleanly unmounted, check forced.
Resize inode not valid.  Recreate? yes

Pass 1: Checking inodes, blocks, and sizes
Reserved inode 3 (<The ACL index inode>) has invalid mode.  Clear? yes

Inode 3, i_blocks is 8, should be 0.  Fix? yes

Reserved inode 4 (<The ACL data inode>) has invalid mode.  Clear? yes

Inode 4, i_blocks is 8, should be 0.  Fix? yes

Reserved inode 5 (<The boot loader inode>) has invalid mode.  Clear? yes

Inode 5, i_blocks is 8, should be 0.  Fix? yes

Reserved inode 6 (<The undelete directory inode>) has invalid mode.  Clear? yes

Inode 6, i_blocks is 16, should be 0.  Fix? yes

Journal inode is not in use, but contains data.  Clear? yes

Pass 2: Checking directory structure
Entry 'mtab' in /etc (15) has deleted/unused inode 1201.  Clear? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -32935 -(65536--69631)
Fix? yes

Free blocks count wrong for group #0 (26110, counted=26111).
Fix? yes

Free blocks count wrong for group #1 (12412, counted=12413).
Fix? yes

Free blocks count wrong for group #2 (3618, counted=7714).
Fix? yes

Free blocks count wrong (111898, counted=115996).
Fix? yes

Inode bitmap differences:  -1201
Fix? yes

Free inodes count wrong for group #0 (0, counted=1).
Fix? yes

Free inodes count wrong (36509, counted=36510).
Fix? yes

Recreate journal? yes

Creating journal (4096 blocks):  Done.

*** journal has been re-created - filesystem is now ext3 again ***

ideflash: ***** FILE SYSTEM WAS MODIFIED *****
ideflash: 24546/61056 files (0.2% non-contiguous), 132325/244224 blocks

Re: ext4 settings in an embedded system

[Eric Sandeen]

On 11/16/12 10:18 AM, Ohlsson, Fredrik (GE Healthcare, consultant) wrote:
> Thank you very much for your helpful response and answers.
> 
> I would like to describe some background for our embedded system. The
> system has recently been upgraded, before the upgrade we did not see
> "filesystem" related problems. Before the upgrade we used  another
> filesystem, kernel and IDE Flash Disk(kernel 2.6.12 kernel, reiserfs and
> a smaller 256 MB IDE Flash Disk).  Today we have kernel 2.6.32,
> ext4(default options) and a 1GB Transend TS1GDOM44H-S IDE Flash Disk.
> We have a very low IO intensity towards the flash disk and low
> requirements on the filesystem performance. 
> 
> In our case with the file of size 0 bytes. We use a bash shell script to
> upgrade our application. The shell script calls the program "tar" and
> tar overwrites/recreates  our application bin file. After some minutes
> the power is cut and our bin file had the new size 0 bytes when the
> system came up again . This particular case was solved by adding a sync
> in the end of our upgrade shell script. I still don't understand why the
> data is not committed to the disk after several minutes? Even if tar
> leaves the file truncated tar must have closed the file and ext4 would
> have done an implicit write-back?

I would have expected all data to make it out after several minutes,
yes (how many is several?)  Background writeout kicks off every
30s by default, I would have expected that after at most a couple of those
cycles you would have seen it all make it to disk.  To investigate, it
might be worth tracing the system to see whether or not it is pushing data
out when the script completes (iostat might be simplest, or blktrace).

I'm curious, did the sync at the end of the script take a very long
time to complete?

> We are worried that this will happen again where we use programs not
> written by ourselves. Will the nodelalloc option solve this behavior?

Doubtful.  For starters, I'd argue that nodelalloc is a less tested
and therefore potentially more bug-prone path in ext4.  

> If I understand you right the corrupted journal superblock (inode #8) is
> most probably a result of a problem in the IDE Flash Disk.  

ok so you got:

> Superblock has an invalid journal (inode 8).
> Clear? yes

after this, of course, fsck is at a disadvantage for recovering the fs
and finds many more errors.  However, things like bad resize inodes
& bad acl index inodes are a bit more suprising; those were written
at mkfs time.

What version of e2fsprogs was this?

You could get the journal message for a few reasons, unfortunately e2fsck
doesn't say which one it was.  An e2image (or maybe just raw dd)
of the fs prior to the repair would offer some clues to someone with
time to investigate.

> I have
> attached dumpe2fs.output from this problem. I booted the system from a
> usb-drive and the filesystem could be repaired by e2fsck, but this is
> not something the customer can do, we have to replace units like this.
> The Transend TS1GDOM44H-S IDE Flash Disk is intended for demanding
> embedded systems that require reliability, I guess it could still be the
> part creating this problem. 
> 
> I think you are saying that ext4 should work fine in our setup were we
> have regular power-cuts if we use sync/fsync and applies the following
> settings:
> -barrier, which we already use by default (Can't see barrier in the
> attached dumpe2fs.output thou).

it's just runtime behavior, so you wouldn't see it there.

> -nodelalloc

I'm a little skeptical of that.  I think you need to get to the root
cause before you start turning more knobs.  I guess I am most skeptical
about the storage, for starters.

I can't tell if the transcend device has a cache or not; presumably
the flash controller does.  I have no idea if it responds to cache flush
requests, or not.

If you look at dmesg, you probably get something like:
[sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

but what the device actually does internally - who knows.

I wonder if a test like this would be interesting:

* Boot system from USB.
* Write a unique pattern directly to the transcend block device, not through the fs.
* Use something like lmdd or xfs_io or some tool which will write a pattern, not
just 0s.
* Wait a minute or two, then cut power (watch iostat maybe to see when all IO is done)
* boot up again and check the pattern

If the pattern is bad, try it again (with a different pattern) and issue
sync prior to the power cut.  See if that behaves differently.

fsync (and, I think, sys_sync) will issue cache flushes to the storage.
Simple writeback won't, AFAIK.  So it'd be interesting to see if data
is being lost inside the device when it loses power; the above test might
be decent to check that.

I'm half tempted to find one of these devices & test it myself ;)

-Eric


> You also advise us to implement or own power-cut tests.
> 
> Are there more settings that could be to our favor, like
> journal_checksum? Tune2fs data=journal?
> 
> Best Regards
> 
> Fredrik Ohlsson
> Software Engineer
> 
> 
> 
> 
> 
>