UBIFS fails & crashes on SheevaPlug

UBIFS fails & crashes on SheevaPlug

[erno]

Hello,

I installed Debian on my SheevaPlug following instructions at 
<http://www.cyrius.com/debian/kirkwood/sheevaplug/> and everything 
seemed to work fine until...

I wanted to try UBIFS on the built-in flash and followed
the instructions for formatting the internal flash
at http://www.plugcomputer.org/plugwiki/index.php/Enabling_UBIFS
except I made an empty image (no -r parameter for mkfs.ubifs).
After the ubiattach command it complained about a read-only fs
and some error messages from the kernel had appeared in the log:

[  190.500000] UBI: physical eraseblock size:   131072 bytes (128 KiB)
[  190.510000] UBI: logical eraseblock size:    129024 bytes
[  190.520000] UBI: smallest flash I/O unit:    2048
[  190.520000] UBI: sub-page size:              512
[  190.530000] UBI: VID header offset:          512 (aligned 512)
[  190.530000] UBI: data offset:                2048
[  191.630000] UBI error: ubi_io_write: error -5 while writing 512 bytes 
to PEB 4054:512, written 0 bytes
[  191.630000] UBI warning: ubi_eba_write_leb: failed to write VID 
header to LEB 2147479551:0, PEB 4054
[  191.640000] UBI: try another PEB
[  191.650000] UBI error: ubi_io_write: error -5 while writing 512 bytes 
to PEB 4053:512, written 0 bytes
[  191.660000] UBI warning: ubi_eba_write_leb: failed to write VID 
header to LEB 2147479551:0, PEB 4053
[  191.670000] UBI: try another PEB
[  191.670000] UBI error: ubi_io_write: error -5 while writing 512 bytes 
to PEB 4052:512, written 0 bytes
[  191.680000] UBI warning: ubi_eba_write_leb: failed to write VID 
header to LEB 2147479551:0, PEB 4052
[  191.690000] UBI: try another PEB
[  191.690000] UBI error: ubi_io_write: error -5 while writing 512 bytes 
to PEB 4051:512, written 0 bytes
[  191.700000] UBI warning: ubi_eba_write_leb: failed to write VID 
header to LEB 2147479551:0, PEB 4051
[  191.710000] UBI warning: ubi_ro_mode: switch to read-only mode
[  191.720000] UBI error: autoresize: cannot auto-resize volume 0

Shortly after this the machine stopped answering ssh/ping and rebooted 
(I had panic on oops and reboot on panic turned on). I didn't have the 
serial console hooked up now so I didn't see messages, but the last time 
I tried it there was some slab corruption oops after the ubiattach 
attempt.

The previous time it was using the stock lenny kernel, this time
it was linux-image-2.6.30-1-kirkwood_2.6.30-3_armel.deb from
http://people.debian.org/~tbm/orion/.

I also get messages like this on boot:

[    4.420000] uncorrectable error : <3>uncorrectable error : 
<3>end_request: I/O error, dev mtdblock0, sector 0
[    4.430000] Buffer I/O error on device mtdblock0, logical block 0
[    4.440000] uncorrectable error : <3>uncorrectable error : 
<3>end_request: I/O error, dev mtdblock0, sector 0
[    4.440000] Buffer I/O error on device mtdblock0, logical block 0
[    4.480000] uncorrectable error : <3>uncorrectable error : 
<3>end_request: I/O error, dev mtdblock0, sector 0

but I'm hopeful it's because something is trying to automatically
access the builtin flash which is in limbo state?

Any ideas welcome,
Erno

Re: UBIFS fails & crashes on SheevaPlug

[Adrian Hunter]

erno@gulp.u--3.com wrote:
> Hello,
> 
> I installed Debian on my SheevaPlug following instructions at 
> <http://www.cyrius.com/debian/kirkwood/sheevaplug/> and everything 
> seemed to work fine until...
> 
> I wanted to try UBIFS on the built-in flash and followed
> the instructions for formatting the internal flash
> at http://www.plugcomputer.org/plugwiki/index.php/Enabling_UBIFS
> except I made an empty image (no -r parameter for mkfs.ubifs).

Isn't the -r parameter mandatory.  For an empty image, use an empty
directory for the -r parameter.

Re: UBIFS fails & crashes on SheevaPlug

[Corentin Chary]

On Thu, Jul 23, 2009 at 10:08 PM, Adrian Hunter<adrian.hunter@nokia.com> wrote:
> erno@gulp.u--3.com wrote:
>>
>> Hello,
>>
>> I installed Debian on my SheevaPlug following instructions at
>> <http://www.cyrius.com/debian/kirkwood/sheevaplug/> and everything seemed to
>> work fine until...
>>
>> I wanted to try UBIFS on the built-in flash and followed
>> the instructions for formatting the internal flash
>> at http://www.plugcomputer.org/plugwiki/index.php/Enabling_UBIFS
>> except I made an empty image (no -r parameter for mkfs.ubifs).
>
> Isn't the -r parameter mandatory.  For an empty image, use an empty
> directory for the -r parameter.

With recent version of mkfs.ubifs you can format directly an ubi volume without
-r parameter (mkfs.ubifs /dev/ubi0_0).

Can you give us the exact commands you used ?

-- 
Corentin Chary
http://xf.iksaif.net - http://uffs.org

Re: UBIFS fails & crashes on SheevaPlug

[erno]

On Thu, Jul 23, 2009 at 11:28:01PM +0200, Corentin Chary wrote:
> Can you give us the exact commands you used ?

Straight copy-paste (except for the -r removal) from 
http://www.plugcomputer.org/plugwiki/index.php/Enabling_UBIFS - so:

   94  cd /var/tmp/
   95  mkfs.ubifs -m 2048 -e 129024 -c 4096 -o ubifs.img -x zlib
   96  /usr/local/usr/sbin/mkfs.ubifs -m 2048 -e 129024 -c 4096 -o 
ubifs.img -x zlib
   99  ubinize -o ubi.img -m 2048 -p 128KiB -s 512 ubi.cfg
  101  cat > ubi.cfg

  102  /usr/local/usr/sbin/ubinize -o ubi.img -m 2048 -p 128KiB -s 512 
ubi.cfg
  105  ubiformat /dev/mtd2 -s 512 -f ubi.img
  106  /usr/local/usr/sbin/ubiformat /dev/mtd2 -s 512 -f ubi.img
  107  /usr/local/usr/sbin/ubiattach /dev/ubi_ctrl -m 2

(funny paths courtesy of mtd-utils "make install")

ubi.cfg is

[ubifs]
mode=ubi
image=ubifs.img
vol_id=0
vol_size=256MiB
vol_type=dynamic
vol_name=rootfs
vol_flags=autoresize

Re: UBIFS fails & crashes on SheevaPlug

[Artem Bityutskiy]

On 07/23/2009 09:59 PM, erno@gulp.u--3.com wrote:
> Hello,
>
> I installed Debian on my SheevaPlug following instructions at
> <http://www.cyrius.com/debian/kirkwood/sheevaplug/>  and everything
> seemed to work fine until...
>
> I wanted to try UBIFS on the built-in flash and followed
> the instructions for formatting the internal flash
> at http://www.plugcomputer.org/plugwiki/index.php/Enabling_UBIFS
> except I made an empty image (no -r parameter for mkfs.ubifs).
> After the ubiattach command it complained about a read-only fs
> and some error messages from the kernel had appeared in the log:
>
> [  190.500000] UBI: physical eraseblock size:   131072 bytes (128 KiB)
> [  190.510000] UBI: logical eraseblock size:    129024 bytes
> [  190.520000] UBI: smallest flash I/O unit:    2048
> [  190.520000] UBI: sub-page size:              512
> [  190.530000] UBI: VID header offset:          512 (aligned 512)
> [  190.530000] UBI: data offset:                2048
> [  191.630000] UBI error: ubi_io_write: error -5 while writing 512 bytes
> to PEB 4054:512, written 0 bytes
> [  191.630000] UBI warning: ubi_eba_write_leb: failed to write VID
> header to LEB 2147479551:0, PEB 4054
> [  191.640000] UBI: try another PEB
> [  191.650000] UBI error: ubi_io_write: error -5 while writing 512 bytes
> to PEB 4053:512, written 0 bytes
> [  191.660000] UBI warning: ubi_eba_write_leb: failed to write VID
> header to LEB 2147479551:0, PEB 4053
> [  191.670000] UBI: try another PEB
> [  191.670000] UBI error: ubi_io_write: error -5 while writing 512 bytes
> to PEB 4052:512, written 0 bytes
> [  191.680000] UBI warning: ubi_eba_write_leb: failed to write VID
> header to LEB 2147479551:0, PEB 4052
> [  191.690000] UBI: try another PEB
> [  191.690000] UBI error: ubi_io_write: error -5 while writing 512 bytes
> to PEB 4051:512, written 0 bytes
> [  191.700000] UBI warning: ubi_eba_write_leb: failed to write VID
> header to LEB 2147479551:0, PEB 4051
> [  191.710000] UBI warning: ubi_ro_mode: switch to read-only mode
> [  191.720000] UBI error: autoresize: cannot auto-resize volume 0

http://www.linux-mtd.infradead.org/faq/ubi.html#L_subpage_verify_fail

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS fails & crashes on SheevaPlug

[Erno Kuusela]

On Fri, Jul 24, 2009 at 08:53:00AM +0300, Artem Bityutskiy wrote:
> On 07/23/2009 09:59 PM, erno@gulp.u--3.com wrote:
>> [  191.690000] UBI error: ubi_io_write: error -5 while writing 512 bytes
>> to PEB 4051:512, written 0 bytes
>> [  191.700000] UBI warning: ubi_eba_write_leb: failed to write VID
>> header to LEB 2147479551:0, PEB 4051
>> [  191.710000] UBI warning: ubi_ro_mode: switch to read-only mode
>> [  191.720000] UBI error: autoresize: cannot auto-resize volume 0
>
> http://www.linux-mtd.infradead.org/faq/ubi.html#L_subpage_verify_fail

Ah, ok. I did google for that error message but searching for 
"ubi_io_write: error -5" doesn't yield anything useful for the first 
screenful of results..

So workaround would be to disable CONFIG_MTD_NAND_VERIFY_WRITE or to 
follow the "ignore sub pages" entry from the faq...

Indeed the latter seems to work, if I give --vid-hdr-offset 2048 option 
to the ubinize, ubiformat & ubiattach commands. Thanks!

Maybe it would be better to disable the verify & subpages combo
until the bug is fixed? Especially since it crashes the kernel...

Also, ubiformat mentioned 1400+ "bad" eraseblocks, what should I make of 
that? Is it really failed flash cells, just some blocks reserved for 
u-boot and stuff, or blocks traumatized from my earlier misadventures 
with ubiformat parameters?

Erno

Re: UBIFS fails & crashes on SheevaPlug

[Artem Bityutskiy]

On 07/24/2009 04:12 PM, Erno Kuusela wrote:
> Ah, ok. I did google for that error message but searching for
> "ubi_io_write: error -5" doesn't yield anything useful for the first
> screenful of results..
>
> So workaround would be to disable CONFIG_MTD_NAND_VERIFY_WRITE or to
> follow the "ignore sub pages" entry from the faq...
>
> Indeed the latter seems to work, if I give --vid-hdr-offset 2048 option
> to the ubinize, ubiformat&  ubiattach commands. Thanks!

It should work without this. Your flash driver reports that your flash
supports 512-byte sub-pages. This means you may save your flash space
and put both UBI headers to the first NAND page. See this explanation:

http://www.linux-mtd.infradead.org/doc/ubi.html#L_overhead

If you follow links from this little section, you should grasp the picture.

So, disabling this option should help without the --vid-hdr-offset 2048
stuff. If it does not, then you should fix your driver.

> Maybe it would be better to disable the verify&  subpages combo
> until the bug is fixed? Especially since it crashes the kernel...

May be, but it is better to just fix this.

> Also, ubiformat mentioned 1400+ "bad" eraseblocks, what should I make of
> that? Is it really failed flash cells, just some blocks reserved for
> u-boot and stuff, or blocks traumatized from my earlier misadventures
> with ubiformat parameters?

Hmm, this is not normal. What is the version of ubiformat you use? Could
you try the latest one from the git tree:

http://www.linux-mtd.infradead.org/doc/ubi.html#L_usptools

Just in case, here are mtd-utils compile issues FAQ:
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_mkfs_ubifs_comp
or you may always cd to ubi-utils and compile only UBI utils.

And what UBI is printing? See "UBI: number of bad PEBs:" message.

And please, send the exact ubiformat pritns.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS fails & crashes on SheevaPlug

[Nicolas Pitre]

On Thu, 23 Jul 2009, erno@gulp.u--3.com wrote:

> I also get messages like this on boot:
> 
> [    4.420000] uncorrectable error : <3>uncorrectable error : 
> <3>end_request: I/O error, dev mtdblock0, sector 0
> [    4.430000] Buffer I/O error on device mtdblock0, logical block 0
> [    4.440000] uncorrectable error : <3>uncorrectable error : 
> <3>end_request: I/O error, dev mtdblock0, sector 0
> [    4.440000] Buffer I/O error on device mtdblock0, logical block 0
> [    4.480000] uncorrectable error : <3>uncorrectable error : 
> <3>end_request: I/O error, dev mtdblock0, sector 0
> 
> but I'm hopeful it's because something is trying to automatically
> access the builtin flash which is in limbo state?

This is because something (udev I think) is trying to automatically 
access every block devices including /dev/mtdblock0. In this case, this 
corresponds to the bootloader.  The problem is that the bootrom 
responsible for fetching u-boot from the flash and copy it to ram does 
understand only one type of ECC, and that ECC format is not (yet) 
supported by Linux.  So this is a known issue which you don't have to 
worry about and this have no relation with your UBIFS issues.

Nicolas

Re: UBIFS fails & crashes on SheevaPlug

[Artem Bityutskiy]

On 07/24/2009 04:12 PM, Erno Kuusela wrote:
> Also, ubiformat mentioned 1400+ "bad" eraseblocks, what should I make of
> that? Is it really failed flash cells, just some blocks reserved for
> u-boot and stuff, or blocks traumatized from my earlier misadventures
> with ubiformat parameters?

Oh, I've just got it. When ubiformat prints:

ubiformat: bad eraseblocks: 1400

this means that you have 1 eraseblock, and it is eraseblock number 1400.
Nothing to worry about. But it is confusing, so I'll change this to

ubiformat: bad eraseblock numbers: A, B, C

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS fails & crashes on SheevaPlug

[Erno Kuusela]

On Fri, Jul 24, 2009 at 05:38:27PM +0300, Artem Bityutskiy wrote:
> On 07/24/2009 04:12 PM, Erno Kuusela wrote:
>> So workaround would be to disable CONFIG_MTD_NAND_VERIFY_WRITE or to
>> follow the "ignore sub pages" entry from the faq...
>>
>> Indeed the latter seems to work, if I give --vid-hdr-offset 2048 option
>> to the ubinize, ubiformat&  ubiattach commands. Thanks!
>
> It should work without this. Your flash driver reports that your flash
> supports 512-byte sub-pages. This means you may save your flash space
> and put both UBI headers to the first NAND page. See this explanation:
>
> http://www.linux-mtd.infradead.org/doc/ubi.html#L_overhead
>
> If you follow links from this little section, you should grasp the picture.
>
> So, disabling this option should help without the --vid-hdr-offset 2048
> stuff. If it does not, then you should fix your driver.

Yeah, but I'm trying to make do without building my own kernels if 
possible.

And yes, the ubiformat "bad eraseblocks" message was indeed
a misunderstanding from my part.

There was one more error (for benefit of anyone else hitting the same 
problem): mkfs.ubifs -e 129024 parameter from 
http://www.plugcomputer.org/plugwiki/index.php/Enabling_UBIFS#Make_ubi_image_out_of_the_ubifs 
didn't work for me (got "validate_sb: LEB size mismatch: 129024 in 
superblock, 126976 real"). Fixing this to the number indicated in the 
error message (126976) fixed it. 

Looks like I have ubifs working now. :)

Re: UBIFS fails & crashes on SheevaPlug

[Artem Bityutskiy]

On 07/26/2009 06:35 PM, Erno Kuusela wrote:
> There was one more error (for benefit of anyone else hitting the same
> problem): mkfs.ubifs -e 129024 parameter from
> http://www.plugcomputer.org/plugwiki/index.php/Enabling_UBIFS#Make_ubi_image_out_of_the_ubifs
> didn't work for me (got "validate_sb: LEB size mismatch: 129024 in
> superblock, 126976 real"). Fixing this to the number indicated in the
> error message (126976) fixed it.

Yes, I've just also created this FAQ entry:
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_lebsz_mismatch

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)