validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Andrew Murray]

Hello,

Is there a way to determine the minimum UBI volume size which will
support a given UBIFS filesystem image?

I've read the infradead.org FAQ's and browsed the source - however can
not find a suitable answer. My current conclusion is that the minimum
UBI volume is somehow dependant on the filesystem size, the log and
the journal. I'm working on an upgrade mechanism which uses UBI/UBIFS
and only creates UBI volumes of the minimum size to support a UBIFS
filesystem and thus struggling to determine the safest minimum size.

This is what I've attempted. In these examples UBIFS is claiming that
the journal size is 71 LEB's - however the filesystem will only mount
if my UBI volume is 77 LEB's - how do I account for this difference?

# ubinfo -a
UBI version:                    1
Count of UBI devices:           1
UBI control device major/minor: 10:63
Present UBI devices:            ubi0
ubi0
Volumes count:                           1
Logical eraseblock size:                 129024 bytes, 126.0 KiB
Total amount of logical eraseblocks:     911 (117540864 bytes, 112.1 MiB)
Amount of available logical eraseblocks: 401 (51738624 bytes, 49.3 MiB)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       1
Count of reserved physical eraseblocks:  9
Current maximum erase counter value:     350
Minimum input/output unit size:          2048 bytes
Character device major/minor:            251:0
Present volumes:                         0
Volume ID:   0 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        497 LEBs (64124928 bytes, 61.2 MiB)
State:       OK
Name:        rootfs
Character device major/minor: 251:1

# ls -l rootfs.ubifs
-rw-r--r--    1 default  default   6838272 Jan 10  2011 rootfs.ubifs
# ubimkvol /dev/ubi0 -N test -s 6838272
Volume ID 1, size 53 LEBs (6838272 bytes, 6.5 MiB), LEB size 129024
bytes (126.0 KiB), dynamic, name "test", alignment 1
# ubiupdatevol /dev/ubi0_1 rootfs.ubifs
# mount -t ubifs ubi0:test /tmp/a/
c->max_bud_bytes: 8388608 > c->leb_size * c->main_lebs: 5419008
UBIFS error (pid 1109): validate_sb: bad superblock, error 8
mount: mounting ubi0:test on /tmp/a/ failed: Invalid argument

# ubirmvol  /dev/ubi0 -N test
# ubimkvol /dev/ubi0 -N test --lebs=77
Volume ID 1, size 77 LEBs (9934848 bytes, 9.5 MiB), LEB size 129024
bytes (126.0 KiB), dynamic, name "test", alignment 1
# ubiupdatevol /dev/ubi0_1 rootfs.ubifs
# mount -t ubifs ubi0:test /tmp/a/
c->max_bud_bytes: 8388608 < c->leb_size * c->main_lebs: 8515584
UBIFS: mounted UBI device 0, volume 1, name "test"
UBIFS: file system size:   8515584 bytes (8316 KiB, 8 MiB, 66 LEBs)
UBIFS: journal size:       9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root:  0 bytes (0 KiB)
# umount /tmp/a
UBIFS: un-mount UBI device 0, volume 1

# ubimkvol /dev/ubi0 -N test --lebs=71
Volume ID 1, size 71 LEBs (9160704 bytes, 8.7 MiB), LEB size 129024
bytes (126.0 KiB), dynamic, name "test", alignment 1
# ubiupdatevol /dev/ubi0_1 rootfs.ubifs
# mount -t ubifs ubi0:test /tmp/a/
c->max_bud_bytes: 8388608 > c->leb_size * c->main_lebs: 7741440
UBIFS error (pid 1136): validate_sb: bad superblock, error 8
mount: mounting ubi0:test on /tmp/a/ failed: Invalid argument

# ubirmvol /dev/ubi0 -N test
# ubimkvol /dev/ubi0 -N test --lebs=72
Volume ID 1, size 72 LEBs (9289728 bytes, 8.9 MiB), LEB size 129024
bytes (126.0 KiB), dynamic, name "test", alignment 1
# ubiupdatevol /dev/ubi0_1 rootfs.ubifs
# mount -t ubifs ubi0:test /tmp/a/
c->max_bud_bytes: 8388608 > c->leb_size * c->main_lebs: 7870464
UBIFS error (pid 1144): validate_sb: bad superblock, error 8
mount: mounting ubi0:test on /tmp/a/ failed: Invalid argument

# ubirmvol /dev/ubi0 -N test
# ubimkvol /dev/ubi0 -N test --lebs=76
Volume ID 1, size 76 LEBs (9805824 bytes, 9.4 MiB), LEB size 129024
bytes (126.0 KiB), dynamic, name "test", alignment 1
# ubiupdatevol /dev/ubi0_1 rootfs.ubifs
# mount -t ubifs ubi0:test /tmp/a/
c->max_bud_bytes: 8388608 > c->leb_size * c->main_lebs: 8386560
UBIFS error (pid 1152): validate_sb: bad superblock, error 8
mount: mounting ubi0:test on /tmp/a/ failed: Invalid argument

# ubirmvol /dev/ubi0 -N test
# ubimkvol /dev/ubi0 -N test --lebs=77
Volume ID 1, size 77 LEBs (9934848 bytes, 9.5 MiB), LEB size 129024
bytes (126.0 KiB), dynamic, name "test", alignment 1
# ubiupdatevol /dev/ubi0_1 rootfs.ubifs
# mount -t ubifs ubi0:test /tmp/a/
c->max_bud_bytes: 8388608 < c->leb_size * c->main_lebs: 8515584
UBIFS: mounted UBI device 0, volume 1, name "test"
UBIFS: file system size:   8515584 bytes (8316 KiB, 8 MiB, 66 LEBs)
UBIFS: journal size:       9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root:  0 bytes (0 KiB)

Many Thanks,

Andrew Murray

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Andrew Murray]

On 11 January 2011 09:47, Andrew Murray <amurray@mpcdata.com> wrote:
>
> Is there a way to determine the minimum UBI volume size which will
> support a given UBIFS filesystem image?
>

My conclusion is this (with my version of kernel and mkfs.ubifs tool):

(Min No. UBI Volume LEBS) = (jrn_lebs) + 3 + (log lebs) + (lpt_lebs) +
(orph_lebs)

And these values can all be determined (or specified) by the
mkfs.ubifs tool by using the verbose -v flag.

This seems to hold true - Does this seem reasonable?

Thanks,

Andrew Murray

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Andrew Murray]

On 15 January 2011 10:48, Andrew Murray <amurray@mpcdata.com> wrote:
> On 11 January 2011 09:47, Andrew Murray <amurray@mpcdata.com> wrote:
>>
>> Is there a way to determine the minimum UBI volume size which will
>> support a given UBIFS filesystem image?
>>
>
> My conclusion is this (with my version of kernel and mkfs.ubifs tool):
>
> (Min No. UBI Volume LEBS) = (jrn_lebs) + 3 + (log lebs) + (lpt_lebs) +
> (orph_lebs)
>
> And these values can all be determined (or specified) by the
> mkfs.ubifs tool by using the verbose -v flag.
>
> This seems to hold true - Does this seem reasonable?

After reading the ODP presentation, white paper and reading the source
- I do not understand the purpose of the second condition in the
following test case in validate_sb:

        if (c->max_bud_bytes < (long long)c->leb_size * UBIFS_MIN_BUD_LEBS ||
            c->max_bud_bytes > (long long)c->leb_size * c->main_lebs

It seems to be asserting that there is enough space on the volume to
cater for the maximum size of the journal. However in the case where
the UBI volume is similar in size to the UBIFS image data -
'main_lebs' will mostly be absent of free space lebs. Therefore there
may not be enough room for max_bud_bytes amounts of journal. And in
any case the journal code seems to cater well for there not being
enough free lebs. (Presumably it copes when the volume is nearly
full).

Therefore is this condition superfluous?

Thanks,

Andrew Murray

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Artem Bityutskiy]

On Sat, 2011-01-15 at 22:09 +0000, Andrew Murray wrote:
> On 15 January 2011 10:48, Andrew Murray <amurray@mpcdata.com> wrote:
> > On 11 January 2011 09:47, Andrew Murray <amurray@mpcdata.com> wrote:
> >>
> >> Is there a way to determine the minimum UBI volume size which will
> >> support a given UBIFS filesystem image?
> >>
> >
> > My conclusion is this (with my version of kernel and mkfs.ubifs tool):
> >
> > (Min No. UBI Volume LEBS) = (jrn_lebs) + 3 + (log lebs) + (lpt_lebs) +
> > (orph_lebs)
> >
> > And these values can all be determined (or specified) by the
> > mkfs.ubifs tool by using the verbose -v flag.
> >
> > This seems to hold true - Does this seem reasonable?
> 
> After reading the ODP presentation, white paper and reading the source
> - I do not understand the purpose of the second condition in the
> following test case in validate_sb:
> 
>         if (c->max_bud_bytes < (long long)c->leb_size * UBIFS_MIN_BUD_LEBS ||
>             c->max_bud_bytes > (long long)c->leb_size * c->main_lebs
> 
> It seems to be asserting that there is enough space on the volume to
> cater for the maximum size of the journal. However in the case where
> the UBI volume is similar in size to the UBIFS image data -
> 'main_lebs' will mostly be absent of free space lebs. Therefore there
> may not be enough room for max_bud_bytes amounts of journal. And in
> any case the journal code seems to cater well for there not being
> enough free lebs. (Presumably it copes when the volume is nearly
> full).
> 
> Therefore is this condition superfluous?

Hi, I'm not able to give you any reasonable answer right now (vacation,
and already have long backlog of e-mails to answer), sorry for this. But
the space calculations may have issues - we did not work on that well
enough, and did not test them with different journal sizes, etc. Just
did not spend much time on those things - wrote the code which
calculates some default sizes and used that all the time. AFSIK,
mkfs.ubifs is mostly copy-paste of the kernel code, including the
default geometry calculations.

Please, feel free to send patches.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Artem Bityutskiy]

On Tue, 2011-01-11 at 09:47 +0000, Andrew Murray wrote:
> Hello,
> 
> Is there a way to determine the minimum UBI volume size which will
> support a given UBIFS filesystem image?
> 
> I've read the infradead.org FAQ's and browsed the source - however can
> not find a suitable answer. My current conclusion is that the minimum
> UBI volume is somehow dependant on the filesystem size, the log and
> the journal. I'm working on an upgrade mechanism which uses UBI/UBIFS
> and only creates UBI volumes of the minimum size to support a UBIFS
> filesystem and thus struggling to determine the safest minimum size.
> 
> This is what I've attempted. In these examples UBIFS is claiming that
> the journal size is 71 LEB's - however the filesystem will only mount
> if my UBI volume is 77 LEB's - how do I account for this difference?

There are 5 (AFAIR) areas in UBIFS:

1. Superblock - always 1 LEB.
2. Log - depends on maximum flash size (-c mkfs.ubifs option) and LEB
   size
3. Lprops - depends on maximum flash size and LEB size.
3. Orphans - I think 1 LEB is enough 
4. Main area - used for various purposes:
   1. the FS index needs at least 3 LEBs for the FS index
   2. 1 LEB for GC
   3. 1 for deletions (because deleting stuff, e.g., file, needs
      writing deletion entries to flash, which also needs space,
      so we need to reserve it to make sure that if your FS is full,
      you still can delete your files).
   4. buds which are part of the journal - need at least 3

Note, Log + buds = the journal.

As you have already found out the journal size is configurable, so you
can make it to be of minimum size.

Also you have already found that there are some functions which try to
calculate default sizes.

In fs/ubifs/ubifs-media.h you can find constants and comments about
minimum required LEBs [1]

Anyway, the only way for the user to affect the minimum FS size is the
--jrn-size option of mkfs.ubifs. How to calculate what will be the
minimum? Not sure, probably UBIFS_MIN_JNL_LEBS = 5 LEBs or something
close to this.

I agree that this is a bit complex, feel free to contribute some piece
of documentation to the mtd web site - you'll find the web site source
here: http://git.infradead.org/mtd-www.git

Just clone it and send a patch.

> 
> # ubinfo -a
> UBI version:                    1
> Count of UBI devices:           1
> UBI control device major/minor: 10:63
> Present UBI devices:            ubi0
> ubi0
> Volumes count:                           1
> Logical eraseblock size:                 129024 bytes, 126.0 KiB
> Total amount of logical eraseblocks:     911 (117540864 bytes, 112.1 MiB)
> Amount of available logical eraseblocks: 401 (51738624 bytes, 49.3 MiB)
> Maximum count of volumes                 128
> Count of bad physical eraseblocks:       1
> Count of reserved physical eraseblocks:  9
> Current maximum erase counter value:     350
> Minimum input/output unit size:          2048 bytes
> Character device major/minor:            251:0
> Present volumes:                         0
> Volume ID:   0 (on ubi0)
> Type:        dynamic
> Alignment:   1
> Size:        497 LEBs (64124928 bytes, 61.2 MiB)
> State:       OK
> Name:        rootfs
> Character device major/minor: 251:1


> 
> # ls -l rootfs.ubifs
> -rw-r--r--    1 default  default   6838272 Jan 10  2011 rootfs.ubifs
> # ubimkvol /dev/ubi0 -N test -s 6838272
> Volume ID 1, size 53 LEBs (6838272 bytes, 6.5 MiB), LEB size 129024
> bytes (126.0 KiB), dynamic, name "test", alignment 1
> # ubiupdatevol /dev/ubi0_1 rootfs.ubifs

If you did not use mkfs.ubifs, then UBIFS is doing "auto-formatting" and
just picks some "reasonable" default journal size, which is not the
minimum possible. You can make it smaller with "mkfs.ubifs --jrn-size".

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Artem Bityutskiy]

On Sat, 2011-01-15 at 10:48 +0000, Andrew Murray wrote:
> On 11 January 2011 09:47, Andrew Murray <amurray@mpcdata.com> wrote:
> >
> > Is there a way to determine the minimum UBI volume size which will
> > support a given UBIFS filesystem image?
> >
> 
> My conclusion is this (with my version of kernel and mkfs.ubifs tool):
> 
> (Min No. UBI Volume LEBS) = (jrn_lebs) + 3 + (log lebs) + (lpt_lebs) +
> (orph_lebs)
> 
> And these values can all be determined (or specified) by the
> mkfs.ubifs tool by using the verbose -v flag.
> 
> This seems to hold true - Does this seem reasonable?

These are definitions for ubifs-media.h kernel header file, and the
absolute minimum is UBIFS_MIN_LEB_CNT.

/*
 * The below constants define the absolute minimum values for various UBIFS
 * media areas. Many of them actually depend of flash geometry and the FS
 * configuration (number of journal heads, orphan LEBs, etc). This means that
 * the smallest volume size which can be used for UBIFS cannot be pre-defined
 * by these constants. The file-system that meets the below limitation will not
 * necessarily mount. UBIFS does run-time calculations and validates the FS
 * size.
 */

/* Minimum number of logical eraseblocks in the log */
#define UBIFS_MIN_LOG_LEBS 2
/* Minimum number of bud logical eraseblocks (one for each head) */
#define UBIFS_MIN_BUD_LEBS 3
/* Minimum number of journal logical eraseblocks */
#define UBIFS_MIN_JNL_LEBS (UBIFS_MIN_LOG_LEBS + UBIFS_MIN_BUD_LEBS)
/* Minimum number of LPT area logical eraseblocks */
#define UBIFS_MIN_LPT_LEBS 2
/* Minimum number of orphan area logical eraseblocks */
#define UBIFS_MIN_ORPH_LEBS 1
/*
 * Minimum number of main area logical eraseblocks (buds, 3 for the index, 1
 * for GC, 1 for deletions, and at least 1 for committed data).
 */
#define UBIFS_MIN_MAIN_LEBS (UBIFS_MIN_BUD_LEBS + 6)

/* Minimum number of logical eraseblocks */
#define UBIFS_MIN_LEB_CNT (UBIFS_SB_LEBS + UBIFS_MST_LEBS + \
                           UBIFS_MIN_LOG_LEBS + UBIFS_MIN_LPT_LEBS + \
                           UBIFS_MIN_ORPH_LEBS + UBIFS_MIN_MAIN_LEBS)
-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Artem Bityutskiy]

On Tue, 2011-01-18 at 11:16 +0200, Artem Bityutskiy wrote:
> Anyway, the only way for the user to affect the minimum FS size is the
> --jrn-size option of mkfs.ubifs. How to calculate what will be the
> minimum? Not sure, probably UBIFS_MIN_JNL_LEBS = 5 LEBs or something
> close to this.

Ah, this is wrong, you can set the log size with --log-lebs as well.

So UBIFS journal = log + buds.

--log-lebs set the log size
--jrn-size sets the buds size.

But AFAICS, you can use --jrn-size and mkfs.ubifs will automatically
calculate minimum amount of log lebs required.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Artem Bityutskiy]

On Sat, 2011-01-15 at 22:09 +0000, Andrew Murray wrote:
> On 15 January 2011 10:48, Andrew Murray <amurray@mpcdata.com> wrote:
> > On 11 January 2011 09:47, Andrew Murray <amurray@mpcdata.com> wrote:
> >>
> >> Is there a way to determine the minimum UBI volume size which will
> >> support a given UBIFS filesystem image?
> >>
> >
> > My conclusion is this (with my version of kernel and mkfs.ubifs tool):
> >
> > (Min No. UBI Volume LEBS) = (jrn_lebs) + 3 + (log lebs) + (lpt_lebs) +
> > (orph_lebs)
> >
> > And these values can all be determined (or specified) by the
> > mkfs.ubifs tool by using the verbose -v flag.
> >
> > This seems to hold true - Does this seem reasonable?
> 
> After reading the ODP presentation, white paper and reading the source
> - I do not understand the purpose of the second condition in the
> following test case in validate_sb:
> 
>         if (c->max_bud_bytes < (long long)c->leb_size * UBIFS_MIN_BUD_LEBS ||
>             c->max_bud_bytes > (long long)c->leb_size * c->main_lebs
> 
> It seems to be asserting that there is enough space on the volume to
> cater for the maximum size of the journal.

Yes.

>  However in the case where
> the UBI volume is similar in size to the UBIFS image data -
> 'main_lebs' will mostly be absent of free space lebs.

Yes, and this is a sanity check to make sure the superblock contains
sane information, not garbage, not crafted attacker's image, but an
image where everything is within sane limits.

IOW, this is about allowing only values we kept in mind during
development, this is about being defensive and playing safe.

>  Therefore there
> may not be enough room for max_bud_bytes amounts of journal.

Yes, and I think this max_bud_bytes is treated as the "journal" size the
user wanted when he created the image, and we kind of committed to cater
the user with this journal, and if we cannot, we fail, we do not
silently keep going.

>  And in
> any case the journal code seems to cater well for there not being
> enough free lebs. (Presumably it copes when the volume is nearly
> full).

Probably it will return -ENOSPC correctly.

> Therefore is this condition superfluous?

No, this is sanity check. And as I said, AFAIR, we considered
max_bud_bytes as a requirement to provide this amount of bytes for buds.
If we cannot do this, we fail.

But if this check makes your life miserable and prevents you from
solving your task, we can consider changing this of course.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: validate_sb: bad superblock, error 8 (Minimum UBI volume size for UBIFS image)

[Artem Bityutskiy]

On Tue, 2011-01-18 at 11:30 +0200, Artem Bityutskiy wrote:
> On Tue, 2011-01-18 at 11:16 +0200, Artem Bityutskiy wrote:
> > Anyway, the only way for the user to affect the minimum FS size is the
> > --jrn-size option of mkfs.ubifs. How to calculate what will be the
> > minimum? Not sure, probably UBIFS_MIN_JNL_LEBS = 5 LEBs or something
> > close to this.
> 
> Ah, this is wrong, you can set the log size with --log-lebs as well.
> 
> So UBIFS journal = log + buds.
> 
> --log-lebs set the log size
> --jrn-size sets the buds size.

And note, strictly speaking --jrn-size is about buds size. This is
confusing naming. But I think we wanted to hide complexities from the
user and make him think that the journal is some area of fixed size, and
--jrn-size defines it. But probably that was a bad idea, not sure.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)