UBI on top of a smaller portion of a bigger MTD device

UBI on top of a smaller portion of a bigger MTD device

[Nam Nguyen]

Hello list,

I am wondering if there is interest to make ubi utils work with a
partial range on the MTD device, like other mtd utils. I would
contribute code to make that happen.

Use case:

We have one big NAND device on which we store kernel (a blob), rootfs
(ext4) and other ubifs. The partition info of that device is stored in
a NOR device. Upon boot, the firmware reads from NOR device, passes
partition values to the kernel via device tree, and the kernel will
set up MTD partitions. The MTD module is compiled in because rootfs is
on it.

In recovery mode, however, the firmware does not read anything from
the (writable) NOR flash for fear of grabbing garbage values and
bricking the device. Hence the kernel has no info about any MTD
partition. Yet the recovery USB stick does have such information on
its rootfs. So we were thinking maybe we could augment ubi-utils with
where the partition starts, where it ends, to work with the raw NAND
device.

I talked to derRichard in #mtd channel. Perhaps this email could
explain it better on why we can't set up MTD partitions.

If there is any other way to achieve the same goal, I'd love to hear
about it too.

Thanks,
Nam

Re: UBI on top of a smaller portion of a bigger MTD device

[Daniel Ehrenberg]

To give slightly more context, we use the same built kernel image in
two modes: One when booting from USB in a "recovery" mode meant to do
a reinstallation to the device, and the other when operating normally.
These have the same kernel command-line arguments, device tree, etc.
In normal boot, the firmware augments the device tree with the
partition table read from NOR, and in USB boot it does not (that might
be wrong anyway--we are recovering, after all). I see two ways that we
could do the installation to the device:
- Make a simple modification to ubiformat to give it offset and length
parameters for a "soft partition" on the device. Another member of our
group has already made a change to nandwrite to do the same thing, and
it seems to have been accepted upstream. This would be very little
code change in mtdutils and not require any kernel code.
- Implement a dynamic MTD partitioning mechanism. We can't simply pass
the partition table through modprobe when loading the NAND driver
because the NAND driver is built-in (since it's what we're loading the
rootfs off of). I thought this option would be a lot more work to
implement. I thought it might be difficult to upstream in part because
there's already a dynamic ubi partitioning mechanism. We're unable to
use ubi partitions for everything, however, since the kernel is read
from the firmware (our firmware is based on depthcharge), and ubi
seems like a lot of code to port to a firmware codebase that prides
itself on being in the low tens of thousands of lines of code.
Instead, we use several MTD partitions, some of which are formatted
with UBI and have one UBI volume on each.

Dan

On Thu, Jan 29, 2015 at 4:58 PM, Nam Nguyen <namnguyen@chromium.org> wrote:
> Hello list,
>
> I am wondering if there is interest to make ubi utils work with a
> partial range on the MTD device, like other mtd utils. I would
> contribute code to make that happen.
>
> Use case:
>
> We have one big NAND device on which we store kernel (a blob), rootfs
> (ext4) and other ubifs. The partition info of that device is stored in
> a NOR device. Upon boot, the firmware reads from NOR device, passes
> partition values to the kernel via device tree, and the kernel will
> set up MTD partitions. The MTD module is compiled in because rootfs is
> on it.
>
> In recovery mode, however, the firmware does not read anything from
> the (writable) NOR flash for fear of grabbing garbage values and
> bricking the device. Hence the kernel has no info about any MTD
> partition. Yet the recovery USB stick does have such information on
> its rootfs. So we were thinking maybe we could augment ubi-utils with
> where the partition starts, where it ends, to work with the raw NAND
> device.
>
> I talked to derRichard in #mtd channel. Perhaps this email could
> explain it better on why we can't set up MTD partitions.
>
> If there is any other way to achieve the same goal, I'd love to hear
> about it too.
>
> Thanks,
> Nam

Re: UBI on top of a smaller portion of a bigger MTD device

[Daniel Ehrenberg]

On Thu, Jan 29, 2015 at 5:16 PM, Daniel Ehrenberg
<dehrenberg@chromium.org> wrote:
> - Implement a dynamic MTD partitioning mechanism.

Oh, my mistake, that's already there, just learned about the
BLKPG_ADD_PARTITION ioctl and the fact that it works on mtd!

We are always partitioning our NAND with mtd0 standing in for the
whole device and further partitions for part of the device. Would it
be possible to delete all partitions but mtd0 and then add the
partitions that we want? It looks like BLKPG_ADD_PARTITION only works
if it's called against an unpartitioned device--in this case, it'd be
a partition whose size is size of the whole device and with offset 0.
How would you feel about a patch to relax this constraint for just
that type of partition? This way, if we had to repartition the device
(in a third mode that I didn't mention, where the partitions are read
from NOR but we're still booting from USB and reinstalling) we don't
have to ask the user to reboot halfway through to get the partition
table reflected.

Thanks,
Dan

Re: UBI on top of a smaller portion of a bigger MTD device

[Richard Weinberger]

On Fri, Jan 30, 2015 at 2:37 AM, Daniel Ehrenberg
<dehrenberg@chromium.org> wrote:
> On Thu, Jan 29, 2015 at 5:16 PM, Daniel Ehrenberg
> <dehrenberg@chromium.org> wrote:
>> - Implement a dynamic MTD partitioning mechanism.
>
> Oh, my mistake, that's already there, just learned about the
> BLKPG_ADD_PARTITION ioctl and the fact that it works on mtd!
>
> We are always partitioning our NAND with mtd0 standing in for the
> whole device and further partitions for part of the device. Would it
> be possible to delete all partitions but mtd0 and then add the
> partitions that we want? It looks like BLKPG_ADD_PARTITION only works
> if it's called against an unpartitioned device--in this case, it'd be
> a partition whose size is size of the whole device and with offset 0.
> How would you feel about a patch to relax this constraint for just
> that type of partition? This way, if we had to repartition the device
> (in a third mode that I didn't mention, where the partitions are read
> from NOR but we're still booting from USB and reinstalling) we don't
> have to ask the user to reboot halfway through to get the partition
> table reflected.

As I explained on IRC to Nam UBI works only on mtd partitions and magic
offset games are something we definitely don't want.

With your mails the use case makes more sense, I agree that you
need some way to partition the mtd at run-time.
If we BLKPG ioctl() does not offer all the functionality you need I don't
you can extend it. Patches are welcome!

-- 
Thanks,
//richard

Re: UBI on top of a smaller portion of a bigger MTD device

[Artem Bityutskiy]

On Thu, 2015-01-29 at 16:58 -0800, Nam Nguyen wrote:
> In recovery mode, however, the firmware does not read anything from
> the (writable) NOR flash for fear of grabbing garbage values and
> bricking the device. Hence the kernel has no info about any MTD
> partition. Yet the recovery USB stick does have such information on
> its rootfs. So we were thinking maybe we could augment ubi-utils with
> where the partition starts, where it ends, to work with the raw NAND
> device.

Hi, I did not read the whole conversation, so I am sorry if my comment
is not 100% relevant.

But it sounds that what you actually want is repartitioning on demand.

IOW, you want an ioctl, which you will use to tell MTD what are the
partition boundaries.

In this case the SW from the USB stick will just do that. You'll get the
right MTD devices.

I do not think it would be a clean architecture to teach UBI tools to
work with offsets instead of /dev/mtdX partitions.

Artem.