[PATCH v5 0/1] ubi: Introduce block devices for UBI volumes

[PATCH v5 0/1] ubi: Introduce block devices for UBI volumes

[Ezequiel Garcia]

This patch adds block device access on top of UBI volumes, which allows
for a ligthweigth solution to mount block-oriented filesystems on top of
flash devices.

The main target of this patch is supporting lighter, read-only filesystem,
by putting a squashfs on top of ubiblock. And since squashfs already provides
a file cache (which is even configurable), a first uncached implementation
it's good enough for now.

You can read this discussion if you are want more historical details:

  http://lkml.indiana.edu/hypermail/linux/kernel/1211.2/02556.html

In particular, it's important to note this already has users:

  1. http://permalink.gmane.org/gmane.linux.drivers.mtd/46221

  2. http://permalink.gmane.org/gmane.linux.drivers.mtd/49385

  3. Myself (Yes, I am now taking my own medicine).

As noted below, and despite being integrated into the UBI module,
the block layer is very losely coupled to the rest of the UBI code.
This is important as it should mean it has a low impact on UBI's stability.

As soon as this gets merged into UBI kernel core, I'll submit the mtd-utils
userspace tools, as well as start preparing some documentation for the mtd
website, and the docbook.

I'd be willing to drop read-support is that makes this feature acceptable;
but I guess that would make it pretty useless :-)

* Changes from v4

  * Dropped write-support

  * Dropped cached access

* Changes from v3

  * Replaced the 2-LEB (read and write) for a 1-LEB cache. This was done
    to prevent any cache-coherency issues and to avoid over-design.
    It's not clear having a separate read and write cache has a dramatical
    impact on wear leveling or performance.

  * Made the 1-LEB cache optional. Based on a suggestion by Piergiorgio
    Beruto, caching at the block level would be redundant and ineffective.
    The filesystem has a better chance of caching.

    Some very simple tests on squashfs (ubiblock's primary target) showed
    that squashfs does a great job on caching its file accesses.

    Given a LEB can be quite big on some devices O(~100KiB), it can be
    interesting to remove the cache altogether on memory-constrained platforms.

  * Make UBI_IOCVOLDETBLK return an error if the detaching wasn't successful,
    as suggested by Mike Frysinger.

  * Fixed a bunch of typos and suggestions made by Mike Frysinger.

* Changes from v2

  * On this 3rd version of the patch I've re-worked the patch considerably,
    mostly integrating it with UBI, instead of making it a separate module.

    Despite this integration, the ubiblock code is still completely independent
    of the UBI core and it can be completely compiled-out through configuration.

  * Making ubiblock part of UBI allows to add ioctl access to attach/detach
    block devices to/from UBI volumes in a very simple way.

  * A new tool 'ubiblkvol' is added to ubi-utils to access these two new ioctls.

  * To address the 'write or not to write' discussion, the write support is
    enabled with a kernel option and there's a big fat 'DANGEROUS' label on it
    to prevent misusage.

Ezequiel Garcia (1):
  ubi: Introduce block devices for UBI volumes

 drivers/mtd/ubi/Kconfig     |  19 ++
 drivers/mtd/ubi/Makefile    |   1 +
 drivers/mtd/ubi/block.c     | 656 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/mtd/ubi/build.c     |   5 +
 drivers/mtd/ubi/cdev.c      |  20 ++
 drivers/mtd/ubi/ubi.h       |  14 +
 include/uapi/mtd/ubi-user.h |  11 +
 7 files changed, 726 insertions(+)
 create mode 100644 drivers/mtd/ubi/block.c

-- 
1.8.1.5

Re: [PATCH v5 0/1] ubi: Introduce block devices for UBI volumes

[Artem Bityutskiy]

Thanks Ezequiel,

On Thu, 2014-02-13 at 16:57 -0300, Ezequiel Garcia wrote:
> As soon as this gets merged into UBI kernel core, I'll submit the mtd-utils
> userspace tools, as well as start preparing some documentation for the mtd
> website, and the docbook.

Ideally the docs should come along with the driver.

> I'd be willing to drop read-support is that makes this feature acceptable;
> but I guess that would make it pretty useless :-)

Well, it was your decision.

I only asked to remove the compile-time option. And if there are
limitations, document them.

I pointed that there is a standard 'blkdev' tool which has '--setro' and
'--setrw' options, and you could try to use that for _run-time_ RO/RW
toggling.

>   * Dropped write-support

Fine with me too, thanks, but please, do not allude you was pushed to do
this.

>   * Dropped cached access

Thanks. This little tiny compile options make more harm than benefit.
They break from time, because people usually use one of the
configurations, and rarely test the changed.

If you want to save people 128KiB of ram or something, do it a better
way. For example, we have memory shrinker in the kernel. Register your
your shrinker and free that buffer in case of memory pressure. Allocate
it back if needed later using "gentle" allocation techniques (NORETRY,
NOFS, etc).

Anyway, thanks for this too.


-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH v5 0/1] ubi: Introduce block devices for UBI volumes

[Ezequiel Garcia]

On Fri, Feb 14, 2014 at 10:11:45AM +0200, Artem Bityutskiy wrote:
> Thanks Ezequiel,
> 

No problem. Thanks for reviewing, I know you're very busy these days.

> On Thu, 2014-02-13 at 16:57 -0300, Ezequiel Garcia wrote:
> > As soon as this gets merged into UBI kernel core, I'll submit the mtd-utils
> > userspace tools, as well as start preparing some documentation for the mtd
> > website, and the docbook.
> 
> Ideally the docs should come along with the driver.
> 

OK.

> > I'd be willing to drop read-support is that makes this feature acceptable;
> > but I guess that would make it pretty useless :-)
> 
> Well, it was your decision.
> 

Of course.

> I only asked to remove the compile-time option. And if there are
> limitations, document them.
> 
> I pointed that there is a standard 'blkdev' tool which has '--setro' and
> '--setrw' options, and you could try to use that for _run-time_ RO/RW
> toggling.
> 

Hm... yes. I just forgot to write it here. The commit log in the patch
mentions it (BLKROSET ioctl). I think it's good idea.

> >   * Dropped write-support
> 
> Fine with me too, thanks, but please, do not allude you was pushed to do
> this.
>

Oh, I don't feel pushed. Sorry if it sounded like that.

The idea was to re-submit with the minimum set of features we all seemed to
agree. Write support can be added (and probably will be), your suggestion
is probably a good idea.

> >   * Dropped cached access
> 
> Thanks. This little tiny compile options make more harm than benefit.
> They break from time, because people usually use one of the
> configurations, and rarely test the changed.
> 

Fair enough.

> If you want to save people 128KiB of ram or something, do it a better
> way. For example, we have memory shrinker in the kernel. Register your
> your shrinker and free that buffer in case of memory pressure. Allocate
> it back if needed later using "gentle" allocation techniques (NORETRY,
> NOFS, etc).
> 

Ah, good. I was trying to find something like this. I also thought about
a custom block ioctl. But this sounds better.

I'll try to cook up a v6, with the docs.

I'd rather not add write-support or cache *now*. Uncached, read-only
is enough for now to support squashfs rootfs, which is something people
ask about from time to time, I think.
-- 
Ezequiel García, Free Electrons
Embedded Linux, Kernel and Android Engineering
http://free-electrons.com

[PATCH v5 0/1] ubi: Introduce block devices for UBI volumes

[Ezequiel Garcia]

This patch adds block device access on top of UBI volumes, which allows
for a ligthweigth solution to mount block-oriented filesystems on top of
flash devices.

The main target of this patch is supporting lighter, read-only filesystem,
by putting a squashfs on top of ubiblock. And since squashfs already provides
a file cache (which is even configurable), a first uncached implementation
it's good enough for now.

You can read this discussion if you are want more historical details:

  http://lkml.indiana.edu/hypermail/linux/kernel/1211.2/02556.html

In particular, it's important to note this already has users:

  1. http://permalink.gmane.org/gmane.linux.drivers.mtd/46221

  2. http://permalink.gmane.org/gmane.linux.drivers.mtd/49385

  3. Myself (Yes, I am now taking my own medicine).

As noted below, and despite being integrated into the UBI module,
the block layer is very losely coupled to the rest of the UBI code.
This is important as it should mean it has a low impact on UBI's stability.

As soon as this gets merged into UBI kernel core, I'll submit the mtd-utils
userspace tools, as well as start preparing some documentation for the mtd
website, and the docbook.

I'd be willing to drop read-support is that makes this feature acceptable;
but I guess that would make it pretty useless :-)

* Changes from v4

  * Dropped write-support

  * Dropped cached access

* Changes from v3

  * Replaced the 2-LEB (read and write) for a 1-LEB cache. This was done
    to prevent any cache-coherency issues and to avoid over-design.
    It's not clear having a separate read and write cache has a dramatical
    impact on wear leveling or performance.

  * Made the 1-LEB cache optional. Based on a suggestion by Piergiorgio
    Beruto, caching at the block level would be redundant and ineffective.
    The filesystem has a better chance of caching.

    Some very simple tests on squashfs (ubiblock's primary target) showed
    that squashfs does a great job on caching its file accesses.

    Given a LEB can be quite big on some devices O(~100KiB), it can be
    interesting to remove the cache altogether on memory-constrained platforms.

  * Make UBI_IOCVOLDETBLK return an error if the detaching wasn't successful,
    as suggested by Mike Frysinger.

  * Fixed a bunch of typos and suggestions made by Mike Frysinger.

* Changes from v2

  * On this 3rd version of the patch I've re-worked the patch considerably,
    mostly integrating it with UBI, instead of making it a separate module.

    Despite this integration, the ubiblock code is still completely independent
    of the UBI core and it can be completely compiled-out through configuration.

  * Making ubiblock part of UBI allows to add ioctl access to attach/detach
    block devices to/from UBI volumes in a very simple way.

  * A new tool 'ubiblkvol' is added to ubi-utils to access these two new ioctls.

  * To address the 'write or not to write' discussion, the write support is
    enabled with a kernel option and there's a big fat 'DANGEROUS' label on it
    to prevent misusage.

Ezequiel Garcia (1):
  ubi: Introduce block devices for UBI volumes

 drivers/mtd/ubi/Kconfig     |  19 ++
 drivers/mtd/ubi/Makefile    |   1 +
 drivers/mtd/ubi/block.c     | 656 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/mtd/ubi/build.c     |   5 +
 drivers/mtd/ubi/cdev.c      |  20 ++
 drivers/mtd/ubi/ubi.h       |  14 +
 include/uapi/mtd/ubi-user.h |  11 +
 7 files changed, 726 insertions(+)
 create mode 100644 drivers/mtd/ubi/block.c

-- 
1.8.1.5