[RFC] Add ubi volume image support, fix ubifs

[RFC] Add ubi volume image support, fix ubifs

[Tom Rini]

Hey all.  I'd like to commit the following patch.  This will add support
for generating single image UBI volumes as well as fix creating ubifs
images.  Going backwards here, the problem with ubifs images right now
is that you must pass in -m -e -c -r and -o args in order to get an
image to come out and only -r / -o are not flash part specific.  To fix
this, on ubifs images, pass in ${MKUBIFS_ARGS}.  Why not just
${EXTRA_IMAGECMD} ? That's because making a UBI volume requires a
different set of mandatory arguments too.  And as there's no way (today)
to have IMAGETYPE=foo require IMAGETYPE=bar to be built previously, we
need to use a separate mechanism here (similar to ext2/3 and blocksize).
To pass in the ubinize args we then go and set UBINIZE_ARGS and then we
get our 1 image UBI volume to come out if mtd-utils-native=1.2.0+git is
being used.

As an example, on a board we have with a Samsung K9F2G08U0M we do:
# Make 256MiB/128KiB
MKUBIFS_ARGS = "-m 2048 -e 126976 -c 2047"
UBINIZE_ARGS = "-m 2048 -p 128KiB -s 512"

Note that for ubinize we get around needing to pass in a vol_size param
by dropping it (causing the image to be exactly the size of the
contents) and then setting the resize flag so that when written it will
take up all the room it can.

diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index 81d4c93..61e8948 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -329,7 +329,8 @@ IMAGE_CMD_tar.gz = "cd ${IMAGE_ROOTFS} && tar -zcvf ${DEPLOY_DIR_IMAGE}/${IMAGE_
 IMAGE_CMD_tar.bz2 = "cd ${IMAGE_ROOTFS} && tar -jcvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar.bz2 ."
 IMAGE_CMD_cpio = "cd ${IMAGE_ROOTFS} && (find . | cpio -o -H newc >${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cpio) ${EXTRA_IMAGECMD}"
 IMAGE_CMD_cpio.gz = "cd ${IMAGE_ROOTFS} && (find . | cpio -o -H newc | gzip -c -9 >${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cpio.gz) ${EXTRA_IMAGECMD}"
-IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.ubifs.img"
+IMAGE_CMD_ubi = "echo \[ubifs\] > ubinize.cfg ; echo mode=ubi >> ubinize.cfg ; echo image=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs >> ubinize.cfg ; echo vol_id=0 >> ubinize.cfg ; echo vol_type=dynamic >> ubinize.cfg ; echo vol_name=rootfs >> ubinize.cfg ; echo vol_flags=autoresize >> ubinize.cfg;mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS} && ubinize -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubi ${UBINIZE_ARGS} ubinize.cfg"
+IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.ubifs.img ${MKUBIFS_ARGS}"
 
 EXTRA_IMAGECMD = ""
 EXTRA_IMAGECMD_jffs2 = ""
@@ -338,6 +339,7 @@ EXTRA_IMAGECMD_squashfs = ""
 EXTRA_IMAGECMD_squashfs-lzma = ""
 EXTRA_IMAGECMD_cpio = ""
 EXTRA_IMAGECMD_cpio.gz = ""
+EXTRA_IMAGECMD_ubi = ""
 EXTRA_IMAGECMD_ubifs = ""
 
 IMAGE_DEPENDS = ""
@@ -350,6 +352,7 @@ IMAGE_DEPENDS_ext3 = "genext2fs-native e2fsprogs-native"
 IMAGE_DEPENDS_ext3.gz = "genext2fs-native e2fsprogs-native"
 IMAGE_DEPENDS_squashfs = "squashfs-tools-native"
 IMAGE_DEPENDS_squashfs-lzma = "squashfs-lzma-tools-native"
+IMAGE_DEPENDS_ubi = "mtd-utils-native"
 IMAGE_DEPENDS_ubifs = "mtd-utils-native"
 
 EXTRA_IMAGEDEPENDS = ""

-- 
Tom Rini

Re: [RFC] Add ubi volume image support, fix ubifs

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]


Op 10 feb 2009, om 21:48 heeft Tom Rini het volgende geschreven:

> Hey all.  I'd like to commit the following patch.

Looks ok, one minor featurerequest:


> +IMAGE_CMD_ubi = "echo \[ubifs\] > ubinize.cfg ; echo mode=ubi >>  
> ubinize.cfg ; echo image=${DEPLOY_DIR_IMAGE}/$ 
> {IMAGE_NAME}.rootfs.ubifs >> ubinize.cfg ; echo vol_id=0 >>  
> ubinize.cfg ; echo vol_type=dynamic >> ubinize.cfg ; echo  
> vol_name=rootfs >> ubinize.cfg ; echo vol_flags=autoresize >>  
> ubinize.cfg;mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/$ 
> {IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS} && ubinize -o $ 
> {DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubi ${UBINIZE_ARGS}  
> ubinize.cfg"

echo vol_name=${MACHINE}-rootfs would be nice :)

regards,

Koen

[-- Attachment #2: Dit deel van het bericht is digitaal ondertekend --]
[-- Type: application/pgp-signature, Size: 193 bytes --]

Re: [RFC] Add ubi volume image support, fix ubifs

[Tom Rini]

On Tue, Feb 10, 2009 at 11:43:07PM +0100, Koen Kooi wrote:
>
> Op 10 feb 2009, om 21:48 heeft Tom Rini het volgende geschreven:
>
>> Hey all.  I'd like to commit the following patch.
>
> Looks ok, one minor featurerequest:
>
>
>> +IMAGE_CMD_ubi = "echo \[ubifs\] > ubinize.cfg ; echo mode=ubi >>  
>> ubinize.cfg ; echo image=${DEPLOY_DIR_IMAGE}/$ 
>> {IMAGE_NAME}.rootfs.ubifs >> ubinize.cfg ; echo vol_id=0 >>  
>> ubinize.cfg ; echo vol_type=dynamic >> ubinize.cfg ; echo  
>> vol_name=rootfs >> ubinize.cfg ; echo vol_flags=autoresize >>  
>> ubinize.cfg;mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/$ 
>> {IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS} && ubinize -o $ 
>> {DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubi ${UBINIZE_ARGS}  
>> ubinize.cfg"
>
> echo vol_name=${MACHINE}-rootfs would be nice :)

Really?  iirc from the mounting commandline stuff that means you have to
pass in machine-rootfs rather than just rootfs, ie
root=ubi0:machine-rootfs not root=ubi0:rootfs

-- 
Tom Rini

Re: [RFC] Add ubi volume image support, fix ubifs

[Koen Kooi]

On 10-02-09 23:59, Tom Rini wrote:
> On Tue, Feb 10, 2009 at 11:43:07PM +0100, Koen Kooi wrote:
>> Op 10 feb 2009, om 21:48 heeft Tom Rini het volgende geschreven:
>>
>>> Hey all.  I'd like to commit the following patch.
>> Looks ok, one minor featurerequest:
>>
>>
>>> +IMAGE_CMD_ubi = "echo \[ubifs\]>  ubinize.cfg ; echo mode=ubi>>
>>> ubinize.cfg ; echo image=${DEPLOY_DIR_IMAGE}/$
>>> {IMAGE_NAME}.rootfs.ubifs>>  ubinize.cfg ; echo vol_id=0>>
>>> ubinize.cfg ; echo vol_type=dynamic>>  ubinize.cfg ; echo
>>> vol_name=rootfs>>  ubinize.cfg ; echo vol_flags=autoresize>>
>>> ubinize.cfg;mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/$
>>> {IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}&&  ubinize -o $
>>> {DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubi ${UBINIZE_ARGS}
>>> ubinize.cfg"
>> echo vol_name=${MACHINE}-rootfs would be nice :)
>
> Really?  iirc from the mounting commandline stuff that means you have to
> pass in machine-rootfs rather than just rootfs, ie
> root=ubi0:machine-rootfs not root=ubi0:rootfs

You'll have to update your bootloaders anyway, so cutting down on 
'flashed wrong machine' errors is a big win :)

regards,

Koen

Re: [RFC] Add ubi volume image support, fix ubifs

[Tom Rini]

On Wed, Feb 11, 2009 at 12:42:34AM +0100, Koen Kooi wrote:
> On 10-02-09 23:59, Tom Rini wrote:
[snip]
>> Really?  iirc from the mounting commandline stuff that means you have to
>> pass in machine-rootfs rather than just rootfs, ie
>> root=ubi0:machine-rootfs not root=ubi0:rootfs
>
> You'll have to update your bootloaders anyway, so cutting down on  
> 'flashed wrong machine' errors is a big win :)

The PoV we have is of starting things from scratch, so that's not
something we worry about.  How about this so distros can take care of
it?

diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index 61e8948..b00d125 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -314,6 +314,8 @@ IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}-${DATETIME}"
 IMAGE_LINK_NAME = "${IMAGE_BASENAME}-${MACHINE}"
 IMAGE_EXTRA_SPACE = 10240
 
+UBI_VOLNAME ?= "${MACHINE}-rootfs"
+
 IMAGE_CMD = ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 -x lzo --root=${IMAGE_ROOTFS} --faketime --output=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.jffs2 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_yaffs2 = "mkyaffs2image ${EXTRA_IMAGECMD} ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.yaffs2"
@@ -329,7 +331,7 @@ IMAGE_CMD_tar.gz = "cd ${IMAGE_ROOTFS} && tar -zcvf ${DEPLOY_DIR_IMAGE}/${IMAGE_
 IMAGE_CMD_tar.bz2 = "cd ${IMAGE_ROOTFS} && tar -jcvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar.bz2 ."
 IMAGE_CMD_cpio = "cd ${IMAGE_ROOTFS} && (find . | cpio -o -H newc >${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cpio) ${EXTRA_IMAGECMD}"
 IMAGE_CMD_cpio.gz = "cd ${IMAGE_ROOTFS} && (find . | cpio -o -H newc | gzip -c -9 >${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cpio.gz) ${EXTRA_IMAGECMD}"
-IMAGE_CMD_ubi = "echo \[ubifs\] > ubinize.cfg ; echo mode=ubi >> ubinize.cfg ; echo image=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs >> ubinize.cfg ; echo vol_id=0 >> ubinize.cfg ; echo vol_type=dynamic >> ubinize.cfg ; echo vol_name=rootfs >> ubinize.cfg ; echo vol_flags=autoresize >> ubinize.cfg;mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS} && ubinize -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubi ${UBINIZE_ARGS} ubinize.cfg"
+IMAGE_CMD_ubi = "echo \[ubifs\] > ubinize.cfg ; echo mode=ubi >> ubinize.cfg ; echo image=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs >> ubinize.cfg ; echo vol_id=0 >> ubinize.cfg ; echo vol_type=dynamic >> ubinize.cfg ; echo vol_name=${UBI_VOLNAME} >> ubinize.cfg ; echo vol_flags=autoresize >> ubinize.cfg;mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS} && ubinize -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubi ${UBINIZE_ARGS} ubinize.cfg"
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.ubifs.img ${MKUBIFS_ARGS}"
 
 EXTRA_IMAGECMD = ""

-- 
Tom Rini

Re: [RFC] Add ubi volume image support, fix ubifs

[Koen Kooi]

On 11-02-09 01:27, Tom Rini wrote:
> On Wed, Feb 11, 2009 at 12:42:34AM +0100, Koen Kooi wrote:
>> On 10-02-09 23:59, Tom Rini wrote:
> [snip]
>>> Really?  iirc from the mounting commandline stuff that means you have to
>>> pass in machine-rootfs rather than just rootfs, ie
>>> root=ubi0:machine-rootfs not root=ubi0:rootfs
>> You'll have to update your bootloaders anyway, so cutting down on
>> 'flashed wrong machine' errors is a big win :)
>
> The PoV we have is of starting things from scratch, so that's not
> something we worry about.  How about this so distros can take care of
> it?

Even better, since I wonder whow ubifs handles 
'somereallylongmachinename-rootfs' :)

regards,

Koen

Re: [RFC] Add ubi volume image support, fix ubifs

[John Willis]

Hi,

Thanks for these patches. Looking good IMHO.

> >> Really?  iirc from the mounting commandline stuff that means you
> have to
> >> pass in machine-rootfs rather than just rootfs, ie
> >> root=ubi0:machine-rootfs not root=ubi0:rootfs
> >
> > You'll have to update your bootloaders anyway, so cutting down on
> > 'flashed wrong machine' errors is a big win :)
> 
> The PoV we have is of starting things from scratch, so that's not
> something we worry about.  How about this so distros can take care of
> it?

Yep, I can't really see many normal OE users will have converted existing
images over to UBI so it seems like the right way to start is as we mean to
go on.

I'll test these patches on the Pandora NAND image later (they are along the
same lines as the patches I have been working on for UBI/UBIFS support but
have the extra bonus feature of actually looking like they work). 
It will make a nice change from post processing a TAR ;).
 
Regards,

John Willis

Re: [RFC] Add ubi volume image support, fix ubifs

[Koen Kooi]

On 10-02-09 21:48, Tom Rini wrote:
> Hey all.  I'd like to commit the following patch.  This will add support
> for generating single image UBI volumes as well as fix creating ubifs
> images.  Going backwards here, the problem with ubifs images right now
> is that you must pass in -m -e -c -r and -o args in order to get an
> image to come out and only -r / -o are not flash part specific.  To fix
> this, on ubifs images, pass in ${MKUBIFS_ARGS}.  Why not just
> ${EXTRA_IMAGECMD} ? That's because making a UBI volume requires a
> different set of mandatory arguments too.  And as there's no way (today)
> to have IMAGETYPE=foo require IMAGETYPE=bar to be built previously, we
> need to use a separate mechanism here (similar to ext2/3 and blocksize).
> To pass in the ubinize args we then go and set UBINIZE_ARGS and then we
> get our 1 image UBI volume to come out if mtd-utils-native=1.2.0+git is
> being used.
>
> As an example, on a board we have with a Samsung K9F2G08U0M we do:
> # Make 256MiB/128KiB
> MKUBIFS_ARGS = "-m 2048 -e 126976 -c 2047"
> UBINIZE_ARGS = "-m 2048 -p 128KiB -s 512"

To get the values for these params:

http://cgit.openembedded.net/cgit.cgi?url=openembedded/commit/&id=76af265d0f1a9ed377dbaba7be03fe23643910fe

# do ubiattach /dev/ubi_ctrl -m 4
# From dmesg:
# UBI: smallest flash I/O unit:    2048
# UBI: logical eraseblock size:    129024 bytes
# from ubiattach stdout:
# UBI device number 0, total 1996 LEBs
MKUBIFS_ARGS = "-m 2048 -e 129024 -c 1996"

# do ubiattach /dev/ubi_ctrl -m 4
# from dmesg:
# UBI: smallest flash I/O unit:    2048
# UBI: physical eraseblock size:   131072 bytes (128 KiB)
# UBI: sub-page size:              512
UBINIZE_ARGS = "-m 2048 -p 128KiB -s 512"

regards,

Koen