[Buildroot] Genimage last partition size issue

[Buildroot] Genimage last partition size issue

[Matthew Weber]

As noted in the partition options, size = 0 should cause the last
partition to take the rest of the space on the device.  However, when
I set size = 0 and then write the image to a SD card using "sudo dd
if=foo.img of=/dev/sdb", the rootfs partition only has 190M (the same
amount as when I don't use the size argument) when there is almost 4G
worth of space available.  I know the size argument works in general
as if I set the size to 3000M, the rootfs partition is 3000M after
writing it to the SD card.


Below is the config file for genimage that I am using.

> image boot.vfat {
>   vfat {
>     files = {
>       "MLO",
>       "u-boot.img"
>     }
>   }
>   size = 32M
> }
> image foo.img {
>   hdimage {
>   }
>   partition boot {
>     partition-type = 0xC
>     bootable = "true"
>     image = "boot.vfat"
>   }
>   partition rootfs {
>     partition-type = 0x83
>     image = "rootfs.ext4"
>     size = 0
>   }
> }




-- 
Matthew L Weber / Pr Software Engineer
Airborne Information Systems / Security Systems and Software / Secure Platforms
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
www.rockwellcollins.com

Note: Any Export License Required Information and License Restricted
Third Party Intellectual Property (TPIP) content must be encrypted and
sent to matthew.weber at corp.rockwellcollins.com.

[Buildroot] Genimage last partition size issue

[Peter Korsgaard]

>>>>> "Matthew" == Matthew Weber <matthew.weber@rockwellcollins.com> writes:

 > As noted in the partition options, size = 0 should cause the last
 > partition to take the rest of the space on the device.  However, when
 > I set size = 0 and then write the image to a SD card using "sudo dd
 > if=foo.img of=/dev/sdb", the rootfs partition only has 190M (the same
 > amount as when I don't use the size argument) when there is almost 4G
 > worth of space available.  I know the size argument works in general
 > as if I set the size to 3000M, the rootfs partition is 3000M after
 > writing it to the SD card.

How should genimage know what the size is of your SD card when you
generate the foo.img image? I guess you would need to add size = 4G
inside image foo.img { }, but notice that this is isn't very efficient
as you will be writing lots of zeros to the sd card.

-- 
Bye, Peter Korsgaard

[Buildroot] Genimage last partition size issue

[Matthew Weber]

On Tue, May 24, 2016 at 10:56 AM, Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "Matthew" == Matthew Weber <matthew.weber@rockwellcollins.com> writes:
>
>  > As noted in the partition options, size = 0 should cause the last
>  > partition to take the rest of the space on the device.  However, when
>  > I set size = 0 and then write the image to a SD card using "sudo dd
>  > if=foo.img of=/dev/sdb", the rootfs partition only has 190M (the same
>  > amount as when I don't use the size argument) when there is almost 4G
>  > worth of space available.  I know the size argument works in general
>  > as if I set the size to 3000M, the rootfs partition is 3000M after
>  > writing it to the SD card.
>
> How should genimage know what the size is of your SD card when you
> generate the foo.img image? I guess you would need to add size = 4G
> inside image foo.img { }, but notice that this is isn't very efficient
> as you will be writing lots of zeros to the sd card.
>
Good point, I should have looked a bit closer before forwarding this
question from one of my developers.  He didn't want to specify the
size anywhere and wanted it to make the last partition fit the disk.
Which with pre-builts is obviously not realistic.  Instead he dd's his
image and then adjusts the partition / re-tunes the fs to make it fill
the rest of the disk. Whick works....

Thanks
Matt

[Buildroot] Genimage last partition size issue

[Gustavo Zacarias]

On 26/05/16 14:42, Matthew Weber wrote:

>> How should genimage know what the size is of your SD card when you
>> generate the foo.img image? I guess you would need to add size = 4G
>> inside image foo.img { }, but notice that this is isn't very efficient
>> as you will be writing lots of zeros to the sd card.
>>
> Good point, I should have looked a bit closer before forwarding this
> question from one of my developers.  He didn't want to specify the
> size anywhere and wanted it to make the last partition fit the disk.
> Which with pre-builts is obviously not realistic.  Instead he dd's his
> image and then adjusts the partition / re-tunes the fs to make it fill
> the rest of the disk. Whick works....
>
> Thanks
> Matt

Hi Matt.
"Autoresize script" - there are some tricks around for this where you 
grow the partition to fit available space on the media, resize2fs and 
voila. Requires some care though.
Regards.

[Buildroot] Genimage last partition size issue

[Thomas Petazzoni]

Hello,

On Tue, 24 May 2016 17:56:50 +0200, Peter Korsgaard wrote:

> How should genimage know what the size is of your SD card when you
> generate the foo.img image? I guess you would need to add size = 4G
> inside image foo.img { }, but notice that this is isn't very efficient
> as you will be writing lots of zeros to the sd card.

Isn't it possible to generate a sparse file, and then use the sparse
capability of dd to avoid writing those lots of zeros?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Genimage last partition size issue

[Gustavo Zacarias]

On 26/05/16 15:35, Thomas Petazzoni wrote:

> Hello,
>
> On Tue, 24 May 2016 17:56:50 +0200, Peter Korsgaard wrote:
>
>> How should genimage know what the size is of your SD card when you
>> generate the foo.img image? I guess you would need to add size = 4G
>> inside image foo.img { }, but notice that this is isn't very efficient
>> as you will be writing lots of zeros to the sd card.
>
> Isn't it possible to generate a sparse file, and then use the sparse
> capability of dd to avoid writing those lots of zeros?
>
> Thomas

You're still dealing with the partition table, it goes beyond sparse files.
Regards.

[Buildroot] Genimage last partition size issue

[Thomas Petazzoni]

Hello,

On Thu, 26 May 2016 15:37:22 -0300, Gustavo Zacarias wrote:

> >> How should genimage know what the size is of your SD card when you
> >> generate the foo.img image? I guess you would need to add size = 4G
> >> inside image foo.img { }, but notice that this is isn't very efficient
> >> as you will be writing lots of zeros to the sd card.  
> >
> > Isn't it possible to generate a sparse file, and then use the sparse
> > capability of dd to avoid writing those lots of zeros?
> >
> > Thomas  
> 
> You're still dealing with the partition table, it goes beyond sparse files.

Well, I was thinking of really creating a 4 GB image file when you have
a 4 GB SD card, so that the partition table is exactly in the state it
will be on the SD card. But by using sparse files, the 4 GB image file
isn't really taking up 4 GB on your disk, and you don't have to dd the
entire 4 GB, since zeros can be skipped.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Genimage last partition size issue

[Gustavo Zacarias]

On 26/05/16 15:41, Thomas Petazzoni wrote:

> Well, I was thinking of really creating a 4 GB image file when you have
> a 4 GB SD card, so that the partition table is exactly in the state it
> will be on the SD card. But by using sparse files, the 4 GB image file
> isn't really taking up 4 GB on your disk, and you don't have to dd the
> entire 4 GB, since zeros can be skipped.
>
> Thomas

Yes, that would be suitable, expect for 8 GB cards where you "loose" 
space :)
The other approach which i briefly mentioned is on first boot running a 
resize script that adjusts the partition table to grow the last 
partition to fit the running block device and resizes the filesystem.
Of course it has some limitations, like ext2/3/4 filesystems (some 
others as well) and having some tools around for the trick.
Regards.

[Buildroot] Genimage last partition size issue

[Thomas Petazzoni]

Hello,

On Thu, 26 May 2016 15:45:17 -0300, Gustavo Zacarias wrote:

> Yes, that would be suitable, expect for 8 GB cards where you "loose" 
> space :)

Well, when you'll be shipping a product, you know what will be the size
of your eMMC / SD card, and it's gonna be the same for all your units,
so you can generate an image with the appropriate size.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Genimage last partition size issue

[Gustavo Zacarias]

On 26/05/16 16:18, Thomas Petazzoni wrote:

> Well, when you'll be shipping a product, you know what will be the size
> of your eMMC / SD card, and it's gonna be the same for all your units,
> so you can generate an image with the appropriate size.

Hi.
Unless you're shipping software images to be burnt by users, where you 
don't ;)
Regards.

[Buildroot] Genimage last partition size issue

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Tue, 24 May 2016 17:56:50 +0200, Peter Korsgaard wrote:

 >> How should genimage know what the size is of your SD card when you
 >> generate the foo.img image? I guess you would need to add size = 4G
 >> inside image foo.img { }, but notice that this is isn't very efficient
 >> as you will be writing lots of zeros to the sd card.

 > Isn't it possible to generate a sparse file, and then use the sparse
 > capability of dd to avoid writing those lots of zeros?

Can dd detect and do something sensible about sparse files? I thought it
was only able to create sparse files (by seeking)? I would guess you
need a tool that understand the filesystem to know if you can really
skip sequences of zeros when writing.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] Genimage last partition size issue

[Peter Korsgaard]

>>>>> "Gustavo" == Gustavo Zacarias <gustavo@zacarias.com.ar> writes:

Hi,

 > Yes, that would be suitable, expect for 8 GB cards where you "loose"
 > space :)

And the fact that 4GB cards are normally a bit less than 4GB big (and
the "bit" afaik varies with the card).

> The other approach which i briefly mentioned is on first boot running
 > a resize script that adjusts the partition table to grow the last
 > partition to fit the running block device and resizes the filesystem.
 > Of course it has some limitations, like ext2/3/4 filesystems (some
 > others as well) and having some tools around for the trick.
 > Regards.

Yes, either at first boot or from the host after writing the image.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] Genimage last partition size issue

[Thomas Petazzoni]

Hello,

On Thu, 26 May 2016 21:54:08 +0200, Peter Korsgaard wrote:

> Can dd detect and do something sensible about sparse files? I thought it
> was only able to create sparse files (by seeking)? I would guess you
> need a tool that understand the filesystem to know if you can really
> skip sequences of zeros when writing.

man dd(1):

       sparse try to seek rather than write the output for NUL input blocks

So it doesn't seem to need a sparse file, it's just that it detects
blocks that are completely zero and does not write them.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Genimage last partition size issue

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Thu, 26 May 2016 21:54:08 +0200, Peter Korsgaard wrote:

 >> Can dd detect and do something sensible about sparse files? I thought it
 >> was only able to create sparse files (by seeking)? I would guess you
 >> need a tool that understand the filesystem to know if you can really
 >> skip sequences of zeros when writing.

 > man dd(1):

 >        sparse try to seek rather than write the output for NUL input blocks

 > So it doesn't seem to need a sparse file, it's just that it detects
 > blocks that are completely zero and does not write them.

Yes, but you cannot safely use that. What if you have a file on your
filesystem containing a block of zeroes? (or as part of the filesystem
meta data like block bitmap or journal) You need to make sure that does
get written to the card, otherwise you will read back whatever was
written before to the card.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] Genimage last partition size issue

[Yann E. MORIN]

All,

On 2016-05-26 22:05 +0200, Peter Korsgaard spake thusly:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
>  > Hello,
>  > On Thu, 26 May 2016 21:54:08 +0200, Peter Korsgaard wrote:
> 
>  >> Can dd detect and do something sensible about sparse files? I thought it
>  >> was only able to create sparse files (by seeking)? I would guess you
>  >> need a tool that understand the filesystem to know if you can really
>  >> skip sequences of zeros when writing.
> 
>  > man dd(1):
> 
>  >        sparse try to seek rather than write the output for NUL input blocks
> 
>  > So it doesn't seem to need a sparse file, it's just that it detects
>  > blocks that are completely zero and does not write them.
> 
> Yes, but you cannot safely use that. What if you have a file on your
> filesystem containing a block of zeroes? (or as part of the filesystem
> meta data like block bitmap or journal) You need to make sure that does
> get written to the card, otherwise you will read back whatever was
> written before to the card.

For the record, I was using the sparse option to dd in the past, and it
was causing a hell of a lot of filesystem corruption at runtime:
ext2/3/4 especially would cause the kernel to PANIC at best, or would
cause massive data corruption at worse.

So no, you can not use sparse when dd-ing or otherwise flashing) your
images to the target.

Beside, we explicitly state so in the manual:

    https://buildroot.org/downloads/manual/manual.html#_dealing_efficiently_with_filesystem_images

    Note however that flashing a filesystem image to a device while using
    the sparse mode of dd may result in a broken filesystem (e.g. the block
    bitmap of an ext2 filesystem may be corrupted; or, if you have sparse
    files in your filesystem, those parts may not be all-zeroes when read
    back). You should only use sparse files when handling files on the build
    machine, not when transferring them to an actual device that will be
    used on the target.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'