[PATCH 0/3] wic: enhanced bootimage + squashfs support

[PATCH 0/3] wic: enhanced bootimage + squashfs support

[Enrico Scholz]

Patchset allows to use different bootimage sources and to reserve more
space for squashfs partitions.

Enrico Scholz (3):
  wic: accept '-' in bitbake variables
  wic: allow multiple /boot partitions with different content
  wic: apply --extra-space + --overhead to squashfs

 scripts/lib/wic/misc.py                             |  2 +-
 scripts/lib/wic/partition.py                        | 13 +++++++++++++
 scripts/lib/wic/plugins/source/bootimg-partition.py | 17 +++++++++++++----
 3 files changed, 27 insertions(+), 5 deletions(-)

-- 
2.9.5

[PATCH 1/3] wic: accept '-' in bitbake variables

[Enrico Scholz]

From: Enrico Scholz <enrico.scholz@ensc.de>

'-' is valid and common in bitbake variables (e.g. 'FOO_pn-bar'). Accept
it and other characters when reading the .env file.

Also, allow variables to be empty.

Signed-off-by: Enrico Scholz <enrico.scholz@ensc.de>
---
 scripts/lib/wic/misc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 4e07cd6..0792480 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -183,7 +183,7 @@ class BitbakeVars(defaultdict):
         self.default_image = None
         self.vars_dir = None
 
-    def _parse_line(self, line, image, matcher=re.compile(r"^(\w+)=(.+)")):
+    def _parse_line(self, line, image, matcher=re.compile(r"^([a-zA-Z0-9\-_+./~]+)=(.*)")):
         """
         Parse one line from bitbake -e output or from .env file.
         Put result key-value pair into the storage.
-- 
2.9.5

[PATCH 2/3] wic: allow multiple /boot partitions with different content

[Enrico Scholz]

From: Enrico Scholz <enrico.scholz@ensc.de>

It can be useful to have multiple partitions with '--source bootimg-partition'
but different content.

E.g. for TI AM335x, one boot partition can contain an first stage
bootloader ("MLO"), while the real bootloader and kernel plus devicetree
are in another one.

Patch allows to specify multiple IMAGE_BOOT_FILES with optional "_label-XXX"
or "_uuid-XXX" overrides.

E.g. with this patch, a .wks file with

| part --source bootimg-partition ... --label=mlo --active
| part --source bootimg-partition ... --label=boot0
| part --source bootimg-partition ... --label=boot1

and a recipe with

| IMAGE_BOOT_FILES_label-mlo = "\
|   MLO-${MACHINE}.img;MLO \
| "
|
| IMAGE_BOOT_FILES_label-boot0 = "\
|   u-boot-${MACHINE}.img;u-boot.img \
|   zImage \
| "
|
| IMAGE_BOOT_FILES_label-boot1 = "${IMAGE_BOOT_FILES_label-boot0}"
|
| WICVARS += " \
|   IMAGE_BOOT_FILES_label-mlo \
|   IMAGE_BOOT_FILES_label-boot0 \
|   IMAGE_BOOT_FILES_label-boot1 \
| "

is possible.  It will create one partition with the MLO and two redundant
ones with the uboot + kernel.

Signed-off-by: Enrico Scholz <enrico.scholz@ensc.de>
---
 scripts/lib/wic/plugins/source/bootimg-partition.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py b/scripts/lib/wic/plugins/source/bootimg-partition.py
index ca074a3..67e5498 100644
--- a/scripts/lib/wic/plugins/source/bootimg-partition.py
+++ b/scripts/lib/wic/plugins/source/bootimg-partition.py
@@ -54,7 +54,7 @@ class BootimgPartitionPlugin(SourcePlugin):
         - sets up a vfat partition
         - copies all files listed in IMAGE_BOOT_FILES variable
         """
-        hdddir = "%s/boot" % cr_workdir
+        hdddir = "%s/boot.%d" % (cr_workdir, part.lineno)
         install_cmd = "install -d %s" % hdddir
         exec_cmd(install_cmd)
 
@@ -65,10 +65,19 @@ class BootimgPartitionPlugin(SourcePlugin):
 
         logger.debug('Kernel dir: %s', bootimg_dir)
 
-        boot_files = get_bitbake_var("IMAGE_BOOT_FILES")
+        boot_files = None
+        for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s", part.label), (None, None)):
+            if fmt:
+                var = fmt % id
+            else:
+                var = ""
+
+            boot_files = get_bitbake_var("IMAGE_BOOT_FILES" + var)
+            if boot_files is not None:
+                break
 
-        if not boot_files:
-            raise WicError('No boot files defined, IMAGE_BOOT_FILES unset')
+        if boot_files is None:
+            raise WicError('No boot files defined, IMAGE_BOOT_FILES unset for entry #%d' % part.lineno)
 
         logger.debug('Boot files: %s', boot_files)
 
-- 
2.9.5

[PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Enrico Scholz]

From: Enrico Scholz <enrico.scholz@ensc.de>

The --extra-space and --overhead option did not had an effect to squashfs
partitions.  Although squashfs is read-only, it can be useful to allocate
more space for the on-disk partition to avoid repartitioning of the whole
disk when a new (and larger) squashfs image is written on later updates.

Patch calls get_rootfs_size() *after* creating the image and truncates
it then.

Signed-off-by: Enrico Scholz <enrico.scholz@ensc.de>
---
 scripts/lib/wic/partition.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b623bb9..da651f8 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -195,6 +195,17 @@ class Partition():
                            "larger (%d kB) than its allowed size %d kB" %
                            (self.mountpoint, self.size, self.fixed_size))
 
+    def __extend_rootfs_image(self, rootfs):
+        """Enlarges the rootfs so that it fulfills size/overhead-factor
+        constraints"""
+
+        sz = (os.stat(rootfs).st_size + 1023) // 1024
+        pad_sz = self.get_rootfs_size(sz)
+
+        if pad_sz > sz:
+            with open(rootfs, 'a') as f:
+                os.ftruncate(f.fileno(), pad_sz * 1024)
+
     def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
                        native_sysroot):
         """
@@ -338,6 +349,8 @@ class Partition():
                        (rootfs_dir, rootfs, extraopts)
         exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo)
 
+        self.__extend_rootfs_image(rootfs)
+
     def prepare_empty_partition_ext(self, rootfs, oe_builddir,
                                     native_sysroot):
         """
-- 
2.9.5

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Ed Bartosh]

On Fri, Sep 08, 2017 at 07:33:03PM +0200, Enrico Scholz wrote:
> From: Enrico Scholz <enrico.scholz@ensc.de>
> 
> The --extra-space and --overhead option did not had an effect to squashfs
> partitions.  Although squashfs is read-only, it can be useful to allocate
> more space for the on-disk partition to avoid repartitioning of the whole
> disk when a new (and larger) squashfs image is written on later updates.
> 

Is it possible to just use --size or --fixed-size in .wks to allocate
partition of certain size?

> Patch calls get_rootfs_size() *after* creating the image and truncates
> it then.
> 
> Signed-off-by: Enrico Scholz <enrico.scholz@ensc.de>
> ---
>  scripts/lib/wic/partition.py | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index b623bb9..da651f8 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -195,6 +195,17 @@ class Partition():
>                             "larger (%d kB) than its allowed size %d kB" %
>                             (self.mountpoint, self.size, self.fixed_size))
>  
> +    def __extend_rootfs_image(self, rootfs):

Do we really need to mangle name of this method?
Please, consider reading this for further details:
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#naming

As this function is not going to be used anywhere I'd just use this code
in prepare_rootfs_squashfs. It would be less generic, but much more
readable and understandable.

> +        """Enlarges the rootfs so that it fulfills size/overhead-factor
> +        constraints"""
> +
> +        sz = (os.stat(rootfs).st_size + 1023) // 1024
> +        pad_sz = self.get_rootfs_size(sz)
> +
> +        if pad_sz > sz:
> +            with open(rootfs, 'a') as f:
> +                os.ftruncate(f.fileno(), pad_sz * 1024)
> +
>      def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
>                         native_sysroot):
>          """
> @@ -338,6 +349,8 @@ class Partition():
>                         (rootfs_dir, rootfs, extraopts)
>          exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo)
>  
> +        self.__extend_rootfs_image(rootfs)
> +
>      def prepare_empty_partition_ext(self, rootfs, oe_builddir,
>                                      native_sysroot):
>          """

--
Regards,
Ed

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Enrico Scholz]

Ed Bartosh <ed.bartosh@linux.intel.com> writes:

>> The --extra-space and --overhead option did not had an effect to squashfs
>> partitions.  Although squashfs is read-only, it can be useful to allocate
>> more space for the on-disk partition to avoid repartitioning of the whole
>> disk when a new (and larger) squashfs image is written on later updates.
>
> Is it possible to just use --size or --fixed-size in .wks to allocate
> partition of certain size?

--fixed-size works with this patch (tested); --size should work.


>> +    def __extend_rootfs_image(self, rootfs):
>
> Do we really need to mangle name of this method?

ok; I will make it _extend_rootfs_image


> As this function is not going to be used anywhere I'd just use this
> code in prepare_rootfs_squashfs. It would be less generic, but much
> more readable and understandable.

I would keep it as is.  It is a non trivial function which is reusable.
Perhaps, somebody adds CRAMFS or another file system which does not
support extending its size.

partition.py is full of redundant code (look for all the "du" calls or
generally at the beginning of the prepare_rootfs_*() methods) where
probably somebody thought that it is not going to be used anywhere else
;)


Enrico

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Ed Bartosh]

On Mon, Sep 11, 2017 at 04:04:40PM +0200, Enrico Scholz wrote:
> Ed Bartosh <ed.bartosh@linux.intel.com> writes:
> 
> >> The --extra-space and --overhead option did not had an effect to squashfs
> >> partitions.  Although squashfs is read-only, it can be useful to allocate
> >> more space for the on-disk partition to avoid repartitioning of the whole
> >> disk when a new (and larger) squashfs image is written on later updates.
> >
> > Is it possible to just use --size or --fixed-size in .wks to allocate
> > partition of certain size?
> 
> --fixed-size works with this patch (tested); --size should work.

Sorry for not being clear. Why do we need this if we can use --size to
explicity specify partition size?

--extra-space and --overhead have the same meaning as
IMAGE_ROOTFS_EXTRA_SPACE and IMAGE_OVERHEAD_SIZE variables. From my
point of view their purpose is to allocate additional space on the file
system. This doesn't make sense for squashfs. That's the reason those
options are not used for squashfs. Using them to implicitly make bigger
partition is a questionable and possible confusing approach.


> 
> >> +    def __extend_rootfs_image(self, rootfs):
> >
> > Do we really need to mangle name of this method?
> 
> ok; I will make it _extend_rootfs_image
> 
> 
> > As this function is not going to be used anywhere I'd just use this
> > code in prepare_rootfs_squashfs. It would be less generic, but much
> > more readable and understandable.
> 
> I would keep it as is.  It is a non trivial function which is reusable.
> Perhaps, somebody adds CRAMFS or another file system which does not
> support extending its size.
> 

I personally find this much more understandable and logical:
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -338,6 +338,14 @@ class Partition():
                        (rootfs_dir, rootfs, extraopts)
         exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo)
 
+        # Enlarges the image to make bigger partition
+        sz = (os.stat(rootfs).st_size + 1023) // 1024
+        pad_sz = self.get_rootfs_size(sz)
+
+        if pad_sz > sz:
+            with open(rootfs, 'a') as fimage:
+                  fimage.truncate(pad_sz * 1024)
+
     def prepare_empty_partition_ext(self, rootfs, oe_builddir,
                                     native_sysroot):

Then introducing a method that will be called for all supported
filesystems except squashfs without any effect.

> partition.py is full of redundant code (look for all the "du" calls or
> generally at the beginning of the prepare_rootfs_*() methods) where
> probably somebody thought that it is not going to be used anywhere else
> ;)
Feel free to send a patch then.
I'd not consider it a good excuse for the above new method though :)

--
Regards,
Ed

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Enrico Scholz]

Ed Bartosh <ed.bartosh@linux.intel.com> writes:

>> >> The --extra-space and --overhead option did not had an effect to squashfs
>> >> partitions.  Although squashfs is read-only, it can be useful to allocate
>> >> more space for the on-disk partition to avoid repartitioning of the whole
>> >> disk when a new (and larger) squashfs image is written on later updates.
>> >
>> > Is it possible to just use --size or --fixed-size in .wks to allocate
>> > partition of certain size?
>> 
>> --fixed-size works with this patch (tested); --size should work.
>
> Sorry for not being clear. Why do we need this if we can use --size to
> explicity specify partition size?

--size and --fixed-size did not had an effect for squashfs with the old
code.


> --extra-space and --overhead have the same meaning as
> IMAGE_ROOTFS_EXTRA_SPACE and IMAGE_OVERHEAD_SIZE variables. From my
> point of view their purpose is to allocate additional space on the
> file system. This doesn't make sense for squashfs. That's the reason
> those options are not used for squashfs. Using them to implicitly make
> bigger partition is a questionable and possible confusing approach.

I want/need it to allow updates of the running system without complete
repartitioning.  E.g. at wic creation time, the squashfs has a certain
size.  Sometime in the future, there are needed e.g. 5 MiB more because
a new package was added or so.

I need to reserve space in the partition so that I can 'dd' the new
image without a complete repartitioning.

--extra-space/--overhead has the meaning which I want (e.g. to say
"reserve +20% for changes in the future").


>> I would keep it as is.  It is a non trivial function which is reusable.
>> Perhaps, somebody adds CRAMFS or another file system which does not
>> support extending its size.
>> 
>
> I personally find this much more understandable and logical:

me not ;) As I said, functionality is perfect for putting into an extra
method and it makes code much more readable.  When somebody implements
CRAMFS he can use this method instead of copy&pasting the code.


> Then introducing a method that will be called for all supported
> filesystems except squashfs without any effect.

You mean, function should be called at the end of prepare_rootfs()
(after "method(...)")?  Yes, it would reduce code duplication but other
methods needs to be touch too (to give information whether rootfs size
has been adjusted already).


Enrico
-- 
SIGMA Chemnitz GmbH       Registergericht:   Amtsgericht Chemnitz HRB 1750
Am Erlenwald 13           Geschaeftsfuehrer: Grit Freitag, Frank Pyritz
09128 Chemnitz

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Ed Bartosh]

On Tue, Sep 12, 2017 at 11:44:02AM +0200, Enrico Scholz wrote:
> Ed Bartosh <ed.bartosh@linux.intel.com> writes:
> 
> >> >> The --extra-space and --overhead option did not had an effect to squashfs
> >> >> partitions.  Although squashfs is read-only, it can be useful to allocate
> >> >> more space for the on-disk partition to avoid repartitioning of the whole
> >> >> disk when a new (and larger) squashfs image is written on later updates.
> >> >
> >> > Is it possible to just use --size or --fixed-size in .wks to allocate
> >> > partition of certain size?
> >> 
> >> --fixed-size works with this patch (tested); --size should work.
> >
> > Sorry for not being clear. Why do we need this if we can use --size to
> > explicity specify partition size?
> 
> --size and --fixed-size did not had an effect for squashfs with the old
> code.
>

I'd propose to fix this instead of applying extra space and overhead
factor to the filesystems that don't support this by design.

The idea is to make size and fixed-size parameters to set partition size
and use extra-space and overhead-factor to extend filesystem size as
they're currently used.

> 
> > --extra-space and --overhead have the same meaning as
> > IMAGE_ROOTFS_EXTRA_SPACE and IMAGE_OVERHEAD_SIZE variables. From my
> > point of view their purpose is to allocate additional space on the
> > file system. This doesn't make sense for squashfs. That's the reason
> > those options are not used for squashfs. Using them to implicitly make
> > bigger partition is a questionable and possible confusing approach.
> 
> I want/need it to allow updates of the running system without complete
> repartitioning.  E.g. at wic creation time, the squashfs has a certain
> size.  Sometime in the future, there are needed e.g. 5 MiB more because
> a new package was added or so.
>

Yep, I understood what you want. I think it's better achieved by setting
partition size with --size option than artificially apply extra-space and
overhead factor to the filesystem that doesn't support this.

> I need to reserve space in the partition so that I can 'dd' the new
> image without a complete repartitioning.
> 
> --extra-space/--overhead has the meaning which I want (e.g. to say
> "reserve +20% for changes in the future").
> 

So far extra-space and overhead factor were used to allocate additional
space on the file system. This is different from the way you want to use
them. You're not allocating space on squashfs, you're allocating
unformatted space on the partition. It's better to use --size for this.

> >> I would keep it as is.  It is a non trivial function which is reusable.
> >> Perhaps, somebody adds CRAMFS or another file system which does not
> >> support extending its size.
> >> 
> >
> > I personally find this much more understandable and logical:
> 
> me not ;) As I said, functionality is perfect for putting into an extra
> method and it makes code much more readable.  When somebody implements
> CRAMFS he can use this method instead of copy&pasting the code.
> 

When somebody implements something that would require this functionality
he/she can create this method instead of copy&pasting the code. Let's
not try to solve problems before they arise as they may never arise :)

> > Then introducing a method that will be called for all supported
> > filesystems except squashfs without any effect.
> 
> You mean, function should be called at the end of prepare_rootfs()
> (after "method(...)")?  Yes, it would reduce code duplication but other
> methods needs to be touch too (to give information whether rootfs size
> has been adjusted already).
> 

I mean that method is called for all possible filesystems, but makes
sense only for squashfs. It makes code less understandable for no
reason. When code is used in the only place it's much more easy to
understand from my point of view.

--
Regards,
Ed

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Enrico Scholz]

Ed Bartosh <ed.bartosh@linux.intel.com> writes:

>> >> >> The --extra-space and --overhead option did not had an effect to squashfs
>> >> >> partitions.  Although squashfs is read-only, it can be useful to allocate
>> >> >> more space for the on-disk partition to avoid repartitioning of the whole
>> >> >> disk when a new (and larger) squashfs image is written on later updates.
>> >> >
>> >> > Is it possible to just use --size or --fixed-size in .wks to allocate
>> >> > partition of certain size?
>> >> 
>> >> --fixed-size works with this patch (tested); --size should work.
>> >
>> > Sorry for not being clear. Why do we need this if we can use --size to
>> > explicity specify partition size?
>> 
>> --size and --fixed-size did not had an effect for squashfs with the old
>> code.
>>
>
> I'd propose to fix this instead of applying extra space and overhead
> factor to the filesystems that don't support this by design.
>
> The idea is to make size and fixed-size parameters to set partition size
> and use extra-space and overhead-factor to extend filesystem size as
> they're currently used.

For what is this overhead to be used?  In most cases, to allow future
updates.  And this is required for squashfs too.

Only difference is, that updates for squashfs are to be applied on
partition level and these for other file systems on file system level.
But the need for extra space exists in both cases.


>> I want/need it to allow updates of the running system without complete
>> repartitioning.  E.g. at wic creation time, the squashfs has a certain
>> size.  Sometime in the future, there are needed e.g. 5 MiB more because
>> a new package was added or so.
>>
>
> Yep, I understood what you want. I think it's better achieved by setting
> partition size with --size option than artificially apply extra-space and
> overhead factor to the filesystem that doesn't support this.

It really does not matter for me whether the filesystem itself can be
extended or whether I need a larger partition to apply future updates.

I just need a partition which provides an absolute or relative amount of
additional space.


>> I need to reserve space in the partition so that I can 'dd' the new
>> image without a complete repartitioning.
>> 
>> --extra-space/--overhead has the meaning which I want (e.g. to say
>> "reserve +20% for changes in the future").
>
> So far extra-space and overhead factor were used to allocate additional
> space on the file system. This is different from the way you want to
> use them. You're not allocating space on squashfs, you're allocating
> unformatted space on the partition. It's better to use --size for this.

No; --size is not what I want. --size (or --fixed-size) assume that I
know the absolute size.

Of course, I can get this size empirically.  But it is highly inflexible
and requires different wks files for different image types.

I want to reserve an extra percentage for future updates.  And these
percentages can be in the range of 1 - 5 MB (for small images with
only test tools) or several hundred MB (for large image types with
e.g. desktop environments).

--size or --fixed-size for these image types would be in completely
different ranges while --extra-space/--overhead fits to all.


> I mean that method is called for all possible filesystems, but makes
> sense only for squashfs. It makes code less understandable for no
> reason. When code is used in the only place it's much more easy to
> understand from my point of view.

Not in my one.



Enrico

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Ed Bartosh]

On Tue, Sep 12, 2017 at 02:18:41PM +0200, Enrico Scholz wrote:
> Ed Bartosh <ed.bartosh@linux.intel.com> writes:
> 
> >> >> >> The --extra-space and --overhead option did not had an effect to squashfs
> >> >> >> partitions.  Although squashfs is read-only, it can be useful to allocate
> >> >> >> more space for the on-disk partition to avoid repartitioning of the whole
> >> >> >> disk when a new (and larger) squashfs image is written on later updates.
> >> >> >
> >> >> > Is it possible to just use --size or --fixed-size in .wks to allocate
> >> >> > partition of certain size?
> >> >> 
> >> >> --fixed-size works with this patch (tested); --size should work.
> >> >
> >> > Sorry for not being clear. Why do we need this if we can use --size to
> >> > explicity specify partition size?
> >> 
> >> --size and --fixed-size did not had an effect for squashfs with the old
> >> code.
> >>
> >
> > I'd propose to fix this instead of applying extra space and overhead
> > factor to the filesystems that don't support this by design.
> >
> > The idea is to make size and fixed-size parameters to set partition size
> > and use extra-space and overhead-factor to extend filesystem size as
> > they're currently used.
> 
> For what is this overhead to be used?  In most cases, to allow future
> updates.  And this is required for squashfs too.
> 

It depends on the usage scenario. Additional space may be required
for other purposes too. It may me needed for the system to be able to
boot and perform what it was built for.

> Only difference is, that updates for squashfs are to be applied on
> partition level and these for other file systems on file system level.
> But the need for extra space exists in both cases.
> 
> 
> >> I want/need it to allow updates of the running system without complete
> >> repartitioning.  E.g. at wic creation time, the squashfs has a certain
> >> size.  Sometime in the future, there are needed e.g. 5 MiB more because
> >> a new package was added or so.
> >>
> >
> > Yep, I understood what you want. I think it's better achieved by setting
> > partition size with --size option than artificially apply extra-space and
> > overhead factor to the filesystem that doesn't support this.
> 
> It really does not matter for me whether the filesystem itself can be
> extended or whether I need a larger partition to apply future updates.
> 

It matters for me. I think I explained why.

> I just need a partition which provides an absolute or relative amount of
> additional space.
> 
> 
> >> I need to reserve space in the partition so that I can 'dd' the new
> >> image without a complete repartitioning.
> >> 
> >> --extra-space/--overhead has the meaning which I want (e.g. to say
> >> "reserve +20% for changes in the future").
> >
> > So far extra-space and overhead factor were used to allocate additional
> > space on the file system. This is different from the way you want to
> > use them. You're not allocating space on squashfs, you're allocating
> > unformatted space on the partition. It's better to use --size for this.
> 
> No; --size is not what I want. --size (or --fixed-size) assume that I
> know the absolute size.
> 
> Of course, I can get this size empirically.  But it is highly inflexible
> and requires different wks files for different image types.
> 
> I want to reserve an extra percentage for future updates.  And these
> percentages can be in the range of 1 - 5 MB (for small images with
> only test tools) or several hundred MB (for large image types with
> e.g. desktop environments).
> 
> --size or --fixed-size for these image types would be in completely
> different ranges while --extra-space/--overhead fits to all.
> 

I agree. --size is less suitable for your needs than extra space and
overhead factor. I still don't like the idea of using them to reserve
non-formatted space. Any other ideas how to do it?

--
Regards,
Ed

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Enrico Scholz]

Ed Bartosh <ed.bartosh@linux.intel.com> writes:

> I agree. --size is less suitable for your needs than extra space and
> overhead factor. I still don't like the idea of using them to reserve
> non-formatted space.

Btw, exactly this is done for extX already.


> Any other ideas how to do it?

Perhaps, Logic can be moved into the disk_size() method:

|    def disk_size(self):
|        method = getattr(self, "_get_disk_size_" + prefix, None)
|        if method:
|            return method(self.size)
|        elif self.fixed_size:
|            return self.fixed_size
|        else:
|            return self.size
|
|    def _get_disk_size_squashfs(size):
|        return self.get_rootfs_size(size)

But I did not checked whether this really works.


Enrico

Re: [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs

[Ed Bartosh]

On Tue, Sep 12, 2017 at 03:23:40PM +0200, Enrico Scholz wrote:
> Ed Bartosh <ed.bartosh@linux.intel.com> writes:
> 
> > I agree. --size is less suitable for your needs than extra space and
> > overhead factor. I still don't like the idea of using them to reserve
> > non-formatted space.
> 
> Btw, exactly this is done for extX already.
> 

Are you sure?
I don't see where we produce unformatted space on extX partitions.
Can you point me out?

--
Regards,
Ed

[PATCH v2 3/3] wic: apply --extra-space + --overhead to squashfs

[Enrico Scholz]

The --extra-space and --overhead option did not had an effect to squashfs
partitions.  Although squashfs is read-only, it can be useful to allocate
more space for the on-disk partition to avoid repartitioning of the whole
disk when a new (and larger) squashfs image is written on later updates.

Patch calls get_rootfs_size() *after* creating the image and truncates
it then.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 scripts/lib/wic/partition.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b623bb9..bf9bbc3 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -195,6 +195,17 @@ class Partition():
                            "larger (%d kB) than its allowed size %d kB" %
                            (self.mountpoint, self.size, self.fixed_size))
 
+    def _extend_rootfs_image(self, rootfs):
+        """Enlarges the rootfs so that it fulfills size/overhead-factor
+        constraints"""
+
+        sz = (os.stat(rootfs).st_size + 1023) // 1024
+        pad_sz = self.get_rootfs_size(sz)
+
+        if pad_sz > sz:
+            with open(rootfs, 'a') as f:
+                os.ftruncate(f.fileno(), pad_sz * 1024)
+
     def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
                        native_sysroot):
         """
@@ -338,6 +349,8 @@ class Partition():
                        (rootfs_dir, rootfs, extraopts)
         exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo)
 
+        self._extend_rootfs_image(rootfs)
+
     def prepare_empty_partition_ext(self, rootfs, oe_builddir,
                                     native_sysroot):
         """
-- 
2.9.5