[wic][PATCH] wic: Implement --build-rootfs command line option

[wic][PATCH] wic: Implement --build-rootfs command line option

[Ed Bartosh]

-f/--build-rootfs option makes wic to run bitbake <image> to
produce rootfs. This option requires image name to be specified
with -e/--image-name.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/image/help.py | 10 ++++++----
 scripts/wic               | 25 ++++++++++++++++++++++---
 2 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index e1eb265..e365a07 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -111,7 +111,7 @@ wic_create_usage = """
             [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
             [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
             [-r, --rootfs-dir] [-b, --bootimg-dir]
-            [-k, --kernel-dir] [-n, --native-sysroot]
+            [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs]
 
  This command creates an OpenEmbedded image based on the 'OE kickstart
  commands' found in the <wks file>.
@@ -132,7 +132,7 @@ SYNOPSIS
         [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
         [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
         [-r, --rootfs-dir] [-b, --bootimg-dir]
-        [-k, --kernel-dir] [-n, --native-sysroot]
+        [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs]
 
 DESCRIPTION
     This command creates an OpenEmbedded image based on the 'OE
@@ -167,6 +167,8 @@ DESCRIPTION
     The -n option is used to specify the path to the native sysroot
     containing the tools to use to build the image.
 
+    The -f option is used to build rootfs by running "bitbake <image>"
+
     The -s option is used to skip the build check.  The build check is
     a simple sanity check used to determine whether the user has
     sourced the build environment so that the -e option can operate
@@ -528,8 +530,8 @@ DESCRIPTION
        usage: wic create <wks file or image name> [-o <DIRNAME> | ...]
             [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
             [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
-            [-r, --rootfs-dir] [-b, --bootimg-dir]
-            [-k, --kernel-dir] [-n, --native-sysroot]
+            [-r, --rootfs-dir] [-b, --bootimg-dir] [-k, --kernel-dir]
+            [-n, --native-sysroot] [-f, --build-rootfs]
 
        This command creates an OpenEmbedded image based on the 'OE
        kickstart commands' found in the <wks file>.
diff --git a/scripts/wic b/scripts/wic
index e7df60f..feff302 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -40,11 +40,15 @@ import logging
 # External modules
 scripts_path = os.path.abspath(os.path.dirname(os.path.abspath(sys.argv[0])))
 lib_path = scripts_path + '/lib'
-sys.path = sys.path + [lib_path]
+bitbake_path = os.path.join(scripts_path, '../bitbake/lib')
+sys.path = sys.path + [lib_path, bitbake_path]
 
 from image.help import *
 from image.engine import *
 
+from bb import cookerdata
+from bb.main import bitbake_main, BitBakeConfigParameters
+
 def rootfs_dir_to_args(krootfs_dir):
     """
     Get a rootfs_dir dict and serialize to string
@@ -94,6 +98,7 @@ def wic_create_subcommand(args, usage_str):
                       action = "store", help = "path to the native sysroot containing the tools to use to build the image")
     parser.add_option("-p", "--skip-build-check", dest = "build_check",
                       action = "store_false", default = True, help = "skip the build check")
+    parser.add_option("-f", "--build-rootfs", action="store_true", help = "build rootfs")
     parser.add_option("-D", "--debug", dest = "debug", action = "store_true",
                       default = False, help = "output debug information")
 
@@ -123,8 +128,6 @@ def wic_create_subcommand(args, usage_str):
         else:
             print "Done.\n"
 
-    print "Creating image(s)...\n"
-
     bitbake_env_lines = find_bitbake_env_lines(options.image_name)
     if not bitbake_env_lines:
         print "Couldn't get bitbake environment, exiting."
@@ -134,9 +137,24 @@ def wic_create_subcommand(args, usage_str):
     bootimg_dir = ""
 
     if options.image_name:
+        if options.build_rootfs:
+            argv = ["bitbake", options.image_name]
+            if options.debug:
+                argv.append("--debug")
+
+            print "Building rootfs...\n"
+            if bitbake_main(BitBakeConfigParameters(argv),
+                            cookerdata.CookerConfiguration()):
+                sys.exit(1)
+
         (rootfs_dir, kernel_dir, bootimg_dir, native_sysroot) \
             = find_artifacts(options.image_name)
 
+    else:
+        if options.build_rootfs:
+            print "Image name is not specified, exiting. (Use -e/--image-name to specify it)\n"
+            sys.exit(1)
+
     wks_file = args[0]
 
     if not wks_file.endswith(".wks"):
@@ -194,6 +212,7 @@ def wic_create_subcommand(args, usage_str):
 
     rootfs_dir = rootfs_dir_to_args(krootfs_dir)
 
+    print "Creating image(s)...\n"
     wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                native_sysroot, scripts_path, image_output_dir,
                options.debug, options.properties_file)
-- 
2.1.4

Re: [wic][PATCH] wic: Implement --build-rootfs command line option

[João Henrique Ferreira de Freitas]

Hi Ed,

I liked.

Two points:

- How about if user sets INHERIT = "rm_work"? I mean the image recipe 
will be rm, right? I always put my images in RM_WORK_EXCLUDE so wic can 
use it.

- What will be the behavior if: wic create test-image-4Gb -e test-image 
--rootfs rootfs1=test-image --rootfs rootfs2=bringup-image -f. A 
believed that only 'test-image' will be build. Right?

Thanks.

On 07/04/2015 06:57, Ed Bartosh wrote:
> -f/--build-rootfs option makes wic to run bitbake <image> to
> produce rootfs. This option requires image name to be specified
> with -e/--image-name.
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>   scripts/lib/image/help.py | 10 ++++++----
>   scripts/wic               | 25 ++++++++++++++++++++++---
>   2 files changed, 28 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
> index e1eb265..e365a07 100644
> --- a/scripts/lib/image/help.py
> +++ b/scripts/lib/image/help.py
> @@ -111,7 +111,7 @@ wic_create_usage = """
>               [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
>               [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
>               [-r, --rootfs-dir] [-b, --bootimg-dir]
> -            [-k, --kernel-dir] [-n, --native-sysroot]
> +            [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs]
>   
>    This command creates an OpenEmbedded image based on the 'OE kickstart
>    commands' found in the <wks file>.
> @@ -132,7 +132,7 @@ SYNOPSIS
>           [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
>           [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
>           [-r, --rootfs-dir] [-b, --bootimg-dir]
> -        [-k, --kernel-dir] [-n, --native-sysroot]
> +        [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs]
>   
>   DESCRIPTION
>       This command creates an OpenEmbedded image based on the 'OE
> @@ -167,6 +167,8 @@ DESCRIPTION
>       The -n option is used to specify the path to the native sysroot
>       containing the tools to use to build the image.
>   
> +    The -f option is used to build rootfs by running "bitbake <image>"
> +
>       The -s option is used to skip the build check.  The build check is
>       a simple sanity check used to determine whether the user has
>       sourced the build environment so that the -e option can operate
> @@ -528,8 +530,8 @@ DESCRIPTION
>          usage: wic create <wks file or image name> [-o <DIRNAME> | ...]
>               [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
>               [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
> -            [-r, --rootfs-dir] [-b, --bootimg-dir]
> -            [-k, --kernel-dir] [-n, --native-sysroot]
> +            [-r, --rootfs-dir] [-b, --bootimg-dir] [-k, --kernel-dir]
> +            [-n, --native-sysroot] [-f, --build-rootfs]
>   
>          This command creates an OpenEmbedded image based on the 'OE
>          kickstart commands' found in the <wks file>.
> diff --git a/scripts/wic b/scripts/wic
> index e7df60f..feff302 100755
> --- a/scripts/wic
> +++ b/scripts/wic
> @@ -40,11 +40,15 @@ import logging
>   # External modules
>   scripts_path = os.path.abspath(os.path.dirname(os.path.abspath(sys.argv[0])))
>   lib_path = scripts_path + '/lib'
> -sys.path = sys.path + [lib_path]
> +bitbake_path = os.path.join(scripts_path, '../bitbake/lib')
> +sys.path = sys.path + [lib_path, bitbake_path]
>   
>   from image.help import *
>   from image.engine import *
>   
> +from bb import cookerdata
> +from bb.main import bitbake_main, BitBakeConfigParameters
> +
>   def rootfs_dir_to_args(krootfs_dir):
>       """
>       Get a rootfs_dir dict and serialize to string
> @@ -94,6 +98,7 @@ def wic_create_subcommand(args, usage_str):
>                         action = "store", help = "path to the native sysroot containing the tools to use to build the image")
>       parser.add_option("-p", "--skip-build-check", dest = "build_check",
>                         action = "store_false", default = True, help = "skip the build check")
> +    parser.add_option("-f", "--build-rootfs", action="store_true", help = "build rootfs")
>       parser.add_option("-D", "--debug", dest = "debug", action = "store_true",
>                         default = False, help = "output debug information")
>   
> @@ -123,8 +128,6 @@ def wic_create_subcommand(args, usage_str):
>           else:
>               print "Done.\n"
>   
> -    print "Creating image(s)...\n"
> -
>       bitbake_env_lines = find_bitbake_env_lines(options.image_name)
>       if not bitbake_env_lines:
>           print "Couldn't get bitbake environment, exiting."
> @@ -134,9 +137,24 @@ def wic_create_subcommand(args, usage_str):
>       bootimg_dir = ""
>   
>       if options.image_name:
> +        if options.build_rootfs:
> +            argv = ["bitbake", options.image_name]
> +            if options.debug:
> +                argv.append("--debug")
> +
> +            print "Building rootfs...\n"
> +            if bitbake_main(BitBakeConfigParameters(argv),
> +                            cookerdata.CookerConfiguration()):
> +                sys.exit(1)
> +
>           (rootfs_dir, kernel_dir, bootimg_dir, native_sysroot) \
>               = find_artifacts(options.image_name)
>   
> +    else:
> +        if options.build_rootfs:
> +            print "Image name is not specified, exiting. (Use -e/--image-name to specify it)\n"
> +            sys.exit(1)
> +
>       wks_file = args[0]
>   
>       if not wks_file.endswith(".wks"):
> @@ -194,6 +212,7 @@ def wic_create_subcommand(args, usage_str):
>   
>       rootfs_dir = rootfs_dir_to_args(krootfs_dir)
>   
> +    print "Creating image(s)...\n"
>       wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
>                  native_sysroot, scripts_path, image_output_dir,
>                  options.debug, options.properties_file)

Re: [wic][PATCH] wic: Implement --build-rootfs command line option

[Ed Bartosh]

On Wed, Apr 08, 2015 at 07:18:40PM -0300, João Henrique Ferreira de Freitas wrote:
> 
> Hi Ed,
> 
> I liked.
> 
> Two points:
> 
> - How about if user sets INHERIT = "rm_work"? I mean the image
> recipe will be rm, right? I always put my images in RM_WORK_EXCLUDE
> so wic can use it.
>
rm_work removes artifacts required by wic. Wic will complain that rootfs
can't be found. There is a bug about i https://bugzilla.yoctoproject.org/show_bug.cgi?id=7042
I closed it as not a bug as wic uses build artifacts by design.

> - What will be the behavior if: wic create test-image-4Gb -e
> test-image --rootfs rootfs1=test-image --rootfs
> rootfs2=bringup-image -f. A believed that only 'test-image' will be
> build. Right?
>
I don't fully understand command line options in your example, but generally you're right.
wic will take image name from -e option and run "bitbake test-image" in this case.

Regards,
Ed

Re: [wic][PATCH] wic: Implement --build-rootfs command line option

[João Henrique Ferreira de Freitas]

Hi

On 09/04/2015 16:12, Ed Bartosh wrote:
> On Wed, Apr 08, 2015 at 07:18:40PM -0300, João Henrique Ferreira de Freitas wrote:
>> Hi Ed,
>>
>> I liked.
>>
>> Two points:
>>
>> - How about if user sets INHERIT = "rm_work"? I mean the image
>> recipe will be rm, right? I always put my images in RM_WORK_EXCLUDE
>> so wic can use it.
>>
> rm_work removes artifacts required by wic. Wic will complain that rootfs
> can't be found. There is a bug about i https://bugzilla.yoctoproject.org/show_bug.cgi?id=7042
> I closed it as not a bug as wic uses build artifacts by design.

I agree.

>> - What will be the behavior if: wic create test-image-4Gb -e
>> test-image --rootfs rootfs1=test-image --rootfs
>> rootfs2=bringup-image -f. A believed that only 'test-image' will be
>> build. Right?
>>
> I don't fully understand command line options in your example, but generally you're right.
> wic will take image name from -e option and run "bitbake test-image" in this case.

This thread explain the above command line options: 
http://lists.openembedded.org/pipermail/openembedded-core/2014-March/091185.html

Usually I run this sequence:

bitbake test-image
bitbate bringup-image

wic create test-image-4Gb -e test-image --rootfs rootfs1=test-image --rootfs rootfs2=bringup-image

So wic will create a directdisk with two images where each one will be 
place on specific partition.

I  guess if '-f' is used, wic could call bitbake to 'test-image' and 
'bringup-image'.

Thanks.

Re: [wic][PATCH] wic: Implement --build-rootfs command line option

[Ed Bartosh]

On Thu, Apr 09, 2015 at 07:24:16PM -0300, João Henrique Ferreira de Freitas wrote:
> 
> >>- What will be the behavior if: wic create test-image-4Gb -e
> >>test-image --rootfs rootfs1=test-image --rootfs
> >>rootfs2=bringup-image -f. A believed that only 'test-image' will be
> >>build. Right?
> >>
> >I don't fully understand command line options in your example, but generally you're right.
> >wic will take image name from -e option and run "bitbake test-image" in this case.
> 
> This thread explain the above command line options: http://lists.openembedded.org/pipermail/openembedded-core/2014-March/091185.html
> 
> Usually I run this sequence:
> 
> bitbake test-image
> bitbate bringup-image
> 
> wic create test-image-4Gb -e test-image --rootfs rootfs1=test-image --rootfs rootfs2=bringup-image
> 
> So wic will create a directdisk with two images where each one will
> be place on specific partition.
> 
> I  guess if '-f' is used, wic could call bitbake to 'test-image' and
> 'bringup-image'.

Thank you for explanations. I missed your patches somehow.

Can we do the same by implementing handling of multiple -e/--image-name options in command line and .wks?
e.g. wic create directdisk-multi-rootfs.wks -e core-image-minimal -e core-image-minimal-dev

directdisk-multi-rootfs.wks:
part / --image-name core-image-minimal --ondisk sda --fstype=ext3 --label primary --align 1024
part /standby --image-name core-image-minimal-dev --ondisk sda --fstype=ext3 --label secondary --align 1024

--
Regards,
Ed

Re: [wic][PATCH] wic: Implement --build-rootfs command line option

[Ed Bartosh]

On Mon, Apr 20, 2015 at 06:13:20PM +0200, Adrian Freihofer wrote:
> Let wic call bitbake seems to be a great idea.
> 
> One question is open to me. What's the recommended approach to resolve
> the dependencies between the image, the bootloader and native tools
> which are additionally required to create the final disk image?
This is open to me too. I was kinda surprised when I saw that no images from 'wic list images' output can be built for not that obvious reasons:
$ for image in `wic list images | cut -f3 -d' '` ; do wic create $image -e core-image-minimal; done
Checking basic build environment...
Done.

Creating image(s)...

Error: Couldn't find HDDDIR, exiting

Checking basic build environment...
Done.

Creating image(s)...

Error: No boot files defined, IMAGE_BOOT_FILES unset
Checking basic build environment...
Done.

Creating image(s)...

Error: Couldn't find HDDDIR, exiting

Checking basic build environment...
Done.

Creating image(s)...

Error: Please build syslinux first

Only last message makes sense to me. The rest two don't give any hints on how to solve the problem.

> Basically the right place to resolve this dependency is the kickstart
> file. Bitbake should not even know that syslinux or any other bootloader
> will be added to the final disk image. In other words
> IMAGE_EXTRA_DEPENDS or similar are not the right place to add the
> bootloader dependency anymore. But if bitbake does not provide e.g.
> syslinux-native wic will fail to use it. This is kind of a chicken egg
> problem...
> A nice solution would be if wic would maintain a simple dependency list
> For example: There is a kickstart file assembling a disk image including
> my-mini-image and syslinux-native. To create the partition layout,
> parted-native is required as well. Hence bitbake should be called like
> this: "bitbake my-mini-image parted-native syslinux-native" before wic
> starts assembling the disk image. my-mini-image is provided by the -e
> parameter of wic. The latter two tools should be provided by the wic
> plugins creating the partitions and the disk image. My proposal is to
> extend the internal API of wic by a dependency list where plugins can
> append required tools e.g. during start up of wic. This would enable the
> disk plugin to request "parted-native" and the boot partition plugin to
> request "syslinux-native" before bitbake is called.

I like the proposal. Just want to point out to a couple of things.
First, parted-native seems to be essential tool for wic, so wic core should check for it from my point of view.
Second, it depends on the environment if wic is able to find bootloader or not. For example, baking bootloader is not always enough for wic to find it. I'm still puzzled by this fact, but it depends on the MACHINE and may be other parameters. For example, wic is not able to find bootloader if core-image-minimal is built for default machine qemux86(see errors above), but it works just fine for intel-corei7-64.

So, I would propose to start from making wic to work for any build. This way we can understand requirements better.

> Further on one more problem could be solved easily. If wic takes care
> about the final disk assembly bitbake --fetch-only might not be
> sufficient to download everything needed to create the whole firmware.
> This makes it hard to create a complete premirror. If wic would support
> the dependency list as mentioned above and provide a command line
> parameter --fetch-only everything would be consistent. Instead of
> calling "bitbake  my-mini-image  --fetch-only" the user just calls the
> corresponding wic command with --fetch-only appended. This would result
> in "bitbake my-mini-image parted-native syslinux-native --fetch-only".

What's fetch-only option? I don't see it among bitbake options.

--
Regards,
Ed

Re: [wic][PATCH] wic: Implement --build-rootfs command line option

[Adrian Freihofer]

Let wic call bitbake seems to be a great idea.

One question is open to me. What's the recommended approach to resolve
the dependencies between the image, the bootloader and native tools
which are additionally required to create the final disk image?
Basically the right place to resolve this dependency is the kickstart
file. Bitbake should not even know that syslinux or any other bootloader
will be added to the final disk image. In other words
IMAGE_EXTRA_DEPENDS or similar are not the right place to add the
bootloader dependency anymore. But if bitbake does not provide e.g.
syslinux-native wic will fail to use it. This is kind of a chicken egg
problem...
A nice solution would be if wic would maintain a simple dependency list
For example: There is a kickstart file assembling a disk image including
my-mini-image and syslinux-native. To create the partition layout,
parted-native is required as well. Hence bitbake should be called like
this: "bitbake my-mini-image parted-native syslinux-native" before wic
starts assembling the disk image. my-mini-image is provided by the -e
parameter of wic. The latter two tools should be provided by the wic
plugins creating the partitions and the disk image. My proposal is to
extend the internal API of wic by a dependency list where plugins can
append required tools e.g. during start up of wic. This would enable the
disk plugin to request "parted-native" and the boot partition plugin to
request "syslinux-native" before bitbake is called.
Further on one more problem could be solved easily. If wic takes care
about the final disk assembly bitbake --fetch-only might not be
sufficient to download everything needed to create the whole firmware.
This makes it hard to create a complete premirror. If wic would support
the dependency list as mentioned above and provide a command line
parameter --fetch-only everything would be consistent. Instead of
calling "bitbake  my-mini-image  --fetch-only" the user just calls the
corresponding wic command with --fetch-only appended. This would result
in "bitbake my-mini-image parted-native syslinux-native --fetch-only".