[PATCH 0/4] use qemuboot.json to replace qemuboot.conf

[PATCH 0/4] use qemuboot.json to replace qemuboot.conf

[Robert Yang]

The qemuboot.conf uses configparser which can't suport upper case as key, and 
json is more clearer than configparser and is widely used in oe-core, so use 
qemuboot.json to replace qemuboot.conf.

// Robert

The following changes since commit a0988c3374e964170d1d24fc230306b887432d31:

  tcmode-default.inc: drop preferred version of gzip-native (2018-01-31 17:01:12 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/json
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/json

Robert Yang (4):
  qemuboot.bbclass: qemuboot.conf -> qemuboot.json
  runqemu: qemuboot.conf -> qemuboot.json
  runqemu: check for qemuboot.conf and raise error
  selftest/cases/runqemu.py: qemuboot.conf -> qemuboot.json

 meta/classes/qemuboot.bbclass           | 17 ++++----
 meta/lib/oeqa/selftest/cases/runqemu.py | 12 +++---
 scripts/runqemu                         | 73 +++++++++++++++++----------------
 3 files changed, 52 insertions(+), 50 deletions(-)

-- 
2.7.4

[PATCH 1/4] qemuboot.bbclass: qemuboot.conf -> qemuboot.json

[Robert Yang]

The qemuboot.conf uses configparser which can't suport upper case as key, and
json is more clearer than configparser and is widely used in oe-core, so use
qemuboot.json to replace qemuboot.conf.

[YOCTO #12503]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/qemuboot.bbclass | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/meta/classes/qemuboot.bbclass b/meta/classes/qemuboot.bbclass
index 15a9e63..7c46a85 100644
--- a/meta/classes/qemuboot.bbclass
+++ b/meta/classes/qemuboot.bbclass
@@ -67,7 +67,7 @@ QB_NETWORK_DEVICE ?= "-device virtio-net-pci,netdev=net0,mac=@MAC@"
 # This should be kept align with ROOT_VM
 QB_DRIVE_TYPE ?= "/dev/sd"
 
-# Create qemuboot.conf
+# Create qemuboot.json
 addtask do_write_qemuboot_conf after do_rootfs before do_image
 IMGDEPLOYDIR ?= "${WORKDIR}/deploy-${PN}-image-complete"
 
@@ -81,14 +81,13 @@ def qemuboot_vars(d):
 do_write_qemuboot_conf[vardeps] += "${@' '.join(qemuboot_vars(d))}"
 do_write_qemuboot_conf[vardepsexclude] += "TOPDIR"
 python do_write_qemuboot_conf() {
-    import configparser
+    import json
 
-    qemuboot = "%s/%s.qemuboot.conf" % (d.getVar('IMGDEPLOYDIR'), d.getVar('IMAGE_NAME'))
-    qemuboot_link = "%s/%s.qemuboot.conf" % (d.getVar('IMGDEPLOYDIR'), d.getVar('IMAGE_LINK_NAME'))
+    qemuboot = "%s/%s.qemuboot.json" % (d.getVar('IMGDEPLOYDIR'), d.getVar('IMAGE_NAME'))
+    qemuboot_link = "%s/%s.qemuboot.json" % (d.getVar('IMGDEPLOYDIR'), d.getVar('IMAGE_LINK_NAME'))
     finalpath = d.getVar("DEPLOY_DIR_IMAGE")
     topdir = d.getVar('TOPDIR')
-    cf = configparser.ConfigParser()
-    cf.add_section('config_bsp')
+    qb_dict = {}
     for k in sorted(qemuboot_vars(d)):
         # qemu-helper-native sysroot is not removed by rm_work and
         # contains all tools required by runqemu
@@ -101,7 +100,7 @@ python do_write_qemuboot_conf() {
         # and still run them
         if val.startswith(topdir):
             val = os.path.relpath(val, finalpath)
-        cf.set('config_bsp', k, '%s' % val)
+        qb_dict[k] = val
 
     # QB_DEFAULT_KERNEL's value of KERNEL_IMAGETYPE is the name of a symlink
     # to the kernel file, which hinders relocatability of the qb conf.
@@ -111,11 +110,11 @@ python do_write_qemuboot_conf() {
     # we only want to write out relative paths so that we can relocate images
     # and still run them
     kernel = os.path.relpath(kernel, finalpath)
-    cf.set('config_bsp', 'QB_DEFAULT_KERNEL', kernel)
+    qb_dict['QB_DEFAULT_KERNEL'] = kernel
 
     bb.utils.mkdirhier(os.path.dirname(qemuboot))
     with open(qemuboot, 'w') as f:
-        cf.write(f)
+        json.dump(qb_dict, f, indent=4, sort_keys=True)
 
     if qemuboot_link != qemuboot:
         if os.path.lexists(qemuboot_link):
-- 
2.7.4

[PATCH 2/4] runqemu: qemuboot.conf -> qemuboot.json

[Robert Yang]

The qemuboot.conf uses configparser which can't suport upper case as key, and
json is more clearer than configparser and is widely used in oe-core, so use
qemuboot.json to replace qemuboot.conf.

[YOCTO #12503]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 scripts/runqemu | 70 ++++++++++++++++++++++++++++-----------------------------
 1 file changed, 35 insertions(+), 35 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index d998494..0ca62f4 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -26,7 +26,7 @@ import re
 import fcntl
 import shutil
 import glob
-import configparser
+import json
 
 class RunQemuError(Exception):
     """Custom exception to raise on known errors."""
@@ -38,8 +38,8 @@ class OEPathError(RunQemuError):
         super().__init__("In order for this script to dynamically infer paths\n \
 kernels or filesystem images, you either need bitbake in your PATH\n \
 or to source oe-init-build-env before running this script.\n\n \
-Dynamic path inference can be avoided by passing a *.qemuboot.conf to\n \
-runqemu, i.e. `runqemu /path/to/my-image-name.qemuboot.conf`\n\n %s" % message)
+Dynamic path inference can be avoided by passing a *.qemuboot.json to\n \
+runqemu, i.e. `runqemu /path/to/my-image-name.qemuboot.json`\n\n %s" % message)
 
 
 def create_logger():
@@ -93,7 +93,7 @@ Examples:
   runqemu
   runqemu qemuarm
   runqemu tmp/deploy/images/qemuarm
-  runqemu tmp/deploy/images/qemux86/<qemuboot.conf>
+  runqemu tmp/deploy/images/qemux86/<qemuboot.json>
   runqemu qemux86-64 core-image-sato ext4
   runqemu qemux86-64 wic-image-minimal wic
   runqemu path/to/bzImage-qemux86.bin path/to/nfsrootdir/ serial
@@ -171,7 +171,7 @@ def check_free_port(host, port):
 
 class BaseConfig(object):
     def __init__(self):
-        # The self.d saved vars from self.set(), part of them are from qemuboot.conf
+        # The self.d saved vars from self.set(), part of them are from qemuboot.json
         self.d = {'QB_KERNEL_ROOT': '/dev/vda'}
 
         # Supported env vars, add it here if a var can be got from env,
@@ -197,7 +197,7 @@ class BaseConfig(object):
         # Setting one also adds "-vga std" because that is all that
         # OVMF supports.
         self.ovmf_bios = []
-        self.qemuboot = ''
+        self.qb_json = ''
         self.qbconfload = False
         self.kernel = ''
         self.kernel_cmdline = ''
@@ -268,8 +268,8 @@ class BaseConfig(object):
 
     def is_deploy_dir_image(self, p):
         if os.path.isdir(p):
-            if not re.search('.qemuboot.conf$', '\n'.join(os.listdir(p)), re.M):
-                logger.debug("Can't find required *.qemuboot.conf in %s" % p)
+            if not re.search('.qemuboot.json$', '\n'.join(os.listdir(p)), re.M):
+                logger.debug("Can't find required *.qemuboot.json in %s" % p)
                 return False
             if not any(map(lambda name: '-image-' in name, os.listdir(p))):
                 logger.debug("Can't find *-image-* in %s" % p)
@@ -309,14 +309,14 @@ class BaseConfig(object):
 
     def check_arg_path(self, p):
         """
-        - Check whether it is <image>.qemuboot.conf or contains <image>.qemuboot.conf
+        - Check whether it is <image>.qemuboot.json or contains <image>.qemuboot.json
         - Check whether is a kernel file
         - Check whether is a image file
         - Check whether it is a nfs dir
         - Check whether it is a OVMF flash file
         """
-        if p.endswith('.qemuboot.conf'):
-            self.qemuboot = p
+        if p.endswith('.qemuboot.json'):
+            self.qb_json = p
             self.qbconfload = True
         elif re.search('\.bin$', p) or re.search('bzImage', p) or \
              re.search('zImage', p) or re.search('vmlinux', p) or \
@@ -338,9 +338,9 @@ class BaseConfig(object):
             if fst:
                 self.check_arg_fstype(fst)
                 qb = re.sub('\.' + fst + "$", '', self.rootfs)
-                qb = '%s%s' % (re.sub('\.rootfs$', '', qb), '.qemuboot.conf')
+                qb = '%s%s' % (re.sub('\.rootfs$', '', qb), '.qemuboot.json')
                 if os.path.exists(qb):
-                    self.qemuboot = qb
+                    self.qb_json = qb
                     self.qbconfload = True
                 else:
                     logger.warn("%s doesn't exist" % qb)
@@ -701,7 +701,7 @@ class BaseConfig(object):
         self.check_tcpserial()
 
     def read_qemuboot(self):
-        if not self.qemuboot:
+        if not self.qb_json:
             if self.get('DEPLOY_DIR_IMAGE'):
                 deploy_dir_image = self.get('DEPLOY_DIR_IMAGE')
             else:
@@ -713,10 +713,10 @@ class BaseConfig(object):
                 machine = self.get('MACHINE')
                 if not machine:
                     machine = os.path.basename(deploy_dir_image)
-                self.qemuboot = "%s/%s-%s.qemuboot.conf" % (deploy_dir_image,
+                self.qb_json = "%s/%s-%s.qemuboot.json" % (deploy_dir_image,
                         self.rootfs, machine)
             else:
-                cmd = 'ls -t %s/*.qemuboot.conf' %  deploy_dir_image
+                cmd = 'ls -t %s/*.qemuboot.json' %  deploy_dir_image
                 logger.debug('Running %s...' % cmd)
                 try:
                     qbs = subprocess.check_output(cmd, shell=True).decode('utf-8')
@@ -727,42 +727,42 @@ class BaseConfig(object):
                         # Don't use initramfs when other choices unless fstype is ramfs
                         if '-initramfs-' in os.path.basename(qb) and self.fstype != 'cpio.gz':
                                 continue
-                        self.qemuboot = qb
+                        self.qb_json = qb
                         break
-                    if not self.qemuboot:
+                    if not self.qb_json:
                         # Use the first one when no choice
-                        self.qemuboot = qbs.split()[0]
+                        self.qb_json = qbs.split()[0]
                     self.qbconfload = True
 
-        if not self.qemuboot:
-            # If we haven't found a .qemuboot.conf at this point it probably
+        if not self.qb_json:
+            # If we haven't found a .qemuboot.json at this point it probably
             # doesn't exist, continue without
             return
 
-        if not os.path.exists(self.qemuboot):
-            raise RunQemuError("Failed to find %s (wrong image name or BSP does not support running under qemu?)." % self.qemuboot)
+        if not os.path.exists(self.qb_json):
+            raise RunQemuError("Failed to find %s (wrong image name or BSP does not support running under qemu?)." % self.qb_json)
 
-        logger.debug('CONFFILE: %s' % self.qemuboot)
+        logger.debug('CONFFILE: %s' % self.qb_json)
 
-        cf = configparser.ConfigParser()
-        cf.read(self.qemuboot)
-        for k, v in cf.items('config_bsp'):
-            k_upper = k.upper()
+        with open(self.qb_json, 'r') as f:
+            qb_json_dict = json.load(f)
+
+        for k, v in qb_json_dict.items():
             if v.startswith("../"):
-                v = os.path.abspath(os.path.dirname(self.qemuboot) + "/" + v)
+                v = os.path.abspath(os.path.dirname(self.qb_json) + "/" + v)
             elif v == ".":
-                v = os.path.dirname(self.qemuboot)
-            self.set(k_upper, v)
+                v = os.path.dirname(self.qb_json)
+            self.set(k, v)
 
     def validate_paths(self):
         """Ensure all relevant path variables are set"""
-        # When we're started with a *.qemuboot.conf arg assume that image
+        # When we're started with a *.qemuboot.json arg assume that image
         # artefacts are relative to that file, rather than in whatever
         # directory DEPLOY_DIR_IMAGE in the conf file points to.
         if self.qbconfload:
-            imgdir = os.path.realpath(os.path.dirname(self.qemuboot))
+            imgdir = os.path.realpath(os.path.dirname(self.qb_json))
             if imgdir != os.path.realpath(self.get('DEPLOY_DIR_IMAGE')):
-                logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir))
+                logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qb_json, imgdir))
                 self.set('DEPLOY_DIR_IMAGE', imgdir)
 
         # If the STAGING_*_NATIVE directories from the config file don't exist
@@ -818,7 +818,7 @@ class BaseConfig(object):
             print('ROOTFS: [%s]' % self.rootfs)
         if self.ovmf_bios:
             print('OVMF: %s' % self.ovmf_bios)
-        print('CONFFILE: [%s]' % self.qemuboot)
+        print('CONFFILE: [%s]' % self.qb_json)
         print('')
 
     def setup_nfs(self):
-- 
2.7.4

[PATCH 3/4] runqemu: check for qemuboot.conf and raise error

[Robert Yang]

[YOCTO #12503]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 scripts/runqemu | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/runqemu b/scripts/runqemu
index 0ca62f4..d50c6d1 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -417,6 +417,9 @@ class BaseConfig(object):
 
         unknown_arg = ""
         for arg in sys.argv[1:]:
+            if arg.endswith('.qemuboot.conf'):
+                raise RunQemuError("qemuboot.conf is not supported any more, use qemuboot.json to instead of it")
+
             if arg in self.fstypes + self.vmtypes:
                 self.check_arg_fstype(arg)
             elif arg == 'nographic':
-- 
2.7.4

[PATCH 4/4] selftest/cases/runqemu.py: qemuboot.conf -> qemuboot.json

[Robert Yang]

[YOCTO #12503]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/lib/oeqa/selftest/cases/runqemu.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/runqemu.py b/meta/lib/oeqa/selftest/cases/runqemu.py
index 47d41f5..cfeb224 100644
--- a/meta/lib/oeqa/selftest/cases/runqemu.py
+++ b/meta/lib/oeqa/selftest/cases/runqemu.py
@@ -117,12 +117,12 @@ SYSLINUX_TIMEOUT = "10"
 
     @OETestID(2010)
     def test_boot_qemu_boot(self):
-        """Test runqemu /path/to/image.qemuboot.conf"""
-        qemuboot_conf = "%s-%s.qemuboot.conf" % (self.recipe, self.machine)
-        qemuboot_conf = os.path.join(self.deploy_dir_image, qemuboot_conf)
-        if not os.path.exists(qemuboot_conf):
-            self.skipTest("%s not found" % qemuboot_conf)
-        cmd = "%s %s" % (self.cmd_common, qemuboot_conf)
+        """Test runqemu /path/to/image.qemuboot.json"""
+        qb_json = "%s-%s.qemuboot.json" % (self.recipe, self.machine)
+        qb_json = os.path.join(self.deploy_dir_image, qb_json)
+        if not os.path.exists(qb_json):
+            self.skipTest("%s not found" % qb_json)
+        cmd = "%s %s" % (self.cmd_common, qb_json)
         with runqemu(self.recipe, ssh=False, launch_cmd=cmd) as qemu:
             self.assertTrue(qemu.runner.logged, "Failed: %s" % cmd)
 
-- 
2.7.4

Re: [PATCH 3/4] runqemu: check for qemuboot.conf and raise error

[Randy MacLeod]

On 2018-02-01 10:23 PM, Robert Yang wrote:
> [YOCTO #12503]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   scripts/runqemu | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/scripts/runqemu b/scripts/runqemu
> index 0ca62f4..d50c6d1 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -417,6 +417,9 @@ class BaseConfig(object):
>   
>           unknown_arg = ""
>           for arg in sys.argv[1:]:
> +            if arg.endswith('.qemuboot.conf'):
> +                raise RunQemuError("qemuboot.conf is not supported any more, use qemuboot.json to instead of it")
> +
better wording:
  qemuboot.conf format is no longer supported, use a qemuboot.json file 
instead.



../Randy

>               if arg in self.fstypes + self.vmtypes:
>                   self.check_arg_fstype(arg)
>               elif arg == 'nographic':
> 


-- 
# Randy MacLeod.  WR Linux
# Wind River an Intel Company

Re: [PATCH 3/4] runqemu: check for qemuboot.conf and raise error

[Robert Yang]

On 02/02/2018 10:49 PM, Randy MacLeod wrote:
> On 2018-02-01 10:23 PM, Robert Yang wrote:
>> [YOCTO #12503]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   scripts/runqemu | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/scripts/runqemu b/scripts/runqemu
>> index 0ca62f4..d50c6d1 100755
>> --- a/scripts/runqemu
>> +++ b/scripts/runqemu
>> @@ -417,6 +417,9 @@ class BaseConfig(object):
>>           unknown_arg = ""
>>           for arg in sys.argv[1:]:
>> +            if arg.endswith('.qemuboot.conf'):
>> +                raise RunQemuError("qemuboot.conf is not supported any more, 
>> use qemuboot.json to instead of it")
>> +
> better wording:
>   qemuboot.conf format is no longer supported, use a qemuboot.json file instead.
> 

Thanks, updated in the repo.

// Robert

> 
> 
> ../Randy
> 
>>               if arg in self.fstypes + self.vmtypes:
>>                   self.check_arg_fstype(arg)
>>               elif arg == 'nographic':
>>
> 
> 

Re: [PATCH 0/4] use qemuboot.json to replace qemuboot.conf

[Robert Yang]

Ping.

// Robert

On 02/02/2018 11:23 AM, Robert Yang wrote:
> The qemuboot.conf uses configparser which can't suport upper case as key, and
> json is more clearer than configparser and is widely used in oe-core, so use
> qemuboot.json to replace qemuboot.conf.
> 
> // Robert
> 
> The following changes since commit a0988c3374e964170d1d24fc230306b887432d31:
> 
>    tcmode-default.inc: drop preferred version of gzip-native (2018-01-31 17:01:12 +0000)
> 
> are available in the git repository at:
> 
>    git://git.openembedded.org/openembedded-core-contrib rbt/json
>    http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/json
> 
> Robert Yang (4):
>    qemuboot.bbclass: qemuboot.conf -> qemuboot.json
>    runqemu: qemuboot.conf -> qemuboot.json
>    runqemu: check for qemuboot.conf and raise error
>    selftest/cases/runqemu.py: qemuboot.conf -> qemuboot.json
> 
>   meta/classes/qemuboot.bbclass           | 17 ++++----
>   meta/lib/oeqa/selftest/cases/runqemu.py | 12 +++---
>   scripts/runqemu                         | 73 +++++++++++++++++----------------
>   3 files changed, 52 insertions(+), 50 deletions(-)
>