[PATCH 0/4] Python3 runqemu

[PATCH 0/4] Python3 runqemu

[Joshua Lock]

This small series against master-next (as the python3 runqemu hasn't made it to
master yet) fixes the new runqemu for a workflow of mine where I build on a
headless machine and copy images locally to test them with runqemu.

The following changes since commit e0f246cefb8b6642e0df98067ade06833e83c1f8:

  gobject-introspection: set GI_SCANNER_DISABLE_CACHE for native (2016-09-05 11:59:01 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib joshuagl/runqemu
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/runqemu

Joshua Lock (4):
  runqemu: fix typos
  runqemu: use glob rather than calling out to ls
  runqemu: assume artefacts are relative to *.qemuboot.conf
  runqemu: get NATIVE dirs from env when in source OE build dir

 scripts/runqemu | 44 ++++++++++++++++++++++++++++++++++++++------
 1 file changed, 38 insertions(+), 6 deletions(-)

-- 
2.7.4

[PATCH 1/4] runqemu: fix typos

[Joshua Lock]

Remove a stray whitespace when accessing a member variable and fix a
spelling mistake in an Exception message.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index a94cc65..72c6352 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -160,7 +160,7 @@ class BaseConfig(object):
         self.custombiosdir = ''
         self.lock = ''
         self.lock_descriptor = ''
-        self. bitbake_e = ''
+        self.bitbake_e = ''
         self.fstypes = ('ext2', 'ext3', 'ext4', 'jffs2', 'nfs', 'btrfs', 'cpio.gz', 'cpio', 'ramfs')
         self.vmtypes = ('hddimg', 'hdddirect', 'wic', 'vmdk', 'qcow2', 'vdi', 'iso')
 
@@ -289,7 +289,7 @@ class BaseConfig(object):
             self.set_machine_deploy_dir(arg, deploy_dir_image)
         else:
             logger.error("%s not a directory valid DEPLOY_DIR_IMAGE" % deploy_dir_image)
-            raise Exception("Falied to set MACHINE to %s. Unknown arg: %s" % (arg, arg))
+            raise Exception("Failed to set MACHINE to %s. Unknown arg: %s" % (arg, arg))
 
     def check_args(self):
         unknown_arg = ""
-- 
2.7.4

[PATCH 2/4] runqemu: use glob rather than calling out to ls

[Joshua Lock]

Use the Python glob module to find files matching a pattern, rather
than calling out to ls and parsing the output.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 72c6352..1599848 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -528,11 +528,10 @@ class BaseConfig(object):
                 self.qemuboot = "%s/%s-%s.qemuboot.conf" % (deploy_dir_image,
                         self.rootfs, machine)
             else:
-                cmd = 'ls -t %s/*.qemuboot.conf' %  deploy_dir_image
-                logger.info('Running %s...' % cmd)
-                qbs = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
+                import glob
+                qbs = glob.glob('%s/*.qemuboot.conf' %  deploy_dir_image)
                 if qbs:
-                    self.qemuboot = qbs.split()[0]
+                    self.qemuboot = qbs.split[0]
 
         if not os.path.exists(self.qemuboot):
             raise Exception("Failed to find <image>.qemuboot.conf!")
-- 
2.7.4

[PATCH 3/4] runqemu: assume artefacts are relative to *.qemuboot.conf

[Joshua Lock]

When runqemu is started with a *.qemuboot.conf arg assume that image
artefacts are relative to that file, rather than in whatever
directory the DEPLOY_DIR_IMAGE variable in the conf file points to.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/scripts/runqemu b/scripts/runqemu
index 1599848..cfe7bff 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -143,6 +143,7 @@ class BaseConfig(object):
         self.nfs_server = ''
         self.rootfs = ''
         self.qemuboot = ''
+        self.qbconfload = False
         self.kernel = ''
         self.kernel_cmdline = ''
         self.kernel_cmdline_script = ''
@@ -239,6 +240,7 @@ class BaseConfig(object):
         """
         if p.endswith('.qemuboot.conf'):
             self.qemuboot = p
+            self.qbconfload = True
         elif re.search('\.bin$', p) or re.search('bzImage', p) or \
              re.search('zImage', p) or re.search('vmlinux', p) or \
              re.search('fitImage', p) or re.search('uImage', p):
@@ -544,6 +546,15 @@ class BaseConfig(object):
             k_upper = k.upper()
             self.set(k_upper, v)
 
+        # When we're started with a *.qemuboot.conf arg assume that image
+        # artefacts are relative to that file, rather than in whatever
+        # directory DEPLOY_DIR_IMAGE in the conf file points to.
+        if self.qbconfload:
+            imgdir = os.path.dirname(self.qemuboot)
+            if imgdir != self.get('DEPLOY_DIR_IMAGE'):
+                logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir))
+                self.set('DEPLOY_DIR_IMAGE', imgdir)
+
     def print_config(self):
         logger.info('Continuing with the following parameters:\n')
         if not self.fstype in self.vmtypes:
-- 
2.7.4

[PATCH 4/4] runqemu: get NATIVE dirs from env when in source OE build dir

[Joshua Lock]

When we're running from a sourced OE build directory parse the
STAGING_* directories from the bitbake environment, rather than
using those hard-coded in the conf file.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/scripts/runqemu b/scripts/runqemu
index cfe7bff..5a97802 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -555,6 +555,16 @@ class BaseConfig(object):
                 logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir))
                 self.set('DEPLOY_DIR_IMAGE', imgdir)
 
+        # When we're running from a sourced OE environment use the STAGING_*
+        # directories from the environment.
+        if self.bitbake_e:
+            native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
+            for nv in native_vars:
+                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
+                if s and s.group(1) != self.get(nv):
+                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
+                    self.set(nv, s.group(1))
+
     def print_config(self):
         logger.info('Continuing with the following parameters:\n')
         if not self.fstype in self.vmtypes:
@@ -815,6 +825,17 @@ class BaseConfig(object):
             shutil.rmtree(self.nfs_dir)
             shutil.rmtree('%s.pseudo_state' % self.nfs_dir)
 
+    def check_sourced_env(self):
+        bitbake = shutil.which('bitbake')
+        if bitbake and not self.bitbake_e:
+            mach = self.get('MACHINE')
+            if mach:
+                cmd = 'MACHINE=%s bitbake -e' % mach
+            else:
+                cmd = 'bitbake -e'
+            logger.info('Running %s...' % cmd)
+            self.bitbake_e = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
+
 def main():
     if len(sys.argv) == 1 or "help" in sys.argv:
         print_usage()
@@ -826,6 +847,7 @@ def main():
         logger.error(esc)
         logger.error("Try 'runqemu help' on how to use it")
         return 1
+    config.check_sourced_env()
     config.read_qemuboot()
     config.check_and_set()
     config.print_config()
-- 
2.7.4

Re: [PATCH 4/4] runqemu: get NATIVE dirs from env when in source OE build dir

[Robert Yang]

Hi Josh,

Thanks for fixing this, I got errors when test this patch:

$ runqemu tmp/deploy/images/qemux86/
runqemu - INFO - Running bitbake -e...
Traceback (most recent call last):
   File "/buildarea/lyang1/poky/scripts/runqemu", line 864, in <module>
     ret = main()
   File "/buildarea/lyang1/poky/scripts/runqemu", line 851, in main
     config.read_qemuboot()
   File "/buildarea/lyang1/poky/scripts/runqemu", line 536, in read_qemuboot
     self.qemuboot = qbs.split[0]
AttributeError: 'list' object has no attribute 'split'

And after your patches, it always need run "bitbake -e", this makes
us can't use runqemu when another build is running:
ERROR: Only one copy of bitbake should be run against a build directory

Maybe we check STAGING_DIR_NATIVE, STAGING_BINDIR_NATIVE
and STAGING_DIR_HOST, if they are existed, then we don't need run bitbake -e.

And please see my comments below.

On 09/05/2016 09:59 PM, Joshua Lock wrote:
> When we're running from a sourced OE build directory parse the
> STAGING_* directories from the bitbake environment, rather than
> using those hard-coded in the conf file.
>
> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> ---
>  scripts/runqemu | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
>
> diff --git a/scripts/runqemu b/scripts/runqemu
> index cfe7bff..5a97802 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -555,6 +555,16 @@ class BaseConfig(object):
>                  logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir))
>                  self.set('DEPLOY_DIR_IMAGE', imgdir)
>
> +        # When we're running from a sourced OE environment use the STAGING_*
> +        # directories from the environment.
> +        if self.bitbake_e:
> +            native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
> +            for nv in native_vars:
> +                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
> +                if s and s.group(1) != self.get(nv):
> +                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
> +                    self.set(nv, s.group(1))
> +
>      def print_config(self):
>          logger.info('Continuing with the following parameters:\n')
>          if not self.fstype in self.vmtypes:
> @@ -815,6 +825,17 @@ class BaseConfig(object):
>              shutil.rmtree(self.nfs_dir)
>              shutil.rmtree('%s.pseudo_state' % self.nfs_dir)
>
> +    def check_sourced_env(self):
> +        bitbake = shutil.which('bitbake')
> +        if bitbake and not self.bitbake_e:
> +            mach = self.get('MACHINE')
> +            if mach:
> +                cmd = 'MACHINE=%s bitbake -e' % mach
> +            else:
> +                cmd = 'bitbake -e'
> +            logger.info('Running %s...' % cmd)
> +            self.bitbake_e = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')

I think that we need check whether the command is successed
or not, and print errors if it is failed to run.

// Robert

> +
>  def main():
>      if len(sys.argv) == 1 or "help" in sys.argv:
>          print_usage()
> @@ -826,6 +847,7 @@ def main():
>          logger.error(esc)
>          logger.error("Try 'runqemu help' on how to use it")
>          return 1
> +    config.check_sourced_env()
>      config.read_qemuboot()
>      config.check_and_set()
>      config.print_config()
>

Re: [PATCH 2/4] runqemu: use glob rather than calling out to ls

[Robert Yang]

On 09/05/2016 09:59 PM, Joshua Lock wrote:
> Use the Python glob module to find files matching a pattern, rather
> than calling out to ls and parsing the output.
>
> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> ---
>  scripts/runqemu | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/runqemu b/scripts/runqemu
> index 72c6352..1599848 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -528,11 +528,10 @@ class BaseConfig(object):
>                  self.qemuboot = "%s/%s-%s.qemuboot.conf" % (deploy_dir_image,
>                          self.rootfs, machine)
>              else:
> -                cmd = 'ls -t %s/*.qemuboot.conf' %  deploy_dir_image
> -                logger.info('Running %s...' % cmd)
> -                qbs = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
> +                import glob
> +                qbs = glob.glob('%s/*.qemuboot.conf' %  deploy_dir_image)

I'm not sure about this, "ls -lt" will sort by time and we will use the
latest one when the user run "runqemu core-image-minimal qemux86", but
glob.glob() seems can't do this.

I have to go to sleep now, have a good day, talk to you tomorrow.

// Robert

>                  if qbs:
> -                    self.qemuboot = qbs.split()[0]
> +                    self.qemuboot = qbs.split[0]
>
>          if not os.path.exists(self.qemuboot):
>              raise Exception("Failed to find <image>.qemuboot.conf!")
>

Re: [PATCH 4/4] runqemu: get NATIVE dirs from env when in source OE build dir

[Joshua Lock]

[-- Attachment #1: Type: text/plain, Size: 3539 bytes --]

On 5 September 2016 at 15:47, Robert Yang <liezhi.yang@windriver.com> wrote:

> Hi Josh,
>
> Thanks for fixing this, I got errors when test this patch:
>
> $ runqemu tmp/deploy/images/qemux86/
> runqemu - INFO - Running bitbake -e...
> Traceback (most recent call last):
>   File "/buildarea/lyang1/poky/scripts/runqemu", line 864, in <module>
>     ret = main()
>   File "/buildarea/lyang1/poky/scripts/runqemu", line 851, in main
>     config.read_qemuboot()
>   File "/buildarea/lyang1/poky/scripts/runqemu", line 536, in
> read_qemuboot
>     self.qemuboot = qbs.split[0]
> AttributeError: 'list' object has no attribute 'split'
>

Thanks for reviewing and catching this.


> And after your patches, it always need run "bitbake -e", this makes
> us can't use runqemu when another build is running:
> ERROR: Only one copy of bitbake should be run against a build directory
>
> Maybe we check STAGING_DIR_NATIVE, STAGING_BINDIR_NATIVE
> and STAGING_DIR_HOST, if they are existed, then we don't need run bitbake
> -e.
>

That's a nice idea, I'll implement that for v2 of the series.


> And please see my comments below.
>
>
> On 09/05/2016 09:59 PM, Joshua Lock wrote:
>
>> When we're running from a sourced OE build directory parse the
>> STAGING_* directories from the bitbake environment, rather than
>> using those hard-coded in the conf file.
>>
>> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
>> ---
>>  scripts/runqemu | 22 ++++++++++++++++++++++
>>  1 file changed, 22 insertions(+)
>>
>> diff --git a/scripts/runqemu b/scripts/runqemu
>> index cfe7bff..5a97802 100755
>> --- a/scripts/runqemu
>> +++ b/scripts/runqemu
>> @@ -555,6 +555,16 @@ class BaseConfig(object):
>>                  logger.info('Setting DEPLOY_DIR_IMAGE to folder
>> containing %s (%s)' % (self.qemuboot, imgdir))
>>                  self.set('DEPLOY_DIR_IMAGE', imgdir)
>>
>> +        # When we're running from a sourced OE environment use the
>> STAGING_*
>> +        # directories from the environment.
>> +        if self.bitbake_e:
>> +            native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
>> +            for nv in native_vars:
>> +                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
>> +                if s and s.group(1) != self.get(nv):
>> +                    logger.info('Overriding conf file setting of %s to
>> %s from Bitbake environment' % (nv, s.group(1)))
>> +                    self.set(nv, s.group(1))
>> +
>>      def print_config(self):
>>          logger.info('Continuing with the following parameters:\n')
>>          if not self.fstype in self.vmtypes:
>> @@ -815,6 +825,17 @@ class BaseConfig(object):
>>              shutil.rmtree(self.nfs_dir)
>>              shutil.rmtree('%s.pseudo_state' % self.nfs_dir)
>>
>> +    def check_sourced_env(self):
>> +        bitbake = shutil.which('bitbake')
>> +        if bitbake and not self.bitbake_e:
>> +            mach = self.get('MACHINE')
>> +            if mach:
>> +                cmd = 'MACHINE=%s bitbake -e' % mach
>> +            else:
>> +                cmd = 'bitbake -e'
>> +            logger.info('Running %s...' % cmd)
>> +            self.bitbake_e = subprocess.Popen(cmd, shell=True,
>> stdout=subprocess.PIPE).stdout.read().decode('utf-8')
>>
>
> I think that we need check whether the command is successed
> or not, and print errors if it is failed to run.
>

Sure, I'll implement that.

Joshua <joshua.g.lock@intel.com>

[-- Attachment #2: Type: text/html, Size: 5296 bytes --]

Re: [PATCH 2/4] runqemu: use glob rather than calling out to ls

[Joshua Lock]

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

On 5 September 2016 at 15:51, Robert Yang <liezhi.yang@windriver.com> wrote:

>
>
> On 09/05/2016 09:59 PM, Joshua Lock wrote:
>
>> Use the Python glob module to find files matching a pattern, rather
>> than calling out to ls and parsing the output.
>>
>> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
>> ---
>>  scripts/runqemu | 7 +++----
>>  1 file changed, 3 insertions(+), 4 deletions(-)
>>
>> diff --git a/scripts/runqemu b/scripts/runqemu
>> index 72c6352..1599848 100755
>> --- a/scripts/runqemu
>> +++ b/scripts/runqemu
>> @@ -528,11 +528,10 @@ class BaseConfig(object):
>>                  self.qemuboot = "%s/%s-%s.qemuboot.conf" %
>> (deploy_dir_image,
>>                          self.rootfs, machine)
>>              else:
>> -                cmd = 'ls -t %s/*.qemuboot.conf' %  deploy_dir_image
>> -                logger.info('Running %s...' % cmd)
>> -                qbs = subprocess.Popen(cmd, shell=True,
>> stdout=subprocess.PIPE).stdout.read().decode('utf-8')
>> +                import glob
>> +                qbs = glob.glob('%s/*.qemuboot.conf' %  deploy_dir_image)
>>
>
> I'm not sure about this, "ls -lt" will sort by time and we will use the
> latest one when the user run "runqemu core-image-minimal qemux86", but
> glob.glob() seems can't do this.
>

We could ensure the list returned by glob is sorted, whilst I prefer to
avoid calling out to the shell unnecessarily I'm not really attached to
this change and can drop it for v2 of the series.


> I have to go to sleep now, have a good day, talk to you tomorrow.
>

Thanks for the review!


> // Robert
>
>
>                  if qbs:
>> -                    self.qemuboot = qbs.split()[0]
>> +                    self.qemuboot = qbs.split[0]
>>
>>          if not os.path.exists(self.qemuboot):
>>              raise Exception("Failed to find <image>.qemuboot.conf!")
>>
>>


-- 
Joshua Lock <joshua.g.lock@intel.com>

[-- Attachment #2: Type: text/html, Size: 3630 bytes --]