[PATCH 1/4] runtime/context: qemu do not need ip and server ip inputs from external

[PATCH 1/4] runtime/context: qemu do not need ip and server ip inputs from external

[Yeoh Ee Peng]

Qemu do not use the ip and server ip inputs from external. It will
assign ip and server ip after initialize qemu.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/lib/oeqa/runtime/context.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/runtime/context.py b/meta/lib/oeqa/runtime/context.py
index a7f3823..fcb8de4 100644
--- a/meta/lib/oeqa/runtime/context.py
+++ b/meta/lib/oeqa/runtime/context.py
@@ -101,7 +101,7 @@ class OERuntimeTestContextExecutor(OETestContextExecutor):
         if target_type == 'simpleremote':
             target = OESSHTarget(logger, target_ip, server_ip, **kwargs)
         elif target_type == 'qemu':
-            target = OEQemuTarget(logger, target_ip, server_ip, **kwargs)
+            target = OEQemuTarget(logger, None, None, **kwargs)
         else:
             # XXX: This code uses the old naming convention for controllers and
             # targets, the idea it is to leave just targets as the controller
-- 
2.7.4

[PATCH 2/4] qemurunner: Add support for slirp

[Yeoh Ee Peng]

Enable qemurunner for slirp. Retrieved the port from localhost
to connect to qemu.

[YOCTO#10713]

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index dfab1bd..c29f6d8 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -28,7 +28,8 @@ re_control_char = re.compile('[%s]' % re.escape("".join(control_chars)))
 
 class QemuRunner:
 
-    def __init__(self, machine, rootfs, display, tmpdir, deploy_dir_image, logfile, boottime, dump_dir, dump_host_cmds, use_kvm, logger):
+    def __init__(self, machine, rootfs, display, tmpdir, deploy_dir_image, logfile, boottime, dump_dir, dump_host_cmds,
+                 use_kvm, use_slirp, logger):
 
         # Popen object for runqemu
         self.runqemu = None
@@ -51,6 +52,7 @@ class QemuRunner:
         self.logged = False
         self.thread = None
         self.use_kvm = use_kvm
+        self.use_slirp = use_slirp
         self.msg = ''
 
         self.runqemutime = 120
@@ -129,6 +131,8 @@ class QemuRunner:
                 self.logger.debug('Not using kvm for runqemu')
             if not self.display:
                 launch_cmd += ' nographic'
+            if self.use_slirp:
+                launch_cmd += ' slirp'
             launch_cmd += ' %s %s' % (self.machine, self.rootfs)
 
         return self.launch(launch_cmd, qemuparams=qemuparams, get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
@@ -238,9 +242,14 @@ class QemuRunner:
                 # because is possible to have control characters
                 cmdline = re_control_char.sub(' ', cmdline)
             try:
-                ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
-                self.ip = ips[0]
-                self.server_ip = ips[1]
+                if self.use_slirp:
+                    tcp_ports = cmdline.split("hostfwd=tcp::")[1]
+                    host_port = tcp_ports[:tcp_ports.find('-')]
+                    self.ip = "localhost:%s" % host_port
+                else:
+                    ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
+                    self.ip = ips[0]
+                    self.server_ip = ips[1]
                 self.logger.debug("qemu cmdline used:\n{}".format(cmdline))
             except (IndexError, ValueError):
                 # Try to get network configuration from runqemu output
-- 
2.7.4

[PATCH 3/4] oeqa/qemu: Add support for slirp

[Yeoh Ee Peng]

Enable qemu for slirp. Initialize Qemurunner with slirp. Setup
ip and port attribute to enable connection with qemu running
with slirp.

[YOCTO#10713]

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/lib/oeqa/core/target/qemu.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/core/target/qemu.py b/meta/lib/oeqa/core/target/qemu.py
index bf3b633..95f14c8 100644
--- a/meta/lib/oeqa/core/target/qemu.py
+++ b/meta/lib/oeqa/core/target/qemu.py
@@ -13,7 +13,7 @@ supported_fstypes = ['ext3', 'ext4', 'cpio.gz', 'wic']
 
 class OEQemuTarget(OESSHTarget):
     def __init__(self, logger, ip, server_ip, timeout=300, user='root',
-            port=None, machine='', rootfs='', kernel='', kvm=False,
+            port=None, machine='', rootfs='', kernel='', kvm=False, slirp=False,
             dump_dir='', dump_host_cmds='', display='', bootlog='',
             tmpdir='', dir_image='', boottime=60, **kwargs):
 
@@ -26,16 +26,25 @@ class OEQemuTarget(OESSHTarget):
         self.rootfs = rootfs
         self.kernel = kernel
         self.kvm = kvm
+        self.slirp = slirp
 
         self.runner = QemuRunner(machine=machine, rootfs=rootfs, tmpdir=tmpdir,
                                  deploy_dir_image=dir_image, display=display,
                                  logfile=bootlog, boottime=boottime,
-                                 use_kvm=kvm, dump_dir=dump_dir,
+                                 use_kvm=kvm, use_slirp=slirp, dump_dir=dump_dir,
                                  dump_host_cmds=dump_host_cmds, logger=logger)
 
     def start(self, params=None, extra_bootparams=None):
         if self.runner.start(params, extra_bootparams=extra_bootparams):
             self.ip = self.runner.ip
+            if self.slirp:
+                target_ip_port = self.runner.ip.split(':')
+                if len(target_ip_port) == 2:
+                    target_ip = target_ip_port[0]
+                    port = target_ip_port[1]
+                    self.ip = target_ip
+                    self.ssh = self.ssh + ['-p', port]
+                    self.scp = self.scp + ['-P', port]
             self.server_ip = self.runner.server_ip
         else:
             self.stop()
-- 
2.7.4

[PATCH 4/4] testimage: Add support for slirp

[Yeoh Ee Peng]

Enable testimage to support qemu slirp. Configure
"QEMU_USE_SLIRP" variable to enable slirp.

[YOCTO#10713]

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/classes/testimage.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index ba2d9c4..2f6a21b 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -256,6 +256,10 @@ def testimage_main(d):
     else:
         kvm = False
 
+    slirp = False
+    if d.getVar("QEMU_USE_SLIRP"):
+        slirp = True
+
     # TODO: We use the current implementatin of qemu runner because of
     # time constrains, qemu runner really needs a refactor too.
     target_kwargs = { 'machine'     : machine,
@@ -267,6 +271,7 @@ def testimage_main(d):
                       'boottime'    : boottime,
                       'bootlog'     : bootlog,
                       'kvm'         : kvm,
+                      'slirp'       : slirp,
                     }
 
     # TODO: Currently BBPATH is needed for custom loading of targets.
-- 
2.7.4

Re: [PATCH 2/4] qemurunner: Add support for slirp

[Richard Purdie]

On Tue, 2018-11-13 at 17:32 +0800, Yeoh Ee Peng wrote:
> Enable qemurunner for slirp. Retrieved the port from localhost
> to connect to qemu.
> 
> [YOCTO#10713]
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
> ---
>  meta/lib/oeqa/utils/qemurunner.py | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)

  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/devtool.py", line 1282, in test_devtool_deploy_target
    with runqemu(testimage) as qemu:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/media/build1/poky/meta/lib/oeqa/utils/commands.py", line 327, in runqemu
    qemu = oeqa.targetcontrol.QemuTarget(recipedata, targetlogger, image_fstype)
  File "/media/build1/poky/meta/lib/oeqa/targetcontrol.py", line 148, in __init__
    logger = logger)
TypeError: __init__() missing 1 required positional argument: 'use_slirp'

I'm therefore not sure how much testing this has had?

Cheers,

Richard

Re: [PATCH 2/4] qemurunner: Add support for slirp

[Yeoh, Ee Peng]

Hi Richard,

I performed testing mainly for ping, ssh, xorg testing. 
Did not expect that targetcontrol has dependency on the qemu. 
Let me look into this and run full runtime tests instead. 
INHERIT += "testimage"
TEST_SUITES = "ping ssh xorg"

Thanks,
Yeoh Ee Peng 

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: Wednesday, November 14, 2018 7:34 AM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 2/4] qemurunner: Add support for slirp

On Tue, 2018-11-13 at 17:32 +0800, Yeoh Ee Peng wrote:
> Enable qemurunner for slirp. Retrieved the port from localhost to 
> connect to qemu.
> 
> [YOCTO#10713]
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
> ---
>  meta/lib/oeqa/utils/qemurunner.py | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)

  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/devtool.py", line 1282, in test_devtool_deploy_target
    with runqemu(testimage) as qemu:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/media/build1/poky/meta/lib/oeqa/utils/commands.py", line 327, in runqemu
    qemu = oeqa.targetcontrol.QemuTarget(recipedata, targetlogger, image_fstype)
  File "/media/build1/poky/meta/lib/oeqa/targetcontrol.py", line 148, in __init__
    logger = logger)
TypeError: __init__() missing 1 required positional argument: 'use_slirp'

I'm therefore not sure how much testing this has had?

Cheers,

Richard

Re: [PATCH 2/4] qemurunner: Add support for slirp

[Yeoh, Ee Peng]

Let me expand the testing to selftest & runtime test as well.
I will perform a tracing on the dependency to decide if this need more testing. 

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: Wednesday, November 14, 2018 7:34 AM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 2/4] qemurunner: Add support for slirp

On Tue, 2018-11-13 at 17:32 +0800, Yeoh Ee Peng wrote:
> Enable qemurunner for slirp. Retrieved the port from localhost to 
> connect to qemu.
> 
> [YOCTO#10713]
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
> ---
>  meta/lib/oeqa/utils/qemurunner.py | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)

  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/devtool.py", line 1282, in test_devtool_deploy_target
    with runqemu(testimage) as qemu:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/media/build1/poky/meta/lib/oeqa/utils/commands.py", line 327, in runqemu
    qemu = oeqa.targetcontrol.QemuTarget(recipedata, targetlogger, image_fstype)
  File "/media/build1/poky/meta/lib/oeqa/targetcontrol.py", line 148, in __init__
    logger = logger)
TypeError: __init__() missing 1 required positional argument: 'use_slirp'

I'm therefore not sure how much testing this has had?

Cheers,

Richard

Re: [PATCH 2/4] qemurunner: Add support for slirp

[Yeoh, Ee Peng]

Hi Richard,

Thank you for sharing on the error related to oeqa.targetcontrol.QemuTarget.
After some analysis, here are my findings:

1. OEQA had classes with very similar name as well as interface for carry out qemu and ssh operations.  Example: "QemuTarget vs OEQemuTarget", "SSHControl vs OESSHTarget".  Where OEQemuTarget & OESSHTarget were used mainly for runtime/testimage and QemuTarget & SSHControl were used for mainly for oe-selftest.  Not only the name and interface were similar, there are some duplicated codes between these two group of classes.  This had introduced additional complexity in both adding new features and testing (eg. now the "slirp" features need to be added to both group of classes and testing both). 

2. Coding design being used were different between these two group of classes. Where OEQemuTarget and OESSHTarget used inheritance design, QemuTarget and SSHControl used composite design.  This further adding more complexity into developing new feature.

Solutions:
1. Short term - I will enable slirp into both group of classes and test them
2. Long term - this require significant refactoring to remove duplication codes and simplify the code base for qemu and ssh operations. 

Please let me know you inputs and recommendation.

Best regards,
Yeoh Ee Peng  

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: Wednesday, November 14, 2018 7:34 AM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 2/4] qemurunner: Add support for slirp

On Tue, 2018-11-13 at 17:32 +0800, Yeoh Ee Peng wrote:
> Enable qemurunner for slirp. Retrieved the port from localhost to 
> connect to qemu.
> 
> [YOCTO#10713]
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
> ---
>  meta/lib/oeqa/utils/qemurunner.py | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)

  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/devtool.py", line 1282, in test_devtool_deploy_target
    with runqemu(testimage) as qemu:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/media/build1/poky/meta/lib/oeqa/utils/commands.py", line 327, in runqemu
    qemu = oeqa.targetcontrol.QemuTarget(recipedata, targetlogger, image_fstype)
  File "/media/build1/poky/meta/lib/oeqa/targetcontrol.py", line 148, in __init__
    logger = logger)
TypeError: __init__() missing 1 required positional argument: 'use_slirp'

I'm therefore not sure how much testing this has had?

Cheers,

Richard