[yocto-autobuilder][PATCH 1/5] buildsteps/BuildImages: Setproperty BUILDIMAGES

[yocto-autobuilder][PATCH 1/5] buildsteps/BuildImages: Setproperty BUILDIMAGES

[Aníbal Limón]

In certain situations is need to know what images was built for
example the DAFT executes flashing over certain image.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
index 30f7a18..7262fd3 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
@@ -146,6 +146,9 @@ class BuildImages(BitbakeShellCommand):
                 if self.minnowExists is None or self.minnowExists == "True":
                     self.command = ". ./oe-init-build-env; bitbake " + bitbakeflags + self.images
                     self.description = ["Building " + str(self.images)]
+        
+        self.setProperty("BUILDIMAGES", self.images)
+
         ShellCommand.start(self)
 
     def describe(self, done=False):
-- 
2.1.4

[yocto-autobuilder][PATCH 2/5] buildsteps/RunSanityTests: Add support for simpleremote target

[Aníbal Limón]

The testimage class supports a simpleremote target specifying the
server and target ip to run tests over SSH.

The main usage of this will in conjunction with the DAFT to make
tests on real HW.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../site-packages/autobuilder/buildsteps/RunSanityTests.py        | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py b/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
index 6fa03bc..0022cb9 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
@@ -60,6 +60,14 @@ class RunSanityTests(BitbakeShellCommand):
         elif layerversion > 1:
             self.command = self.command + "echo 'INHERIT += \"testimage\"' >> ./conf/auto.conf;"
             self.command = self.command + "echo 'TEST_QEMUBOOT_TIMEOUT = \"1500\"' >> ./conf/auto.conf;"
+
+            target_ip = self.getProperty('target_ip')
+            server_ip = self.getProperty('server_ip')
+            if target_ip and server_ip:
+                self.command = self.command +  "echo 'TEST_TARGET=\"simpleremote\"' >> ./conf/auto.conf;"
+                self.command = self.command +  "echo 'TEST_TARGET_IP=\"%s\"' >> ./conf/auto.conf;" % target_ip
+                self.command = self.command +  "echo 'TEST_SERVER_IP=\"%s\"' >> ./conf/auto.conf;" % server_ip
+
             if self.suites:
                 if layerversion > 3:
                     self.command = self.command + "echo 'TEST_SUITES = \"" + self.suites + "\"'" + " >> ./conf/auto.conf;"
-- 
2.1.4

[yocto-autobuilder][PATCH 3/5] autobuilder/buildsteps: Add DaftGetDevices step

[Aníbal Limón]

This buildstep will copy the devices configuration from the
worker and read it, in order to get daft ip an ssh port, to
use later in RunSanityTests

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
---
 config/autobuilder.conf.example                    |   3 +
 .../autobuilder/buildsteps/DaftGetDevices.py       |  44 ++++++++
 lib/python2.7/site-packages/autobuilder/config.py  |   1 +
 .../site-packages/autobuilder/lib/daft.py          | 115 +++++++++++++++++++++
 4 files changed, 163 insertions(+)
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py
 create mode 100644 lib/python2.7/site-packages/autobuilder/lib/daft.py

diff --git a/config/autobuilder.conf.example b/config/autobuilder.conf.example
index 2ee11e6..e5ec16b 100644
--- a/config/autobuilder.conf.example
+++ b/config/autobuilder.conf.example
@@ -95,3 +95,6 @@ PERFORMANCE_MAIL_TO = "root@localhost otherperson@localhost"
 PERFORMANCE_MAIL_CC = "buildcc@localhost"
 PERFORMANCE_MAIL_BCC = "buildbcc@localhost"
 PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
+
+[Daft]
+DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py
new file mode 100644
index 0000000..133a4d6
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py
@@ -0,0 +1,44 @@
+import os
+from buildbot.steps.transfer import FileUpload
+from buildbot.process.buildstep import BuildStep
+
+from lib.daft import DeployScanner
+from autobuilder.config import DAFT_WORKER_DEVICES_CFG
+
+class DaftGetDevices(FileUpload):
+    haltOnFailure = True
+
+    name = "DaftGetDevices"
+
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.tests = None
+        self.factory = factory
+        for k, v in argdict.iteritems():
+            setattr(self, k, v)
+        self.description = "Getting devices configuration"
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+
+        super(DaftGetDevices, self).__init__(DAFT_WORKER_DEVICES_CFG,
+                os.path.join('/tmp', 'devices.cfg'))
+
+    def finished(self, result):
+        if self.cmd:
+            ds = DeployScanner(devsconf_file = self.masterdest)
+            devices = ds()
+
+            found = False
+            dut_label = self.getProperty('custom_dut')
+            for d in devices:
+                if d['dut_label'] == dut_label:
+                    self.setProperty('dut_name', d['dut_label'], 'DaftGetDevices')
+                    self.setProperty('server_ip', d['server_address'], 'DaftGetDevices')
+                    target_ip = "%s:%s" % (d['ctrl_address'], d['dut_sshport'])
+                    self.setProperty('target_ip', target_ip, 'DaftGetDevices')
+
+                    found = True
+
+            if not found:
+                 return BuildStep.finished(self, FAILURE)
+
+        return super(DaftGetDevices, self).finished(result)
diff --git a/lib/python2.7/site-packages/autobuilder/config.py b/lib/python2.7/site-packages/autobuilder/config.py
index 9d945b1..5bcf6c6 100644
--- a/lib/python2.7/site-packages/autobuilder/config.py
+++ b/lib/python2.7/site-packages/autobuilder/config.py
@@ -23,3 +23,4 @@ RPM_PUBLISH_DIR = os.environ.get("RPM_PUBLISH_DIR")
 IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
 DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
 PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
+DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
diff --git a/lib/python2.7/site-packages/autobuilder/lib/daft.py b/lib/python2.7/site-packages/autobuilder/lib/daft.py
new file mode 100644
index 0000000..eef4bab
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/lib/daft.py
@@ -0,0 +1,115 @@
+import os
+import ConfigParser as configparser
+
+class DeployScanner(object):
+    '''
+    In charge of scanning deployed daft bbb devices
+    '''
+
+    __MAGIC_SERVER_ADDRESS = '192.168.30.1' # DAFT uses this address for this internal network
+    __MAGIC_SSH_PORT = 2233 # it is hardcoded as per DAFT implementation manual
+
+    def  __init__(self, *args, **kwargs):
+        self.devsconf_fp = kwargs.get('devsconf_file', None)
+        if not self.devsconf_fp:
+            raise Exception('not fed devsconf file')
+        self.ign_leases = kwargs.get('ign_leases', True)
+        if not self.ign_leases:
+            self.leases_file_path =  kwargs.get('leases_file', None)
+            if not self.leases_file_path:
+                raise Exception('not fed leases file')
+
+    def __call__(self):
+        '''
+        Creates relation of deployed devices
+        Returns:
+            List of dictionaries containing info about devices deployed.
+        '''
+        def create_relation(i,j):
+            r = []
+            for conf in i:
+                for active in j:
+                    if conf['bb_ip'] == active['ip']:
+                        r.append({
+                            'dut_label': conf['device'].lower(),
+                            'dut_family': conf['device_type'].lower(),
+                            'dut_sshport': str(self.__MAGIC_SSH_PORT),
+                            'ctrl_address': conf['bb_ip'],
+                            'server_address': self.__MAGIC_SERVER_ADDRESS
+                        })
+            return r
+
+        def slack_relation(i):
+            r = []
+            for conf in i:
+                r.append({
+                    'dut_label': conf['device'],
+                    'dut_family': conf['device_type'],
+                    'dut_sshport': str(self.__MAGIC_SSH_PORT),
+                    'ctrl_address': conf['bb_ip'],
+                    'server_address': self.__MAGIC_SERVER_ADDRESS
+                })
+            return r
+
+        fc = self.__fetch_confs()
+        if not fc:
+            raise Exception('There are no configurations as per BBB devices')
+
+        if self.ign_leases:
+            # Devices that nobody knows if were deployed
+            return slack_relation(fc)
+
+        als = self.__active_leases()
+        if not als:
+            raise Exception('DHCP server has not registered any host yet')
+
+        return create_relation(fc, als)
+
+    def __fetch_confs(self):
+        '''
+        Read and parse BBB configuration file and return result as dictionary
+        '''
+        config = configparser.SafeConfigParser()
+        config.read(self.devsconf_fp)
+        configurations = []
+        for device in config.sections():
+            device_config = dict(config.items(device))
+            device_config["device"] = device
+            device_config["device_type"] = device.rstrip('1234567890_')
+            configurations.append(device_config)
+        return configurations
+
+    def __active_leases(self):
+        """
+        Read the active leases from dnsmasq leases file and return a list of
+        active leases as dictionaries.
+        Args:
+            file_name (str): Path to leases file, e.g. /path/to/file/dnsmasq.leases
+        Returns:
+            List of dictionaries containing the active leases.
+            The dictionaries have the following format:
+            {
+                "mac": "device_mac_address",
+                "ip": "device_ip_address",
+                "hostname": "device_host_name",
+                "client_id": "client_id_or_*_if_unset"
+            }
+        """
+        with open(self.leases_file_path) as lease_file:
+            leases = lease_file.readlines()
+
+        leases_list = []
+
+        # dnsmasq.leases contains rows with the following format:
+        # <lease_expiry_time_as_epoch_format> <mac> <ip> <hostname> <domain>
+        # See:
+        #http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2005q1/000143.html
+        for lease in leases:
+            lease = lease.split()
+            leases_list.append({
+                "mac": lease[1],
+                "ip": lease[2],
+                "hostname": lease[3],
+                "client_id": lease[4],
+            })
+        return leases_list
-- 
2.1.4

[yocto-autobuilder][PATCH 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep

[Aníbal Limón]

This will execute daft flash cycle without reboot and prepare the dut
for sanity tests.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
---
 config/autobuilder.conf.example                    |  1 +
 .../autobuilder/buildsteps/DaftFlash.py            | 42 ++++++++++++++++++++++
 lib/python2.7/site-packages/autobuilder/config.py  |  1 +
 3 files changed, 44 insertions(+)
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py

diff --git a/config/autobuilder.conf.example b/config/autobuilder.conf.example
index e5ec16b..2e992cf 100644
--- a/config/autobuilder.conf.example
+++ b/config/autobuilder.conf.example
@@ -98,3 +98,4 @@ PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
 
 [Daft]
 DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
+DAFT_WORKER_WORKSPACE = "/home/ab/workspace"
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
new file mode 100644
index 0000000..c16cb9e
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
@@ -0,0 +1,42 @@
+import os
+
+from autobuilder.config import DAFT_WORKER_WORKSPACE_DIR
+from buildbot.steps.shell import ShellCommand
+
+class DaftFlash(ShellCommand):
+    haltOnFailure = True
+
+    name = "DaftFlash"
+
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.tests = None
+        self.factory = factory
+        for k, v in argdict.iteritems():
+            setattr(self, k, v)
+        self.description = "DUT Flashing"
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+        ShellCommand.__init__(self, **kwargs)
+
+    def start(self):
+        workspace_dir = DAFT_WORKER_WORKSPACE_DIR
+
+        dut_name = self.getProperty('dut_name')
+        workdir = self.getProperty('workdir')
+        image = self.getProperty('BUILDIMAGES')
+        machine = self.getProperty('MACHINE')
+
+        image_path = os.path.join(workdir, 'build', 'build', 'tmp', 'deploy',
+                'images', machine, '%s-%s.hddimg' % (image, machine))
+        image_name = os.path.basename(image_path)
+
+        # XXX: DAFT needs to have the image in a shared workspace 
+        self.command = "cp %s %s; " % (image_path, workspace_dir)
+
+        self.command += "cd %s; " % (workspace_dir)
+
+        # XXX: DAFT needs to be executed by root user
+        daft_cmd = "daft --setout %s %s" % (dut_name, image_name)
+        self.command += "sudo su -l -c 'cd {} && {}'; ".format(workspace_dir, daft_cmd)
+
+        ShellCommand.start(self)
diff --git a/lib/python2.7/site-packages/autobuilder/config.py b/lib/python2.7/site-packages/autobuilder/config.py
index 5bcf6c6..fcc08f0 100644
--- a/lib/python2.7/site-packages/autobuilder/config.py
+++ b/lib/python2.7/site-packages/autobuilder/config.py
@@ -24,3 +24,4 @@ IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
 DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
 PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
 DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
+DAFT_WORKER_WORKSPACE = os.environ.get("DAFT_WORKER_WORKSPACE")
-- 
2.1.4

[yocto-autobuilder][PATCH 5/5] buildset-config.yocto-qa: Add intel-corei7-64-daft buildset

[Aníbal Limón]

This buildset is for support running sanity tests over a DUT like
Minnowboard for the moment only builds core-image-sato, there is a need
to make it configurable.

Also the listo of the available devices are fixed into the UI as
parameters.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 buildset-config.yocto-qa/intel-corei7-64-daft.conf | 59 ++++++++++++++++++++++
 1 file changed, 59 insertions(+)
 create mode 100644 buildset-config.yocto-qa/intel-corei7-64-daft.conf

diff --git a/buildset-config.yocto-qa/intel-corei7-64-daft.conf b/buildset-config.yocto-qa/intel-corei7-64-daft.conf
new file mode 100644
index 0000000..cb3bcda
--- /dev/null
+++ b/buildset-config.yocto-qa/intel-corei7-64-daft.conf
@@ -0,0 +1,59 @@
+[intel-corei7-64-daft]
+builders: 'example-worker'
+repos: [{'poky':
+            {'repourl':'git://git.yoctoproject.org/poky',
+             'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
+             'branch':'master'}},
+        {'meta-intel':
+            {'repourl':'git://git.yoctoproject.org/meta-intel',
+             'layerversion':{'intel':'meta-intel'},
+             'branch':'master'}}]
+props: [{'kmeta':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'kmeta',
+                       'default': '',
+                       'label':'<hr><h3> branch for KMETA_${MACHINE} (default is blank):</h3>'}},
+        {'srcrev_meta':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'srcrev_meta',
+                       'default': '${AUTOREV}',
+                       'label':'<hr><h3> SRCREV for KMETA_${MACHINE} (default is ${AUTOREV}):<br></h3>'}},
+        {'srcuri_meta':{'prop_type':'StringParameter',
+                       'size': 50,
+                       'name': 'srcuri_meta',
+                       'default': '',
+                       'label':'<hr><h3> SRC_URI_pn-linux-yocto KMETA append. (default is blank).<br> This should be constructed as *just* the git uri portion of the string. <br>(git://git.yoctoproject.org/linux-yocto-3.19.git</h3>'}},
+        {'kbranch':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'kbranch',
+                       'default': '',
+                       'label':'<hr><h3> branch for KBRANCH_${MACHINE} (default is blank):</h3>'}},
+        {'srcrev_machine':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'srcrev_machine',
+                       'default': '${AUTOREV}',
+                       'label':'<hr><h3> SRCREV for KBRANCH. (default is ${AUTOREV}):<br></h3>'}},
+        {'srcuri_machine':{'prop_type':'StringParameter',
+                       'size': 50,
+                       'name': 'srcuri_machine',
+                       'default': '',
+                       'label':'<hr><h3> SRC_URI_pn-linux-yocto KBRANCH append. (default is blank).<br> This should be constructed as *just* the git uri portion of the string. <br>(git://git.yoctoproject.org/linux-yocto-3.19.git</h3>'}},
+	{'dut':{'prop_type':'ChoiceStringParameter',
+		   'choices': ['Minnowboard1'],
+		   'name': 'dut',
+		   'label':'<hr><h3>Selects DUT for flashing.<br></h3>'}}]
+steps: [{'SetDest':{}},
+        {'CheckOutLayers': {}},
+        {'RunPreamble': {}},
+        {'CheckBSPExists': {}},
+        {'GetDistroVersion' : {'distro': 'poky'}},
+        {'CreateAutoConf': {'machine': 'intel-corei7-64', 'SDKMACHINE' : 'x86_64', 
+                            'distro': 'poky'}},
+        {'CreateBBLayersConf': {'buildprovider' : 'yocto',
+                                'bsplayer': True, 'bspprovider': 'intel',
+                                'layerdirs': ['meta-intel', 'meta-intel/meta-tlk']}},
+        {'AddKernelProps': {}},
+        {'DaftGetDevices': {}},
+        {'BuildImages': {'images': 'core-image-sato'}},
+        {'DaftFlash':{}},
+        {'RunSanityTests': {'images': 'core-image-sato'}},]
-- 
2.1.4