[yocto-autobuilder][PATCH] ScrapeTargets.py: improve target search algorithm

[yocto-autobuilder][PATCH] ScrapeTargets.py: improve target search algorithm

[Stephano Cetola]

When scraping the build targets from refkit-ci.inc, grep would only
return the first line of the search result. By replacing grep with
awk, we are now searching for multi-line variables, and should now
scrape build targets even if they have line breaks between them.

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
---
 .../site-packages/autobuilder/buildsteps/ScrapeTargets.py        | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py b/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
index 80153fe85..c5a2f8d6d 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
@@ -41,7 +41,9 @@ class ScrapeTargets(ShellCommand):
         workerdir = os.path.join(os.path.join(YOCTO_ABBASE, "yocto-worker"))
         buildername = self.getProperty("buildername")
         src = os.path.join(workerdir, buildername, "build", self.source)
-        self.command = "cat " + src + " | grep " + self.targetsvar
+        # find targetsvar=", skip a line, then return lines up to a quote
+        self.command = ["awk",
+                '/%s="/{flag=1;next}/"/{flag=0}flag' % self.targetsvar, src]
         ShellCommand.start(self)
 
     def commandComplete(self, cmd):
@@ -49,10 +51,7 @@ class ScrapeTargets(ShellCommand):
             return
 
         result = cmd.logs['stdio'].getText()
-        targets = result.replace(self.targetsvar, "")
-        targets = targets.strip()
-        targets = targets.replace('=', '')
-        targets = targets.strip('"')
+        targets = result.strip()
         self.setProperty("scraped_targets",
                          targets,
                          'Targets "%s" scraped from %s' % (targets,
-- 
2.13.1

Re: [yocto-autobuilder][PATCH] ScrapeTargets.py: improve target search algorithm

[Joshua Lock]

On Sat, 2017-07-01 at 15:05 -0700, Stephano Cetola wrote:
> When scraping the build targets from refkit-ci.inc, grep would only
> return the first line of the search result. By replacing grep with
> awk, we are now searching for multi-line variables, and should now
> scrape build targets even if they have line breaks between them.
> 
> Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
> ---
>  .../site-packages/autobuilder/buildsteps/ScrapeTargets.py        | 9
> ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/python2.7/site-
> packages/autobuilder/buildsteps/ScrapeTargets.py
> b/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
> index 80153fe85..c5a2f8d6d 100644
> --- a/lib/python2.7/site-
> packages/autobuilder/buildsteps/ScrapeTargets.py
> +++ b/lib/python2.7/site-
> packages/autobuilder/buildsteps/ScrapeTargets.py
> @@ -41,7 +41,9 @@ class ScrapeTargets(ShellCommand):
>          workerdir = os.path.join(os.path.join(YOCTO_ABBASE, "yocto-
> worker"))
>          buildername = self.getProperty("buildername")
>          src = os.path.join(workerdir, buildername, "build",
> self.source)
> -        self.command = "cat " + src + " | grep " + self.targetsvar
> +        # find targetsvar=", skip a line, then return lines up to a
> quote

What if the first line includes a desired target? Can we change this
not to skip the first line? 

Does including the line continuation character in the property cause
BuildImages to choke later?

> +        self.command = ["awk",
> +                '/%s="/{flag=1;next}/"/{flag=0}flag' %
> self.targetsvar, src]
>          ShellCommand.start(self)
>  
>      def commandComplete(self, cmd):
> @@ -49,10 +51,7 @@ class ScrapeTargets(ShellCommand):
>              return
>  
>          result = cmd.logs['stdio'].getText()
> -        targets = result.replace(self.targetsvar, "")
> -        targets = targets.strip()
> -        targets = targets.replace('=', '')
> -        targets = targets.strip('"')
> +        targets = result.strip()
>          self.setProperty("scraped_targets",
>                           targets,
>                           'Targets "%s" scraped from %s' % (targets,
> -- 
> 2.13.1
>