All of lore.kernel.org
 help / color / mirror / Atom feed
* [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script
@ 2020-11-18  7:05 Srivatsan S
  2020-11-18 20:54 ` Bird, Tim
  0 siblings, 1 reply; 4+ messages in thread
From: Srivatsan S @ 2020-11-18  7:05 UTC (permalink / raw)
  To: Bird, Tim, tbird20; +Cc: fuego

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]

Tim,

While we were looking at eliminating the usage of jenkins-cli.jar in the
ftc script, we came across create_job() API in the Jenkins Python module
which can be used to create jobs in Jenkins, thus eliminating the usage of
jenkins-cli.jar in the ftc script.

The below patch works for job creation in Jenkins through ftc script. We
shall get back to you on ways to programmatically update other job
properties.

git diff ftc

diff --git a/scripts/ftc b/scripts/ftc
index 30195fb..a193c4d 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec}
\\
     job_name = board + "." + test.spec + "." + test.name
     print("Creating job " + job_name)
     try:
-        rcode = subprocess.call('java -jar
/var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
create-job ' +
-            job_name + ' < ' + tmp, shell=True)
-        os.unlink(tmp)
+        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
         return rcode
     except Exception as e:
         print("Job already exists")
@@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan,
plan_tests):
     print("Creating batch job ")
     try:
         job_name = board+'.'+testplan+'.batch'
-        rcode = subprocess.call('java -jar
/var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
create-job ' +
-             job_name + '< ' + tmp, shell=True)
-        os.unlink(tmp)
+        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
         return rcode
     except Exception as e:
         print("Job already exists")

Thanks,
Srivatsan


~

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #2: Type: text/html, Size: 3055 bytes --]

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script
  2020-11-18  7:05 [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script Srivatsan S
@ 2020-11-18 20:54 ` Bird, Tim
  2020-11-19  3:07   ` Srivatsan S
  0 siblings, 1 reply; 4+ messages in thread
From: Bird, Tim @ 2020-11-18 20:54 UTC (permalink / raw)
  To: Srivatsan S, tbird20; +Cc: fuego



> -----Original Message-----
> From: Srivatsan S <srivatsan.s@pathpartnertech.com>
> 
> Tim,
> 
> While we were looking at eliminating the usage of jenkins-cli.jar in the ftc script, we came across create_job() API in the Jenkins Python
> module which can be used to create jobs in Jenkins, thus eliminating the usage of jenkins-cli.jar in the ftc script.
> 
> The below patch works for job creation in Jenkins through ftc script. We shall get back to you on ways to programmatically update other
> job properties.

Thanks for sending this.  It helped me work the issues with this change.
There were a few problems with the implementation.  Did you test this?
It doesn’t seem like it would have worked, since it would have created
jobs with no shell command data.

I ended up using the basic concept, but handling other aspects required for
this change that you-all missed.

> 
> git diff ftc
> 
> diff --git a/scripts/ftc b/scripts/ftc
> index 30195fb..a193c4d 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
>      job_name = board + "." + test.spec + "." + test.name <http://test.name>
>      print("Creating job " + job_name)
>      try:
> -        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> -            job_name + ' < ' + tmp, shell=True)

In this case, the tmp file contains the data for config.xml for the new job.  The code above this needed
to be refactored to submit the xml data as a string rather than piping in to jenkins-cli.jar from a file.

> -        os.unlink(tmp)
> +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
This EMPTY_CONFIG_XML will end up creating a stub job, with no shell command
to invoke the appropriate Fuego test.

>          return rcode
server.create_job returns "None" on success.  The caller can only handle a numeric
return result (it does a bitwise |, in the case of creating multiple jobs).

>      except Exception as e:
>          print("Job already exists")
> @@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan, plan_tests):
>      print("Creating batch job ")
>      try:
>          job_name = board+'.'+testplan+'.batch'
> -        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> -             job_name + '< ' + tmp, shell=True)
> -        os.unlink(tmp)
> +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
Same issues here as above.

>          return rcode
>      except Exception as e:
>          print("Job already exists")
> 
> 
> Thanks,
> Srivatsan
> 

FYI - Below is the patch I ended up with, that seems to work.  I tested it with
both existing and non-existing regular jobs and batch jobs.  I also tested creating
jobs for multiple boards on the same command line (some already existing
and some not).

Thanks for the inspiration for the change.  I think this is a nice improvement
to Fuego.
  -- Tim


commit b019ac2a2e7d16b03d24083c2fce868d736a61ec
Author: Tim Bird <tim.bird@sony.com>
Date:   Wed Nov 18 13:26:39 2020 -0700

    ftc: eliminate call to jenkins-cli.jar for creating jobs
    
    Use the python jenkins module to use the Jenkins REST API
    to perform job creation, instead of using jenkins-cli.jar.
    This eliminates all references to jenkins-cli.jar from ftc.
    Theoretically, we could remove the installation of jenkins-cli.jar
    from the Docker container, but let's hold off and make sure it's
    not needed for anything else.
    
    The call to create jobs via the jenkins module is much simpler.
    We don't need to create a temp file to hold the config xml for the
    job.  We do have to convert a non-numeric return code into a
    number since callers do bitwise or on it.
    
    Signed-off-by: Tim Bird <tim.bird@sony.com>

diff --git a/scripts/ftc b/scripts/ftc
index 30195fb..d6c9033 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -1438,6 +1438,7 @@ def link_key(item):
         return "zzz+"+key
 
 
+# return 0 on success
 def create_job(conf, board, test):
     flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
 
@@ -1505,9 +1506,7 @@ def create_job(conf, board, test):
         for cover, file_ref in link_tuples:
             fail_links += ' ' + template_link % (str(file_ref), str(cover))
 
-    tmp = "/tmp/fuego_tmp_job"
-    fd = open(tmp, "w+")
-    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
+    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
 <project>
     <actions/>
     <description></description>
@@ -1553,28 +1552,28 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
 """.format(board=board, reboot=test.reboot, rebuild=test.rebuild,
         precleanup=test.precleanup, postcleanup=test.postcleanup,
         testdir=test.name, testspec=test.spec, timeout=test.timeout,
-        flot_link=flot_link, success_links=success_links, fail_links=fail_links))
-    fd.close()
+        flot_link=flot_link, success_links=success_links,
+        fail_links=fail_links)
 
     job_name = board + "." + test.spec + "." + test.name
     print("Creating job " + job_name)
     try:
-        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
-            job_name + ' < ' + tmp, shell=True)
-        os.unlink(tmp)
+        rcode = server.create_job(job_name, job_xml)
+        if not rcode:
+            return 0
+        else:
+            return 1
         return rcode
     except Exception as e:
         print("Job already exists")
         print(e)
-        sys.exit(1)
+        return 1
 
 
 def create_batch_job(conf, board, testplan, plan_tests):
-    tmp = "/tmp/fuego_tmp_batch"
     job_list = [board+'.'+test.spec+'.'+test.name for test in plan_tests]
 
-    fd = open(tmp, "w+")
-    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
+    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
 <project>
   <actions/>
   <description></description>
@@ -1602,21 +1601,20 @@ def create_batch_job(conf, board, testplan, plan_tests):
   </publishers>
   <buildWrappers/>
 </project>
-""".format(board=board, alljobs=','.join(job_list)))
-
-    fd.close()
+""".format(board=board, alljobs=','.join(job_list))
 
     print("Creating batch job ")
     try:
         job_name = board+'.'+testplan+'.batch'
-        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
-             job_name + '< ' + tmp, shell=True)
-        os.unlink(tmp)
-        return rcode
+        rcode = server.create_job(job_name, job_xml)
+        if not rcode:
+            return 0
+        else:
+            return 1
     except Exception as e:
         print("Job already exists")
         print(e)
-        sys.exit(1)
+        return 1
 
 
 # parse the testplan and returns a list of test_class instances.

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script
  2020-11-18 20:54 ` Bird, Tim
@ 2020-11-19  3:07   ` Srivatsan S
  2020-11-19 17:27     ` Bird, Tim
  0 siblings, 1 reply; 4+ messages in thread
From: Srivatsan S @ 2020-11-19  3:07 UTC (permalink / raw)
  To: Bird, Tim; +Cc: tbird20, fuego

[-- Attachment #1: Type: text/plain, Size: 8687 bytes --]

Tim,

I hadn't tested with the change, but this was a work in progress as it
created the job with empty XML config. I had the job_xml stuff in progress
to pass the configs to Jenkins. Noticed that you've committed this change.
Shall test with your latest changes.

Thanks,
Srivatsan

On Thu, Nov 19, 2020 at 2:24 AM Bird, Tim <Tim.Bird@sony.com> wrote:

>
>
> > -----Original Message-----
> > From: Srivatsan S <srivatsan.s@pathpartnertech.com>
> >
> > Tim,
> >
> > While we were looking at eliminating the usage of jenkins-cli.jar in the
> ftc script, we came across create_job() API in the Jenkins Python
> > module which can be used to create jobs in Jenkins, thus eliminating the
> usage of jenkins-cli.jar in the ftc script.
> >
> > The below patch works for job creation in Jenkins through ftc script. We
> shall get back to you on ways to programmatically update other
> > job properties.
>
> Thanks for sending this.  It helped me work the issues with this change.
> There were a few problems with the implementation.  Did you test this?
> It doesn’t seem like it would have worked, since it would have created
> jobs with no shell command data.
>
> I ended up using the basic concept, but handling other aspects required for
> this change that you-all missed.
>
> >
> > git diff ftc
> >
> > diff --git a/scripts/ftc b/scripts/ftc
> > index 30195fb..a193c4d 100755
> > --- a/scripts/ftc
> > +++ b/scripts/ftc
> > @@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s
> {testspec} \\
> >      job_name = board + "." + test.spec + "." + test.name <
> http://test.name>
> >      print("Creating job " + job_name)
> >      try:
> > -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> > -            job_name + ' < ' + tmp, shell=True)
>
> In this case, the tmp file contains the data for config.xml for the new
> job.  The code above this needed
> to be refactored to submit the xml data as a string rather than piping in
> to jenkins-cli.jar from a file.
>
> > -        os.unlink(tmp)
> > +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> This EMPTY_CONFIG_XML will end up creating a stub job, with no shell
> command
> to invoke the appropriate Fuego test.
>
> >          return rcode
> server.create_job returns "None" on success.  The caller can only handle a
> numeric
> return result (it does a bitwise |, in the case of creating multiple jobs).
>
> >      except Exception as e:
> >          print("Job already exists")
> > @@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan,
> plan_tests):
> >      print("Creating batch job ")
> >      try:
> >          job_name = board+'.'+testplan+'.batch'
> > -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> > -             job_name + '< ' + tmp, shell=True)
> > -        os.unlink(tmp)
> > +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> Same issues here as above.
>
> >          return rcode
> >      except Exception as e:
> >          print("Job already exists")
> >
> >
> > Thanks,
> > Srivatsan
> >
>
> FYI - Below is the patch I ended up with, that seems to work.  I tested it
> with
> both existing and non-existing regular jobs and batch jobs.  I also tested
> creating
> jobs for multiple boards on the same command line (some already existing
> and some not).
>
> Thanks for the inspiration for the change.  I think this is a nice
> improvement
> to Fuego.
>   -- Tim
>
>
> commit b019ac2a2e7d16b03d24083c2fce868d736a61ec
> Author: Tim Bird <tim.bird@sony.com>
> Date:   Wed Nov 18 13:26:39 2020 -0700
>
>     ftc: eliminate call to jenkins-cli.jar for creating jobs
>
>     Use the python jenkins module to use the Jenkins REST API
>     to perform job creation, instead of using jenkins-cli.jar.
>     This eliminates all references to jenkins-cli.jar from ftc.
>     Theoretically, we could remove the installation of jenkins-cli.jar
>     from the Docker container, but let's hold off and make sure it's
>     not needed for anything else.
>
>     The call to create jobs via the jenkins module is much simpler.
>     We don't need to create a temp file to hold the config xml for the
>     job.  We do have to convert a non-numeric return code into a
>     number since callers do bitwise or on it.
>
>     Signed-off-by: Tim Bird <tim.bird@sony.com>
>
> diff --git a/scripts/ftc b/scripts/ftc
> index 30195fb..d6c9033 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -1438,6 +1438,7 @@ def link_key(item):
>          return "zzz+"+key
>
>
> +# return 0 on success
>  def create_job(conf, board, test):
>      flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
>
> @@ -1505,9 +1506,7 @@ def create_job(conf, board, test):
>          for cover, file_ref in link_tuples:
>              fail_links += ' ' + template_link % (str(file_ref),
> str(cover))
>
> -    tmp = "/tmp/fuego_tmp_job"
> -    fd = open(tmp, "w+")
> -    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> +    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
>  <project>
>      <actions/>
>      <description></description>
> @@ -1553,28 +1552,28 @@ ftc run-test -b $NODE_NAME -t {testdir} -s
> {testspec} \\
>  """.format(board=board, reboot=test.reboot, rebuild=test.rebuild,
>          precleanup=test.precleanup, postcleanup=test.postcleanup,
>          testdir=test.name, testspec=test.spec, timeout=test.timeout,
> -        flot_link=flot_link, success_links=success_links,
> fail_links=fail_links))
> -    fd.close()
> +        flot_link=flot_link, success_links=success_links,
> +        fail_links=fail_links)
>
>      job_name = board + "." + test.spec + "." + test.name
>      print("Creating job " + job_name)
>      try:
> -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> -            job_name + ' < ' + tmp, shell=True)
> -        os.unlink(tmp)
> +        rcode = server.create_job(job_name, job_xml)
> +        if not rcode:
> +            return 0
> +        else:
> +            return 1
>          return rcode
>      except Exception as e:
>          print("Job already exists")
>          print(e)
> -        sys.exit(1)
> +        return 1
>
>
>  def create_batch_job(conf, board, testplan, plan_tests):
> -    tmp = "/tmp/fuego_tmp_batch"
>      job_list = [board+'.'+test.spec+'.'+test.name for test in plan_tests]
>
> -    fd = open(tmp, "w+")
> -    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> +    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
>  <project>
>    <actions/>
>    <description></description>
> @@ -1602,21 +1601,20 @@ def create_batch_job(conf, board, testplan,
> plan_tests):
>    </publishers>
>    <buildWrappers/>
>  </project>
> -""".format(board=board, alljobs=','.join(job_list)))
> -
> -    fd.close()
> +""".format(board=board, alljobs=','.join(job_list))
>
>      print("Creating batch job ")
>      try:
>          job_name = board+'.'+testplan+'.batch'
> -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> -             job_name + '< ' + tmp, shell=True)
> -        os.unlink(tmp)
> -        return rcode
> +        rcode = server.create_job(job_name, job_xml)
> +        if not rcode:
> +            return 0
> +        else:
> +            return 1
>      except Exception as e:
>          print("Job already exists")
>          print(e)
> -        sys.exit(1)
> +        return 1
>
>
>  # parse the testplan and returns a list of test_class instances.
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #2: Type: text/html, Size: 11115 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script
  2020-11-19  3:07   ` Srivatsan S
@ 2020-11-19 17:27     ` Bird, Tim
  0 siblings, 0 replies; 4+ messages in thread
From: Bird, Tim @ 2020-11-19 17:27 UTC (permalink / raw)
  To: Srivatsan S; +Cc: tbird20, fuego

> -----Original Message-----
> From: Srivatsan S <srivatsan.s@pathpartnertech.com>
> 
> Tim,
> 
> I hadn't tested with the change, but this was a work in progress as it created the job with empty XML config. I had the job_xml stuff in
> progress to pass the configs to Jenkins. Noticed that you've committed this change. Shall test with your latest changes.

Sorry.  I didn't realize this was a work in progress.  I would have given you feedback and more time
to complete it if I had known.

Please do test the changes I committed.  Thanks.

 -- Tim

 
> Thanks,
> Srivatsan
> 
> On Thu, Nov 19, 2020 at 2:24 AM Bird, Tim <Tim.Bird@sony.com <mailto:Tim.Bird@sony.com> > wrote:
> 
> 
> 
> 
> 	> -----Original Message-----
> 	> From: Srivatsan S <srivatsan.s@pathpartnertech.com <mailto:srivatsan.s@pathpartnertech.com> >
> 	>
> 	> Tim,
> 	>
> 	> While we were looking at eliminating the usage of jenkins-cli.jar in the ftc script, we came across create_job() API in the Jenkins
> Python
> 	> module which can be used to create jobs in Jenkins, thus eliminating the usage of jenkins-cli.jar in the ftc script.
> 	>
> 	> The below patch works for job creation in Jenkins through ftc script. We shall get back to you on ways to programmatically
> update other
> 	> job properties.
> 
> 	Thanks for sending this.  It helped me work the issues with this change.
> 	There were a few problems with the implementation.  Did you test this?
> 	It doesn’t seem like it would have worked, since it would have created
> 	jobs with no shell command data.
> 
> 	I ended up using the basic concept, but handling other aspects required for
> 	this change that you-all missed.
> 
> 	>
> 	> git diff ftc
> 	>
> 	> diff --git a/scripts/ftc b/scripts/ftc
> 	> index 30195fb..a193c4d 100755
> 	> --- a/scripts/ftc
> 	> +++ b/scripts/ftc
> 	> @@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
> 	>      job_name = board + "." + test.spec + "." + test.name <http://test.name>  <http://test.name>
> 	>      print("Creating job " + job_name)
> 	>      try:
> 	> -        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	> -            job_name + ' < ' + tmp, shell=True)
> 
> 	In this case, the tmp file contains the data for config.xml for the new job.  The code above this needed
> 	to be refactored to submit the xml data as a string rather than piping in to jenkins-cli.jar from a file.
> 
> 	> -        os.unlink(tmp)
> 	> +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> 	This EMPTY_CONFIG_XML will end up creating a stub job, with no shell command
> 	to invoke the appropriate Fuego test.
> 
> 	>          return rcode
> 	server.create_job returns "None" on success.  The caller can only handle a numeric
> 	return result (it does a bitwise |, in the case of creating multiple jobs).
> 
> 	>      except Exception as e:
> 	>          print("Job already exists")
> 	> @@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan, plan_tests):
> 	>      print("Creating batch job ")
> 	>      try:
> 	>          job_name = board+'.'+testplan+'.batch'
> 	> -        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	> -             job_name + '< ' + tmp, shell=True)
> 	> -        os.unlink(tmp)
> 	> +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> 	Same issues here as above.
> 
> 	>          return rcode
> 	>      except Exception as e:
> 	>          print("Job already exists")
> 	>
> 	>
> 	> Thanks,
> 	> Srivatsan
> 	>
> 
> 	FYI - Below is the patch I ended up with, that seems to work.  I tested it with
> 	both existing and non-existing regular jobs and batch jobs.  I also tested creating
> 	jobs for multiple boards on the same command line (some already existing
> 	and some not).
> 
> 	Thanks for the inspiration for the change.  I think this is a nice improvement
> 	to Fuego.
> 	  -- Tim
> 
> 
> 	commit b019ac2a2e7d16b03d24083c2fce868d736a61ec
> 	Author: Tim Bird <tim.bird@sony.com <mailto:tim.bird@sony.com> >
> 	Date:   Wed Nov 18 13:26:39 2020 -0700
> 
> 	    ftc: eliminate call to jenkins-cli.jar for creating jobs
> 
> 	    Use the python jenkins module to use the Jenkins REST API
> 	    to perform job creation, instead of using jenkins-cli.jar.
> 	    This eliminates all references to jenkins-cli.jar from ftc.
> 	    Theoretically, we could remove the installation of jenkins-cli.jar
> 	    from the Docker container, but let's hold off and make sure it's
> 	    not needed for anything else.
> 
> 	    The call to create jobs via the jenkins module is much simpler.
> 	    We don't need to create a temp file to hold the config xml for the
> 	    job.  We do have to convert a non-numeric return code into a
> 	    number since callers do bitwise or on it.
> 
> 	    Signed-off-by: Tim Bird <tim.bird@sony.com <mailto:tim.bird@sony.com> >
> 
> 	diff --git a/scripts/ftc b/scripts/ftc
> 	index 30195fb..d6c9033 100755
> 	--- a/scripts/ftc
> 	+++ b/scripts/ftc
> 	@@ -1438,6 +1438,7 @@ def link_key(item):
> 	         return "zzz+"+key
> 
> 
> 	+# return 0 on success
> 	 def create_job(conf, board, test):
> 	     flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
> 
> 	@@ -1505,9 +1506,7 @@ def create_job(conf, board, test):
> 	         for cover, file_ref in link_tuples:
> 	             fail_links += ' ' + template_link % (str(file_ref), str(cover))
> 
> 	-    tmp = "/tmp/fuego_tmp_job"
> 	-    fd = open(tmp, "w+")
> 	-    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> 	+    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
> 	 <project>
> 	     <actions/>
> 	     <description></description>
> 	@@ -1553,28 +1552,28 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
> 	 """.format(board=board, reboot=test.reboot, rebuild=test.rebuild,
> 	         precleanup=test.precleanup, postcleanup=test.postcleanup,
> 	         testdir=test.name <http://test.name> , testspec=test.spec, timeout=test.timeout,
> 	-        flot_link=flot_link, success_links=success_links, fail_links=fail_links))
> 	-    fd.close()
> 	+        flot_link=flot_link, success_links=success_links,
> 	+        fail_links=fail_links)
> 
> 	     job_name = board + "." + test.spec + "." + test.name <http://test.name>
> 	     print("Creating job " + job_name)
> 	     try:
> 	-        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	-            job_name + ' < ' + tmp, shell=True)
> 	-        os.unlink(tmp)
> 	+        rcode = server.create_job(job_name, job_xml)
> 	+        if not rcode:
> 	+            return 0
> 	+        else:
> 	+            return 1
> 	         return rcode
> 	     except Exception as e:
> 	         print("Job already exists")
> 	         print(e)
> 	-        sys.exit(1)
> 	+        return 1
> 
> 
> 	 def create_batch_job(conf, board, testplan, plan_tests):
> 	-    tmp = "/tmp/fuego_tmp_batch"
> 	     job_list = [board+'.'+test.spec+'.'+test.name <http://test.name>  for test in plan_tests]
> 
> 	-    fd = open(tmp, "w+")
> 	-    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> 	+    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
> 	 <project>
> 	   <actions/>
> 	   <description></description>
> 	@@ -1602,21 +1601,20 @@ def create_batch_job(conf, board, testplan, plan_tests):
> 	   </publishers>
> 	   <buildWrappers/>
> 	 </project>
> 	-""".format(board=board, alljobs=','.join(job_list)))
> 	-
> 	-    fd.close()
> 	+""".format(board=board, alljobs=','.join(job_list))
> 
> 	     print("Creating batch job ")
> 	     try:
> 	         job_name = board+'.'+testplan+'.batch'
> 	-        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	-             job_name + '< ' + tmp, shell=True)
> 	-        os.unlink(tmp)
> 	-        return rcode
> 	+        rcode = server.create_job(job_name, job_xml)
> 	+        if not rcode:
> 	+            return 0
> 	+        else:
> 	+            return 1
> 	     except Exception as e:
> 	         print("Job already exists")
> 	         print(e)
> 	-        sys.exit(1)
> 	+        return 1
> 
> 
> 	 # parse the testplan and returns a list of test_class instances.
> 
> 
> 
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2020-11-19 17:27 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-18  7:05 [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script Srivatsan S
2020-11-18 20:54 ` Bird, Tim
2020-11-19  3:07   ` Srivatsan S
2020-11-19 17:27     ` Bird, Tim

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.