* [Fuego] patch re-send and a couple of extra patches
@ 2017-08-04 2:17 Daniel Sangorrin
2017-08-04 2:17 ` [Fuego] [PATCH 1/4] parser: add toshiba copyright Daniel Sangorrin
` (4 more replies)
0 siblings, 5 replies; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-04 2:17 UTC (permalink / raw)
To: fuego
Hi Tim,
I have added the --remove-logs flag and Toshiba copyright
as you suggested.
Additionally I ported Dhrystone to the new format and
fixed the python path on the parser.py files.
[PATCH 1/4] parser: add toshiba copyright
[PATCH 2/4] parsers: fix the python path on the header
[PATCH 3/4] Dhrystone: fix for the new parser
[PATCH 4/4] ftc: add remove-logs flag to rm-jobs
Thanks,
Daniel
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Fuego] [PATCH 1/4] parser: add toshiba copyright
2017-08-04 2:17 [Fuego] patch re-send and a couple of extra patches Daniel Sangorrin
@ 2017-08-04 2:17 ` Daniel Sangorrin
2017-08-04 3:07 ` Bird, Timothy
2017-08-04 2:17 ` [Fuego] [PATCH 2/4] parsers: fix the python path on the header Daniel Sangorrin
` (3 subsequent siblings)
4 siblings, 1 reply; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-04 2:17 UTC (permalink / raw)
To: fuego
Update the copyright notice because I rewrote the parser to
introduce the unified output format.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/scripts/parser/common.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/engine/scripts/parser/common.py b/engine/scripts/parser/common.py
index ffabd2a..3be5d32 100644
--- a/engine/scripts/parser/common.py
+++ b/engine/scripts/parser/common.py
@@ -1,6 +1,8 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
+# Copyright (c) 2017 Toshiba corp.
+#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
--
2.7.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Fuego] [PATCH 2/4] parsers: fix the python path on the header
2017-08-04 2:17 [Fuego] patch re-send and a couple of extra patches Daniel Sangorrin
2017-08-04 2:17 ` [Fuego] [PATCH 1/4] parser: add toshiba copyright Daniel Sangorrin
@ 2017-08-04 2:17 ` Daniel Sangorrin
2017-08-04 3:08 ` Bird, Timothy
2017-08-04 2:17 ` [Fuego] [PATCH 3/4] Dhrystone: fix for the new parser Daniel Sangorrin
` (2 subsequent siblings)
4 siblings, 1 reply; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-04 2:17 UTC (permalink / raw)
To: fuego
This replaces /bin/python by /usr/bin/python which is where
the binary actually is.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/scripts/generic_parser.py | 2 +-
engine/tests/Benchmark.Dhrystone/parser.py | 2 +-
engine/tests/Benchmark.GLMark/parser.py | 2 +-
engine/tests/Benchmark.Interbench/parser.py | 2 +-
engine/tests/Benchmark.Java/parser.py | 2 +-
engine/tests/Benchmark.OpenSSL/parser.py | 2 +-
engine/tests/Benchmark.Stream/parser.py | 2 +-
engine/tests/Benchmark.aim7/parser.py | 2 +-
engine/tests/Benchmark.blobsallad/parser.py | 2 +-
engine/tests/Benchmark.cyclictest/parser.py | 2 +-
engine/tests/Benchmark.ebizzy/parser.py | 2 +-
engine/tests/Benchmark.fuego_check_plots/parser.py | 2 +-
engine/tests/Benchmark.gtkperf/parser.py | 2 +-
engine/tests/Benchmark.himeno/parser.py | 2 +-
engine/tests/Benchmark.iperf/parser.py | 2 +-
engine/tests/Benchmark.linpack/parser.py | 2 +-
engine/tests/Benchmark.lmbench2/parser.py | 2 +-
engine/tests/Benchmark.nbench_byte/parser.py | 2 +-
engine/tests/Benchmark.netperf/parser.py | 2 +-
engine/tests/Benchmark.reboot/parser.py | 2 +-
engine/tests/Benchmark.signaltest/parser.py | 2 +-
engine/tests/Benchmark.tiobench/parser.py | 2 +-
engine/tests/Benchmark.x11perf/parser.py | 2 +-
23 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/engine/scripts/generic_parser.py b/engine/scripts/generic_parser.py
index 377e02e..5e280a0 100755
--- a/engine/scripts/generic_parser.py
+++ b/engine/scripts/generic_parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
#
# generic_parser.py
# This is not really a parser, but it is used as a stub
diff --git a/engine/tests/Benchmark.Dhrystone/parser.py b/engine/tests/Benchmark.Dhrystone/parser.py
index 05bfd85..f410b2c 100755
--- a/engine/tests/Benchmark.Dhrystone/parser.py
+++ b/engine/tests/Benchmark.Dhrystone/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.GLMark/parser.py b/engine/tests/Benchmark.GLMark/parser.py
index d676ee3..c0e5a89 100755
--- a/engine/tests/Benchmark.GLMark/parser.py
+++ b/engine/tests/Benchmark.GLMark/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
# See common.py for description of command-line arguments
import os, re, sys
diff --git a/engine/tests/Benchmark.Interbench/parser.py b/engine/tests/Benchmark.Interbench/parser.py
index a9e1099..6615d68 100755
--- a/engine/tests/Benchmark.Interbench/parser.py
+++ b/engine/tests/Benchmark.Interbench/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.Java/parser.py b/engine/tests/Benchmark.Java/parser.py
index 08a4d1c..15c06e4 100755
--- a/engine/tests/Benchmark.Java/parser.py
+++ b/engine/tests/Benchmark.Java/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.OpenSSL/parser.py b/engine/tests/Benchmark.OpenSSL/parser.py
index 5ab1da9..7533da7 100755
--- a/engine/tests/Benchmark.OpenSSL/parser.py
+++ b/engine/tests/Benchmark.OpenSSL/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
# Parameters for this script are: JOB_NAME, PLATFORM, TARBALL_TEMPLATE, BUILD_ID, BUILD_NUMBER, Number of last builds to plot
import os, re, sys
diff --git a/engine/tests/Benchmark.Stream/parser.py b/engine/tests/Benchmark.Stream/parser.py
index e8fff68..1a91990 100755
--- a/engine/tests/Benchmark.Stream/parser.py
+++ b/engine/tests/Benchmark.Stream/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.aim7/parser.py b/engine/tests/Benchmark.aim7/parser.py
index 3201e0e..f28b8f5 100755
--- a/engine/tests/Benchmark.aim7/parser.py
+++ b/engine/tests/Benchmark.aim7/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
# See common.py for description of command-line arguments
import os, re, sys
diff --git a/engine/tests/Benchmark.blobsallad/parser.py b/engine/tests/Benchmark.blobsallad/parser.py
index c58cc3f..2efb5ae 100755
--- a/engine/tests/Benchmark.blobsallad/parser.py
+++ b/engine/tests/Benchmark.blobsallad/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.cyclictest/parser.py b/engine/tests/Benchmark.cyclictest/parser.py
index a491b25..44425ea 100755
--- a/engine/tests/Benchmark.cyclictest/parser.py
+++ b/engine/tests/Benchmark.cyclictest/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
# See common.py for description of command-line arguments
import os, re, sys
diff --git a/engine/tests/Benchmark.ebizzy/parser.py b/engine/tests/Benchmark.ebizzy/parser.py
index cca3907..8f8f120 100755
--- a/engine/tests/Benchmark.ebizzy/parser.py
+++ b/engine/tests/Benchmark.ebizzy/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.fuego_check_plots/parser.py b/engine/tests/Benchmark.fuego_check_plots/parser.py
index 0e07c58..cabe228 100755
--- a/engine/tests/Benchmark.fuego_check_plots/parser.py
+++ b/engine/tests/Benchmark.fuego_check_plots/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys, random
diff --git a/engine/tests/Benchmark.gtkperf/parser.py b/engine/tests/Benchmark.gtkperf/parser.py
index d3af83e..e4e623c 100755
--- a/engine/tests/Benchmark.gtkperf/parser.py
+++ b/engine/tests/Benchmark.gtkperf/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
# See common.py for description of command-line arguments
import os, re, sys
diff --git a/engine/tests/Benchmark.himeno/parser.py b/engine/tests/Benchmark.himeno/parser.py
index a776768..61fb029 100755
--- a/engine/tests/Benchmark.himeno/parser.py
+++ b/engine/tests/Benchmark.himeno/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.iperf/parser.py b/engine/tests/Benchmark.iperf/parser.py
index 77dbdf8..7720d9c 100755
--- a/engine/tests/Benchmark.iperf/parser.py
+++ b/engine/tests/Benchmark.iperf/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.linpack/parser.py b/engine/tests/Benchmark.linpack/parser.py
index eaa2472..7006ac3 100755
--- a/engine/tests/Benchmark.linpack/parser.py
+++ b/engine/tests/Benchmark.linpack/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.lmbench2/parser.py b/engine/tests/Benchmark.lmbench2/parser.py
index 8a14bfa..337ea0e 100755
--- a/engine/tests/Benchmark.lmbench2/parser.py
+++ b/engine/tests/Benchmark.lmbench2/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.nbench_byte/parser.py b/engine/tests/Benchmark.nbench_byte/parser.py
index 9f814e4..66a0f7f 100755
--- a/engine/tests/Benchmark.nbench_byte/parser.py
+++ b/engine/tests/Benchmark.nbench_byte/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.netperf/parser.py b/engine/tests/Benchmark.netperf/parser.py
index 37132da..2fce982 100755
--- a/engine/tests/Benchmark.netperf/parser.py
+++ b/engine/tests/Benchmark.netperf/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.reboot/parser.py b/engine/tests/Benchmark.reboot/parser.py
index 65a208f..2a7bdce 100755
--- a/engine/tests/Benchmark.reboot/parser.py
+++ b/engine/tests/Benchmark.reboot/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.signaltest/parser.py b/engine/tests/Benchmark.signaltest/parser.py
index a491b25..44425ea 100755
--- a/engine/tests/Benchmark.signaltest/parser.py
+++ b/engine/tests/Benchmark.signaltest/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
# See common.py for description of command-line arguments
import os, re, sys
diff --git a/engine/tests/Benchmark.tiobench/parser.py b/engine/tests/Benchmark.tiobench/parser.py
index a5c2f9f..bfbaeb4 100755
--- a/engine/tests/Benchmark.tiobench/parser.py
+++ b/engine/tests/Benchmark.tiobench/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
import os, re, sys
diff --git a/engine/tests/Benchmark.x11perf/parser.py b/engine/tests/Benchmark.x11perf/parser.py
index f8c7a7a..9efb063 100755
--- a/engine/tests/Benchmark.x11perf/parser.py
+++ b/engine/tests/Benchmark.x11perf/parser.py
@@ -1,4 +1,4 @@
-#!/bin/python
+#!/usr/bin/python
# See common.py for description of command-line arguments
import os, re, sys
--
2.7.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Fuego] [PATCH 3/4] Dhrystone: fix for the new parser
2017-08-04 2:17 [Fuego] patch re-send and a couple of extra patches Daniel Sangorrin
2017-08-04 2:17 ` [Fuego] [PATCH 1/4] parser: add toshiba copyright Daniel Sangorrin
2017-08-04 2:17 ` [Fuego] [PATCH 2/4] parsers: fix the python path on the header Daniel Sangorrin
@ 2017-08-04 2:17 ` Daniel Sangorrin
2017-08-04 3:09 ` Bird, Timothy
2017-08-04 2:17 ` [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs Daniel Sangorrin
2017-08-04 3:06 ` [Fuego] patch re-send and a couple of extra patches Bird, Timothy
4 siblings, 1 reply; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-04 2:17 UTC (permalink / raw)
To: fuego
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/tests/Benchmark.Dhrystone/criteria.json | 16 ++++++++++++++++
engine/tests/Benchmark.Dhrystone/parser.py | 20 ++++++--------------
engine/tests/Benchmark.Dhrystone/reference.json | 18 ++++++++++++++++++
engine/tests/Benchmark.Dhrystone/reference.log | 2 --
4 files changed, 40 insertions(+), 16 deletions(-)
create mode 100644 engine/tests/Benchmark.Dhrystone/criteria.json
create mode 100644 engine/tests/Benchmark.Dhrystone/reference.json
delete mode 100644 engine/tests/Benchmark.Dhrystone/reference.log
diff --git a/engine/tests/Benchmark.Dhrystone/criteria.json b/engine/tests/Benchmark.Dhrystone/criteria.json
new file mode 100644
index 0000000..fe55b3b
--- /dev/null
+++ b/engine/tests/Benchmark.Dhrystone/criteria.json
@@ -0,0 +1,16 @@
+{
+ "schema_version":"1.0",
+ "criteria":[
+ {
+ "tguid":"default.Dhrystone.Score",
+ "reference":{
+ "value":0,
+ "comparison":"gt"
+ }
+ },
+ {
+ "tguid":"default.Dhrystone",
+ "min_pass":1
+ }
+ ]
+}
diff --git a/engine/tests/Benchmark.Dhrystone/parser.py b/engine/tests/Benchmark.Dhrystone/parser.py
index f410b2c..6664506 100755
--- a/engine/tests/Benchmark.Dhrystone/parser.py
+++ b/engine/tests/Benchmark.Dhrystone/parser.py
@@ -5,20 +5,12 @@ import os, re, sys
sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
import common as plib
+regex_string = "^(Dhrystones.per.Second:)(\ *)([\d]{1,8}.?[\d]{1,3})(.*)$"
-ref_section_pat = "\[[\w]+.[gle]{2}\]"
-cur_search_str = "^(Dhrystones.per.Second:)(\ *)([\d]{1,8}.?[\d]{1,3})(.*)$"
+measurements = {}
+matches = plib.parse_log(regex_string)
-cur_dict = {}
-cur_file = open(plib.TEST_LOG,'r')
-print "Reading current values from " + plib.TEST_LOG
-cur_raw_values = cur_file.readlines()
-cur_file.close()
-
-for cur_item in cur_raw_values:
- cur_match = re.match(cur_search_str, cur_item)
- if cur_match:
- cur_dict["Dhrystone"] = cur_match.group(3)
-
-sys.exit(plib.process_data(ref_section_pat, cur_dict, 's', 'FPS'))
+if matches:
+ measurements['default.Dhrystone'] = [{"name": "Score", "measure" : float(matches[0][2])}]
+sys.exit(plib.process(measurements))
diff --git a/engine/tests/Benchmark.Dhrystone/reference.json b/engine/tests/Benchmark.Dhrystone/reference.json
new file mode 100644
index 0000000..532ac16
--- /dev/null
+++ b/engine/tests/Benchmark.Dhrystone/reference.json
@@ -0,0 +1,18 @@
+{
+ "test_sets":[
+ {
+ "name":"default",
+ "test_cases":[
+ {
+ "name":"Dhrystone",
+ "measurements":[
+ {
+ "name":"Score",
+ "unit":"Dhrystones/s"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}
diff --git a/engine/tests/Benchmark.Dhrystone/reference.log b/engine/tests/Benchmark.Dhrystone/reference.log
deleted file mode 100644
index 7cdba53..0000000
--- a/engine/tests/Benchmark.Dhrystone/reference.log
+++ /dev/null
@@ -1,2 +0,0 @@
-[Dhrystone|ge]
-1
--
2.7.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
2017-08-04 2:17 [Fuego] patch re-send and a couple of extra patches Daniel Sangorrin
` (2 preceding siblings ...)
2017-08-04 2:17 ` [Fuego] [PATCH 3/4] Dhrystone: fix for the new parser Daniel Sangorrin
@ 2017-08-04 2:17 ` Daniel Sangorrin
2017-08-04 3:14 ` Bird, Timothy
2017-08-04 3:06 ` [Fuego] patch re-send and a couple of extra patches Bird, Timothy
4 siblings, 1 reply; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-04 2:17 UTC (permalink / raw)
To: fuego
When removing jobs from jenkins this flag additionally allows
deleting the corresponding log files.
For developers, ftc rm-jobs --remove-logs, will leave Fuego
in a pristine state without jobs nor old logs.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/scripts/ftc | 45 +++++++++++++++++++++++++++++++++++++++------
1 file changed, 39 insertions(+), 6 deletions(-)
diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index e885c5f..d6cede5 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -44,7 +44,7 @@
import os, sys, string, re
import time
import commands
-from shutil import copy2
+from shutil import copy2, rmtree
from lxml import etree
import io
import subprocess
@@ -57,6 +57,7 @@ import shutil
import tempfile
import yaml
import jenkins
+import glob
# MAJOR, MINOR, REVISION
VERSION = (1,2,0)
@@ -94,15 +95,17 @@ command_help = {
This interface may change in the future."""),
"rm-jobs":("Removes jobs from Jenkins.",
- """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
+ """Usage: ftc rm-jobs [--remove-logs] <target>.<testspec>.<testtype>.<testcase>
Use list-jobs to see the existing jobs. A wildcard can be used to
specify which jobs to remove (just make sure you have 4 words):
Example: ftc rm-jobs "docker.testplan_docker.*.*"
Example: ftc rm-jobs "docker.*.F*.*stress"
Multiple combinations of the <target>.<testspec>.<testtype>.<testcase>
- pattern can be passed as well.
-
- If no option is provided all existing jobs will be removed."""),
+ pattern can be passed as well. The option --remove-logs will additionally
+ remove the corresponding log files in the log directory. If no job is
+ provided all existing jobs will be removed.
+ Example: ftc rm-jobs --remove-logs
+ """),
"add-nodes":("Adds new nodes to Jenkins.",
"""Usage: ftc add-nodes [-f] <board1> <board2> ...
@@ -1077,8 +1080,25 @@ def do_build_jobs(conf, options):
def do_rm_jobs(conf, options):
count = 0
+
+ if '--remove-logs' in options:
+ remove_logs = True
+ options.remove('--remove-logs')
+ raw_input("WARNING: this will remove logs. Press ENTER to confirm...")
+ else:
+ remove_logs = False
+
if len(options) == 0:
- # FIXTHIS: warn user that all jobs will be removed
+ if remove_logs:
+ # remove files and folders in LOGDIR
+ path = conf.FUEGO_RW + "/logs/*"
+ files = glob.glob(path)
+ for f in files:
+ if os.path.isfile(f):
+ os.remove(f)
+ else:
+ shutil.rmtree(f, ignore_errors=True)
+ # remove jenkins jobs
jobs = [job['name'] for job in server.get_jobs()]
for job in jobs:
server.delete_job(job)
@@ -1097,6 +1117,19 @@ def do_rm_jobs(conf, options):
else:
for match in matches:
server.delete_job(match)
+ if remove_logs:
+ # remove the per-build_number log folders for the job
+ path = conf.FUEGO_RW + '/logs/' + '.'.join(match.split('.')[2:]) + '/' + '.'.join(match.split('.')[:2]) + '*'
+ folders = glob.glob(path)
+ for folder in folders:
+ print 'Removing folder ' + folder
+ shutil.rmtree(folder, ignore_errors=True)
+ # remove the log folder itself in case it became empty
+ folders = glob.glob(path)
+ if not folders:
+ path = conf.FUEGO_RW + '/logs/' + '.'.join(match.split('.')[2:])
+ print 'Removing folder ' + path
+ shutil.rmtree(path, ignore_errors=True)
count = count + 1
if not quiet:
print "Deleted %d jobs." % count
--
2.7.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [Fuego] patch re-send and a couple of extra patches
2017-08-04 2:17 [Fuego] patch re-send and a couple of extra patches Daniel Sangorrin
` (3 preceding siblings ...)
2017-08-04 2:17 ` [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs Daniel Sangorrin
@ 2017-08-04 3:06 ` Bird, Timothy
4 siblings, 0 replies; 15+ messages in thread
From: Bird, Timothy @ 2017-08-04 3:06 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
> -----Original Message-----
> From: Daniel Sangorrin on Thursday, August 03, 2017 7:17 PM
>
> Hi Tim,
>
> I have added the --remove-logs flag and Toshiba copyright
> as you suggested.
> Additionally I ported Dhrystone to the new format and
> fixed the python path on the parser.py files.
>
> [PATCH 1/4] parser: add toshiba copyright
> [PATCH 2/4] parsers: fix the python path on the header
> [PATCH 3/4] Dhrystone: fix for the new parser
> [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
I looked over these. They look good.
I have a few comments on some of them, that I'll put on the
individual patches. I won't have time to integrate these
tonight (at least I don't thinkso), but any that I ack today I'll
get into my 'next' branch tomorrow.
-- Tim
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 1/4] parser: add toshiba copyright
2017-08-04 2:17 ` [Fuego] [PATCH 1/4] parser: add toshiba copyright Daniel Sangorrin
@ 2017-08-04 3:07 ` Bird, Timothy
0 siblings, 0 replies; 15+ messages in thread
From: Bird, Timothy @ 2017-08-04 3:07 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
Ack
> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Thursday, August 03, 2017 7:17 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 1/4] parser: add toshiba copyright
>
> Update the copyright notice because I rewrote the parser to
> introduce the unified output format.
>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> engine/scripts/parser/common.py | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/engine/scripts/parser/common.py
> b/engine/scripts/parser/common.py
> index ffabd2a..3be5d32 100644
> --- a/engine/scripts/parser/common.py
> +++ b/engine/scripts/parser/common.py
> @@ -1,6 +1,8 @@
> #!/usr/bin/env python
> # -*- coding: utf-8 -*-
> #
> +# Copyright (c) 2017 Toshiba corp.
> +#
> # Permission is hereby granted, free of charge, to any person obtaining a
> copy
> # of this software and associated documentation files (the "Software"), to
> deal
> # in the Software without restriction, including without limitation the rights
> --
> 2.7.4
>
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 2/4] parsers: fix the python path on the header
2017-08-04 2:17 ` [Fuego] [PATCH 2/4] parsers: fix the python path on the header Daniel Sangorrin
@ 2017-08-04 3:08 ` Bird, Timothy
2017-08-04 3:57 ` Daniel Sangorrin
0 siblings, 1 reply; 15+ messages in thread
From: Bird, Timothy @ 2017-08-04 3:08 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Thursday, August 03, 2017 7:17 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 2/4] parsers: fix the python path on the header
>
> This replaces /bin/python by /usr/bin/python which is where
> the binary actually is.
Is this with the intent to execute these directly, instead of through run_python?
It doesn't matter - I'll accept the patch either way. I'm just curious if
this is functional or just cosmetic.
-- Tim
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> engine/scripts/generic_parser.py | 2 +-
> engine/tests/Benchmark.Dhrystone/parser.py | 2 +-
> engine/tests/Benchmark.GLMark/parser.py | 2 +-
> engine/tests/Benchmark.Interbench/parser.py | 2 +-
> engine/tests/Benchmark.Java/parser.py | 2 +-
> engine/tests/Benchmark.OpenSSL/parser.py | 2 +-
> engine/tests/Benchmark.Stream/parser.py | 2 +-
> engine/tests/Benchmark.aim7/parser.py | 2 +-
> engine/tests/Benchmark.blobsallad/parser.py | 2 +-
> engine/tests/Benchmark.cyclictest/parser.py | 2 +-
> engine/tests/Benchmark.ebizzy/parser.py | 2 +-
> engine/tests/Benchmark.fuego_check_plots/parser.py | 2 +-
> engine/tests/Benchmark.gtkperf/parser.py | 2 +-
> engine/tests/Benchmark.himeno/parser.py | 2 +-
> engine/tests/Benchmark.iperf/parser.py | 2 +-
> engine/tests/Benchmark.linpack/parser.py | 2 +-
> engine/tests/Benchmark.lmbench2/parser.py | 2 +-
> engine/tests/Benchmark.nbench_byte/parser.py | 2 +-
> engine/tests/Benchmark.netperf/parser.py | 2 +-
> engine/tests/Benchmark.reboot/parser.py | 2 +-
> engine/tests/Benchmark.signaltest/parser.py | 2 +-
> engine/tests/Benchmark.tiobench/parser.py | 2 +-
> engine/tests/Benchmark.x11perf/parser.py | 2 +-
> 23 files changed, 23 insertions(+), 23 deletions(-)
>
> diff --git a/engine/scripts/generic_parser.py
> b/engine/scripts/generic_parser.py
> index 377e02e..5e280a0 100755
> --- a/engine/scripts/generic_parser.py
> +++ b/engine/scripts/generic_parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> #
> # generic_parser.py
> # This is not really a parser, but it is used as a stub
> diff --git a/engine/tests/Benchmark.Dhrystone/parser.py
> b/engine/tests/Benchmark.Dhrystone/parser.py
> index 05bfd85..f410b2c 100755
> --- a/engine/tests/Benchmark.Dhrystone/parser.py
> +++ b/engine/tests/Benchmark.Dhrystone/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.GLMark/parser.py
> b/engine/tests/Benchmark.GLMark/parser.py
> index d676ee3..c0e5a89 100755
> --- a/engine/tests/Benchmark.GLMark/parser.py
> +++ b/engine/tests/Benchmark.GLMark/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> # See common.py for description of command-line arguments
>
> import os, re, sys
> diff --git a/engine/tests/Benchmark.Interbench/parser.py
> b/engine/tests/Benchmark.Interbench/parser.py
> index a9e1099..6615d68 100755
> --- a/engine/tests/Benchmark.Interbench/parser.py
> +++ b/engine/tests/Benchmark.Interbench/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.Java/parser.py
> b/engine/tests/Benchmark.Java/parser.py
> index 08a4d1c..15c06e4 100755
> --- a/engine/tests/Benchmark.Java/parser.py
> +++ b/engine/tests/Benchmark.Java/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.OpenSSL/parser.py
> b/engine/tests/Benchmark.OpenSSL/parser.py
> index 5ab1da9..7533da7 100755
> --- a/engine/tests/Benchmark.OpenSSL/parser.py
> +++ b/engine/tests/Benchmark.OpenSSL/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> # Parameters for this script are: JOB_NAME, PLATFORM,
> TARBALL_TEMPLATE, BUILD_ID, BUILD_NUMBER, Number of last builds to
> plot
>
> import os, re, sys
> diff --git a/engine/tests/Benchmark.Stream/parser.py
> b/engine/tests/Benchmark.Stream/parser.py
> index e8fff68..1a91990 100755
> --- a/engine/tests/Benchmark.Stream/parser.py
> +++ b/engine/tests/Benchmark.Stream/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.aim7/parser.py
> b/engine/tests/Benchmark.aim7/parser.py
> index 3201e0e..f28b8f5 100755
> --- a/engine/tests/Benchmark.aim7/parser.py
> +++ b/engine/tests/Benchmark.aim7/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> # See common.py for description of command-line arguments
>
> import os, re, sys
> diff --git a/engine/tests/Benchmark.blobsallad/parser.py
> b/engine/tests/Benchmark.blobsallad/parser.py
> index c58cc3f..2efb5ae 100755
> --- a/engine/tests/Benchmark.blobsallad/parser.py
> +++ b/engine/tests/Benchmark.blobsallad/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.cyclictest/parser.py
> b/engine/tests/Benchmark.cyclictest/parser.py
> index a491b25..44425ea 100755
> --- a/engine/tests/Benchmark.cyclictest/parser.py
> +++ b/engine/tests/Benchmark.cyclictest/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> # See common.py for description of command-line arguments
>
> import os, re, sys
> diff --git a/engine/tests/Benchmark.ebizzy/parser.py
> b/engine/tests/Benchmark.ebizzy/parser.py
> index cca3907..8f8f120 100755
> --- a/engine/tests/Benchmark.ebizzy/parser.py
> +++ b/engine/tests/Benchmark.ebizzy/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.fuego_check_plots/parser.py
> b/engine/tests/Benchmark.fuego_check_plots/parser.py
> index 0e07c58..cabe228 100755
> --- a/engine/tests/Benchmark.fuego_check_plots/parser.py
> +++ b/engine/tests/Benchmark.fuego_check_plots/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys, random
>
> diff --git a/engine/tests/Benchmark.gtkperf/parser.py
> b/engine/tests/Benchmark.gtkperf/parser.py
> index d3af83e..e4e623c 100755
> --- a/engine/tests/Benchmark.gtkperf/parser.py
> +++ b/engine/tests/Benchmark.gtkperf/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> # See common.py for description of command-line arguments
>
> import os, re, sys
> diff --git a/engine/tests/Benchmark.himeno/parser.py
> b/engine/tests/Benchmark.himeno/parser.py
> index a776768..61fb029 100755
> --- a/engine/tests/Benchmark.himeno/parser.py
> +++ b/engine/tests/Benchmark.himeno/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.iperf/parser.py
> b/engine/tests/Benchmark.iperf/parser.py
> index 77dbdf8..7720d9c 100755
> --- a/engine/tests/Benchmark.iperf/parser.py
> +++ b/engine/tests/Benchmark.iperf/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.linpack/parser.py
> b/engine/tests/Benchmark.linpack/parser.py
> index eaa2472..7006ac3 100755
> --- a/engine/tests/Benchmark.linpack/parser.py
> +++ b/engine/tests/Benchmark.linpack/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.lmbench2/parser.py
> b/engine/tests/Benchmark.lmbench2/parser.py
> index 8a14bfa..337ea0e 100755
> --- a/engine/tests/Benchmark.lmbench2/parser.py
> +++ b/engine/tests/Benchmark.lmbench2/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.nbench_byte/parser.py
> b/engine/tests/Benchmark.nbench_byte/parser.py
> index 9f814e4..66a0f7f 100755
> --- a/engine/tests/Benchmark.nbench_byte/parser.py
> +++ b/engine/tests/Benchmark.nbench_byte/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.netperf/parser.py
> b/engine/tests/Benchmark.netperf/parser.py
> index 37132da..2fce982 100755
> --- a/engine/tests/Benchmark.netperf/parser.py
> +++ b/engine/tests/Benchmark.netperf/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.reboot/parser.py
> b/engine/tests/Benchmark.reboot/parser.py
> index 65a208f..2a7bdce 100755
> --- a/engine/tests/Benchmark.reboot/parser.py
> +++ b/engine/tests/Benchmark.reboot/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.signaltest/parser.py
> b/engine/tests/Benchmark.signaltest/parser.py
> index a491b25..44425ea 100755
> --- a/engine/tests/Benchmark.signaltest/parser.py
> +++ b/engine/tests/Benchmark.signaltest/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> # See common.py for description of command-line arguments
>
> import os, re, sys
> diff --git a/engine/tests/Benchmark.tiobench/parser.py
> b/engine/tests/Benchmark.tiobench/parser.py
> index a5c2f9f..bfbaeb4 100755
> --- a/engine/tests/Benchmark.tiobench/parser.py
> +++ b/engine/tests/Benchmark.tiobench/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
>
> import os, re, sys
>
> diff --git a/engine/tests/Benchmark.x11perf/parser.py
> b/engine/tests/Benchmark.x11perf/parser.py
> index f8c7a7a..9efb063 100755
> --- a/engine/tests/Benchmark.x11perf/parser.py
> +++ b/engine/tests/Benchmark.x11perf/parser.py
> @@ -1,4 +1,4 @@
> -#!/bin/python
> +#!/usr/bin/python
> # See common.py for description of command-line arguments
>
> import os, re, sys
> --
> 2.7.4
>
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 3/4] Dhrystone: fix for the new parser
2017-08-04 2:17 ` [Fuego] [PATCH 3/4] Dhrystone: fix for the new parser Daniel Sangorrin
@ 2017-08-04 3:09 ` Bird, Timothy
0 siblings, 0 replies; 15+ messages in thread
From: Bird, Timothy @ 2017-08-04 3:09 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
Ack.
> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Thursday, August 03, 2017 7:17 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 3/4] Dhrystone: fix for the new parser
>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> engine/tests/Benchmark.Dhrystone/criteria.json | 16 ++++++++++++++++
> engine/tests/Benchmark.Dhrystone/parser.py | 20 ++++++--------------
> engine/tests/Benchmark.Dhrystone/reference.json | 18
> ++++++++++++++++++
> engine/tests/Benchmark.Dhrystone/reference.log | 2 --
> 4 files changed, 40 insertions(+), 16 deletions(-)
> create mode 100644 engine/tests/Benchmark.Dhrystone/criteria.json
> create mode 100644 engine/tests/Benchmark.Dhrystone/reference.json
> delete mode 100644 engine/tests/Benchmark.Dhrystone/reference.log
>
> diff --git a/engine/tests/Benchmark.Dhrystone/criteria.json
> b/engine/tests/Benchmark.Dhrystone/criteria.json
> new file mode 100644
> index 0000000..fe55b3b
> --- /dev/null
> +++ b/engine/tests/Benchmark.Dhrystone/criteria.json
> @@ -0,0 +1,16 @@
> +{
> + "schema_version":"1.0",
> + "criteria":[
> + {
> + "tguid":"default.Dhrystone.Score",
> + "reference":{
> + "value":0,
> + "comparison":"gt"
> + }
> + },
> + {
> + "tguid":"default.Dhrystone",
> + "min_pass":1
> + }
> + ]
> +}
> diff --git a/engine/tests/Benchmark.Dhrystone/parser.py
> b/engine/tests/Benchmark.Dhrystone/parser.py
> index f410b2c..6664506 100755
> --- a/engine/tests/Benchmark.Dhrystone/parser.py
> +++ b/engine/tests/Benchmark.Dhrystone/parser.py
> @@ -5,20 +5,12 @@ import os, re, sys
> sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> import common as plib
>
> +regex_string = "^(Dhrystones.per.Second:)(\ *)([\d]{1,8}.?[\d]{1,3})(.*)$"
>
> -ref_section_pat = "\[[\w]+.[gle]{2}\]"
> -cur_search_str = "^(Dhrystones.per.Second:)(\ *)([\d]{1,8}.?[\d]{1,3})(.*)$"
> +measurements = {}
> +matches = plib.parse_log(regex_string)
>
> -cur_dict = {}
> -cur_file = open(plib.TEST_LOG,'r')
> -print "Reading current values from " + plib.TEST_LOG
> -cur_raw_values = cur_file.readlines()
> -cur_file.close()
> -
> -for cur_item in cur_raw_values:
> - cur_match = re.match(cur_search_str, cur_item)
> - if cur_match:
> - cur_dict["Dhrystone"] = cur_match.group(3)
> -
> -sys.exit(plib.process_data(ref_section_pat, cur_dict, 's', 'FPS'))
> +if matches:
> + measurements['default.Dhrystone'] = [{"name": "Score", "measure" :
> float(matches[0][2])}]
>
> +sys.exit(plib.process(measurements))
> diff --git a/engine/tests/Benchmark.Dhrystone/reference.json
> b/engine/tests/Benchmark.Dhrystone/reference.json
> new file mode 100644
> index 0000000..532ac16
> --- /dev/null
> +++ b/engine/tests/Benchmark.Dhrystone/reference.json
> @@ -0,0 +1,18 @@
> +{
> + "test_sets":[
> + {
> + "name":"default",
> + "test_cases":[
> + {
> + "name":"Dhrystone",
> + "measurements":[
> + {
> + "name":"Score",
> + "unit":"Dhrystones/s"
> + }
> + ]
> + }
> + ]
> + }
> + ]
> +}
> diff --git a/engine/tests/Benchmark.Dhrystone/reference.log
> b/engine/tests/Benchmark.Dhrystone/reference.log
> deleted file mode 100644
> index 7cdba53..0000000
> --- a/engine/tests/Benchmark.Dhrystone/reference.log
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -[Dhrystone|ge]
> -1
> --
> 2.7.4
>
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
2017-08-04 2:17 ` [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs Daniel Sangorrin
@ 2017-08-04 3:14 ` Bird, Timothy
2017-08-04 3:59 ` Daniel Sangorrin
0 siblings, 1 reply; 15+ messages in thread
From: Bird, Timothy @ 2017-08-04 3:14 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
See comment below.
> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Thursday, August 03, 2017 7:17 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
>
> When removing jobs from jenkins this flag additionally allows
> deleting the corresponding log files.
>
> For developers, ftc rm-jobs --remove-logs, will leave Fuego
> in a pristine state without jobs nor old logs.
>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> engine/scripts/ftc | 45 +++++++++++++++++++++++++++++++++++++++---
> ---
> 1 file changed, 39 insertions(+), 6 deletions(-)
>
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index e885c5f..d6cede5 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -44,7 +44,7 @@
> import os, sys, string, re
> import time
> import commands
> -from shutil import copy2
> +from shutil import copy2, rmtree
> from lxml import etree
> import io
> import subprocess
> @@ -57,6 +57,7 @@ import shutil
> import tempfile
> import yaml
> import jenkins
> +import glob
>
> # MAJOR, MINOR, REVISION
> VERSION = (1,2,0)
> @@ -94,15 +95,17 @@ command_help = {
> This interface may change in the future."""),
>
> "rm-jobs":("Removes jobs from Jenkins.",
> - """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
> + """Usage: ftc rm-jobs [--remove-logs]
> <target>.<testspec>.<testtype>.<testcase>
> Use list-jobs to see the existing jobs. A wildcard can be used to
> specify which jobs to remove (just make sure you have 4 words):
> Example: ftc rm-jobs "docker.testplan_docker.*.*"
> Example: ftc rm-jobs "docker.*.F*.*stress"
> Multiple combinations of the <target>.<testspec>.<testtype>.<testcase>
> - pattern can be passed as well.
> -
> - If no option is provided all existing jobs will be removed."""),
> + pattern can be passed as well. The option --remove-logs will additionally
> + remove the corresponding log files in the log directory. If no job is
> + provided all existing jobs will be removed.
> + Example: ftc rm-jobs --remove-logs
> + """),
>
> "add-nodes":("Adds new nodes to Jenkins.",
> """Usage: ftc add-nodes [-f] <board1> <board2> ...
> @@ -1077,8 +1080,25 @@ def do_build_jobs(conf, options):
>
> def do_rm_jobs(conf, options):
> count = 0
> +
> + if '--remove-logs' in options:
> + remove_logs = True
> + options.remove('--remove-logs')
> + raw_input("WARNING: this will remove logs. Press ENTER to
> confirm...")
I don't think this is needed. If you really want it, I can leave it,
but I think the user adding '--remove-logs' is specific enough to
not warrant asking for confirmation. Let me know. If you're
OK with me taking it out, I'll do so when I add the patch to
my next branch. If you want to keep it, though, just let
me know.
> + else:
> + remove_logs = False
> +
> if len(options) == 0:
> - # FIXTHIS: warn user that all jobs will be removed
> + if remove_logs:
> + # remove files and folders in LOGDIR
> + path = conf.FUEGO_RW + "/logs/*"
I can't tell, but I assume this is in the 'remove all jobs' path?
> + files = glob.glob(path)
> + for f in files:
> + if os.path.isfile(f):
> + os.remove(f)
> + else:
> + shutil.rmtree(f, ignore_errors=True)
> + # remove jenkins jobs
> jobs = [job['name'] for job in server.get_jobs()]
> for job in jobs:
> server.delete_job(job)
> @@ -1097,6 +1117,19 @@ def do_rm_jobs(conf, options):
> else:
> for match in matches:
> server.delete_job(match)
> + if remove_logs:
> + # remove the per-build_number log folders for the job
> + path = conf.FUEGO_RW + '/logs/' + '.'.join(match.split('.')[2:]) +
> '/' + '.'.join(match.split('.')[:2]) + '*'
> + folders = glob.glob(path)
> + for folder in folders:
> + print 'Removing folder ' + folder
> + shutil.rmtree(folder, ignore_errors=True)
> + # remove the log folder itself in case it became empty
> + folders = glob.glob(path)
> + if not folders:
> + path = conf.FUEGO_RW + '/logs/' + '.'.join(match.split('.')[2:])
> + print 'Removing folder ' + path
> + shutil.rmtree(path, ignore_errors=True)
> count = count + 1
> if not quiet:
> print "Deleted %d jobs." % count
> --
> 2.7.4
Ack.
-- Tim
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 2/4] parsers: fix the python path on the header
2017-08-04 3:08 ` Bird, Timothy
@ 2017-08-04 3:57 ` Daniel Sangorrin
0 siblings, 0 replies; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-04 3:57 UTC (permalink / raw)
To: 'Bird, Timothy', fuego
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Friday, August 04, 2017 12:09 PM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 2/4] parsers: fix the python path on the header
>
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > Sent: Thursday, August 03, 2017 7:17 PM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH 2/4] parsers: fix the python path on the header
> >
> > This replaces /bin/python by /usr/bin/python which is where
> > the binary actually is.
>
> Is this with the intent to execute these directly, instead of through run_python?
> It doesn't matter - I'll accept the patch either way. I'm just curious if
> this is functional or just cosmetic.
It's just a cosmetic thing.
Daniel
> -- Tim
>
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> > engine/scripts/generic_parser.py | 2 +-
> > engine/tests/Benchmark.Dhrystone/parser.py | 2 +-
> > engine/tests/Benchmark.GLMark/parser.py | 2 +-
> > engine/tests/Benchmark.Interbench/parser.py | 2 +-
> > engine/tests/Benchmark.Java/parser.py | 2 +-
> > engine/tests/Benchmark.OpenSSL/parser.py | 2 +-
> > engine/tests/Benchmark.Stream/parser.py | 2 +-
> > engine/tests/Benchmark.aim7/parser.py | 2 +-
> > engine/tests/Benchmark.blobsallad/parser.py | 2 +-
> > engine/tests/Benchmark.cyclictest/parser.py | 2 +-
> > engine/tests/Benchmark.ebizzy/parser.py | 2 +-
> > engine/tests/Benchmark.fuego_check_plots/parser.py | 2 +-
> > engine/tests/Benchmark.gtkperf/parser.py | 2 +-
> > engine/tests/Benchmark.himeno/parser.py | 2 +-
> > engine/tests/Benchmark.iperf/parser.py | 2 +-
> > engine/tests/Benchmark.linpack/parser.py | 2 +-
> > engine/tests/Benchmark.lmbench2/parser.py | 2 +-
> > engine/tests/Benchmark.nbench_byte/parser.py | 2 +-
> > engine/tests/Benchmark.netperf/parser.py | 2 +-
> > engine/tests/Benchmark.reboot/parser.py | 2 +-
> > engine/tests/Benchmark.signaltest/parser.py | 2 +-
> > engine/tests/Benchmark.tiobench/parser.py | 2 +-
> > engine/tests/Benchmark.x11perf/parser.py | 2 +-
> > 23 files changed, 23 insertions(+), 23 deletions(-)
> >
> > diff --git a/engine/scripts/generic_parser.py
> > b/engine/scripts/generic_parser.py
> > index 377e02e..5e280a0 100755
> > --- a/engine/scripts/generic_parser.py
> > +++ b/engine/scripts/generic_parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > #
> > # generic_parser.py
> > # This is not really a parser, but it is used as a stub
> > diff --git a/engine/tests/Benchmark.Dhrystone/parser.py
> > b/engine/tests/Benchmark.Dhrystone/parser.py
> > index 05bfd85..f410b2c 100755
> > --- a/engine/tests/Benchmark.Dhrystone/parser.py
> > +++ b/engine/tests/Benchmark.Dhrystone/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.GLMark/parser.py
> > b/engine/tests/Benchmark.GLMark/parser.py
> > index d676ee3..c0e5a89 100755
> > --- a/engine/tests/Benchmark.GLMark/parser.py
> > +++ b/engine/tests/Benchmark.GLMark/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > # See common.py for description of command-line arguments
> >
> > import os, re, sys
> > diff --git a/engine/tests/Benchmark.Interbench/parser.py
> > b/engine/tests/Benchmark.Interbench/parser.py
> > index a9e1099..6615d68 100755
> > --- a/engine/tests/Benchmark.Interbench/parser.py
> > +++ b/engine/tests/Benchmark.Interbench/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.Java/parser.py
> > b/engine/tests/Benchmark.Java/parser.py
> > index 08a4d1c..15c06e4 100755
> > --- a/engine/tests/Benchmark.Java/parser.py
> > +++ b/engine/tests/Benchmark.Java/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.OpenSSL/parser.py
> > b/engine/tests/Benchmark.OpenSSL/parser.py
> > index 5ab1da9..7533da7 100755
> > --- a/engine/tests/Benchmark.OpenSSL/parser.py
> > +++ b/engine/tests/Benchmark.OpenSSL/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > # Parameters for this script are: JOB_NAME, PLATFORM,
> > TARBALL_TEMPLATE, BUILD_ID, BUILD_NUMBER, Number of last builds to
> > plot
> >
> > import os, re, sys
> > diff --git a/engine/tests/Benchmark.Stream/parser.py
> > b/engine/tests/Benchmark.Stream/parser.py
> > index e8fff68..1a91990 100755
> > --- a/engine/tests/Benchmark.Stream/parser.py
> > +++ b/engine/tests/Benchmark.Stream/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.aim7/parser.py
> > b/engine/tests/Benchmark.aim7/parser.py
> > index 3201e0e..f28b8f5 100755
> > --- a/engine/tests/Benchmark.aim7/parser.py
> > +++ b/engine/tests/Benchmark.aim7/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > # See common.py for description of command-line arguments
> >
> > import os, re, sys
> > diff --git a/engine/tests/Benchmark.blobsallad/parser.py
> > b/engine/tests/Benchmark.blobsallad/parser.py
> > index c58cc3f..2efb5ae 100755
> > --- a/engine/tests/Benchmark.blobsallad/parser.py
> > +++ b/engine/tests/Benchmark.blobsallad/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.cyclictest/parser.py
> > b/engine/tests/Benchmark.cyclictest/parser.py
> > index a491b25..44425ea 100755
> > --- a/engine/tests/Benchmark.cyclictest/parser.py
> > +++ b/engine/tests/Benchmark.cyclictest/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > # See common.py for description of command-line arguments
> >
> > import os, re, sys
> > diff --git a/engine/tests/Benchmark.ebizzy/parser.py
> > b/engine/tests/Benchmark.ebizzy/parser.py
> > index cca3907..8f8f120 100755
> > --- a/engine/tests/Benchmark.ebizzy/parser.py
> > +++ b/engine/tests/Benchmark.ebizzy/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.fuego_check_plots/parser.py
> > b/engine/tests/Benchmark.fuego_check_plots/parser.py
> > index 0e07c58..cabe228 100755
> > --- a/engine/tests/Benchmark.fuego_check_plots/parser.py
> > +++ b/engine/tests/Benchmark.fuego_check_plots/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys, random
> >
> > diff --git a/engine/tests/Benchmark.gtkperf/parser.py
> > b/engine/tests/Benchmark.gtkperf/parser.py
> > index d3af83e..e4e623c 100755
> > --- a/engine/tests/Benchmark.gtkperf/parser.py
> > +++ b/engine/tests/Benchmark.gtkperf/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > # See common.py for description of command-line arguments
> >
> > import os, re, sys
> > diff --git a/engine/tests/Benchmark.himeno/parser.py
> > b/engine/tests/Benchmark.himeno/parser.py
> > index a776768..61fb029 100755
> > --- a/engine/tests/Benchmark.himeno/parser.py
> > +++ b/engine/tests/Benchmark.himeno/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.iperf/parser.py
> > b/engine/tests/Benchmark.iperf/parser.py
> > index 77dbdf8..7720d9c 100755
> > --- a/engine/tests/Benchmark.iperf/parser.py
> > +++ b/engine/tests/Benchmark.iperf/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.linpack/parser.py
> > b/engine/tests/Benchmark.linpack/parser.py
> > index eaa2472..7006ac3 100755
> > --- a/engine/tests/Benchmark.linpack/parser.py
> > +++ b/engine/tests/Benchmark.linpack/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.lmbench2/parser.py
> > b/engine/tests/Benchmark.lmbench2/parser.py
> > index 8a14bfa..337ea0e 100755
> > --- a/engine/tests/Benchmark.lmbench2/parser.py
> > +++ b/engine/tests/Benchmark.lmbench2/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.nbench_byte/parser.py
> > b/engine/tests/Benchmark.nbench_byte/parser.py
> > index 9f814e4..66a0f7f 100755
> > --- a/engine/tests/Benchmark.nbench_byte/parser.py
> > +++ b/engine/tests/Benchmark.nbench_byte/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.netperf/parser.py
> > b/engine/tests/Benchmark.netperf/parser.py
> > index 37132da..2fce982 100755
> > --- a/engine/tests/Benchmark.netperf/parser.py
> > +++ b/engine/tests/Benchmark.netperf/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.reboot/parser.py
> > b/engine/tests/Benchmark.reboot/parser.py
> > index 65a208f..2a7bdce 100755
> > --- a/engine/tests/Benchmark.reboot/parser.py
> > +++ b/engine/tests/Benchmark.reboot/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.signaltest/parser.py
> > b/engine/tests/Benchmark.signaltest/parser.py
> > index a491b25..44425ea 100755
> > --- a/engine/tests/Benchmark.signaltest/parser.py
> > +++ b/engine/tests/Benchmark.signaltest/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > # See common.py for description of command-line arguments
> >
> > import os, re, sys
> > diff --git a/engine/tests/Benchmark.tiobench/parser.py
> > b/engine/tests/Benchmark.tiobench/parser.py
> > index a5c2f9f..bfbaeb4 100755
> > --- a/engine/tests/Benchmark.tiobench/parser.py
> > +++ b/engine/tests/Benchmark.tiobench/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> >
> > import os, re, sys
> >
> > diff --git a/engine/tests/Benchmark.x11perf/parser.py
> > b/engine/tests/Benchmark.x11perf/parser.py
> > index f8c7a7a..9efb063 100755
> > --- a/engine/tests/Benchmark.x11perf/parser.py
> > +++ b/engine/tests/Benchmark.x11perf/parser.py
> > @@ -1,4 +1,4 @@
> > -#!/bin/python
> > +#!/usr/bin/python
> > # See common.py for description of command-line arguments
> >
> > import os, re, sys
> > --
> > 2.7.4
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
2017-08-04 3:14 ` Bird, Timothy
@ 2017-08-04 3:59 ` Daniel Sangorrin
2017-08-04 23:32 ` Bird, Timothy
0 siblings, 1 reply; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-04 3:59 UTC (permalink / raw)
To: 'Bird, Timothy', fuego
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Friday, August 04, 2017 12:14 PM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
>
> See comment below.
>
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > Sent: Thursday, August 03, 2017 7:17 PM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> >
> > When removing jobs from jenkins this flag additionally allows
> > deleting the corresponding log files.
> >
> > For developers, ftc rm-jobs --remove-logs, will leave Fuego
> > in a pristine state without jobs nor old logs.
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> > engine/scripts/ftc | 45 +++++++++++++++++++++++++++++++++++++++---
> > ---
> > 1 file changed, 39 insertions(+), 6 deletions(-)
> >
> > diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> > index e885c5f..d6cede5 100755
> > --- a/engine/scripts/ftc
> > +++ b/engine/scripts/ftc
> > @@ -44,7 +44,7 @@
> > import os, sys, string, re
> > import time
> > import commands
> > -from shutil import copy2
> > +from shutil import copy2, rmtree
> > from lxml import etree
> > import io
> > import subprocess
> > @@ -57,6 +57,7 @@ import shutil
> > import tempfile
> > import yaml
> > import jenkins
> > +import glob
> >
> > # MAJOR, MINOR, REVISION
> > VERSION = (1,2,0)
> > @@ -94,15 +95,17 @@ command_help = {
> > This interface may change in the future."""),
> >
> > "rm-jobs":("Removes jobs from Jenkins.",
> > - """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
> > + """Usage: ftc rm-jobs [--remove-logs]
> > <target>.<testspec>.<testtype>.<testcase>
> > Use list-jobs to see the existing jobs. A wildcard can be used to
> > specify which jobs to remove (just make sure you have 4 words):
> > Example: ftc rm-jobs "docker.testplan_docker.*.*"
> > Example: ftc rm-jobs "docker.*.F*.*stress"
> > Multiple combinations of the <target>.<testspec>.<testtype>.<testcase>
> > - pattern can be passed as well.
> > -
> > - If no option is provided all existing jobs will be removed."""),
> > + pattern can be passed as well. The option --remove-logs will additionally
> > + remove the corresponding log files in the log directory. If no job is
> > + provided all existing jobs will be removed.
> > + Example: ftc rm-jobs --remove-logs
> > + """),
> >
> > "add-nodes":("Adds new nodes to Jenkins.",
> > """Usage: ftc add-nodes [-f] <board1> <board2> ...
> > @@ -1077,8 +1080,25 @@ def do_build_jobs(conf, options):
> >
> > def do_rm_jobs(conf, options):
> > count = 0
> > +
> > + if '--remove-logs' in options:
> > + remove_logs = True
> > + options.remove('--remove-logs')
> > + raw_input("WARNING: this will remove logs. Press ENTER to
> > confirm...")
>
> I don't think this is needed. If you really want it, I can leave it,
> but I think the user adding '--remove-logs' is specific enough to
> not warrant asking for confirmation. Let me know. If you're
> OK with me taking it out, I'll do so when I add the patch to
> my next branch. If you want to keep it, though, just let
> me know.
Maybe I was being too conservative. Please remove the line if you want.
> > + else:
> > + remove_logs = False
> > +
> > if len(options) == 0:
> > - # FIXTHIS: warn user that all jobs will be removed
> > + if remove_logs:
> > + # remove files and folders in LOGDIR
> > + path = conf.FUEGO_RW + "/logs/*"
>
> I can't tell, but I assume this is in the 'remove all jobs' path?
Yes, that's right.
>
> > + files = glob.glob(path)
> > + for f in files:
> > + if os.path.isfile(f):
> > + os.remove(f)
> > + else:
> > + shutil.rmtree(f, ignore_errors=True)
> > + # remove jenkins jobs
> > jobs = [job['name'] for job in server.get_jobs()]
> > for job in jobs:
> > server.delete_job(job)
> > @@ -1097,6 +1117,19 @@ def do_rm_jobs(conf, options):
> > else:
> > for match in matches:
> > server.delete_job(match)
> > + if remove_logs:
> > + # remove the per-build_number log folders for the job
> > + path = conf.FUEGO_RW + '/logs/' + '.'.join(match.split('.')[2:]) +
> > '/' + '.'.join(match.split('.')[:2]) + '*'
> > + folders = glob.glob(path)
> > + for folder in folders:
> > + print 'Removing folder ' + folder
> > + shutil.rmtree(folder, ignore_errors=True)
> > + # remove the log folder itself in case it became empty
> > + folders = glob.glob(path)
> > + if not folders:
> > + path = conf.FUEGO_RW + '/logs/' + '.'.join(match.split('.')[2:])
> > + print 'Removing folder ' + path
> > + shutil.rmtree(path, ignore_errors=True)
> > count = count + 1
> > if not quiet:
> > print "Deleted %d jobs." % count
> > --
> > 2.7.4
>
> Ack.
> -- Tim
>
Thanks for the quick review of the patches.
Daniel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
2017-08-04 3:59 ` Daniel Sangorrin
@ 2017-08-04 23:32 ` Bird, Timothy
2017-08-07 3:05 ` Daniel Sangorrin
0 siblings, 1 reply; 15+ messages in thread
From: Bird, Timothy @ 2017-08-04 23:32 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
Daniel,
I added this patch to my 'next' branch, but on testing it found that it removes a bit too much.
This sequence shows the problem:
ftc add-job -b board1 -t Functional.hello_world
ftc add-job -b board2 -t Functional.hello_world -s hello-fail
ftc build-job board1.default.Functional.hello_world
ftc build-job board2.hello-fail.Functional.hello_world
At this point there are 2 jobs and 2 log directories
ftc rm-job --remove-logs board2.hello-fail.Functional.hello_world
At this point the directory /fuego-rw/logs/Functional.hello_world is removed
(including the board1.default.1.1 log directory)
Also, what happens to the results.json file when only some of the
logs for a test are removed? (for example, some for only one spec, or
some for only one board). It seems like it would need to be re-written.)
Can you please fix this?
Thanks,
-- Tim
> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Thursday, August 03, 2017 9:00 PM
> To: Bird, Timothy <Tim.Bird@sony.com>; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
>
>
>
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > Sent: Friday, August 04, 2017 12:14 PM
> > To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> >
> > See comment below.
> >
> > > -----Original Message-----
> > > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > Sent: Thursday, August 03, 2017 7:17 PM
> > > To: fuego@lists.linuxfoundation.org
> > > Subject: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> > >
> > > When removing jobs from jenkins this flag additionally allows
> > > deleting the corresponding log files.
> > >
> > > For developers, ftc rm-jobs --remove-logs, will leave Fuego
> > > in a pristine state without jobs nor old logs.
> > >
> > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > ---
> > > engine/scripts/ftc | 45
> +++++++++++++++++++++++++++++++++++++++---
> > > ---
> > > 1 file changed, 39 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> > > index e885c5f..d6cede5 100755
> > > --- a/engine/scripts/ftc
> > > +++ b/engine/scripts/ftc
> > > @@ -44,7 +44,7 @@
> > > import os, sys, string, re
> > > import time
> > > import commands
> > > -from shutil import copy2
> > > +from shutil import copy2, rmtree
> > > from lxml import etree
> > > import io
> > > import subprocess
> > > @@ -57,6 +57,7 @@ import shutil
> > > import tempfile
> > > import yaml
> > > import jenkins
> > > +import glob
> > >
> > > # MAJOR, MINOR, REVISION
> > > VERSION = (1,2,0)
> > > @@ -94,15 +95,17 @@ command_help = {
> > > This interface may change in the future."""),
> > >
> > > "rm-jobs":("Removes jobs from Jenkins.",
> > > - """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
> > > + """Usage: ftc rm-jobs [--remove-logs]
> > > <target>.<testspec>.<testtype>.<testcase>
> > > Use list-jobs to see the existing jobs. A wildcard can be used to
> > > specify which jobs to remove (just make sure you have 4 words):
> > > Example: ftc rm-jobs "docker.testplan_docker.*.*"
> > > Example: ftc rm-jobs "docker.*.F*.*stress"
> > > Multiple combinations of the
> <target>.<testspec>.<testtype>.<testcase>
> > > - pattern can be passed as well.
> > > -
> > > - If no option is provided all existing jobs will be removed."""),
> > > + pattern can be passed as well. The option --remove-logs will
> additionally
> > > + remove the corresponding log files in the log directory. If no job is
> > > + provided all existing jobs will be removed.
> > > + Example: ftc rm-jobs --remove-logs
> > > + """),
> > >
> > > "add-nodes":("Adds new nodes to Jenkins.",
> > > """Usage: ftc add-nodes [-f] <board1> <board2> ...
> > > @@ -1077,8 +1080,25 @@ def do_build_jobs(conf, options):
> > >
> > > def do_rm_jobs(conf, options):
> > > count = 0
> > > +
> > > + if '--remove-logs' in options:
> > > + remove_logs = True
> > > + options.remove('--remove-logs')
> > > + raw_input("WARNING: this will remove logs. Press ENTER to
> > > confirm...")
> >
> > I don't think this is needed. If you really want it, I can leave it,
> > but I think the user adding '--remove-logs' is specific enough to
> > not warrant asking for confirmation. Let me know. If you're
> > OK with me taking it out, I'll do so when I add the patch to
> > my next branch. If you want to keep it, though, just let
> > me know.
>
> Maybe I was being too conservative. Please remove the line if you want.
>
> > > + else:
> > > + remove_logs = False
> > > +
> > > if len(options) == 0:
> > > - # FIXTHIS: warn user that all jobs will be removed
> > > + if remove_logs:
> > > + # remove files and folders in LOGDIR
> > > + path = conf.FUEGO_RW + "/logs/*"
> >
> > I can't tell, but I assume this is in the 'remove all jobs' path?
>
> Yes, that's right.
> >
> > > + files = glob.glob(path)
> > > + for f in files:
> > > + if os.path.isfile(f):
> > > + os.remove(f)
> > > + else:
> > > + shutil.rmtree(f, ignore_errors=True)
> > > + # remove jenkins jobs
> > > jobs = [job['name'] for job in server.get_jobs()]
> > > for job in jobs:
> > > server.delete_job(job)
> > > @@ -1097,6 +1117,19 @@ def do_rm_jobs(conf, options):
> > > else:
> > > for match in matches:
> > > server.delete_job(match)
> > > + if remove_logs:
> > > + # remove the per-build_number log folders for the job
> > > + path = conf.FUEGO_RW + '/logs/' +
> '.'.join(match.split('.')[2:]) +
> > > '/' + '.'.join(match.split('.')[:2]) + '*'
> > > + folders = glob.glob(path)
> > > + for folder in folders:
> > > + print 'Removing folder ' + folder
> > > + shutil.rmtree(folder, ignore_errors=True)
> > > + # remove the log folder itself in case it became empty
> > > + folders = glob.glob(path)
> > > + if not folders:
> > > + path = conf.FUEGO_RW + '/logs/' +
> '.'.join(match.split('.')[2:])
> > > + print 'Removing folder ' + path
> > > + shutil.rmtree(path, ignore_errors=True)
> > > count = count + 1
> > > if not quiet:
> > > print "Deleted %d jobs." % count
> > > --
> > > 2.7.4
> >
> > Ack.
> > -- Tim
> >
>
> Thanks for the quick review of the patches.
> Daniel
>
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
2017-08-04 23:32 ` Bird, Timothy
@ 2017-08-07 3:05 ` Daniel Sangorrin
2017-08-09 1:11 ` Bird, Timothy
0 siblings, 1 reply; 15+ messages in thread
From: Daniel Sangorrin @ 2017-08-07 3:05 UTC (permalink / raw)
To: 'Bird, Timothy', fuego
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Saturday, August 05, 2017 8:33 AM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
>
> Daniel,
>
> I added this patch to my 'next' branch, but on testing it found that it removes a bit too much.
>
> This sequence shows the problem:
> ftc add-job -b board1 -t Functional.hello_world
> ftc add-job -b board2 -t Functional.hello_world -s hello-fail
> ftc build-job board1.default.Functional.hello_world
> ftc build-job board2.hello-fail.Functional.hello_world
> At this point there are 2 jobs and 2 log directories
> ftc rm-job --remove-logs board2.hello-fail.Functional.hello_world
> At this point the directory /fuego-rw/logs/Functional.hello_world is removed
> (including the board1.default.1.1 log directory)
Oops sorry about that. It should be working with the new patch I just sent.
> Also, what happens to the results.json file when only some of the
> logs for a test are removed? (for example, some for only one spec, or
> some for only one board). It seems like it would need to be re-written.)
This was harder than what it sounds but it should work with the new patch.
I had to separate the results.json merging function from the common.py
parser so that I could update the results.json without having to setup lots
of environment variables.
Thanks,
Daniel
>
> Can you please fix this?
>
> Thanks,
> -- Tim
>
>
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > Sent: Thursday, August 03, 2017 9:00 PM
> > To: Bird, Timothy <Tim.Bird@sony.com>; fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> >
> >
> >
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > > Sent: Friday, August 04, 2017 12:14 PM
> > > To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> > > Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> > >
> > > See comment below.
> > >
> > > > -----Original Message-----
> > > > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > > > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > > Sent: Thursday, August 03, 2017 7:17 PM
> > > > To: fuego@lists.linuxfoundation.org
> > > > Subject: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> > > >
> > > > When removing jobs from jenkins this flag additionally allows
> > > > deleting the corresponding log files.
> > > >
> > > > For developers, ftc rm-jobs --remove-logs, will leave Fuego
> > > > in a pristine state without jobs nor old logs.
> > > >
> > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > > ---
> > > > engine/scripts/ftc | 45
> > +++++++++++++++++++++++++++++++++++++++---
> > > > ---
> > > > 1 file changed, 39 insertions(+), 6 deletions(-)
> > > >
> > > > diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> > > > index e885c5f..d6cede5 100755
> > > > --- a/engine/scripts/ftc
> > > > +++ b/engine/scripts/ftc
> > > > @@ -44,7 +44,7 @@
> > > > import os, sys, string, re
> > > > import time
> > > > import commands
> > > > -from shutil import copy2
> > > > +from shutil import copy2, rmtree
> > > > from lxml import etree
> > > > import io
> > > > import subprocess
> > > > @@ -57,6 +57,7 @@ import shutil
> > > > import tempfile
> > > > import yaml
> > > > import jenkins
> > > > +import glob
> > > >
> > > > # MAJOR, MINOR, REVISION
> > > > VERSION = (1,2,0)
> > > > @@ -94,15 +95,17 @@ command_help = {
> > > > This interface may change in the future."""),
> > > >
> > > > "rm-jobs":("Removes jobs from Jenkins.",
> > > > - """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
> > > > + """Usage: ftc rm-jobs [--remove-logs]
> > > > <target>.<testspec>.<testtype>.<testcase>
> > > > Use list-jobs to see the existing jobs. A wildcard can be used to
> > > > specify which jobs to remove (just make sure you have 4 words):
> > > > Example: ftc rm-jobs "docker.testplan_docker.*.*"
> > > > Example: ftc rm-jobs "docker.*.F*.*stress"
> > > > Multiple combinations of the
> > <target>.<testspec>.<testtype>.<testcase>
> > > > - pattern can be passed as well.
> > > > -
> > > > - If no option is provided all existing jobs will be removed."""),
> > > > + pattern can be passed as well. The option --remove-logs will
> > additionally
> > > > + remove the corresponding log files in the log directory. If no job is
> > > > + provided all existing jobs will be removed.
> > > > + Example: ftc rm-jobs --remove-logs
> > > > + """),
> > > >
> > > > "add-nodes":("Adds new nodes to Jenkins.",
> > > > """Usage: ftc add-nodes [-f] <board1> <board2> ...
> > > > @@ -1077,8 +1080,25 @@ def do_build_jobs(conf, options):
> > > >
> > > > def do_rm_jobs(conf, options):
> > > > count = 0
> > > > +
> > > > + if '--remove-logs' in options:
> > > > + remove_logs = True
> > > > + options.remove('--remove-logs')
> > > > + raw_input("WARNING: this will remove logs. Press ENTER to
> > > > confirm...")
> > >
> > > I don't think this is needed. If you really want it, I can leave it,
> > > but I think the user adding '--remove-logs' is specific enough to
> > > not warrant asking for confirmation. Let me know. If you're
> > > OK with me taking it out, I'll do so when I add the patch to
> > > my next branch. If you want to keep it, though, just let
> > > me know.
> >
> > Maybe I was being too conservative. Please remove the line if you want.
> >
> > > > + else:
> > > > + remove_logs = False
> > > > +
> > > > if len(options) == 0:
> > > > - # FIXTHIS: warn user that all jobs will be removed
> > > > + if remove_logs:
> > > > + # remove files and folders in LOGDIR
> > > > + path = conf.FUEGO_RW + "/logs/*"
> > >
> > > I can't tell, but I assume this is in the 'remove all jobs' path?
> >
> > Yes, that's right.
> > >
> > > > + files = glob.glob(path)
> > > > + for f in files:
> > > > + if os.path.isfile(f):
> > > > + os.remove(f)
> > > > + else:
> > > > + shutil.rmtree(f, ignore_errors=True)
> > > > + # remove jenkins jobs
> > > > jobs = [job['name'] for job in server.get_jobs()]
> > > > for job in jobs:
> > > > server.delete_job(job)
> > > > @@ -1097,6 +1117,19 @@ def do_rm_jobs(conf, options):
> > > > else:
> > > > for match in matches:
> > > > server.delete_job(match)
> > > > + if remove_logs:
> > > > + # remove the per-build_number log folders for the job
> > > > + path = conf.FUEGO_RW + '/logs/' +
> > '.'.join(match.split('.')[2:]) +
> > > > '/' + '.'.join(match.split('.')[:2]) + '*'
> > > > + folders = glob.glob(path)
> > > > + for folder in folders:
> > > > + print 'Removing folder ' + folder
> > > > + shutil.rmtree(folder, ignore_errors=True)
> > > > + # remove the log folder itself in case it became empty
> > > > + folders = glob.glob(path)
> > > > + if not folders:
> > > > + path = conf.FUEGO_RW + '/logs/' +
> > '.'.join(match.split('.')[2:])
> > > > + print 'Removing folder ' + path
> > > > + shutil.rmtree(path, ignore_errors=True)
> > > > count = count + 1
> > > > if not quiet:
> > > > print "Deleted %d jobs." % count
> > > > --
> > > > 2.7.4
> > >
> > > Ack.
> > > -- Tim
> > >
> >
> > Thanks for the quick review of the patches.
> > Daniel
> >
> >
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
2017-08-07 3:05 ` Daniel Sangorrin
@ 2017-08-09 1:11 ` Bird, Timothy
0 siblings, 0 replies; 15+ messages in thread
From: Bird, Timothy @ 2017-08-09 1:11 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
> -----Original Message-----
> From: Daniel Sangorrin on Sunday, August 06, 2017 8:06 PM
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > Sent: Saturday, August 05, 2017 8:33 AM
> > To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> >
> > Daniel,
> >
> > I added this patch to my 'next' branch, but on testing it found that it
> removes a bit too much.
> >
> > This sequence shows the problem:
> > ftc add-job -b board1 -t Functional.hello_world
> > ftc add-job -b board2 -t Functional.hello_world -s hello-fail
> > ftc build-job board1.default.Functional.hello_world
> > ftc build-job board2.hello-fail.Functional.hello_world
> > At this point there are 2 jobs and 2 log directories
> > ftc rm-job --remove-logs board2.hello-fail.Functional.hello_world
> > At this point the directory /fuego-rw/logs/Functional.hello_world is
> removed
> > (including the board1.default.1.1 log directory)
>
> Oops sorry about that. It should be working with the new patch I just sent.
>
> > Also, what happens to the results.json file when only some of the
> > logs for a test are removed? (for example, some for only one spec, or
> > some for only one board). It seems like it would need to be re-written.)
>
> This was harder than what it sounds but it should work with the new patch.
> I had to separate the results.json merging function from the common.py
> parser so that I could update the results.json without having to setup lots
> of environment variables.
Thanks for the quick response. I looked at the patch, and it seems OK to me.
I haven't tested it yet, but I pulled it into my 'next' branch.
-- Tim
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2017-08-09 1:11 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-04 2:17 [Fuego] patch re-send and a couple of extra patches Daniel Sangorrin
2017-08-04 2:17 ` [Fuego] [PATCH 1/4] parser: add toshiba copyright Daniel Sangorrin
2017-08-04 3:07 ` Bird, Timothy
2017-08-04 2:17 ` [Fuego] [PATCH 2/4] parsers: fix the python path on the header Daniel Sangorrin
2017-08-04 3:08 ` Bird, Timothy
2017-08-04 3:57 ` Daniel Sangorrin
2017-08-04 2:17 ` [Fuego] [PATCH 3/4] Dhrystone: fix for the new parser Daniel Sangorrin
2017-08-04 3:09 ` Bird, Timothy
2017-08-04 2:17 ` [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs Daniel Sangorrin
2017-08-04 3:14 ` Bird, Timothy
2017-08-04 3:59 ` Daniel Sangorrin
2017-08-04 23:32 ` Bird, Timothy
2017-08-07 3:05 ` Daniel Sangorrin
2017-08-09 1:11 ` Bird, Timothy
2017-08-04 3:06 ` [Fuego] patch re-send and a couple of extra patches Bird, Timothy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.