Xen-Devel Archive on lore.kernel.org
 help / color / Atom feed
* [OSSTEST PATCH v2 00/41] Performance work
@ 2020-07-31 11:37 Ian Jackson
  2020-07-31 11:37 ` [OSSTEST PATCH v2 01/41] Add cperl-indent-level to .dir-locals.el Ian Jackson
                   ` (40 more replies)
  0 siblings, 41 replies; 48+ messages in thread
From: Ian Jackson @ 2020-07-31 11:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Ian Jackson, George Dunlap

This is a combination of two series and some new work:
 * [OSSTEST PATCH 00/14] Flight report performance improvements
 * [OSSTEST PATCH 00/11] Improve performance of sg-report-host-history
 * New work to improve the performance of cs-bisection-step
 * Fixes to usage of SQL LIKE

Thanks to George for reviews of some of the most critical patches to
the regression analyser in sg-report-flight.  Where necessary I have
rebased those reviewed patches over the SQL LIKE fixes, and in those
cases I retained the reviewed-by.  I hope that's OK.

Outstanding in my perf programme is sg-report-job-history.

Ian Jackson (41):
  Add cperl-indent-level to .dir-locals.el
  SQL: Use "LIKE" rather than "like", etc.
  SQL: Fix incorrect LIKE pattern syntax (literals)
  SQL: Fix incorrect LIKE pattern syntax (program variables)
  sg-report-flight: Add a comment re same-flight search narrowing
  sg-report-flight: Sort failures by job name as last resort
  schema: Provide indices for sg-report-flight
  sg-report-flight: Ask the db for flights of interest
  sg-report-flight: Use WITH to use best index use for $flightsq
  sg-report-flight: Use WITH clause to use index for $anypassq
  sg-report-flight: Use the job row from the intitial query
  Executive: Use index for report__find_test
  duration_estimator: Ignore truncated jobs unless we know the step
  duration_estimator: Introduce some _qtxt variables
  duration_estimator: Explicitly provide null in general host q
  duration_estimator: Return job column in first query
  duration_estimator: Move $uptincl_testid to separate @x_params
  duration_estimator: Move duration query loop into database
  Executive: Drop redundant AND clause
  schema: Add index for quick lookup by host
  sg-report-host-history: Find flight limit by flight start date
  sg-report-host-history: Drop per-job debug etc.
  Executive: Export opendb_tests
  sg-report-host-history: Add a debug print after sorting jobs
  sg-report-host-history: Do the main query per host
  sg-report-host-history: Rerganisation: Make mainquery per-host
  sg-report-host-history: Rerganisation: Read old logs later
  sg-report-host-history: Rerganisation: Change loops
  sg-report-host-history: Drop a redundznt AND clause
  sg-report-host-history: Fork
  schema: Add index to help cs-bisection-step
  adhoc-revtuple-generator: Fix an undef warning in a debug print
  cs-bisection-step: Generalise qtxt_common_rev_ok
  cs-bisection-step: Move an AND
  cs-bisection-step: Break out qtxt_common_ok
  cs-bisection-step: Use db_prepare a few times instead of ->do
  cs-bisection-step: temporary table: Insert only rows we care about
  SQL: Change LIKE E'...\\_...' to LIKE '...\_...'
  cs-bisection-step: Add a debug print when we run dot(1)
  cs-bisection-step: Lay out the revision tuple graph once
  duration_estimator: Clarify recentflights query a bit

 .dir-locals.el                    |   3 +-
 Osstest.pm                        |   8 +-
 Osstest/Executive.pm              |  79 +++++++++------
 Osstest/JobDB/Executive.pm        |   2 +-
 adhoc-revtuple-generator          |   2 +-
 cr-ensure-disk-space              |   4 +-
 cs-adjust-flight                  |   2 +-
 cs-bisection-step                 |  51 +++++++---
 mg-force-push                     |   2 +-
 mg-report-host-usage-collect      |  10 +-
 ms-planner                        |   2 +-
 schema/runvars-built-index.sql    |   7 ++
 schema/runvars-host-index.sql     |   8 ++
 schema/runvars-revision-index.sql |   7 ++
 schema/steps-broken-index.sql     |   7 ++
 schema/steps-job-index.sql        |   7 ++
 sg-report-flight                  | 129 ++++++++++++++++++++----
 sg-report-host-history            | 161 +++++++++++++++++-------------
 sg-report-job-history             |   4 +-
 ts-logs-capture                   |   2 +-
 20 files changed, 344 insertions(+), 153 deletions(-)
 create mode 100644 schema/runvars-built-index.sql
 create mode 100644 schema/runvars-host-index.sql
 create mode 100644 schema/runvars-revision-index.sql
 create mode 100644 schema/steps-broken-index.sql
 create mode 100644 schema/steps-job-index.sql

-- 
2.20.1



^ permalink raw reply	[flat|nested] 48+ messages in thread

end of thread, back to index

Thread overview: 48+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-31 11:37 [OSSTEST PATCH v2 00/41] Performance work Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 01/41] Add cperl-indent-level to .dir-locals.el Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 02/41] SQL: Use "LIKE" rather than "like", etc Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 03/41] SQL: Fix incorrect LIKE pattern syntax (literals) Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 04/41] SQL: Fix incorrect LIKE pattern syntax (program variables) Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 05/41] sg-report-flight: Add a comment re same-flight search narrowing Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 06/41] sg-report-flight: Sort failures by job name as last resort Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 07/41] schema: Provide indices for sg-report-flight Ian Jackson
2020-07-31 14:21   ` George Dunlap
2020-07-31 14:55     ` Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 08/41] sg-report-flight: Ask the db for flights of interest Ian Jackson
2020-07-31 14:17   ` George Dunlap
2020-07-31 15:43     ` Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 09/41] sg-report-flight: Use WITH to use best index use for $flightsq Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 10/41] sg-report-flight: Use WITH clause to use index for $anypassq Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 11/41] sg-report-flight: Use the job row from the intitial query Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 12/41] Executive: Use index for report__find_test Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 13/41] duration_estimator: Ignore truncated jobs unless we know the step Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 14/41] duration_estimator: Introduce some _qtxt variables Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 15/41] duration_estimator: Explicitly provide null in general host q Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 16/41] duration_estimator: Return job column in first query Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 17/41] duration_estimator: Move $uptincl_testid to separate @x_params Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 18/41] duration_estimator: Move duration query loop into database Ian Jackson
2020-07-31 11:37 ` [OSSTEST PATCH v2 19/41] Executive: Drop redundant AND clause Ian Jackson
2020-07-31 14:17   ` George Dunlap
2020-07-31 11:37 ` [OSSTEST PATCH v2 20/41] schema: Add index for quick lookup by host Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 21/41] sg-report-host-history: Find flight limit by flight start date Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 22/41] sg-report-host-history: Drop per-job debug etc Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 23/41] Executive: Export opendb_tests Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 24/41] sg-report-host-history: Add a debug print after sorting jobs Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 25/41] sg-report-host-history: Do the main query per host Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 26/41] sg-report-host-history: Rerganisation: Make mainquery per-host Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 27/41] sg-report-host-history: Rerganisation: Read old logs later Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 28/41] sg-report-host-history: Rerganisation: Change loops Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 29/41] sg-report-host-history: Drop a redundznt AND clause Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 30/41] sg-report-host-history: Fork Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 31/41] schema: Add index to help cs-bisection-step Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 32/41] adhoc-revtuple-generator: Fix an undef warning in a debug print Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 33/41] cs-bisection-step: Generalise qtxt_common_rev_ok Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 34/41] cs-bisection-step: Move an AND Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 35/41] cs-bisection-step: Break out qtxt_common_ok Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 36/41] cs-bisection-step: Use db_prepare a few times instead of ->do Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 37/41] cs-bisection-step: temporary table: Insert only rows we care about Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 38/41] SQL: Change LIKE E'...\\_...' to LIKE '...\_...' Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 39/41] cs-bisection-step: Add a debug print when we run dot(1) Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 40/41] cs-bisection-step: Lay out the revision tuple graph once Ian Jackson
2020-07-31 11:38 ` [OSSTEST PATCH v2 41/41] duration_estimator: Clarify recentflights query a bit Ian Jackson
2020-07-31 14:04   ` George Dunlap

Xen-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/xen-devel/0 xen-devel/git/0.git
	git clone --mirror https://lore.kernel.org/xen-devel/1 xen-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 xen-devel xen-devel/ https://lore.kernel.org/xen-devel \
		xen-devel@lists.xenproject.org xen-devel@lists.xen.org
	public-inbox-index xen-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.xenproject.lists.xen-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git