[PATCH OSSTEST] README: Add some core concepts and terminology

[PATCH OSSTEST] README: Add some core concepts and terminology

[Ian Campbell]

---
 README | 202 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 201 insertions(+), 1 deletion(-)

diff --git a/README b/README
index 29c9d45..60379c4 100644
--- a/README
+++ b/README
@@ -1,3 +1,197 @@
+Introduction
+============
+
+OSStest is the Xen Project automated test infrastructure.
+
+Terminology
+===========
+
+"flight":
+
+    Each run of osstest is referred to as a "flight". Each flight is
+    given a unique ID (a number or name).
+
+"job":
+
+    Each flight consists of one or more "jobs". These are a sequence
+    of test steps run in order and correspond to a column in the test
+    report grid. They have names like "build-amd64" or
+    "test-amd64-amd64-pv". A job can depend on the output of another
+    job in the flight -- e.g. most test-* jobs depend on one or more
+    build-* jobs.
+
+"step":
+
+    Each job consists of multiple "steps" which is an individual test
+    operation, such as "build the hypervisor", "install a guest",
+    "start a guest", "migrate a guest", etc. A step corresponds to a
+    cell in the results grid. A given step can be reused in multiple
+    different jobs, e.g. the "xen build" step is used in several
+    different build-* jobs. This reuse can be seen in the rows of the
+    results grid.
+
+"runvar":
+
+    A runvar is a named textual variable associated with each job in a
+    given flight. They serve as both the inputs and outputs to the
+    job.
+
+    For example a Xen build job may have input runvars "tree_xen" (the
+    tree to clone) and "revision_xen" (the version to test). As output
+    it sets "path_xendist" which is the path to the tarball of the
+    resulting binary.
+
+    As a further example a test job may have an input runvar
+    "xenbuildjob" which specifies which build job produced the binary
+    to be tested. The "xen install" step can then read this runvar in
+    order to find the binary to install.
+
+    Other runvars also exist covering things such as:
+
+        * constraints on which machines in the test pool a job can be
+          run on (e.g. the architecure, the need for a particular
+          processor class, the presence of SR-IOV etc).
+
+        * the parameters of the guest to test (e.g. distro, PV vs HVM
+          etc).
+
+Operation
+=========
+
+A flight is constructed by the "make-flight" script.
+
+"make-flight" will allocate a new flight number, create a set of jobs
+with input runvars depending on the configuration (e.g. branch/version
+to test).
+
+A flight is run by the "mg-execute-flight" script, which in turn calls
+"sg-execute-flight". "sg-execute-flight" then spawns an instance of
+"sg-run-job" for each job in the flight.
+
+"sg-run-job" encodes various recipes (sequences of steps) which are
+referenced by each job's configuration. It then runs each of these in
+turn, taking into account the prerequisites etc, by calling the
+relevant "ts-*" scripts.
+
+When running in standalone mode it is possible to run any of these
+steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although
+you will need to find the correct inputs (some of which are documented
+below) and perhaps take care of prerequisites yourself (e.g. running
+"./sg-run-job test-armhf-armhf-xl" means you must have done
+"./sg-runjob build-armhf" and "build-armhf-pvops" first.
+
+Results
+=======
+
+For flights run automatically by the infrastructure an email report is
+produced. For most normal flights this is mailed to the xen-devel
+mailing list. The report for flight 24438 can be seen at
+
+    http://lists.xen.org/archives/html/xen-devel/2014-01/msg01614.html
+
+The report will link to a set of complete logs. Since these are
+regularly expired due to space constraints the logs for flight 24438
+have been archived to
+
+    http://xenbits.xen.org/docs/osstest-output-example/24438/
+
+NB: to save space any files larger than 100K have been replaced with a
+placeholder.
+
+The results grid contains an overview of the flight's execution.
+
+The results for each job are reached by clicking the header of the
+column in the results grid which will lead to reports such as:
+
+    http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/info.html
+    http://xenbits.xen.org/docs/osstest-output-example/24438/build-amd64/info.html
+
+The job report contains all of the logs and build outputs associated
+with this job.
+
+The logs for a step are reached by clicking the individual cells of
+the results grid, or by clicking the list of steps on the job
+report. In either case this will lead to a report such as
+
+    http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/4.ts-xen-install.log
+
+Additional details (e.g. serial logs, guest cfg files, etc) will be
+available in the complete logs associated with the containing job.
+
+The runvars are listed in the results page for the job as "Test
+control variables". e.g. See the end of:
+
+    http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/info.html
+
+In order to find the binaries which went into a test job you should
+consult the results page for that job and find the relevant build
+job. e.g.
+
+    http://www.chiark.greenend.org.uk/~xensrcts/logs/24438/test-amd64-amd64-xl/info.html
+
+lists "xenbuildjob" as "build-amd64". Therefore the input binaries are
+found at
+
+    http://www.chiark.greenend.org.uk/~xensrcts/logs/24438/build-amd64/info.html
+
+which is linked from the top of the relevant column in the overview
+grid.
+
+Script Naming Conventions
+=========================
+
+Most of the scripts follow a naming convention:
+
+ap-*:   Adhoc push scripts
+
+cr-*:   Cron scripts
+
+cri-*:  Cron scripts (internal)
+
+cs-*:   Control Scripts
+
+mg-*:   Management scripts
+
+ms-*:   Management Services
+
+sg-*:   ?
+
+ts-*:   Test Step scripts.
+
+Jobs
+====
+
+The names of jobs follow some common patterns:
+
+    build-$ARCH
+
+        Build Xen for $ARCH
+
+    build-$ARCH-xend
+
+        Build Xen for $ARCH, with xend enabled
+
+    build-$ARCH-pvops
+
+        Build an upstream ("pvops") kernel for $ARCH
+
+build-$ARCH-oldkern
+
+        Build the old "linux-2.6.18-xen" tree for $ARCH
+
+test-$XENARCH-$DOM0ARCH-<CASE>
+
+        A test <CASE> running a $XENARCH hypervisor with a $DOM0ARCH
+        dom0.
+
+        Some tests also have a -$DOMUARCH suffix indicating the
+        obvious thing.
+
+NB: $ARCH (and $XENARCH etc) are Debian arch names, i386, amd64, armhf.
+
+Standalone Mode
+===============
+
 To run osstest in standalone mode:
 
  - You need to install
@@ -18,7 +212,7 @@ To run osstest in standalone mode:
    gives you the "branch" consisting of tests run for the xen-unstable
    push gate.  You need to select a job.  The list of available jobs
    is that shown in the publicly emailed test reports on xen-devel, eg
-     http://lists.xen.org/archives/html/xen-devel/2013-08/msg02529.html
+     http://lists.xen.org/archives/html/xen-devel/2014-01/msg01614.html
 
    If you don't want to repro one of those and don't know how to
    choose a job, choose one of
@@ -26,6 +220,12 @@ To run osstest in standalone mode:
 
  - Run ./standalone-reset
 
+   This will call "make-flight" for you to create a flight targetting
+   xen-unstable (this can be adjusted by passing parameters to
+   standalone-reset). By default the flight identifier is
+   "standalone". standalone-reset will also make sure that certain
+   bits of static data are available (e.g. Debian installer images)
+
  - Then you can run
       ./sg-run-job <job>
    to run that job on the default host.  NB in most cases this will
-- 
1.8.5.2

Re: [PATCH OSSTEST] README: Add some core concepts and terminology

[Ian Campbell]

On Mon, 2014-01-20 at 13:20 +0000, Ian Campbell wrote:

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

(I don't know why I find this so hard to remember...)

Re: [PATCH OSSTEST] README: Add some core concepts and terminology

[Ian Campbell]

I should have said this yesterday on the docs day, but: Ping

On Mon, 2014-01-20 at 13:20 +0000, Ian Campbell wrote:
> ---
>  README | 202 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 201 insertions(+), 1 deletion(-)
> 
> diff --git a/README b/README
> index 29c9d45..60379c4 100644
> --- a/README
> +++ b/README
> @@ -1,3 +1,197 @@
> +Introduction
> +============
> +
> +OSStest is the Xen Project automated test infrastructure.
> +
> +Terminology
> +===========
> +
> +"flight":
> +
> +    Each run of osstest is referred to as a "flight". Each flight is
> +    given a unique ID (a number or name).
> +
> +"job":
> +
> +    Each flight consists of one or more "jobs". These are a sequence
> +    of test steps run in order and correspond to a column in the test
> +    report grid. They have names like "build-amd64" or
> +    "test-amd64-amd64-pv". A job can depend on the output of another
> +    job in the flight -- e.g. most test-* jobs depend on one or more
> +    build-* jobs.
> +
> +"step":
> +
> +    Each job consists of multiple "steps" which is an individual test
> +    operation, such as "build the hypervisor", "install a guest",
> +    "start a guest", "migrate a guest", etc. A step corresponds to a
> +    cell in the results grid. A given step can be reused in multiple
> +    different jobs, e.g. the "xen build" step is used in several
> +    different build-* jobs. This reuse can be seen in the rows of the
> +    results grid.
> +
> +"runvar":
> +
> +    A runvar is a named textual variable associated with each job in a
> +    given flight. They serve as both the inputs and outputs to the
> +    job.
> +
> +    For example a Xen build job may have input runvars "tree_xen" (the
> +    tree to clone) and "revision_xen" (the version to test). As output
> +    it sets "path_xendist" which is the path to the tarball of the
> +    resulting binary.
> +
> +    As a further example a test job may have an input runvar
> +    "xenbuildjob" which specifies which build job produced the binary
> +    to be tested. The "xen install" step can then read this runvar in
> +    order to find the binary to install.
> +
> +    Other runvars also exist covering things such as:
> +
> +        * constraints on which machines in the test pool a job can be
> +          run on (e.g. the architecure, the need for a particular
> +          processor class, the presence of SR-IOV etc).
> +
> +        * the parameters of the guest to test (e.g. distro, PV vs HVM
> +          etc).
> +
> +Operation
> +=========
> +
> +A flight is constructed by the "make-flight" script.
> +
> +"make-flight" will allocate a new flight number, create a set of jobs
> +with input runvars depending on the configuration (e.g. branch/version
> +to test).
> +
> +A flight is run by the "mg-execute-flight" script, which in turn calls
> +"sg-execute-flight". "sg-execute-flight" then spawns an instance of
> +"sg-run-job" for each job in the flight.
> +
> +"sg-run-job" encodes various recipes (sequences of steps) which are
> +referenced by each job's configuration. It then runs each of these in
> +turn, taking into account the prerequisites etc, by calling the
> +relevant "ts-*" scripts.
> +
> +When running in standalone mode it is possible to run any of these
> +steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although
> +you will need to find the correct inputs (some of which are documented
> +below) and perhaps take care of prerequisites yourself (e.g. running
> +"./sg-run-job test-armhf-armhf-xl" means you must have done
> +"./sg-runjob build-armhf" and "build-armhf-pvops" first.
> +
> +Results
> +=======
> +
> +For flights run automatically by the infrastructure an email report is
> +produced. For most normal flights this is mailed to the xen-devel
> +mailing list. The report for flight 24438 can be seen at
> +
> +    http://lists.xen.org/archives/html/xen-devel/2014-01/msg01614.html
> +
> +The report will link to a set of complete logs. Since these are
> +regularly expired due to space constraints the logs for flight 24438
> +have been archived to
> +
> +    http://xenbits.xen.org/docs/osstest-output-example/24438/
> +
> +NB: to save space any files larger than 100K have been replaced with a
> +placeholder.
> +
> +The results grid contains an overview of the flight's execution.
> +
> +The results for each job are reached by clicking the header of the
> +column in the results grid which will lead to reports such as:
> +
> +    http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/info.html
> +    http://xenbits.xen.org/docs/osstest-output-example/24438/build-amd64/info.html
> +
> +The job report contains all of the logs and build outputs associated
> +with this job.
> +
> +The logs for a step are reached by clicking the individual cells of
> +the results grid, or by clicking the list of steps on the job
> +report. In either case this will lead to a report such as
> +
> +    http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/4.ts-xen-install.log
> +
> +Additional details (e.g. serial logs, guest cfg files, etc) will be
> +available in the complete logs associated with the containing job.
> +
> +The runvars are listed in the results page for the job as "Test
> +control variables". e.g. See the end of:
> +
> +    http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/info.html
> +
> +In order to find the binaries which went into a test job you should
> +consult the results page for that job and find the relevant build
> +job. e.g.
> +
> +    http://www.chiark.greenend.org.uk/~xensrcts/logs/24438/test-amd64-amd64-xl/info.html
> +
> +lists "xenbuildjob" as "build-amd64". Therefore the input binaries are
> +found at
> +
> +    http://www.chiark.greenend.org.uk/~xensrcts/logs/24438/build-amd64/info.html
> +
> +which is linked from the top of the relevant column in the overview
> +grid.
> +
> +Script Naming Conventions
> +=========================
> +
> +Most of the scripts follow a naming convention:
> +
> +ap-*:   Adhoc push scripts
> +
> +cr-*:   Cron scripts
> +
> +cri-*:  Cron scripts (internal)
> +
> +cs-*:   Control Scripts
> +
> +mg-*:   Management scripts
> +
> +ms-*:   Management Services
> +
> +sg-*:   ?
> +
> +ts-*:   Test Step scripts.
> +
> +Jobs
> +====
> +
> +The names of jobs follow some common patterns:
> +
> +    build-$ARCH
> +
> +        Build Xen for $ARCH
> +
> +    build-$ARCH-xend
> +
> +        Build Xen for $ARCH, with xend enabled
> +
> +    build-$ARCH-pvops
> +
> +        Build an upstream ("pvops") kernel for $ARCH
> +
> +build-$ARCH-oldkern
> +
> +        Build the old "linux-2.6.18-xen" tree for $ARCH
> +
> +test-$XENARCH-$DOM0ARCH-<CASE>
> +
> +        A test <CASE> running a $XENARCH hypervisor with a $DOM0ARCH
> +        dom0.
> +
> +        Some tests also have a -$DOMUARCH suffix indicating the
> +        obvious thing.
> +
> +NB: $ARCH (and $XENARCH etc) are Debian arch names, i386, amd64, armhf.
> +
> +Standalone Mode
> +===============
> +
>  To run osstest in standalone mode:
>  
>   - You need to install
> @@ -18,7 +212,7 @@ To run osstest in standalone mode:
>     gives you the "branch" consisting of tests run for the xen-unstable
>     push gate.  You need to select a job.  The list of available jobs
>     is that shown in the publicly emailed test reports on xen-devel, eg
> -     http://lists.xen.org/archives/html/xen-devel/2013-08/msg02529.html
> +     http://lists.xen.org/archives/html/xen-devel/2014-01/msg01614.html
>  
>     If you don't want to repro one of those and don't know how to
>     choose a job, choose one of
> @@ -26,6 +220,12 @@ To run osstest in standalone mode:
>  
>   - Run ./standalone-reset
>  
> +   This will call "make-flight" for you to create a flight targetting
> +   xen-unstable (this can be adjusted by passing parameters to
> +   standalone-reset). By default the flight identifier is
> +   "standalone". standalone-reset will also make sure that certain
> +   bits of static data are available (e.g. Debian installer images)
> +
>   - Then you can run
>        ./sg-run-job <job>
>     to run that job on the default host.  NB in most cases this will

Re: [PATCH OSSTEST] README: Add some core concepts and terminology

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 573 bytes --]

On mar, 2014-01-28 at 10:46 +0000, Ian Campbell wrote:
> I should have said this yesterday on the docs day, but: Ping
> 
Actually, is it ok for someone to ping on someone else's patch?
Well, I really think this patch would be a super-cool one to have in,
so, I'll take my chances.

Ping. :-)

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH OSSTEST] README: Add some core concepts and terminology

[Ian Jackson]

Dario Faggioli writes ("Re: [Xen-devel] [PATCH OSSTEST] README: Add some core concepts and terminology"):
> On mar, 2014-01-28 at 10:46 +0000, Ian Campbell wrote:
> > I should have said this yesterday on the docs day, but: Ping
> > 
> Actually, is it ok for someone to ping on someone else's patch?
> Well, I really think this patch would be a super-cool one to have in,
> so, I'll take my chances.

That's fine.  I have made a new branch for generally-good patches
which are not related to (a) the 4.4 release and (b) my efforts to
evacuate the dying server "woking".

This branch is here:
  http://xenbits.xen.org/gitweb/?p=people/iwj/osstest.git;a=shortlog;h=refs/heads/wip.rebasing

I have pushed this patch onto it.  And I'll throw it (and anything
else this branch accumulates) into the osstest push gate when the
current crop of stuff is done.

Thanks,
Ian.