From: Andrew Cooper <andrew.cooper3@citrix.com>
To: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Xen-devel <xen-devel@lists.xenproject.org>,
Wei Liu <wei.liu2@citrix.com>
Subject: Re: [PATCH XTF 3/3] xtf-runner: regularise runner exit code
Date: Mon, 25 Jul 2016 13:09:10 +0100 [thread overview]
Message-ID: <9a4ad5ca-d0c4-5ac8-f18e-a39df869b05b@citrix.com> (raw)
In-Reply-To: <22421.63298.204024.286027@mariner.uk.xensource.com>
On 25/07/16 12:25, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [PATCH XTF 3/3] xtf-runner: regularise runner exit code"):
>> On 22/07/16 10:29, Wei Liu wrote:
>>> This would cause a FAILURE to overwrite previous ERROR result. Is that
>>> what you want?
>> When running more than one test, the overall result should be the most
>> severe. So yes, a subsequent FAILURE should override an ERROR.
> "Error" is surely a more severe problem than "Failure".
No. Error means "something unexpected went wrong", while Failure is
"the test worked, and identified that the area under test is defective".
Error is typically a test case bug or infrastructure issue.
> In particular, Error might mask a failure.
Indeed it could. Either should cause a prompt investigation and effort
to resolve the issues, but one must be more severe than the other and
this is the way things are specified.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
next prev parent reply other threads:[~2016-07-25 12:09 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-21 15:44 [PATCH XTF 0/3] Runner exit code clean up Wei Liu
2016-07-21 15:44 ` [PATCH XTF 1/3] xtf-runner: sync all test states Wei Liu
2016-07-21 18:49 ` Andrew Cooper
2016-07-21 15:44 ` [PATCH XTF 2/3] xtf-runner: provide a set of exit codes for different states Wei Liu
2016-07-21 16:27 ` Ian Jackson
2016-07-21 16:29 ` Wei Liu
2016-07-21 16:37 ` Wei Liu
2016-07-21 18:56 ` Andrew Cooper
2016-07-21 15:44 ` [PATCH XTF 3/3] xtf-runner: regularise runner exit code Wei Liu
2016-07-21 19:04 ` Andrew Cooper
2016-07-22 9:29 ` Wei Liu
2016-07-22 9:35 ` Andrew Cooper
2016-07-25 11:25 ` Ian Jackson
2016-07-25 12:09 ` Andrew Cooper [this message]
2016-07-25 17:05 ` Ian Jackson
2016-07-26 8:57 ` Wei Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9a4ad5ca-d0c4-5ac8-f18e-a39df869b05b@citrix.com \
--to=andrew.cooper3@citrix.com \
--cc=ian.jackson@eu.citrix.com \
--cc=wei.liu2@citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.