From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:49895)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <peter.maydell@linaro.org>) id 1YhevP-0003xf-N3
	for qemu-devel@nongnu.org; Mon, 13 Apr 2015 10:01:25 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <peter.maydell@linaro.org>) id 1YhevL-0005cM-W1
	for qemu-devel@nongnu.org; Mon, 13 Apr 2015 10:01:19 -0400
Received: from mail-ig0-f169.google.com ([209.85.213.169]:36218)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <peter.maydell@linaro.org>) id 1YhevL-0005cC-On
	for qemu-devel@nongnu.org; Mon, 13 Apr 2015 10:01:15 -0400
Received: by igblo3 with SMTP id lo3so46105038igb.1
	for <qemu-devel@nongnu.org>; Mon, 13 Apr 2015 07:01:15 -0700 (PDT)
MIME-Version: 1.0
In-Reply-To: <CAFEAcA_1FdF9CAOfGxL2oF2BhCAyjQaHCcR6TBjQf7KW3b3yrg@mail.gmail.com>
References: <1428777292-24628-1-git-send-email-afaerber@suse.de>
	<CAFEAcA_1FdF9CAOfGxL2oF2BhCAyjQaHCcR6TBjQf7KW3b3yrg@mail.gmail.com>
From: Peter Maydell <peter.maydell@linaro.org>
Date: Mon, 13 Apr 2015 15:00:54 +0100
Message-ID: <CAFEAcA9ZUrwuvNKX9jgpT78MgVu3NCoYwqis-bQg9RQpGh6QQg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [Qemu-devel] [PATCH for-2.3? 0/7] tests: Fix TCG make test
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: =?UTF-8?Q?Andreas_F=C3=A4rber?= <afaerber@suse.de>
Cc: QEMU Developers <qemu-devel@nongnu.org>, Richard Henderson <rth@twiddle.net>

On 13 April 2015 at 12:36, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 11 April 2015 at 19:34, Andreas F=C3=A4rber <afaerber@suse.de> wrote:
>> Hello,
>>
>> In my quest to improve our test coverage, in light of the recent cpu_cop=
y() breakage,
>> I've taken a stab at make test and actually managed to get most of it to=
 work on x86_64.
>
> Thanks for tackling this cleanup; to answer the question
> in the subject, I think that at this point in the release
> cycle and given that 'make test' has always been somewhere
> between flaky and broken we should not put this in 2.3.

[summary of a comment from IRC]
Also, even with this patchset it looks like obvious test failures
(eg "reference output differs from actual" and "test segvs")
don't turn into "'make test' returns an error status, so it
doesn't seem to me that "make test" is actually testing
anything usefully at all. I would favour for 2.3 sticking
with our current approach (viz, telling people to use 'make
check', not 'make test'), and for 2.4 trying to integrate
these tests into 'make check', fixing them up in the
process.

thanks
-- PMM