Re: [PATCH 1/2] Add latest LTP test in autotest

From: Martin Bligh <mbligh@google.com>
To: subrata@linux.vnet.ibm.com
Cc: Mike Frysinger <vapier@gentoo.org>,
	ltp-list <ltp-list@lists.sf.net>,
	sudhir kumar <smalikphy@gmail.com>,
	Autotest mailing list <autotest@test.kernel.org>,
	Lucas Meneghel Rodrigues <mrodrigu@redhat.com>,
	Uri Lublin <uril@redhat.com>, kvm-devel <kvm@vger.kernel.org>
Subject: Re: [PATCH 1/2] Add latest LTP test in autotest
Date: Wed, 8 Jul 2009 08:05:33 -0700	[thread overview]
Message-ID: <33307c790907080805r78016b3cl21b065e2a0d50c3b@mail.gmail.com> (raw)
In-Reply-To: <1247048369.5405.35.camel@subratamodak.linux.ibm.com>

>> Yup, we can pass an excluded test list. I really wish they'd fix their
>> tests, but I've been saying that for 6 years now, and it hasn't happened
>> yet ;-(
>
> I would slightly disagree to that. 6 years is history. But, have you
> recently checked with LTP ?

I hate to be completely cynical about this, but that's exactly the same
message I get every year.

Yes, absolute, the best thing would be for someone to run all the tests,
work through all the problems, categorize them as kernel / library / distro,
and get each of them fixed. However, it's a fair chunk of work that I don't
have time to do.

So all I'm saying is that I know which of the current tests we have issues
with, and I don't want to upgrade LTP without a new set of data, and that
work being done. From previous experience, I would be extremely
surprised if there's not at least one new problem, and I'm not just going
to dump that on users.

Does the LTP project do this itself on a regular basis ... ie are you running
LTP against the latest kernel (or even some known stable kernel) and
seeing which tests are broken? If you can point me to that, I'd have much
more faith about picking this up ...

Up until this point we've not even managed to agree that PASS means
"ran as expected" and FAIL meant "something is wrong". LTP always
had "expected failures" which seems like a completely broken model
to me.

M.