Re: [PATCH v5 9/9] drm: selftest: convert drm_mm selftest to KUnit

[David Gow <davidgow@google.com>]

[-- Attachment #1: Type: text/plain, Size: 2987 bytes --]

On Tue, 25 Jul 2023 at 16:38, Maxime Ripard <mripard@kernel.org> wrote:
>
> Hi,
>
> On Thu, Apr 27, 2023 at 03:14:39PM +0200, Maxime Ripard wrote:
> > Hi,
> >
> > On Fri, Jul 08, 2022 at 05:30:52PM -0300, Maíra Canal wrote:
> > > From: Arthur Grillo <arthur.grillo@usp.br>
> > >
> > > Considering the current adoption of the KUnit framework, convert the
> > > DRM mm selftest to the KUnit API.
> > >
> > > Signed-off-by: Arthur Grillo <arthur.grillo@usp.br>
> > > Tested-by: David Gow <davidgow@google.com>
> > > Acked-by: Daniel Latypov <dlatypov@google.com>
> > > Reviewed-by: Javier Martinez Canillas <javierm@redhat.com>
> > > Signed-off-by: Maíra Canal <maira.canal@usp.br>
> >
> > I'm very late to the party, but I'd like to discuss that patch some more.
> >
> > Two tests (drm_test_mm_reserve, drm_test_mm_insert) in it take a super
> > long time to run (about 30s each on my machine).
> >
> > If we run all the DRM tests and VC4 tests, each of those two are longer
> > to run than all the ~300 tests combined. About 100 times longer.
> >
> > I don't think that running for so long is reasonable, and for multiple
> > reasons:
> >
> >   - While I don't know drm_mm well, it doesn't look like any of those
> >     tests do something that really should take this long. I'm especially
> >     skeptical about the fact that we test each operation 8192 times by
> >     default.
> >
> >   - It makes using kunit more tedious than it should be. Like I said, on
> >     a very capable machine, running the all the DRM and VC4 tests takes
> >     about 50s with those two tests, ~0.4s without.
> >
> >   - The corollary is that it will get in the way of people that really
> >     want to use kunit will just remove those tests before doing so,
> >     defeating the original intent.
> >
> >
> > I understand that it came from selftests initially, but I think we
> > should rewrite the tests entirely to have smaller, faster tests. It's
> > not clear to me why those tests are as complicated as they are though.
> >
> > Also, going forward we should probably put disencourage tests running
> > that long. Could Kunit timeout/warn after a while if a test is taking
> > more than X seconds to run?
>
> I'd still like to address this. We spend ~90% of the DRM kunit tests
> execution time executing those two tests, which doesn't seem like a
> reasonable thing to do.

FWIW, KUnit is going to add a "speed" attribute for tests, so that
it's easy to skip tests which are slow:
https://lore.kernel.org/linux-kselftest/20230724162834.1354164-3-rmoar@google.com/T/#u

This would allow the slow tests to be marked using KUNIT_CASE_SLOW(),
and then be run via kunit.py --filter "speed>slow".

It obviously doesn't make the tests themselves any faster, but could
at least make it possible to run only the fast tests during
development, and the full, slower set before sending the patches out
(or in CI), for example.

-- David

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]