From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
To: Rob Clark <robdclark@gmail.com>, Daniel Vetter <daniel@ffwll.ch>
Cc: Jessica Zhang <quic_jesszhan@quicinc.com>,
"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
linux-arm-msm <linux-arm-msm@vger.kernel.org>,
Maarten Lankhorst <maarten.lankhorst@linux.intel.com>,
Maxime Ripard <mripard@kernel.org>,
Thomas Zimmermann <tzimmermann@suse.de>,
David Airlie <airlied@linux.ie>, Jonathan Corbet <corbet@lwn.net>,
Sean Paul <sean@poorly.run>,
Abhinav Kumar <quic_abhinavk@quicinc.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
dri-devel <dri-devel@lists.freedesktop.org>,
freedreno <freedreno@lists.freedesktop.org>
Subject: Re: [Freedreno] [RFC v2] drm/msm: Add initial ci/ subdirectory
Date: Thu, 12 May 2022 15:28:16 +0200 [thread overview]
Message-ID: <79d79110-9fbc-0e96-d17e-68a1f8f2c224@collabora.com> (raw)
In-Reply-To: <CAF6AEGthpxPLxyt_i-aUFgW485hA5qw+xXcJ3gKQUJ+fM=ZBhg@mail.gmail.com>
On 5/11/22 7:46 PM, Rob Clark wrote:
> On Wed, May 11, 2022 at 10:12 AM Daniel Vetter <daniel@ffwll.ch> wrote:
>>
>> On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
>>>
>>> On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
>>> <quic_jesszhan@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
>>>>> And use it to store expectations about what the drm/msm driver is
>>>>> supposed to pass in the IGT test suite.
>>>>>
>>>>> Also include a configuration file that points to the out-of-tree CI
>>>>> scripts.
>>>>>
>>>>> By storing the test expectations along the code we can make sure both
>>>>> stay in sync with each other, and so we can know when a code change
>>>>> breaks those expectations.
>>>>>
>>>>> This will allow all contributors to drm/msm to reuse the infrastructure
>>>>> already in gitlab.freedesktop.org to test the driver on several
>>>>> generations of the hardware.
>>>>>
>>>>> v2:
>>>>> - Fix names of result expectation files to match SoC
>>>>> - Don't execute tests that are going to skip on all boards
>>>>>
>>>>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>>>>> ---
>>>>> Documentation/gpu/msm_automated_testing.rst | 70 +++++++++
>>>>> drivers/gpu/drm/msm/ci/gitlab-ci.yml | 11 ++
>>>>> drivers/gpu/drm/msm/ci/msm.testlist | 148 ++++++++++++++++++
>>>>> .../gpu/drm/msm/ci/msm_apq8016_results.txt | 140 +++++++++++++++++
>>>>> .../gpu/drm/msm/ci/msm_apq8096_results.txt | 140 +++++++++++++++++
>>>>> drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
>>>>> drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
>>>>> 7 files changed, 791 insertions(+)
>>>>> create mode 100644 Documentation/gpu/msm_automated_testing.rst
>>>>> create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
>>>>> create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
>>>>> create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
>>>>> create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
>>>>> create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>>>>> create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
>>>>>
>
> [snip]
>
>>>>> diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>>>>> new file mode 100644
>>>>> index 000000000000..01f7b4b399b5
>>>>> --- /dev/null
>>>>> +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>>>>> @@ -0,0 +1,141 @@
>>>>> +igt@core_auth@getclient-simple,dmesg-warn
>>>>> +igt@core_auth@getclient-master-drop,pass
>>>>> +igt@core_auth@basic-auth,pass
>>>>> +igt@core_auth@many-magics,pass
>>>>> +igt@core_getclient,pass
>>>>> +igt@core_getstats,pass
>>>>> +igt@core_getversion,pass
>>>>> +igt@core_setmaster_vs_auth,pass
>>>>> +igt@drm_read@invalid-buffer,pass
>>>>> +igt@drm_read@fault-buffer,pass
>>>>> +igt@drm_read@empty-block,pass
>>>>> +igt@drm_read@empty-nonblock,pass
>>>>> +igt@drm_read@short-buffer-block,pass
>>>>> +igt@drm_read@short-buffer-nonblock,pass
>>>>> +igt@drm_read@short-buffer-wakeup,pass
>>>>> +igt@kms_addfb_basic@unused-handle,pass
>>>>> +igt@kms_addfb_basic@unused-pitches,pass
>>>>> +igt@kms_addfb_basic@unused-offsets,pass
>>>>> +igt@kms_addfb_basic@unused-modifier,pass
>>>>> +igt@kms_addfb_basic@legacy-format,dmesg-warn
>>>>> +igt@kms_addfb_basic@no-handle,pass
>>>>> +igt@kms_addfb_basic@basic,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-0,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-32,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-63,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-128,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-256,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-1024,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-999,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-65536,pass
>>>>> +igt@kms_addfb_basic@size-max,pass
>>>>> +igt@kms_addfb_basic@too-wide,pass
>>>>> +igt@kms_addfb_basic@too-high,dmesg-warn
>>>>
>>>> For test results on Trogdor, is is possible to have them be
>>>> success/fail/skip only?
>>>>
>>>> Results such as dmesg-warn/dmesg-fail are igt_runner specific and
>>>> because there isn't support for igt_runner on ChromeOS, they will be
>>>> difficult to replicate and debug.
>>>
>>> Actually, I wonder if it would be better to just treat
>>> dmesg-warn/dmesg-fail as pass/fail? I'd noticed some flakes on
>>> rockchip which looked just like unrelated dmesg msg which just
>>> happened to show up while the test was running.
>>
>> This is kinda the reason behind standardizing on drm dmesg logging, so
>> that we have some chances at filtering stuff out. Not sure that's a
>> good idea, since when your entire box splats and lockdep is dead, then
>> continuing to run drm tests is still fairly pointless.
>
> I'm not sure if we are using it yet for drm-ci, but for mesa-ci we
> monitor dmesg (over serial port, from the controller) for splats, so
> we already have the tech for restarting or aborting the CI run. We
> don't need igt-runner to tell us.
Yep, these scripts are currently being used as-is from Mesa, so we got
that functionality for free.
>> I think this is another reason why trying at least to standardize this
>> stuff over drivers would be pretty good idea.
>>
>>> Additionally, some of the tests, like msm_recovery, are *expected* to
>>> generate some dmesg spam since they are intentionally triggering GPU
>>> hangs to test the recovery mechanism.
>>
>> Uh I don't like that. It just allows userspace to spam dmesg, which
>> doesn't seem like a great idea. That's at least why i915 dumps these
>> at a lower level, and in the past had a special "I'm going to whack
>> the gpu real hard expect hangs" knob in debugfs.
>>
>> Having tests which intentionally spam dmesg above info level isn't
>> really good since then you need endless amounts of test-specific
>> encoding of what is considered a success and what not. Like when a
>> backmerge breaks a testcases which is already at dmesg-fail, is that
>> bad or not? Probably bad, but was the situation before that really
>> good or already kinda on fire?
>
> I guess I could add some debugfs knobs to squelch the dmesg msgs on
> gpu hangs. In the normal case, I'd prefer that gpu hangs are not
> silent.. since that is something we get in feedback reports if a user
> (or dogfooder) reports a bug.
>
> The rockchip case I mentioned was some unrelated dmesg about
> linktraining failing.. presumably because there was no display
> attached? IDK, I didn't look too closely. But my point is we could
> be getting unrelated and asynchronous dmesg spam, even from other
> kernel subsystems. Letting that be part of the test results just
> sounds like asking for flakes.
I think some drivers are currently a bit too buggy to behave reliably
under CI unless one reduces coverage (rockchip on rk3399, for example).
And some other drivers (in other subsystems as well) could do with a
review of what they print to the console. I guess these are things we
could and probably should fix?
Cheers,
Tomeu
> BR,
> -R
>
>> -Daniel
>>
>>> BR,
>>> -R
>>>
next prev parent reply other threads:[~2022-05-12 13:28 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20220510070140.45407-1-tomeu.vizoso@collabora.com>
2022-05-10 14:13 ` [RFC v2] drm/msm: Add initial ci/ subdirectory Tomeu Vizoso
2022-05-10 19:39 ` [Freedreno] " Jessica Zhang
2022-05-10 20:25 ` Rob Clark
2022-05-11 17:12 ` Daniel Vetter
2022-05-11 17:46 ` Rob Clark
2022-05-11 19:14 ` Daniel Vetter
2022-05-11 20:32 ` Rob Clark
2022-05-11 21:09 ` Daniel Vetter
2022-05-12 13:28 ` Tomeu Vizoso [this message]
2022-05-12 14:02 ` Daniel Vetter
2022-05-11 4:25 ` Tomeu Vizoso
2022-05-11 5:06 ` Adding CI results to the kernel tree was " Dave Airlie
2022-05-11 6:22 ` Greg Kroah-Hartman
2022-05-11 10:26 ` Michel Dänzer
2022-05-11 11:50 ` Greg Kroah-Hartman
2022-05-11 13:33 ` [Freedreno] " Rob Clark
2022-05-11 16:43 ` Daniel Vetter
2022-05-11 17:23 ` Rob Clark
2022-05-12 2:24 ` Theodore Ts'o
2022-05-11 17:33 ` Linus Torvalds
2022-05-11 18:39 ` Rob Clark
2022-05-11 19:08 ` Linus Torvalds
2022-05-11 19:12 ` Linus Torvalds
2022-05-11 20:14 ` [Freedreno] " Rob Clark
2022-05-11 20:06 ` Rob Clark
2022-05-11 19:39 ` Daniel Vetter
2022-05-11 6:15 ` [RFC v3] " Tomeu Vizoso
2022-05-11 13:20 ` Rob Clark
2022-05-11 14:03 ` Tomeu Vizoso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=79d79110-9fbc-0e96-d17e-68a1f8f2c224@collabora.com \
--to=tomeu.vizoso@collabora.com \
--cc=airlied@linux.ie \
--cc=corbet@lwn.net \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=freedreno@lists.freedesktop.org \
--cc=linux-arm-msm@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=maarten.lankhorst@linux.intel.com \
--cc=mripard@kernel.org \
--cc=quic_abhinavk@quicinc.com \
--cc=quic_jesszhan@quicinc.com \
--cc=robdclark@gmail.com \
--cc=sean@poorly.run \
--cc=tzimmermann@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).