From: Steven Price <steven.price@arm.com>
To: "Koenig, Christian" <Christian.Koenig@amd.com>
Cc: Hillf Danton <hdanton@sina.com>,
Tomeu Vizoso <tomeu.vizoso@collabora.com>,
Neil Armstrong <narmstrong@baylibre.com>,
"airlied@linux.ie" <airlied@linux.ie>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"dri-devel@lists.freedesktop.org"
<dri-devel@lists.freedesktop.org>,
"open list:ARM/Amlogic Meson..."
<linux-amlogic@lists.infradead.org>,
Erico Nunes <nunes.erico@gmail.com>
Subject: Re: drm_sched with panfrost crash on T820
Date: Mon, 7 Oct 2019 13:47:30 +0100 [thread overview]
Message-ID: <cf6c7130-51ea-155b-cbe7-9f05281be360@arm.com> (raw)
In-Reply-To: <8e003dfd-2761-4941-8b5b-ecc186222229@email.android.com>
On 04/10/2019 17:33, Koenig, Christian wrote:
>
>
> Am 04.10.2019 18:02 schrieb Steven Price <steven.price@arm.com>:
> On 04/10/2019 16:34, Koenig, Christian wrote:
>> Am 04.10.19 um 17:27 schrieb Steven Price:
>>> On 04/10/2019 16:03, Neil Armstrong wrote:
>>>> On 04/10/2019 16:53, Grodzovsky, Andrey wrote:
>>>>> On 10/3/19 4:34 AM, Neil Armstrong wrote:
>>>>>> Hi Andrey,
>>>>>>
>>>>>> Le 02/10/2019 à 16:40, Grodzovsky, Andrey a écrit :
>>>>>>> On 9/30/19 10:52 AM, Hillf Danton wrote:
>>>>>>>> On Mon, 30 Sep 2019 11:17:45 +0200 Neil Armstrong wrote:
>>>>>>>>> Did a new run from 5.3:
>>>>>>>>>
>>>>>>>>> [ 35.971972] Call trace:
>>>>>>>>> [ 35.974391] drm_sched_increase_karma+0x5c/0xf0
>>>>>>>>> ffff000010667f38 FFFF000010667F94
>>>>>>>>> drivers/gpu/drm/scheduler/sched_main.c:335
>>>>>>>>>
>>>>>>>>> The crashing line is :
>>>>>>>>> if (bad->s_fence->scheduled.context ==
>>>>>>>>> entity->fence_context) {
>>>>>>>>>
>>>>>>>>> Doesn't seem related to guilty job.
>>>>>>>> Bail out if s_fence is no longer fresh.
>>>>>>>>
>>>>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>>>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>>>>>>> @@ -333,6 +333,10 @@ void drm_sched_increase_karma(struct drm
>>>>>>>>
>>>>>>>> spin_lock(&rq->lock);
>>>>>>>> list_for_each_entry_safe(entity, tmp, &rq->entities, list) {
>>>>>>>> + if (!smp_load_acquire(&bad->s_fence)) {
>>>>>>>> + spin_unlock(&rq->lock);
>>>>>>>> + return;
>>>>>>>> + }
>>>>>>>> if (bad->s_fence->scheduled.context ==
>>>>>>>> entity->fence_context) {
>>>>>>>> if (atomic_read(&bad->karma) >
>>>>>>>> @@ -543,7 +547,7 @@ EXPORT_SYMBOL(drm_sched_job_init);
>>>>>>>> void drm_sched_job_cleanup(struct drm_sched_job *job)
>>>>>>>> {
>>>>>>>> dma_fence_put(&job->s_fence->finished);
>>>>>>>> - job->s_fence = NULL;
>>>>>>>> + smp_store_release(&job->s_fence, 0);
>>>>>>>> }
>>>>>>>> EXPORT_SYMBOL(drm_sched_job_cleanup);
>>>>>> This fixed the problem on the 10 CI runs.
>>>>>>
>>>>>> Neil
>>>>>
>>>>> These are good news but I still fail to see how this fixes the problem -
>>>>> Hillf, do you mind explaining how you came up with this particular fix -
>>>>> what was the bug you saw ?
>>>> As Steven explained, seems the same job was submitted on both HW slots,
>>>> and then when timeout occurs each thread calls panfrost_job_timedout
>>>> which leads to drm_sched_stop() on the first call and on the
>>>> second call the job was already freed.
>>>>
>>>> Steven proposed a working fix, and this one does the same but on
>>>> the drm_sched side. This one looks cleaner, but panfrost should
>>>> not call drm_sched_stop() twice for the same job.
>>> I'm not sure that Hillf's fix is sufficient. In particular in
>>> drm_sched_increase_karma() I don't understand how the smp_load_acquire()
>>> call prevents bad->s_fence becoming NULL immediately afterwards (but
>>> admittedly the window is much smaller). But really this is just a
>>> Panfrost bug (calling drm_sched_stop() twice on the same job).
>>>
>>> The part of my change that I'd welcome feedback on is changing
>>> cancel_delayed_work() to cancel_delayed_work_sync() in drm_sched_stop()
>>> when called on different scheduler to the bad job. It's not clear to me
>>> exactly what the semantics of the function should be, and I haven't
>>> tested the effect of the change on drivers other than Panfrost.
>>
>> Yeah, at least of hand that change doesn't seem to make sense to me.
>
> We need to ensure that any other timeouts that might have started
> processing are complete before actually resetting the hardware.
> Otherwise after the reset another thread could come along and attempt to
> reset the hardware again (and cause a double free of a job).
>
> This is intentional behaviour. If you don't want the double reset in Panfrost you should probably call the cancel_sync yourself.
It's less the double reset that is the problem, more that the job gets
cleaned up twice: drm_sched_stop() will either free the job or mark it
to be freed later. By having two threads both drm_sched_stop()ing all
slots you end up with the guilty job(s) potentially being double freed.
I've move the call to cancel_delayed_work_sync() into Panfrost since I'm
not sure whether this is generically useful to other drivers.
Steve
next parent reply other threads:[~2019-10-07 12:47 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <8e003dfd-2761-4941-8b5b-ecc186222229@email.android.com>
2019-10-07 12:47 ` Steven Price [this message]
2019-09-27 8:12 drm_sched with panfrost crash on T820 Neil Armstrong
2019-09-27 9:55 ` Steven Price
2019-09-27 10:48 ` Steven Price
2019-09-27 11:27 ` Neil Armstrong
2019-09-27 11:48 ` Neil Armstrong
2019-09-27 15:00 ` Steven Price
2019-09-27 15:20 ` Neil Armstrong
2019-09-30 13:18 ` Neil Armstrong
2019-09-27 20:55 ` Grodzovsky, Andrey
2019-09-30 9:17 ` Neil Armstrong
2019-10-02 16:53 ` Grodzovsky, Andrey
2019-10-03 8:36 ` Neil Armstrong
[not found] ` <20190930145228.14000-1-hdanton@sina.com>
2019-10-02 14:40 ` Grodzovsky, Andrey
2019-10-02 14:44 ` Neil Armstrong
2019-10-03 8:34 ` Neil Armstrong
2019-10-04 14:53 ` Grodzovsky, Andrey
2019-10-04 15:03 ` Neil Armstrong
2019-10-04 15:27 ` Steven Price
2019-10-04 15:34 ` Koenig, Christian
2019-10-04 16:02 ` Steven Price
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cf6c7130-51ea-155b-cbe7-9f05281be360@arm.com \
--to=steven.price@arm.com \
--cc=Christian.Koenig@amd.com \
--cc=airlied@linux.ie \
--cc=dri-devel@lists.freedesktop.org \
--cc=hdanton@sina.com \
--cc=linux-amlogic@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=narmstrong@baylibre.com \
--cc=nunes.erico@gmail.com \
--cc=tomeu.vizoso@collabora.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).