From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46B8CC4360C for ; Fri, 27 Sep 2019 10:48:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1F0DC20673 for ; Fri, 27 Sep 2019 10:48:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726902AbfI0Ksd (ORCPT ); Fri, 27 Sep 2019 06:48:33 -0400 Received: from foss.arm.com ([217.140.110.172]:48924 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725890AbfI0Ksc (ORCPT ); Fri, 27 Sep 2019 06:48:32 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E949628; Fri, 27 Sep 2019 03:48:31 -0700 (PDT) Received: from [10.1.196.133] (e112269-lin.cambridge.arm.com [10.1.196.133]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8EE103F67D; Fri, 27 Sep 2019 03:48:30 -0700 (PDT) From: Steven Price Subject: Re: drm_sched with panfrost crash on T820 To: Neil Armstrong , daniel@ffwll.ch, airlied@linux.ie, =?UTF-8?Q?Christian_K=c3=b6nig?= Cc: Tomeu Vizoso , linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, "open list:ARM/Amlogic Meson..." , Erico Nunes , Rob Herring References: <3fb178d8-f069-0ae2-1ed3-4ded84a71951@arm.com> Message-ID: <26ae2a4d-8df1-e8db-3060-41638ed63e2a@arm.com> Date: Fri, 27 Sep 2019 11:48:29 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <3fb178d8-f069-0ae2-1ed3-4ded84a71951@arm.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 27/09/2019 10:55, Steven Price wrote: [...] > One obvious issue with the DRM scheduler is that there is a call to > cancel_delayed_work() in drm_sched_stop() which to me looks like it > should be cancel_delayed_work_sync() to ensure that the timeout handling > has completed. > > However in the above scenario a _sync() variety would then cause a > deadlock (one thread has pfdev->reset_lock and is waiting for the other > thread which is trying to take the lock). > > So we need to update Panfrost so that it can coordinate the reset > between schedulers. Can you try something like the following (untested): And actually testing it I of course discover it doesn't quite work. We do need the cancel_delayed_work_sync() in the DRM scheduler (when stopping a different scheduler) and we need to avoid holding the reset_lock during the drm_sched_stop() call to prevent deadlocking with another thread handling a timeout. Can you give the following patch a spin? I don't have a great reproduction case, so it would be good to get some confidence it fixes the problem. ----8<---- >From 521a286789260197ae94f698932ebf369efc45ad Mon Sep 17 00:00:00 2001 From: Steven Price Date: Fri, 27 Sep 2019 11:42:40 +0100 Subject: [PATCH] drm/panfrost: Handle resetting on timeout better Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use. Modify drm_sched_stop() to call cancel_delayed_work_sync() when stopping a different scheduler to the one belonging to the passed in job. panfrost_job_timedout() is also modified to only allow one thread at a time to handle the reset. Any subsequent threads simply return assuming that the first thread will handle the situation. Signed-off-by: Steven Price --- drivers/gpu/drm/panfrost/panfrost_device.h | 2 ++ drivers/gpu/drm/panfrost/panfrost_job.c | 11 ++++++++++- drivers/gpu/drm/scheduler/sched_main.c | 5 ++++- 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/panfrost/panfrost_device.h b/drivers/gpu/drm/panfrost/panfrost_device.h index f503c566e99f..6441c7fba6c4 100644 --- a/drivers/gpu/drm/panfrost/panfrost_device.h +++ b/drivers/gpu/drm/panfrost/panfrost_device.h @@ -99,6 +99,8 @@ struct panfrost_device { unsigned long cur_volt; struct panfrost_devfreq_slot slot[NUM_JOB_SLOTS]; } devfreq; + + bool is_resetting; }; struct panfrost_mmu { diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index 05c85f45a0de..1b2019e08b43 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -388,13 +388,21 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) mutex_lock(&pfdev->reset_lock); + if (pfdev->is_resetting) { + mutex_unlock(&pfdev->reset_lock); + return; + } + pfdev->is_resetting = true; + + mutex_unlock(&pfdev->reset_lock); + for (i = 0; i < NUM_JOB_SLOTS; i++) drm_sched_stop(&pfdev->js->queue[i].sched, sched_job); if (sched_job) drm_sched_increase_karma(sched_job); - /* panfrost_core_dump(pfdev); */ + mutex_lock(&pfdev->reset_lock); panfrost_devfreq_record_transition(pfdev, js); panfrost_device_reset(pfdev); @@ -406,6 +414,7 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) for (i = 0; i < NUM_JOB_SLOTS; i++) drm_sched_start(&pfdev->js->queue[i].sched, true); + pfdev->is_resetting = false; mutex_unlock(&pfdev->reset_lock); } diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 148468447ba9..bc6d1862ec8a 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -415,7 +415,10 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) * this TDR finished and before the newly restarted jobs had a * chance to complete. */ - cancel_delayed_work(&sched->work_tdr); + if (bad->sched != sched) + cancel_delayed_work_sync(&sched->work_tdr); + else + cancel_delayed_work(&sched->work_tdr); } EXPORT_SYMBOL(drm_sched_stop); -- 2.20.1 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Price Subject: Re: drm_sched with panfrost crash on T820 Date: Fri, 27 Sep 2019 11:48:29 +0100 Message-ID: <26ae2a4d-8df1-e8db-3060-41638ed63e2a@arm.com> References: <3fb178d8-f069-0ae2-1ed3-4ded84a71951@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by gabe.freedesktop.org (Postfix) with ESMTP id A19096EE4C for ; Fri, 27 Sep 2019 10:48:32 +0000 (UTC) In-Reply-To: <3fb178d8-f069-0ae2-1ed3-4ded84a71951@arm.com> Content-Language: en-GB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Neil Armstrong , daniel@ffwll.ch, airlied@linux.ie, =?UTF-8?Q?Christian_K=c3=b6nig?= Cc: Tomeu Vizoso , linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, "open list:ARM/Amlogic Meson..." , Erico Nunes List-Id: dri-devel@lists.freedesktop.org T24gMjcvMDkvMjAxOSAxMDo1NSwgU3RldmVuIFByaWNlIHdyb3RlOgpbLi4uXQo+IE9uZSBvYnZp b3VzIGlzc3VlIHdpdGggdGhlIERSTSBzY2hlZHVsZXIgaXMgdGhhdCB0aGVyZSBpcyBhIGNhbGwg dG8KPiBjYW5jZWxfZGVsYXllZF93b3JrKCkgaW4gZHJtX3NjaGVkX3N0b3AoKSB3aGljaCB0byBt ZSBsb29rcyBsaWtlIGl0Cj4gc2hvdWxkIGJlIGNhbmNlbF9kZWxheWVkX3dvcmtfc3luYygpIHRv IGVuc3VyZSB0aGF0IHRoZSB0aW1lb3V0IGhhbmRsaW5nCj4gaGFzIGNvbXBsZXRlZC4KPiAKPiBI b3dldmVyIGluIHRoZSBhYm92ZSBzY2VuYXJpbyBhIF9zeW5jKCkgdmFyaWV0eSB3b3VsZCB0aGVu IGNhdXNlIGEKPiBkZWFkbG9jayAob25lIHRocmVhZCBoYXMgcGZkZXYtPnJlc2V0X2xvY2sgYW5k IGlzIHdhaXRpbmcgZm9yIHRoZSBvdGhlcgo+IHRocmVhZCB3aGljaCBpcyB0cnlpbmcgdG8gdGFr ZSB0aGUgbG9jaykuCj4gCj4gU28gd2UgbmVlZCB0byB1cGRhdGUgUGFuZnJvc3Qgc28gdGhhdCBp dCBjYW4gY29vcmRpbmF0ZSB0aGUgcmVzZXQKPiBiZXR3ZWVuIHNjaGVkdWxlcnMuIENhbiB5b3Ug dHJ5IHNvbWV0aGluZyBsaWtlIHRoZSBmb2xsb3dpbmcgKHVudGVzdGVkKToKCkFuZCBhY3R1YWxs eSB0ZXN0aW5nIGl0IEkgb2YgY291cnNlIGRpc2NvdmVyIGl0IGRvZXNuJ3QgcXVpdGUgd29yay4g V2UKZG8gbmVlZCB0aGUgY2FuY2VsX2RlbGF5ZWRfd29ya19zeW5jKCkgaW4gdGhlIERSTSBzY2hl ZHVsZXIgKHdoZW4Kc3RvcHBpbmcgYSBkaWZmZXJlbnQgc2NoZWR1bGVyKSBhbmQgd2UgbmVlZCB0 byBhdm9pZCBob2xkaW5nIHRoZQpyZXNldF9sb2NrIGR1cmluZyB0aGUgZHJtX3NjaGVkX3N0b3Ao KSBjYWxsIHRvIHByZXZlbnQgZGVhZGxvY2tpbmcgd2l0aAphbm90aGVyIHRocmVhZCBoYW5kbGlu ZyBhIHRpbWVvdXQuCgpDYW4geW91IGdpdmUgdGhlIGZvbGxvd2luZyBwYXRjaCBhIHNwaW4/IEkg ZG9uJ3QgaGF2ZSBhIGdyZWF0CnJlcHJvZHVjdGlvbiBjYXNlLCBzbyBpdCB3b3VsZCBiZSBnb29k IHRvIGdldCBzb21lIGNvbmZpZGVuY2UgaXQgZml4ZXMKdGhlIHByb2JsZW0uCgotLS0tODwtLS0t CkZyb20gNTIxYTI4Njc4OTI2MDE5N2FlOTRmNjk4OTMyZWJmMzY5ZWZjNDVhZCBNb24gU2VwIDE3 IDAwOjAwOjAwIDIwMDEKRnJvbTogU3RldmVuIFByaWNlIDxzdGV2ZW4ucHJpY2VAYXJtLmNvbT4K RGF0ZTogRnJpLCAyNyBTZXAgMjAxOSAxMTo0Mjo0MCArMDEwMApTdWJqZWN0OiBbUEFUQ0hdIGRy bS9wYW5mcm9zdDogSGFuZGxlIHJlc2V0dGluZyBvbiB0aW1lb3V0IGJldHRlcgoKUGFuZnJvc3Qg dXNlcyBtdWx0aXBsZSBzY2hlZHVsZXJzIChvbmUgZm9yIGVhY2ggc2xvdCwgc28gMiBpbiByZWFs aXR5KSwKYW5kIG9uIGEgdGltZW91dCBoYXMgdG8gc3RvcCBhbGwgdGhlIHNjaGVkdWxlcnMgdG8g c2FmZWx5IHBlcmZvcm0gYQpyZXNldC4gSG93ZXZlciBtb3JlIHRoYW4gb25lIHNjaGVkdWxlciBj YW4gdHJpZ2dlciBhIHRpbWVvdXQgYXQgdGhlIHNhbWUKdGltZS4gVGhpcyByYWNlIGNvbmRpdGlv biByZXN1bHRzIGluIGpvYnMgYmVpbmcgZnJlZWQgd2hpbGUgdGhleSBhcmUKc3RpbGwgaW4gdXNl LgoKTW9kaWZ5IGRybV9zY2hlZF9zdG9wKCkgdG8gY2FsbCBjYW5jZWxfZGVsYXllZF93b3JrX3N5 bmMoKSB3aGVuIHN0b3BwaW5nCmEgZGlmZmVyZW50IHNjaGVkdWxlciB0byB0aGUgb25lIGJlbG9u Z2luZyB0byB0aGUgcGFzc2VkIGluIGpvYi4KcGFuZnJvc3Rfam9iX3RpbWVkb3V0KCkgaXMgYWxz byBtb2RpZmllZCB0byBvbmx5IGFsbG93IG9uZSB0aHJlYWQgYXQgYQp0aW1lIHRvIGhhbmRsZSB0 aGUgcmVzZXQuIEFueSBzdWJzZXF1ZW50IHRocmVhZHMgc2ltcGx5IHJldHVybiBhc3N1bWluZwp0 aGF0IHRoZSBmaXJzdCB0aHJlYWQgd2lsbCBoYW5kbGUgdGhlIHNpdHVhdGlvbi4KClNpZ25lZC1v ZmYtYnk6IFN0ZXZlbiBQcmljZSA8c3RldmVuLnByaWNlQGFybS5jb20+Ci0tLQogZHJpdmVycy9n cHUvZHJtL3BhbmZyb3N0L3BhbmZyb3N0X2RldmljZS5oIHwgIDIgKysKIGRyaXZlcnMvZ3B1L2Ry bS9wYW5mcm9zdC9wYW5mcm9zdF9qb2IuYyAgICB8IDExICsrKysrKysrKystCiBkcml2ZXJzL2dw dS9kcm0vc2NoZWR1bGVyL3NjaGVkX21haW4uYyAgICAgfCAgNSArKysrLQogMyBmaWxlcyBjaGFu Z2VkLCAxNiBpbnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2RyaXZl cnMvZ3B1L2RybS9wYW5mcm9zdC9wYW5mcm9zdF9kZXZpY2UuaCBiL2RyaXZlcnMvZ3B1L2RybS9w YW5mcm9zdC9wYW5mcm9zdF9kZXZpY2UuaAppbmRleCBmNTAzYzU2NmU5OWYuLjY0NDFjN2ZiYTZj NCAxMDA2NDQKLS0tIGEvZHJpdmVycy9ncHUvZHJtL3BhbmZyb3N0L3BhbmZyb3N0X2RldmljZS5o CisrKyBiL2RyaXZlcnMvZ3B1L2RybS9wYW5mcm9zdC9wYW5mcm9zdF9kZXZpY2UuaApAQCAtOTks NiArOTksOCBAQCBzdHJ1Y3QgcGFuZnJvc3RfZGV2aWNlIHsKIAkJdW5zaWduZWQgbG9uZyBjdXJf dm9sdDsKIAkJc3RydWN0IHBhbmZyb3N0X2RldmZyZXFfc2xvdCBzbG90W05VTV9KT0JfU0xPVFNd OwogCX0gZGV2ZnJlcTsKKworCWJvb2wgaXNfcmVzZXR0aW5nOwogfTsKIAogc3RydWN0IHBhbmZy b3N0X21tdSB7CmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vcGFuZnJvc3QvcGFuZnJvc3Rf am9iLmMgYi9kcml2ZXJzL2dwdS9kcm0vcGFuZnJvc3QvcGFuZnJvc3Rfam9iLmMKaW5kZXggMDVj ODVmNDVhMGRlLi4xYjIwMTllMDhiNDMgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1L2RybS9wYW5m cm9zdC9wYW5mcm9zdF9qb2IuYworKysgYi9kcml2ZXJzL2dwdS9kcm0vcGFuZnJvc3QvcGFuZnJv c3Rfam9iLmMKQEAgLTM4OCwxMyArMzg4LDIxIEBAIHN0YXRpYyB2b2lkIHBhbmZyb3N0X2pvYl90 aW1lZG91dChzdHJ1Y3QgZHJtX3NjaGVkX2pvYiAqc2NoZWRfam9iKQogCiAJbXV0ZXhfbG9jaygm cGZkZXYtPnJlc2V0X2xvY2spOwogCisJaWYgKHBmZGV2LT5pc19yZXNldHRpbmcpIHsKKwkJbXV0 ZXhfdW5sb2NrKCZwZmRldi0+cmVzZXRfbG9jayk7CisJCXJldHVybjsKKwl9CisJcGZkZXYtPmlz X3Jlc2V0dGluZyA9IHRydWU7CisKKwltdXRleF91bmxvY2soJnBmZGV2LT5yZXNldF9sb2NrKTsK KwogCWZvciAoaSA9IDA7IGkgPCBOVU1fSk9CX1NMT1RTOyBpKyspCiAJCWRybV9zY2hlZF9zdG9w KCZwZmRldi0+anMtPnF1ZXVlW2ldLnNjaGVkLCBzY2hlZF9qb2IpOwogCiAJaWYgKHNjaGVkX2pv YikKIAkJZHJtX3NjaGVkX2luY3JlYXNlX2thcm1hKHNjaGVkX2pvYik7CiAKLQkvKiBwYW5mcm9z dF9jb3JlX2R1bXAocGZkZXYpOyAqLworCW11dGV4X2xvY2soJnBmZGV2LT5yZXNldF9sb2NrKTsK IAogCXBhbmZyb3N0X2RldmZyZXFfcmVjb3JkX3RyYW5zaXRpb24ocGZkZXYsIGpzKTsKIAlwYW5m cm9zdF9kZXZpY2VfcmVzZXQocGZkZXYpOwpAQCAtNDA2LDYgKzQxNCw3IEBAIHN0YXRpYyB2b2lk IHBhbmZyb3N0X2pvYl90aW1lZG91dChzdHJ1Y3QgZHJtX3NjaGVkX2pvYiAqc2NoZWRfam9iKQog CWZvciAoaSA9IDA7IGkgPCBOVU1fSk9CX1NMT1RTOyBpKyspCiAJCWRybV9zY2hlZF9zdGFydCgm cGZkZXYtPmpzLT5xdWV1ZVtpXS5zY2hlZCwgdHJ1ZSk7CiAKKwlwZmRldi0+aXNfcmVzZXR0aW5n ID0gZmFsc2U7CiAJbXV0ZXhfdW5sb2NrKCZwZmRldi0+cmVzZXRfbG9jayk7CiB9CiAKZGlmZiAt LWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfbWFpbi5jIGIvZHJpdmVycy9n cHUvZHJtL3NjaGVkdWxlci9zY2hlZF9tYWluLmMKaW5kZXggMTQ4NDY4NDQ3YmE5Li5iYzZkMTg2 MmVjOGEgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfbWFpbi5j CisrKyBiL2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfbWFpbi5jCkBAIC00MTUsNyAr NDE1LDEwIEBAIHZvaWQgZHJtX3NjaGVkX3N0b3Aoc3RydWN0IGRybV9ncHVfc2NoZWR1bGVyICpz Y2hlZCwgc3RydWN0IGRybV9zY2hlZF9qb2IgKmJhZCkKIAkgKiB0aGlzIFREUiBmaW5pc2hlZCBh bmQgYmVmb3JlIHRoZSBuZXdseSByZXN0YXJ0ZWQgam9icyBoYWQgYQogCSAqIGNoYW5jZSB0byBj b21wbGV0ZS4KIAkgKi8KLQljYW5jZWxfZGVsYXllZF93b3JrKCZzY2hlZC0+d29ya190ZHIpOwor CWlmIChiYWQtPnNjaGVkICE9IHNjaGVkKQorCQljYW5jZWxfZGVsYXllZF93b3JrX3N5bmMoJnNj aGVkLT53b3JrX3Rkcik7CisJZWxzZQorCQljYW5jZWxfZGVsYXllZF93b3JrKCZzY2hlZC0+d29y a190ZHIpOwogfQogCiBFWFBPUlRfU1lNQk9MKGRybV9zY2hlZF9zdG9wKTsKLS0gCjIuMjAuMQoK X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVs IG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlz dHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2705CC4360C for ; Fri, 27 Sep 2019 10:48:46 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E76EF20673 for ; Fri, 27 Sep 2019 10:48:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="UavQYzTe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E76EF20673 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-amlogic-bounces+linux-amlogic=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:References:To:Subject:From:Reply-To:Content-ID:Content-Description :Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hagZrF16LwrMSk6n/4JV28JOFXyLUJj1cc0NwZzhj04=; b=UavQYzTeragpNP hjGcmnswOdnGz4Zq7Vojj5xewPOtrwVkDxdlRnRHjEpjHkiZYEIlGG2jzwZeIcphNVGFML4yCZXre 4I36L5ydWAxHdTnLtJVVhBUUSuGD49YNzj5nrjd0UQ4EvbwLpWf4E7cjorpIc9cCnzzMEV8IiIeM8 fExIeKijP7I9em+K7NPuIFJ4zLxqx+7/cAd9U2es309A2FXITs6A4rk8uiWCb2iBj005zj0nt59dV +EpTK+a81RhTAzBFrw/s2tpZjbEEu3N+A8OA6Hpvg2rOxc3xCQwQSkHM1mXTSbV8yLU2Jah9Tv47e Q3uR28wo+dzUFNmw3F8w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iDnnj-0001F3-7Y; Fri, 27 Sep 2019 10:48:39 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iDnnc-0001DT-Is for linux-amlogic@lists.infradead.org; Fri, 27 Sep 2019 10:48:34 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E949628; Fri, 27 Sep 2019 03:48:31 -0700 (PDT) Received: from [10.1.196.133] (e112269-lin.cambridge.arm.com [10.1.196.133]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8EE103F67D; Fri, 27 Sep 2019 03:48:30 -0700 (PDT) From: Steven Price Subject: Re: drm_sched with panfrost crash on T820 To: Neil Armstrong , daniel@ffwll.ch, airlied@linux.ie, =?UTF-8?Q?Christian_K=c3=b6nig?= References: <3fb178d8-f069-0ae2-1ed3-4ded84a71951@arm.com> Message-ID: <26ae2a4d-8df1-e8db-3060-41638ed63e2a@arm.com> Date: Fri, 27 Sep 2019 11:48:29 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <3fb178d8-f069-0ae2-1ed3-4ded84a71951@arm.com> Content-Language: en-GB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190927_034832_709637_8903B3EB X-CRM114-Status: GOOD ( 22.95 ) X-BeenThere: linux-amlogic@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Herring , Tomeu Vizoso , linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, "open list:ARM/Amlogic Meson..." , Erico Nunes Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-amlogic" Errors-To: linux-amlogic-bounces+linux-amlogic=archiver.kernel.org@lists.infradead.org On 27/09/2019 10:55, Steven Price wrote: [...] > One obvious issue with the DRM scheduler is that there is a call to > cancel_delayed_work() in drm_sched_stop() which to me looks like it > should be cancel_delayed_work_sync() to ensure that the timeout handling > has completed. > > However in the above scenario a _sync() variety would then cause a > deadlock (one thread has pfdev->reset_lock and is waiting for the other > thread which is trying to take the lock). > > So we need to update Panfrost so that it can coordinate the reset > between schedulers. Can you try something like the following (untested): And actually testing it I of course discover it doesn't quite work. We do need the cancel_delayed_work_sync() in the DRM scheduler (when stopping a different scheduler) and we need to avoid holding the reset_lock during the drm_sched_stop() call to prevent deadlocking with another thread handling a timeout. Can you give the following patch a spin? I don't have a great reproduction case, so it would be good to get some confidence it fixes the problem. ----8<---- >From 521a286789260197ae94f698932ebf369efc45ad Mon Sep 17 00:00:00 2001 From: Steven Price Date: Fri, 27 Sep 2019 11:42:40 +0100 Subject: [PATCH] drm/panfrost: Handle resetting on timeout better Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use. Modify drm_sched_stop() to call cancel_delayed_work_sync() when stopping a different scheduler to the one belonging to the passed in job. panfrost_job_timedout() is also modified to only allow one thread at a time to handle the reset. Any subsequent threads simply return assuming that the first thread will handle the situation. Signed-off-by: Steven Price --- drivers/gpu/drm/panfrost/panfrost_device.h | 2 ++ drivers/gpu/drm/panfrost/panfrost_job.c | 11 ++++++++++- drivers/gpu/drm/scheduler/sched_main.c | 5 ++++- 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/panfrost/panfrost_device.h b/drivers/gpu/drm/panfrost/panfrost_device.h index f503c566e99f..6441c7fba6c4 100644 --- a/drivers/gpu/drm/panfrost/panfrost_device.h +++ b/drivers/gpu/drm/panfrost/panfrost_device.h @@ -99,6 +99,8 @@ struct panfrost_device { unsigned long cur_volt; struct panfrost_devfreq_slot slot[NUM_JOB_SLOTS]; } devfreq; + + bool is_resetting; }; struct panfrost_mmu { diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index 05c85f45a0de..1b2019e08b43 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -388,13 +388,21 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) mutex_lock(&pfdev->reset_lock); + if (pfdev->is_resetting) { + mutex_unlock(&pfdev->reset_lock); + return; + } + pfdev->is_resetting = true; + + mutex_unlock(&pfdev->reset_lock); + for (i = 0; i < NUM_JOB_SLOTS; i++) drm_sched_stop(&pfdev->js->queue[i].sched, sched_job); if (sched_job) drm_sched_increase_karma(sched_job); - /* panfrost_core_dump(pfdev); */ + mutex_lock(&pfdev->reset_lock); panfrost_devfreq_record_transition(pfdev, js); panfrost_device_reset(pfdev); @@ -406,6 +414,7 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) for (i = 0; i < NUM_JOB_SLOTS; i++) drm_sched_start(&pfdev->js->queue[i].sched, true); + pfdev->is_resetting = false; mutex_unlock(&pfdev->reset_lock); } diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 148468447ba9..bc6d1862ec8a 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -415,7 +415,10 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) * this TDR finished and before the newly restarted jobs had a * chance to complete. */ - cancel_delayed_work(&sched->work_tdr); + if (bad->sched != sched) + cancel_delayed_work_sync(&sched->work_tdr); + else + cancel_delayed_work(&sched->work_tdr); } EXPORT_SYMBOL(drm_sched_stop); -- 2.20.1 _______________________________________________ linux-amlogic mailing list linux-amlogic@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-amlogic