amdgpu: ratelimit on vm faults to avoid spaming console

amdgpu: ratelimit on vm faults to avoid spaming console

[Edward O'Callaghan]

These are rather minor however should help stop some folks
machines grinding to a halt when a userspace application somehow
gets the GPU into some horrible state causing the console to fill
very quickly. Applies on top of master.

Please kindly review,

Edward O'Callaghan (2):
 [PATCH v2 1/2] amdgpu: Use dev_err() over vanilla printk() in
 [PATCH v2 2/2] amdgpu: Wrap dev_err() calls on vm faults with
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH v2 1/2] amdgpu: Use dev_err() over vanilla printk() in vm_decode_fault()

[Edward O'Callaghan]

Signed-off-by: Edward O'Callaghan <funfunctor@folklore1984.net>
---
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 2 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
index aa0c4b9..a0df431 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
@@ -711,7 +711,7 @@ static void gmc_v7_0_vm_decode_fault(struct amdgpu_device *adev,
 	mc_id = REG_GET_FIELD(status, VM_CONTEXT1_PROTECTION_FAULT_STATUS,
 			      MEMORY_CLIENT_ID);
 
-	printk("VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n",
+	dev_err(adev->dev, "VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n",
 	       protections, vmid, addr,
 	       REG_GET_FIELD(status, VM_CONTEXT1_PROTECTION_FAULT_STATUS,
 			     MEMORY_CLIENT_RW) ?
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
index a16b220..7285294 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
@@ -837,7 +837,7 @@ static void gmc_v8_0_vm_decode_fault(struct amdgpu_device *adev,
 	mc_id = REG_GET_FIELD(status, VM_CONTEXT1_PROTECTION_FAULT_STATUS,
 			      MEMORY_CLIENT_ID);
 
-	printk("VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n",
+	dev_err(adev->dev, "VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n",
 	       protections, vmid, addr,
 	       REG_GET_FIELD(status, VM_CONTEXT1_PROTECTION_FAULT_STATUS,
 			     MEMORY_CLIENT_RW) ?
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH v2 2/2] amdgpu: Wrap dev_err() calls on vm faults with printk_ratelimit()

[Edward O'Callaghan]

It can be the case that upon GPU page faults we start trashing
the logs, and so let us ratelimit here to avoid that.

V2. Fix issue where calling dev_err_ratelimited separately for
    each line means that some lines corresponding to a single
    VM fault may or may not appear depending on the rate.
    - Michel Dänzer.

Signed-off-by: Edward O'Callaghan <funfunctor@folklore1984.net>
---
 drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c | 16 +++++++++-------
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 16 +++++++++-------
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 16 +++++++++-------
 3 files changed, 27 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
index b13c8aa..1721c4a 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
@@ -1007,13 +1007,15 @@ static int gmc_v6_0_process_interrupt(struct amdgpu_device *adev,
 	if (amdgpu_vm_fault_stop == AMDGPU_VM_FAULT_STOP_FIRST)
 		gmc_v6_0_set_fault_enable_default(adev, false);
 
-	dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n",
-		entry->src_id, entry->src_data);
-	dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x%08X\n",
-		addr);
-	dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n",
-		status);
-	gmc_v6_0_vm_decode_fault(adev, status, addr, 0);
+	if (printk_ratelimit()) {
+		dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n",
+			entry->src_id, entry->src_data);
+		dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x%08X\n",
+			addr);
+		dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n",
+			status);
+		gmc_v6_0_vm_decode_fault(adev, status, addr, 0);
+	}
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
index a0df431..940857e 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
@@ -1198,13 +1198,15 @@ static int gmc_v7_0_process_interrupt(struct amdgpu_device *adev,
 	if (amdgpu_vm_fault_stop == AMDGPU_VM_FAULT_STOP_FIRST)
 		gmc_v7_0_set_fault_enable_default(adev, false);
 
-	dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n",
-		entry->src_id, entry->src_data);
-	dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x%08X\n",
-		addr);
-	dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n",
-		status);
-	gmc_v7_0_vm_decode_fault(adev, status, addr, mc_client);
+	if (printk_ratelimit()) {
+		dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n",
+			entry->src_id, entry->src_data);
+		dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x%08X\n",
+			addr);
+		dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n",
+			status);
+		gmc_v7_0_vm_decode_fault(adev, status, addr, mc_client);
+	}
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
index 7285294..77690f8 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
@@ -1242,13 +1242,15 @@ static int gmc_v8_0_process_interrupt(struct amdgpu_device *adev,
 	if (amdgpu_vm_fault_stop == AMDGPU_VM_FAULT_STOP_FIRST)
 		gmc_v8_0_set_fault_enable_default(adev, false);
 
-	dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n",
-		entry->src_id, entry->src_data);
-	dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x%08X\n",
-		addr);
-	dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n",
-		status);
-	gmc_v8_0_vm_decode_fault(adev, status, addr, mc_client);
+	if (printk_ratelimit()) {
+		dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n",
+			entry->src_id, entry->src_data);
+		dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x%08X\n",
+			addr);
+		dev_err(adev->dev, "  VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n",
+			status);
+		gmc_v8_0_vm_decode_fault(adev, status, addr, mc_client);
+	}
 
 	return 0;
 }
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: amdgpu: ratelimit on vm faults to avoid spaming console

[Christian König]

Am 07.11.2016 um 05:35 schrieb Edward O'Callaghan:
> These are rather minor however should help stop some folks
> machines grinding to a halt when a userspace application somehow
> gets the GPU into some horrible state causing the console to fill
> very quickly. Applies on top of master.
>
> Please kindly review,

Clearly looks like a good idea to me. Patch are Reviewed-by: Christian 
König <christian.koenig@amd.com>.

Regards,
Christian.

>
> Edward O'Callaghan (2):
>   [PATCH v2 1/2] amdgpu: Use dev_err() over vanilla printk() in
>   [PATCH v2 2/2] amdgpu: Wrap dev_err() calls on vm faults with
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx


_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: amdgpu: ratelimit on vm faults to avoid spaming console

[Alex Deucher]

On Sun, Nov 6, 2016 at 11:35 PM, Edward O'Callaghan
<funfunctor@folklore1984.net> wrote:
> These are rather minor however should help stop some folks
> machines grinding to a halt when a userspace application somehow
> gets the GPU into some horrible state causing the console to fill
> very quickly. Applies on top of master.

I'm generally ok with the idea.  My only concern would be if this
would adversely affect the radeon GPUVM debugging options in mesa and
piglit. Nicolai? Marek?

Alex
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: amdgpu: ratelimit on vm faults to avoid spaming console

[StDenis, Tom]

[-- Attachment #1.1: Type: text/plain, Size: 1500 bytes --]

Could we provide fault information through a ring buffer and a debugfs or drm ioctl interface?


Tom


________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of Alex Deucher <alexdeucher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Sent: Monday, November 7, 2016 13:35
To: Edward O'Callaghan; Nicolai Hähnle; Marek Olsak
Cc: Michel Dänzer; amd-gfx list
Subject: Re: amdgpu: ratelimit on vm faults to avoid spaming console

On Sun, Nov 6, 2016 at 11:35 PM, Edward O'Callaghan
<funfunctor-dczkZgxz+BNUPWh3PAxdjQ@public.gmane.org> wrote:
> These are rather minor however should help stop some folks
> machines grinding to a halt when a userspace application somehow
> gets the GPU into some horrible state causing the console to fill
> very quickly. Applies on top of master.

I'm generally ok with the idea.  My only concern would be if this
would adversely affect the radeon GPUVM debugging options in mesa and
piglit. Nicolai? Marek?

Alex
_______________________________________________
amd-gfx mailing list
amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
amd-gfx Info Page - lists.freedesktop.org<https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives. Using amd-gfx: To post a message to all the list members, send email ...




[-- Attachment #1.2: Type: text/html, Size: 4239 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: amdgpu: ratelimit on vm faults to avoid spaming console

[Christian König]

[-- Attachment #1.1: Type: text/plain, Size: 2135 bytes --]

I would rather say that we need to add this to the per context lockup 
informations.

Marek had a rather good start for this, but from the VM fault handler it 
is actually rather tricky to figure out reliable who caused it.

Because of this we didn't followed this approach to the end.

Christian.

Am 07.11.2016 um 19:42 schrieb StDenis, Tom:
>
> Could we provide fault information through a ring buffer and a debugfs 
> or drm ioctl interface?
>
>
> Tom
>
>
>
> ------------------------------------------------------------------------
> *From:* amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of 
> Alex Deucher <alexdeucher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> *Sent:* Monday, November 7, 2016 13:35
> *To:* Edward O'Callaghan; Nicolai Hähnle; Marek Olsak
> *Cc:* Michel Dänzer; amd-gfx list
> *Subject:* Re: amdgpu: ratelimit on vm faults to avoid spaming console
> On Sun, Nov 6, 2016 at 11:35 PM, Edward O'Callaghan
> <funfunctor-dczkZgxz+BNUPWh3PAxdjQ@public.gmane.org> wrote:
> > These are rather minor however should help stop some folks
> > machines grinding to a halt when a userspace application somehow
> > gets the GPU into some horrible state causing the console to fill
> > very quickly. Applies on top of master.
>
> I'm generally ok with the idea.  My only concern would be if this
> would adversely affect the radeon GPUVM debugging options in mesa and
> piglit. Nicolai? Marek?
>
> Alex
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> amd-gfx Info Page - lists.freedesktop.org 
> <https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
> lists.freedesktop.org
> To see the collection of prior postings to the list, visit the amd-gfx 
> Archives. Using amd-gfx: To post a message to all the list members, 
> send email ...
>
>
>
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx



[-- Attachment #1.2: Type: text/html, Size: 8126 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: amdgpu: ratelimit on vm faults to avoid spaming console

[Edward O'Callaghan]

[-- Attachment #1.1.1: Type: text/plain, Size: 3321 bytes --]

I had a mini discussion about this issue the other day on the IRC
channel. Essentially yes - we would want to map back from the offending
context back to the pid and I suppose kill it to restore a 'good state'.
I don't know yet how to go about implementing such a mechanism so any
suggestions would be really helpful to me!

I think, when the driver does winds up in these kinds of state, for us
to have the ability to provide something meaningful for the user to do
to file a bug report much like how the intel driver does would be a nice
addition.

In any case, this point of those patches were just to give some
breathing room and allow for machines to stay responsive enough to allow
for further debug and maintain access to a working console for the user.

Cheers,
Edward.

On 11/08/2016 09:10 PM, Christian König wrote:
> I would rather say that we need to add this to the per context lockup
> informations.
> 
> Marek had a rather good start for this, but from the VM fault handler it
> is actually rather tricky to figure out reliable who caused it.
> 
> Because of this we didn't followed this approach to the end.
> 
> Christian.
> 
> Am 07.11.2016 um 19:42 schrieb StDenis, Tom:
>>
>> Could we provide fault information through a ring buffer and a debugfs
>> or drm ioctl interface?
>>
>>
>> Tom
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of
>> Alex Deucher <alexdeucher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>> *Sent:* Monday, November 7, 2016 13:35
>> *To:* Edward O'Callaghan; Nicolai Hähnle; Marek Olsak
>> *Cc:* Michel Dänzer; amd-gfx list
>> *Subject:* Re: amdgpu: ratelimit on vm faults to avoid spaming console
>>  
>> On Sun, Nov 6, 2016 at 11:35 PM, Edward O'Callaghan
>> <funfunctor-dczkZgxz+BNUPWh3PAxdjQ@public.gmane.org> wrote:
>> > These are rather minor however should help stop some folks
>> > machines grinding to a halt when a userspace application somehow
>> > gets the GPU into some horrible state causing the console to fill
>> > very quickly. Applies on top of master.
>>
>> I'm generally ok with the idea.  My only concern would be if this
>> would adversely affect the radeon GPUVM debugging options in mesa and
>> piglit. Nicolai? Marek?
>>
>> Alex
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>> amd-gfx Info Page - lists.freedesktop.org
>> <https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
>> lists.freedesktop.org
>> To see the collection of prior postings to the list, visit the amd-gfx
>> Archives. Using amd-gfx: To post a message to all the list members,
>> send email ...
>>
>>
>>
>>
>>
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> 
> 
> 
> 
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> 


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amdgpu: ratelimit on vm faults to avoid spaming console

[Edward O'Callaghan]

These are rather minor however should help stop some folks
machines grinding to a halt when a userspace application somehow
gets the GPU into some horrible state causing the console to fill
very quickly. Applies on top of master.

Please kindly review,

Edward O'Callaghan (2):
 [PATCH 1/2] amdgpu: Use dev_err() over vanilla printk() in
 [PATCH 2/2] amdgpu: Use 'dev_err_ratelimited()' over 'dev_err()' on
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx