* [PATCH] nouveau: fix instmem race condition around ptr stores
@ 2024-04-11 1:15 Dave Airlie
2024-04-15 9:06 ` Danilo Krummrich
0 siblings, 1 reply; 6+ messages in thread
From: Dave Airlie @ 2024-04-11 1:15 UTC (permalink / raw)
To: dri-devel; +Cc: nouveau
From: Dave Airlie <airlied@redhat.com>
Running a lot of VK CTS in parallel against nouveau, once every
few hours you might see something like this crash.
BUG: kernel NULL pointer dereference, address: 0000000000000008
PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
FS: 00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
...
? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
nvkm_vmm_iter+0x351/0xa20 [nouveau]
? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
? __lock_acquire+0x3ed/0x2170
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]
Adding any sort of useful debug usually makes it go away, so I hand
wrote the function in a line, and debugged the asm.
Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in
the nv50_instobj_acquire called from nvkm_kmap.
If Thread A and Thread B both get to nv50_instobj_acquire around
the same time, and Thread A hits the refcount_set line, and in
lockstep thread B succeeds at refcount_inc_not_zero, there is a
chance the ptrs value won't have been stored since refcount_set
is unordered. Force a memory barrier here, I picked smp_mb, since
we want it on all CPUs and it's write followed by a read.
v2: use paired smp_rmb/smp_wmb.
Cc: linux-stable
Signed-off-by: Dave Airlie <airlied@redhat.com>
---
drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
index a7f3fc342d87..dd5b5a17ece0 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
@@ -222,8 +222,11 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
void __iomem *map = NULL;
/* Already mapped? */
- if (refcount_inc_not_zero(&iobj->maps))
+ if (refcount_inc_not_zero(&iobj->maps)) {
+ /* read barrier match the wmb on refcount set */
+ smp_rmb();
return iobj->map;
+ }
/* Take the lock, and re-check that another thread hasn't
* already mapped the object in the meantime.
@@ -250,6 +253,8 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
iobj->base.memory.ptrs = &nv50_instobj_fast;
else
iobj->base.memory.ptrs = &nv50_instobj_slow;
+ /* barrier to ensure the ptrs are written before refcount is set */
+ smp_wmb();
refcount_set(&iobj->maps, 1);
}
--
2.43.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] nouveau: fix instmem race condition around ptr stores
2024-04-11 1:15 [PATCH] nouveau: fix instmem race condition around ptr stores Dave Airlie
@ 2024-04-15 9:06 ` Danilo Krummrich
0 siblings, 0 replies; 6+ messages in thread
From: Danilo Krummrich @ 2024-04-15 9:06 UTC (permalink / raw)
To: Dave Airlie; +Cc: nouveau, dri-devel
On 4/11/24 03:15, Dave Airlie wrote:
> From: Dave Airlie <airlied@redhat.com>
>
> Running a lot of VK CTS in parallel against nouveau, once every
> few hours you might see something like this crash.
>
> BUG: kernel NULL pointer dereference, address: 0000000000000008
> PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
> Oops: 0000 [#1] PREEMPT SMP PTI
> CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
> Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
> RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
> Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
> RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
> RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
> RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
> RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
> R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
> R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
> FS: 00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>
> ...
>
> ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
> ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
> nvkm_vmm_iter+0x351/0xa20 [nouveau]
> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> ? __lock_acquire+0x3ed/0x2170
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]
>
> Adding any sort of useful debug usually makes it go away, so I hand
> wrote the function in a line, and debugged the asm.
>
> Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in
> the nv50_instobj_acquire called from nvkm_kmap.
>
> If Thread A and Thread B both get to nv50_instobj_acquire around
> the same time, and Thread A hits the refcount_set line, and in
> lockstep thread B succeeds at refcount_inc_not_zero, there is a
> chance the ptrs value won't have been stored since refcount_set
> is unordered. Force a memory barrier here, I picked smp_mb, since
> we want it on all CPUs and it's write followed by a read.
>
> v2: use paired smp_rmb/smp_wmb.
>
> Cc: linux-stable
> Signed-off-by: Dave Airlie <airlied@redhat.com>
Added a "Fixes:" tag and applied to drm-misc-fixes.
> ---
> drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> index a7f3fc342d87..dd5b5a17ece0 100644
> --- a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> @@ -222,8 +222,11 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
> void __iomem *map = NULL;
>
> /* Already mapped? */
> - if (refcount_inc_not_zero(&iobj->maps))
> + if (refcount_inc_not_zero(&iobj->maps)) {
> + /* read barrier match the wmb on refcount set */
> + smp_rmb();
> return iobj->map;
> + }
>
> /* Take the lock, and re-check that another thread hasn't
> * already mapped the object in the meantime.
> @@ -250,6 +253,8 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
> iobj->base.memory.ptrs = &nv50_instobj_fast;
> else
> iobj->base.memory.ptrs = &nv50_instobj_slow;
> + /* barrier to ensure the ptrs are written before refcount is set */
> + smp_wmb();
> refcount_set(&iobj->maps, 1);
> }
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] nouveau: fix instmem race condition around ptr stores
2024-04-09 11:33 ` Danilo Krummrich
@ 2024-04-09 19:37 ` Dave Airlie
0 siblings, 0 replies; 6+ messages in thread
From: Dave Airlie @ 2024-04-09 19:37 UTC (permalink / raw)
To: Danilo Krummrich; +Cc: Lucas Stach, nouveau, dri-devel
On Tue, 9 Apr 2024 at 21:33, Danilo Krummrich <me@dakr.org> wrote:
>
> On 4/9/24 10:27, Lucas Stach wrote:
> > Am Dienstag, dem 09.04.2024 um 10:34 +1000 schrieb Dave Airlie:
> >> From: Dave Airlie <airlied@redhat.com>
> >>
> >> Running a lot of VK CTS in parallel against nouveau, once every
> >> few hours you might see something like this crash.
> >>
> >> BUG: kernel NULL pointer dereference, address: 0000000000000008
> >> PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
> >> Oops: 0000 [#1] PREEMPT SMP PTI
> >> CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
> >> Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
> >> RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
> >> Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
> >> RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
> >> RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
> >> RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
> >> RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
> >> R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
> >> R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
> >> FS: 00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
> >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >> CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
> >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> >> Call Trace:
> >>
> >> ...
> >>
> >> ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
> >> ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
> >> nvkm_vmm_iter+0x351/0xa20 [nouveau]
> >> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
> >> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> >> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> >> ? __lock_acquire+0x3ed/0x2170
> >> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> >> nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
> >> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
> >> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> >> nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]
> >>
> >> Adding any sort of useful debug usually makes it go away, so I hand
> >> wrote the function in a line, and debugged the asm.
> >>
> >> Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in
> >> the nv50_instobj_acquire called from nvkm_kmap.
> >>
> >> If Thread A and Thread B both get to nv50_instobj_acquire around
> >> the same time, and Thread A hits the refcount_set line, and in
> >> lockstep thread B succeeds at refcount_inc_not_zero, there is a
> >> chance the ptrs value won't have been stored since refcount_set
> >> is unordered. Force a memory barrier here, I picked smp_mb, since
> >> we want it on all CPUs and it's write followed by a read.
>
> Good catch!
>
> >>
> >> Cc: linux-stable
> >> Signed-off-by: Dave Airlie <airlied@redhat.com>
> >> ---
> >> drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
> >> 1 file changed, 3 insertions(+)
> >>
> >> diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> >> index a7f3fc342d87..cbacc7b11f8c 100644
> >> --- a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> >> +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> >> @@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
> >> iobj->base.memory.ptrs = &nv50_instobj_fast;
> >> else
> >> iobj->base.memory.ptrs = &nv50_instobj_slow;
> >> + /* barrier to ensure ptrs is written before another thread
> >> + does refcount_inc_not_zero successfully. */
> >> + smp_mb();
> >
> > Doesn't this miss the corresponding smp_rmb after
> > refcount_inc_not_zero? Without it a sufficiently speculating CPU might
> > still hoist the NULL ptr load across the refcount increase.
>
> Agree, also think this one could be smp_wmb() only.
>
> I also think it's reasonable to keep "the fast path refcount_inc_not_zero
> that doesn't take the lock", since the scope for this being potentially racy
> is limited to this function only.
I've been retesting with just barrier() here, since this seems at
least to be compiler related, but probably the smp_rmb/smp_wmb combo
is the safest answer across arches.
Dave.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] nouveau: fix instmem race condition around ptr stores
2024-04-09 8:27 ` Lucas Stach
@ 2024-04-09 11:33 ` Danilo Krummrich
2024-04-09 19:37 ` Dave Airlie
0 siblings, 1 reply; 6+ messages in thread
From: Danilo Krummrich @ 2024-04-09 11:33 UTC (permalink / raw)
To: Lucas Stach, Dave Airlie; +Cc: nouveau, dri-devel
On 4/9/24 10:27, Lucas Stach wrote:
> Am Dienstag, dem 09.04.2024 um 10:34 +1000 schrieb Dave Airlie:
>> From: Dave Airlie <airlied@redhat.com>
>>
>> Running a lot of VK CTS in parallel against nouveau, once every
>> few hours you might see something like this crash.
>>
>> BUG: kernel NULL pointer dereference, address: 0000000000000008
>> PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
>> Oops: 0000 [#1] PREEMPT SMP PTI
>> CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
>> Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
>> RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
>> Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
>> RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
>> RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
>> RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
>> RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
>> R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
>> R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
>> FS: 00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Call Trace:
>>
>> ...
>>
>> ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
>> ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
>> nvkm_vmm_iter+0x351/0xa20 [nouveau]
>> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
>> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
>> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
>> ? __lock_acquire+0x3ed/0x2170
>> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
>> nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
>> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
>> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
>> nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]
>>
>> Adding any sort of useful debug usually makes it go away, so I hand
>> wrote the function in a line, and debugged the asm.
>>
>> Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in
>> the nv50_instobj_acquire called from nvkm_kmap.
>>
>> If Thread A and Thread B both get to nv50_instobj_acquire around
>> the same time, and Thread A hits the refcount_set line, and in
>> lockstep thread B succeeds at refcount_inc_not_zero, there is a
>> chance the ptrs value won't have been stored since refcount_set
>> is unordered. Force a memory barrier here, I picked smp_mb, since
>> we want it on all CPUs and it's write followed by a read.
Good catch!
>>
>> Cc: linux-stable
>> Signed-off-by: Dave Airlie <airlied@redhat.com>
>> ---
>> drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
>> index a7f3fc342d87..cbacc7b11f8c 100644
>> --- a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
>> +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
>> @@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
>> iobj->base.memory.ptrs = &nv50_instobj_fast;
>> else
>> iobj->base.memory.ptrs = &nv50_instobj_slow;
>> + /* barrier to ensure ptrs is written before another thread
>> + does refcount_inc_not_zero successfully. */
>> + smp_mb();
>
> Doesn't this miss the corresponding smp_rmb after
> refcount_inc_not_zero? Without it a sufficiently speculating CPU might
> still hoist the NULL ptr load across the refcount increase.
Agree, also think this one could be smp_wmb() only.
I also think it's reasonable to keep "the fast path refcount_inc_not_zero
that doesn't take the lock", since the scope for this being potentially racy
is limited to this function only.
>
> Regards,
> Lucas
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] nouveau: fix instmem race condition around ptr stores
2024-04-09 0:34 Dave Airlie
@ 2024-04-09 8:27 ` Lucas Stach
2024-04-09 11:33 ` Danilo Krummrich
0 siblings, 1 reply; 6+ messages in thread
From: Lucas Stach @ 2024-04-09 8:27 UTC (permalink / raw)
To: Dave Airlie, dri-devel; +Cc: nouveau
Am Dienstag, dem 09.04.2024 um 10:34 +1000 schrieb Dave Airlie:
> From: Dave Airlie <airlied@redhat.com>
>
> Running a lot of VK CTS in parallel against nouveau, once every
> few hours you might see something like this crash.
>
> BUG: kernel NULL pointer dereference, address: 0000000000000008
> PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
> Oops: 0000 [#1] PREEMPT SMP PTI
> CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
> Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
> RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
> Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
> RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
> RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
> RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
> RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
> R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
> R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
> FS: 00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>
> ...
>
> ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
> ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
> nvkm_vmm_iter+0x351/0xa20 [nouveau]
> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> ? __lock_acquire+0x3ed/0x2170
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
> ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
> ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
> nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]
>
> Adding any sort of useful debug usually makes it go away, so I hand
> wrote the function in a line, and debugged the asm.
>
> Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in
> the nv50_instobj_acquire called from nvkm_kmap.
>
> If Thread A and Thread B both get to nv50_instobj_acquire around
> the same time, and Thread A hits the refcount_set line, and in
> lockstep thread B succeeds at refcount_inc_not_zero, there is a
> chance the ptrs value won't have been stored since refcount_set
> is unordered. Force a memory barrier here, I picked smp_mb, since
> we want it on all CPUs and it's write followed by a read.
>
> Cc: linux-stable
> Signed-off-by: Dave Airlie <airlied@redhat.com>
> ---
> drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> index a7f3fc342d87..cbacc7b11f8c 100644
> --- a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
> @@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
> iobj->base.memory.ptrs = &nv50_instobj_fast;
> else
> iobj->base.memory.ptrs = &nv50_instobj_slow;
> + /* barrier to ensure ptrs is written before another thread
> + does refcount_inc_not_zero successfully. */
> + smp_mb();
Doesn't this miss the corresponding smp_rmb after
refcount_inc_not_zero? Without it a sufficiently speculating CPU might
still hoist the NULL ptr load across the refcount increase.
Regards,
Lucas
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] nouveau: fix instmem race condition around ptr stores
@ 2024-04-09 0:34 Dave Airlie
2024-04-09 8:27 ` Lucas Stach
0 siblings, 1 reply; 6+ messages in thread
From: Dave Airlie @ 2024-04-09 0:34 UTC (permalink / raw)
To: dri-devel; +Cc: nouveau
From: Dave Airlie <airlied@redhat.com>
Running a lot of VK CTS in parallel against nouveau, once every
few hours you might see something like this crash.
BUG: kernel NULL pointer dereference, address: 0000000000000008
PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
FS: 00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
...
? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
nvkm_vmm_iter+0x351/0xa20 [nouveau]
? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
? __lock_acquire+0x3ed/0x2170
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]
Adding any sort of useful debug usually makes it go away, so I hand
wrote the function in a line, and debugged the asm.
Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in
the nv50_instobj_acquire called from nvkm_kmap.
If Thread A and Thread B both get to nv50_instobj_acquire around
the same time, and Thread A hits the refcount_set line, and in
lockstep thread B succeeds at refcount_inc_not_zero, there is a
chance the ptrs value won't have been stored since refcount_set
is unordered. Force a memory barrier here, I picked smp_mb, since
we want it on all CPUs and it's write followed by a read.
Cc: linux-stable
Signed-off-by: Dave Airlie <airlied@redhat.com>
---
drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
index a7f3fc342d87..cbacc7b11f8c 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c
@@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
iobj->base.memory.ptrs = &nv50_instobj_fast;
else
iobj->base.memory.ptrs = &nv50_instobj_slow;
+ /* barrier to ensure ptrs is written before another thread
+ does refcount_inc_not_zero successfully. */
+ smp_mb();
refcount_set(&iobj->maps, 1);
}
--
2.43.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-04-15 9:06 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-11 1:15 [PATCH] nouveau: fix instmem race condition around ptr stores Dave Airlie
2024-04-15 9:06 ` Danilo Krummrich
-- strict thread matches above, loose matches on Subject: below --
2024-04-09 0:34 Dave Airlie
2024-04-09 8:27 ` Lucas Stach
2024-04-09 11:33 ` Danilo Krummrich
2024-04-09 19:37 ` Dave Airlie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).