Re: [PATCH v2 5/8] drm/amdgpu: Refactor sysfs removal

[Andrey Grodzovsky <Andrey.Grodzovsky@amd.com>]

On 6/22/20 12:45 PM, Greg KH wrote:
> On Mon, Jun 22, 2020 at 12:07:25PM -0400, Andrey Grodzovsky wrote:
>> On 6/22/20 7:21 AM, Greg KH wrote:
>>> On Mon, Jun 22, 2020 at 11:51:24AM +0200, Daniel Vetter wrote:
>>>> On Sun, Jun 21, 2020 at 02:03:05AM -0400, Andrey Grodzovsky wrote:
>>>>> Track sysfs files in a list so they all can be removed during pci remove
>>>>> since otherwise their removal after that causes crash because parent
>>>>> folder was already removed during pci remove.
>>> Huh?  That should not happen, do you have a backtrace of that crash?
>>
>> 2 examples in the attached trace.
> Odd, how did you trigger these?


By manually triggering PCI remove from sysfs

cd /sys/bus/pci/devices/0000\:05\:00.0 && echo 1 > remove


>
>
>> [  925.738225 <    0.188086>] BUG: kernel NULL pointer dereference, address: 0000000000000090
>> [  925.738232 <    0.000007>] #PF: supervisor read access in kernel mode
>> [  925.738236 <    0.000004>] #PF: error_code(0x0000) - not-present page
>> [  925.738240 <    0.000004>] PGD 0 P4D 0
>> [  925.738245 <    0.000005>] Oops: 0000 [#1] SMP PTI
>> [  925.738249 <    0.000004>] CPU: 7 PID: 2547 Comm: amdgpu_test Tainted: G        W  OE     5.5.0-rc7-dev-kfd+ #50
>> [  925.738256 <    0.000007>] Hardware name: System manufacturer System Product Name/RAMPAGE IV FORMULA, BIOS 4804 12/30/2013
>> [  925.738266 <    0.000010>] RIP: 0010:kernfs_find_ns+0x18/0x110
>> [  925.738270 <    0.000004>] Code: a6 cf ff 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 49 89 f6 41 55 41 54 49 89 fd 55 53 49 89 d4 <0f> b7 af 90 00 00 00 8b 05 8f ee 6b 01 48 8b 5f 68 66 83 e5 20 41
>> [  925.738282 <    0.000012>] RSP: 0018:ffffad6d0118fb00 EFLAGS: 00010246
>> [  925.738287 <    0.000005>] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 2098a12076864b7e
>> [  925.738292 <    0.000005>] RDX: 0000000000000000 RSI: ffffffffb6606b31 RDI: 0000000000000000
>> [  925.738297 <    0.000005>] RBP: ffffffffb6606b31 R08: ffffffffb5379d10 R09: 0000000000000000
>> [  925.738302 <    0.000005>] R10: ffffad6d0118fb38 R11: ffff9a75f64820a8 R12: 0000000000000000
>> [  925.738307 <    0.000005>] R13: 0000000000000000 R14: ffffffffb6606b31 R15: ffff9a7612b06130
>> [  925.738313 <    0.000006>] FS:  00007f3eca4e8700(0000) GS:ffff9a763dbc0000(0000) knlGS:0000000000000000
>> [  925.738319 <    0.000006>] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [  925.738323 <    0.000004>] CR2: 0000000000000090 CR3: 0000000035e5a005 CR4: 00000000000606e0
>> [  925.738329 <    0.000006>] Call Trace:
>> [  925.738334 <    0.000005>]  kernfs_find_and_get_ns+0x2e/0x50
>> [  925.738339 <    0.000005>]  sysfs_remove_group+0x25/0x80
>> [  925.738344 <    0.000005>]  sysfs_remove_groups+0x29/0x40
>> [  925.738350 <    0.000006>]  free_msi_irqs+0xf5/0x190
>> [  925.738354 <    0.000004>]  pci_disable_msi+0xe9/0x120
> So the PCI core is trying to clean up attributes that it had registered,
> which is fine.  But we can't seem to find the attributes?  Were they
> already removed somewhere else?
>
> that's odd.


Yes, as i pointed above i am emulating device remove from sysfs and this 
triggers pci device remove sequence and as part of that my specific device 
folder (05:00.0) is removed from the sysfs tree.


>
>> [  925.738406 <    0.000052>]  amdgpu_irq_fini+0xe3/0xf0 [amdgpu]
>> [  925.738453 <    0.000047>]  tonga_ih_sw_fini+0xe/0x30 [amdgpu]
>> [  925.738490 <    0.000037>]  amdgpu_device_fini_late+0x14b/0x440 [amdgpu]
>> [  925.738529 <    0.000039>]  amdgpu_driver_release_kms+0x16/0x40 [amdgpu]
>> [  925.738548 <    0.000019>]  drm_dev_put+0x5b/0x80 [drm]
>> [  925.738558 <    0.000010>]  drm_release+0xc6/0xd0 [drm]
>> [  925.738563 <    0.000005>]  __fput+0xc6/0x260
>> [  925.738568 <    0.000005>]  task_work_run+0x79/0xb0
>> [  925.738573 <    0.000005>]  do_exit+0x3d0/0xc60
>> [  925.738578 <    0.000005>]  do_group_exit+0x47/0xb0
>> [  925.738583 <    0.000005>]  get_signal+0x18b/0xc30
>> [  925.738589 <    0.000006>]  do_signal+0x36/0x6a0
>> [  925.738593 <    0.000004>]  ? force_sig_info_to_task+0xbc/0xd0
>> [  925.738597 <    0.000004>]  ? signal_wake_up_state+0x15/0x30
>> [  925.738603 <    0.000006>]  exit_to_usermode_loop+0x6f/0xc0
>> [  925.738608 <    0.000005>]  prepare_exit_to_usermode+0xc7/0x110
>> [  925.738613 <    0.000005>]  ret_from_intr+0x25/0x35
>> [  925.738617 <    0.000004>] RIP: 0033:0x417369
>> [  925.738621 <    0.000004>] Code: Bad RIP value.
>> [  925.738625 <    0.000004>] RSP: 002b:00007ffdd6bf0900 EFLAGS: 00010246
>> [  925.738629 <    0.000004>] RAX: 00007f3eca509000 RBX: 000000000000001e RCX: 00007f3ec95ba260
>> [  925.738634 <    0.000005>] RDX: 00007f3ec9889790 RSI: 000000000000000a RDI: 0000000000000000
>> [  925.738639 <    0.000005>] RBP: 00007ffdd6bf0990 R08: 00007f3ec9889780 R09: 00007f3eca4e8700
>> [  925.738645 <    0.000006>] R10: 000000000000035c R11: 0000000000000246 R12: 00000000021c6170
>> [  925.738650 <    0.000005>] R13: 00007ffdd6bf0c00 R14: 0000000000000000 R15: 0000000000000000
>>
>>
>>
>>
>> [   40.880899 <    0.000004>] BUG: kernel NULL pointer dereference, address: 0000000000000090
>> [   40.880906 <    0.000007>] #PF: supervisor read access in kernel mode
>> [   40.880910 <    0.000004>] #PF: error_code(0x0000) - not-present page
>> [   40.880915 <    0.000005>] PGD 0 P4D 0
>> [   40.880920 <    0.000005>] Oops: 0000 [#1] SMP PTI
>> [   40.880924 <    0.000004>] CPU: 1 PID: 2526 Comm: amdgpu_test Tainted: G        W  OE     5.5.0-rc7-dev-kfd+ #50
>> [   40.880932 <    0.000008>] Hardware name: System manufacturer System Product Name/RAMPAGE IV FORMULA, BIOS 4804 12/30/2013
>> [   40.880941 <    0.000009>] RIP: 0010:kernfs_find_ns+0x18/0x110
>> [   40.880945 <    0.000004>] Code: a6 cf ff 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 49 89 f6 41 55 41 54 49 89 fd 55 53 49 89 d4 <0f> b7 af 90 00 00 00 8b 05 8f ee 6b 01 48 8b 5f 68 66 83 e5 20 41
>> [   40.880957 <    0.000012>] RSP: 0018:ffffaf3380467ba8 EFLAGS: 00010246
>> [   40.880963 <    0.000006>] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 2098a12076864b7e
>> [   40.880968 <    0.000005>] RDX: 0000000000000000 RSI: ffffffffc0678cfc RDI: 0000000000000000
>> [   40.880973 <    0.000005>] RBP: ffffffffc0678cfc R08: ffffffffaa379d10 R09: 0000000000000000
>> [   40.880979 <    0.000006>] R10: ffffaf3380467be0 R11: ffff93547615d128 R12: 0000000000000000
>> [   40.880984 <    0.000005>] R13: 0000000000000000 R14: ffffffffc0678cfc R15: ffff93549be86130
>> [   40.880990 <    0.000006>] FS:  00007fd9ecb10700(0000) GS:ffff9354bd840000(0000) knlGS:0000000000000000
>> [   40.880996 <    0.000006>] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [   40.881001 <    0.000005>] CR2: 0000000000000090 CR3: 0000000072866001 CR4: 00000000000606e0
>> [   40.881006 <    0.000005>] Call Trace:
>> [   40.881011 <    0.000005>]  kernfs_find_and_get_ns+0x2e/0x50
>> [   40.881016 <    0.000005>]  sysfs_remove_group+0x25/0x80
>> [   40.881055 <    0.000039>]  amdgpu_device_fini_late+0x3eb/0x440 [amdgpu]
>> [   40.881095 <    0.000040>]  amdgpu_driver_release_kms+0x16/0x40 [amdgpu]
> Here is this is your driver doing the same thing, removing attributes it
> created.  But again they are not there.
>
> So something went through and wiped the tree clean, which if I'm reading
> this correctly, your patch would not solve as you would try to also
> remove attributes that were already removed, right?


I don't think so, the stack here is from a later stage (after pci remove) where 
the last user process holding a reference to the device file decides to die and 
thus triggering drm_dev_release sequence after drm dev refcount dropped to zero. 
And this why my patch helps, i am expediting all amdgpu sysfs attributes removal 
to the pci remove stage when the device folder is still present in the sysfs 
hierarchy. At least this is my  understanding to why it helped. I admit i am not 
an expert on sysfs internals.


>
> And 5.5-rc7 is a bit old (6 months and many thousands of changes ago),
> does this still happen on a modern, released, kernel?


I will give it a try with the latest and greatest but it might take some time as 
I have to make a temporary context switch to some urgent task.

Andrey


>
> thanks,
>
> greg k-h
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx