Re: [PATCH v2 5/8] drm/amdgpu: Refactor sysfs removal

[Andrey Grodzovsky <Andrey.Grodzovsky@amd.com>]

On 6/23/20 2:05 AM, Greg KH wrote:
> On Tue, Jun 23, 2020 at 12:51:00AM -0400, Andrey Grodzovsky wrote:
>> On 6/22/20 12:45 PM, Greg KH wrote:
>>> On Mon, Jun 22, 2020 at 12:07:25PM -0400, Andrey Grodzovsky wrote:
>>>> On 6/22/20 7:21 AM, Greg KH wrote:
>>>>> On Mon, Jun 22, 2020 at 11:51:24AM +0200, Daniel Vetter wrote:
>>>>>> On Sun, Jun 21, 2020 at 02:03:05AM -0400, Andrey Grodzovsky wrote:
>>>>>>> Track sysfs files in a list so they all can be removed during pci remove
>>>>>>> since otherwise their removal after that causes crash because parent
>>>>>>> folder was already removed during pci remove.
>>>>> Huh?  That should not happen, do you have a backtrace of that crash?
>>>> 2 examples in the attached trace.
>>> Odd, how did you trigger these?
>>
>> By manually triggering PCI remove from sysfs
>>
>> cd /sys/bus/pci/devices/0000\:05\:00.0 && echo 1 > remove
> For some reason, I didn't think that video/drm devices could handle
> hot-remove like this.  The "old" PCI hotplug specification explicitly
> said that video devices were not supported, has that changed?
>
> And this whole issue is probably tied to the larger issue that Daniel
> was asking me about, when it came to device lifetimes and the drm layer,
> so odds are we need to fix that up first before worrying about trying to
> support this crazy request, right?  :)
>
>>>> [  925.738225 <    0.188086>] BUG: kernel NULL pointer dereference, address: 0000000000000090
>>>> [  925.738232 <    0.000007>] #PF: supervisor read access in kernel mode
>>>> [  925.738236 <    0.000004>] #PF: error_code(0x0000) - not-present page
>>>> [  925.738240 <    0.000004>] PGD 0 P4D 0
>>>> [  925.738245 <    0.000005>] Oops: 0000 [#1] SMP PTI
>>>> [  925.738249 <    0.000004>] CPU: 7 PID: 2547 Comm: amdgpu_test Tainted: G        W  OE     5.5.0-rc7-dev-kfd+ #50
>>>> [  925.738256 <    0.000007>] Hardware name: System manufacturer System Product Name/RAMPAGE IV FORMULA, BIOS 4804 12/30/2013
>>>> [  925.738266 <    0.000010>] RIP: 0010:kernfs_find_ns+0x18/0x110
>>>> [  925.738270 <    0.000004>] Code: a6 cf ff 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 49 89 f6 41 55 41 54 49 89 fd 55 53 49 89 d4 <0f> b7 af 90 00 00 00 8b 05 8f ee 6b 01 48 8b 5f 68 66 83 e5 20 41
>>>> [  925.738282 <    0.000012>] RSP: 0018:ffffad6d0118fb00 EFLAGS: 00010246
>>>> [  925.738287 <    0.000005>] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 2098a12076864b7e
>>>> [  925.738292 <    0.000005>] RDX: 0000000000000000 RSI: ffffffffb6606b31 RDI: 0000000000000000
>>>> [  925.738297 <    0.000005>] RBP: ffffffffb6606b31 R08: ffffffffb5379d10 R09: 0000000000000000
>>>> [  925.738302 <    0.000005>] R10: ffffad6d0118fb38 R11: ffff9a75f64820a8 R12: 0000000000000000
>>>> [  925.738307 <    0.000005>] R13: 0000000000000000 R14: ffffffffb6606b31 R15: ffff9a7612b06130
>>>> [  925.738313 <    0.000006>] FS:  00007f3eca4e8700(0000) GS:ffff9a763dbc0000(0000) knlGS:0000000000000000
>>>> [  925.738319 <    0.000006>] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [  925.738323 <    0.000004>] CR2: 0000000000000090 CR3: 0000000035e5a005 CR4: 00000000000606e0
>>>> [  925.738329 <    0.000006>] Call Trace:
>>>> [  925.738334 <    0.000005>]  kernfs_find_and_get_ns+0x2e/0x50
>>>> [  925.738339 <    0.000005>]  sysfs_remove_group+0x25/0x80
>>>> [  925.738344 <    0.000005>]  sysfs_remove_groups+0x29/0x40
>>>> [  925.738350 <    0.000006>]  free_msi_irqs+0xf5/0x190
>>>> [  925.738354 <    0.000004>]  pci_disable_msi+0xe9/0x120
>>> So the PCI core is trying to clean up attributes that it had registered,
>>> which is fine.  But we can't seem to find the attributes?  Were they
>>> already removed somewhere else?
>>>
>>> that's odd.
>>
>> Yes, as i pointed above i am emulating device remove from sysfs and this
>> triggers pci device remove sequence and as part of that my specific device
>> folder (05:00.0) is removed from the sysfs tree.
> But why are things being removed twice?


Not sure I understand what removed twice ? I remove only once per sysfs attribute.

Andrey


>
>>>> [  925.738406 <    0.000052>]  amdgpu_irq_fini+0xe3/0xf0 [amdgpu]
>>>> [  925.738453 <    0.000047>]  tonga_ih_sw_fini+0xe/0x30 [amdgpu]
>>>> [  925.738490 <    0.000037>]  amdgpu_device_fini_late+0x14b/0x440 [amdgpu]
>>>> [  925.738529 <    0.000039>]  amdgpu_driver_release_kms+0x16/0x40 [amdgpu]
>>>> [  925.738548 <    0.000019>]  drm_dev_put+0x5b/0x80 [drm]
>>>> [  925.738558 <    0.000010>]  drm_release+0xc6/0xd0 [drm]
>>>> [  925.738563 <    0.000005>]  __fput+0xc6/0x260
>>>> [  925.738568 <    0.000005>]  task_work_run+0x79/0xb0
>>>> [  925.738573 <    0.000005>]  do_exit+0x3d0/0xc60
>>>> [  925.738578 <    0.000005>]  do_group_exit+0x47/0xb0
>>>> [  925.738583 <    0.000005>]  get_signal+0x18b/0xc30
>>>> [  925.738589 <    0.000006>]  do_signal+0x36/0x6a0
>>>> [  925.738593 <    0.000004>]  ? force_sig_info_to_task+0xbc/0xd0
>>>> [  925.738597 <    0.000004>]  ? signal_wake_up_state+0x15/0x30
>>>> [  925.738603 <    0.000006>]  exit_to_usermode_loop+0x6f/0xc0
>>>> [  925.738608 <    0.000005>]  prepare_exit_to_usermode+0xc7/0x110
>>>> [  925.738613 <    0.000005>]  ret_from_intr+0x25/0x35
>>>> [  925.738617 <    0.000004>] RIP: 0033:0x417369
>>>> [  925.738621 <    0.000004>] Code: Bad RIP value.
>>>> [  925.738625 <    0.000004>] RSP: 002b:00007ffdd6bf0900 EFLAGS: 00010246
>>>> [  925.738629 <    0.000004>] RAX: 00007f3eca509000 RBX: 000000000000001e RCX: 00007f3ec95ba260
>>>> [  925.738634 <    0.000005>] RDX: 00007f3ec9889790 RSI: 000000000000000a RDI: 0000000000000000
>>>> [  925.738639 <    0.000005>] RBP: 00007ffdd6bf0990 R08: 00007f3ec9889780 R09: 00007f3eca4e8700
>>>> [  925.738645 <    0.000006>] R10: 000000000000035c R11: 0000000000000246 R12: 00000000021c6170
>>>> [  925.738650 <    0.000005>] R13: 00007ffdd6bf0c00 R14: 0000000000000000 R15: 0000000000000000
>>>>
>>>>
>>>>
>>>>
>>>> [   40.880899 <    0.000004>] BUG: kernel NULL pointer dereference, address: 0000000000000090
>>>> [   40.880906 <    0.000007>] #PF: supervisor read access in kernel mode
>>>> [   40.880910 <    0.000004>] #PF: error_code(0x0000) - not-present page
>>>> [   40.880915 <    0.000005>] PGD 0 P4D 0
>>>> [   40.880920 <    0.000005>] Oops: 0000 [#1] SMP PTI
>>>> [   40.880924 <    0.000004>] CPU: 1 PID: 2526 Comm: amdgpu_test Tainted: G        W  OE     5.5.0-rc7-dev-kfd+ #50
>>>> [   40.880932 <    0.000008>] Hardware name: System manufacturer System Product Name/RAMPAGE IV FORMULA, BIOS 4804 12/30/2013
>>>> [   40.880941 <    0.000009>] RIP: 0010:kernfs_find_ns+0x18/0x110
>>>> [   40.880945 <    0.000004>] Code: a6 cf ff 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 49 89 f6 41 55 41 54 49 89 fd 55 53 49 89 d4 <0f> b7 af 90 00 00 00 8b 05 8f ee 6b 01 48 8b 5f 68 66 83 e5 20 41
>>>> [   40.880957 <    0.000012>] RSP: 0018:ffffaf3380467ba8 EFLAGS: 00010246
>>>> [   40.880963 <    0.000006>] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 2098a12076864b7e
>>>> [   40.880968 <    0.000005>] RDX: 0000000000000000 RSI: ffffffffc0678cfc RDI: 0000000000000000
>>>> [   40.880973 <    0.000005>] RBP: ffffffffc0678cfc R08: ffffffffaa379d10 R09: 0000000000000000
>>>> [   40.880979 <    0.000006>] R10: ffffaf3380467be0 R11: ffff93547615d128 R12: 0000000000000000
>>>> [   40.880984 <    0.000005>] R13: 0000000000000000 R14: ffffffffc0678cfc R15: ffff93549be86130
>>>> [   40.880990 <    0.000006>] FS:  00007fd9ecb10700(0000) GS:ffff9354bd840000(0000) knlGS:0000000000000000
>>>> [   40.880996 <    0.000006>] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [   40.881001 <    0.000005>] CR2: 0000000000000090 CR3: 0000000072866001 CR4: 00000000000606e0
>>>> [   40.881006 <    0.000005>] Call Trace:
>>>> [   40.881011 <    0.000005>]  kernfs_find_and_get_ns+0x2e/0x50
>>>> [   40.881016 <    0.000005>]  sysfs_remove_group+0x25/0x80
>>>> [   40.881055 <    0.000039>]  amdgpu_device_fini_late+0x3eb/0x440 [amdgpu]
>>>> [   40.881095 <    0.000040>]  amdgpu_driver_release_kms+0x16/0x40 [amdgpu]
>>> Here is this is your driver doing the same thing, removing attributes it
>>> created.  But again they are not there.
>>>
>>> So something went through and wiped the tree clean, which if I'm reading
>>> this correctly, your patch would not solve as you would try to also
>>> remove attributes that were already removed, right?
>>
>> I don't think so, the stack here is from a later stage (after pci remove)
>> where the last user process holding a reference to the device file decides
>> to die and thus triggering drm_dev_release sequence after drm dev refcount
>> dropped to zero. And this why my patch helps, i am expediting all amdgpu
>> sysfs attributes removal to the pci remove stage when the device folder is
>> still present in the sysfs hierarchy. At least this is my  understanding to
>> why it helped. I admit i am not an expert on sysfs internals.
> Ok, yeah, I think this is back to the drm lifecycle issues mentioned
> above.
>
> {sigh}, I'll get to that once I deal with the -rc1/-rc2 merge fallout,
> that will take me a week or so, sorry...
>
> thanks,
>
> greg k-h
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx