Re: possible IO map leak in drm/gem

Re: possible IO map leak in drm/gem

[Thomas Zimmermann]

[-- Attachment #1.1.1: Type: text/plain, Size: 2142 bytes --]

(cc'ing dri-devel)

Hi,

thanks for reporting the bug.

Am 21.01.21 um 15:35 schrieb Chuck Lever:
> Hi Thomas-
> 
> I was not able to find an appropriate mailing list entry in MAINTAINERS,

That would be dri-devel@lists.freedesktop.org

> so I'm mailing you directly as committer of record for:
> 
> 43676605f890 ("drm/ttm: Add vmap/vunmap to TTM and TTM GEM helpers")
> 
> I've noticed that since putting v5.11-rc on my test systems, overnight
> on an otherwise idle system the load average seems to grow as the result
> of a kernel worker thread.

Earlier this week I fixed a couple of leaks in that code. Could you 
please apply the patch at [1] and report back if it fixes the issue.

If it's a separate problem, I'll take a closer look.

Best regards
Thomas

[1] 
https://lore.kernel.org/dri-devel/20210118144639.27307-1-tzimmermann@suse.de/

> 
> I used "perf top" to see what it had gotten up to, and it appears that
> it was spending lots of time walking an interval tree on behalf of
> memtype_reserve().
> 
> The most frequently-observed stack trace seems to be:
> 
>       kworker/3:1-2355  [003] 60950.150928: function:             memtype_reserve
>       kworker/3:1-2355  [003] 60950.150942: kernel_stack:         <stack trace>
> => ffffffffc0c66083
> => memtype_reserve (ffffffffa005f9d5)
> => __ioremap_caller (ffffffffa005aac1)
> => ttm_bo_vmap (ffffffffc040f266)
> => drm_gem_vram_vmap (ffffffffc042c5cd)
> => drm_gem_vmap (ffffffffc0506a7f)
> => drm_client_buffer_vmap (ffffffffc0523741)
> => drm_fb_helper_damage_work (ffffffffc049a34a)
> => process_one_work (ffffffffa00dd92e)
> => worker_thread (ffffffffa00dde46)
> => kthread (ffffffffa00e22c4)
> => ret_from_fork (ffffffffa0004192)
> 
> I see a regular call to memtype_reserve(), but never a matching call to
> memtype_free(), thus I suspect a leak of I/O maps in this code.
> 
> --
> Chuck Lever
> 
> 
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: possible IO map leak in drm/gem

[Chuck Lever]

> On Jan 21, 2021, at 9:47 AM, Thomas Zimmermann <tzimmermann@suse.de> wrote:
> 
> (cc'ing dri-devel)
> 
> Hi,
> 
> thanks for reporting the bug.
> 
> Am 21.01.21 um 15:35 schrieb Chuck Lever:
>> Hi Thomas-
>> I was not able to find an appropriate mailing list entry in MAINTAINERS,
> 
> That would be dri-devel@lists.freedesktop.org
> 
>> so I'm mailing you directly as committer of record for:
>> 43676605f890 ("drm/ttm: Add vmap/vunmap to TTM and TTM GEM helpers")
>> I've noticed that since putting v5.11-rc on my test systems, overnight
>> on an otherwise idle system the load average seems to grow as the result
>> of a kernel worker thread.
> 
> Earlier this week I fixed a couple of leaks in that code. Could you please apply the patch at [1] and report back if it fixes the issue.
> 
> If it's a separate problem, I'll take a closer look.

Thomas, thank you for your quick response. I've installed your patch
on one system and it looks promising already. I'll let it soak overnight.


> Best regards
> Thomas
> 
> [1] https://lore.kernel.org/dri-devel/20210118144639.27307-1-tzimmermann@suse.de/
> 
>> I used "perf top" to see what it had gotten up to, and it appears that
>> it was spending lots of time walking an interval tree on behalf of
>> memtype_reserve().
>> The most frequently-observed stack trace seems to be:
>>      kworker/3:1-2355  [003] 60950.150928: function:             memtype_reserve
>>      kworker/3:1-2355  [003] 60950.150942: kernel_stack:         <stack trace>
>> => ffffffffc0c66083
>> => memtype_reserve (ffffffffa005f9d5)
>> => __ioremap_caller (ffffffffa005aac1)
>> => ttm_bo_vmap (ffffffffc040f266)
>> => drm_gem_vram_vmap (ffffffffc042c5cd)
>> => drm_gem_vmap (ffffffffc0506a7f)
>> => drm_client_buffer_vmap (ffffffffc0523741)
>> => drm_fb_helper_damage_work (ffffffffc049a34a)
>> => process_one_work (ffffffffa00dd92e)
>> => worker_thread (ffffffffa00dde46)
>> => kthread (ffffffffa00e22c4)
>> => ret_from_fork (ffffffffa0004192)
>> I see a regular call to memtype_reserve(), but never a matching call to
>> memtype_free(), thus I suspect a leak of I/O maps in this code.
>> --
>> Chuck Lever
> 
> -- 
> Thomas Zimmermann
> Graphics Driver Developer
> SUSE Software Solutions Germany GmbH
> Maxfeldstr. 5, 90409 Nürnberg, Germany
> (HRB 36809, AG Nürnberg)
> Geschäftsführer: Felix Imendörffer

--
Chuck Lever



_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: possible IO map leak in drm/gem

[Chuck Lever]

> On Jan 21, 2021, at 10:05 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
> 
> 
> 
>> On Jan 21, 2021, at 9:47 AM, Thomas Zimmermann <tzimmermann@suse.de> wrote:
>> 
>> (cc'ing dri-devel)
>> 
>> Hi,
>> 
>> thanks for reporting the bug.
>> 
>> Am 21.01.21 um 15:35 schrieb Chuck Lever:
>>> Hi Thomas-
>>> I was not able to find an appropriate mailing list entry in MAINTAINERS,
>> 
>> That would be dri-devel@lists.freedesktop.org
>> 
>>> so I'm mailing you directly as committer of record for:
>>> 43676605f890 ("drm/ttm: Add vmap/vunmap to TTM and TTM GEM helpers")
>>> I've noticed that since putting v5.11-rc on my test systems, overnight
>>> on an otherwise idle system the load average seems to grow as the result
>>> of a kernel worker thread.
>> 
>> Earlier this week I fixed a couple of leaks in that code. Could you please apply the patch at [1] and report back if it fixes the issue.
>> 
>> If it's a separate problem, I'll take a closer look.
> 
> Thomas, thank you for your quick response. I've installed your patch
> on one system and it looks promising already. I'll let it soak overnight.

All symptoms appear to be gone. Fwiw,

Tested-by: Chuck Lever <chuck.lever@oracle.com>


>> Best regards
>> Thomas
>> 
>> [1] https://lore.kernel.org/dri-devel/20210118144639.27307-1-tzimmermann@suse.de/
>> 
>>> I used "perf top" to see what it had gotten up to, and it appears that
>>> it was spending lots of time walking an interval tree on behalf of
>>> memtype_reserve().
>>> The most frequently-observed stack trace seems to be:
>>>     kworker/3:1-2355  [003] 60950.150928: function:             memtype_reserve
>>>     kworker/3:1-2355  [003] 60950.150942: kernel_stack:         <stack trace>
>>> => ffffffffc0c66083
>>> => memtype_reserve (ffffffffa005f9d5)
>>> => __ioremap_caller (ffffffffa005aac1)
>>> => ttm_bo_vmap (ffffffffc040f266)
>>> => drm_gem_vram_vmap (ffffffffc042c5cd)
>>> => drm_gem_vmap (ffffffffc0506a7f)
>>> => drm_client_buffer_vmap (ffffffffc0523741)
>>> => drm_fb_helper_damage_work (ffffffffc049a34a)
>>> => process_one_work (ffffffffa00dd92e)
>>> => worker_thread (ffffffffa00dde46)
>>> => kthread (ffffffffa00e22c4)
>>> => ret_from_fork (ffffffffa0004192)
>>> I see a regular call to memtype_reserve(), but never a matching call to
>>> memtype_free(), thus I suspect a leak of I/O maps in this code.
>>> --
>>> Chuck Lever
>> 
>> -- 
>> Thomas Zimmermann
>> Graphics Driver Developer
>> SUSE Software Solutions Germany GmbH
>> Maxfeldstr. 5, 90409 Nürnberg, Germany
>> (HRB 36809, AG Nürnberg)
>> Geschäftsführer: Felix Imendörffer
> 
> --
> Chuck Lever

--
Chuck Lever



_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: possible IO map leak in drm/gem

[Thomas Zimmermann]

[-- Attachment #1.1.1: Type: text/plain, Size: 3095 bytes --]

Hi

Am 22.01.21 um 15:27 schrieb Chuck Lever:
> 
> 
>> On Jan 21, 2021, at 10:05 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
>>
>>
>>
>>> On Jan 21, 2021, at 9:47 AM, Thomas Zimmermann <tzimmermann@suse.de> wrote:
>>>
>>> (cc'ing dri-devel)
>>>
>>> Hi,
>>>
>>> thanks for reporting the bug.
>>>
>>> Am 21.01.21 um 15:35 schrieb Chuck Lever:
>>>> Hi Thomas-
>>>> I was not able to find an appropriate mailing list entry in MAINTAINERS,
>>>
>>> That would be dri-devel@lists.freedesktop.org
>>>
>>>> so I'm mailing you directly as committer of record for:
>>>> 43676605f890 ("drm/ttm: Add vmap/vunmap to TTM and TTM GEM helpers")
>>>> I've noticed that since putting v5.11-rc on my test systems, overnight
>>>> on an otherwise idle system the load average seems to grow as the result
>>>> of a kernel worker thread.
>>>
>>> Earlier this week I fixed a couple of leaks in that code. Could you please apply the patch at [1] and report back if it fixes the issue.
>>>
>>> If it's a separate problem, I'll take a closer look.
>>
>> Thomas, thank you for your quick response. I've installed your patch
>> on one system and it looks promising already. I'll let it soak overnight.
> 
> All symptoms appear to be gone. Fwiw,
> 
> Tested-by: Chuck Lever <chuck.lever@oracle.com>

Great. Thanks a lot for testing.

Best regards
Thomas

> 
> 
>>> Best regards
>>> Thomas
>>>
>>> [1] https://lore.kernel.org/dri-devel/20210118144639.27307-1-tzimmermann@suse.de/
>>>
>>>> I used "perf top" to see what it had gotten up to, and it appears that
>>>> it was spending lots of time walking an interval tree on behalf of
>>>> memtype_reserve().
>>>> The most frequently-observed stack trace seems to be:
>>>>      kworker/3:1-2355  [003] 60950.150928: function:             memtype_reserve
>>>>      kworker/3:1-2355  [003] 60950.150942: kernel_stack:         <stack trace>
>>>> => ffffffffc0c66083
>>>> => memtype_reserve (ffffffffa005f9d5)
>>>> => __ioremap_caller (ffffffffa005aac1)
>>>> => ttm_bo_vmap (ffffffffc040f266)
>>>> => drm_gem_vram_vmap (ffffffffc042c5cd)
>>>> => drm_gem_vmap (ffffffffc0506a7f)
>>>> => drm_client_buffer_vmap (ffffffffc0523741)
>>>> => drm_fb_helper_damage_work (ffffffffc049a34a)
>>>> => process_one_work (ffffffffa00dd92e)
>>>> => worker_thread (ffffffffa00dde46)
>>>> => kthread (ffffffffa00e22c4)
>>>> => ret_from_fork (ffffffffa0004192)
>>>> I see a regular call to memtype_reserve(), but never a matching call to
>>>> memtype_free(), thus I suspect a leak of I/O maps in this code.
>>>> --
>>>> Chuck Lever
>>>
>>> -- 
>>> Thomas Zimmermann
>>> Graphics Driver Developer
>>> SUSE Software Solutions Germany GmbH
>>> Maxfeldstr. 5, 90409 Nürnberg, Germany
>>> (HRB 36809, AG Nürnberg)
>>> Geschäftsführer: Felix Imendörffer
>>
>> --
>> Chuck Lever
> 
> --
> Chuck Lever
> 
> 
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel