[PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[Yang Wei]

From: Yang Wei <yang.wei@linux.alibaba.com>

We found _set_memory_array() and set_memory_array_wb() takes more than
500ms in kernel space in the following scenario.
So use this patch to trigger schedule for each page, to avoid other
threads getting stuck.

    0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
    0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
    0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
    0xffffffff8107796f reserve_memtype  ([kernel.kallsyms])
    0xffffffff81075e98 _set_memory_array  ([kernel.kallsyms])
    0xffffffffc0ef6083 nv_alloc_system_pages  [nvidia] ([kernel.kallsyms])

    0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
    0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
    0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
    0xffffffff8107745a free_memtype.part.7  ([kernel.kallsyms])
    0xffffffff8107606e set_memory_array_wb  ([kernel.kallsyms])
    0xffffffffc0ef6291 nv_free_system_pages  [nvidia]([kernel.kallsyms])

Signed-off-by: Yang Wei <yang.wei@linux.alibaba.com>
Tested-by: Yang Wei <yang.wei@linux.alibaba.com>
---
 arch/x86/mm/pageattr.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index 101f3ad..f7c67b8 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -1625,6 +1625,7 @@ static int _set_memory_array(unsigned long *addr, int addrinarray,
 					new_type, NULL);
 		if (ret)
 			goto out_free;
+		cond_resched();
 	}
 
 	/* If WC, set to UC- first and then WC */
@@ -1989,6 +1990,7 @@ int set_pages_array_wb(struct page **pages, int addrinarray)
 		start = page_to_pfn(pages[i]) << PAGE_SHIFT;
 		end = start + PAGE_SIZE;
 		free_memtype(start, end);
+		cond_resched();
 	}
 
 	return 0;
-- 
1.8.3.1

Re: [PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[Greg KH]

On Fri, Nov 26, 2021 at 03:32:26PM +0800, Yang Wei wrote:
> From: Yang Wei <yang.wei@linux.alibaba.com>
> 
> We found _set_memory_array() and set_memory_array_wb() takes more than
> 500ms in kernel space in the following scenario.
> So use this patch to trigger schedule for each page, to avoid other
> threads getting stuck.
> 
>     0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
>     0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
>     0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
>     0xffffffff8107796f reserve_memtype  ([kernel.kallsyms])
>     0xffffffff81075e98 _set_memory_array  ([kernel.kallsyms])
>     0xffffffffc0ef6083 nv_alloc_system_pages  [nvidia] ([kernel.kallsyms])
> 
>     0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
>     0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
>     0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
>     0xffffffff8107745a free_memtype.part.7  ([kernel.kallsyms])
>     0xffffffff8107606e set_memory_array_wb  ([kernel.kallsyms])
>     0xffffffffc0ef6291 nv_free_system_pages  [nvidia]([kernel.kallsyms])
> 
> Signed-off-by: Yang Wei <yang.wei@linux.alibaba.com>
> Tested-by: Yang Wei <yang.wei@linux.alibaba.com>
> ---
>  arch/x86/mm/pageattr.c | 2 ++
>  1 file changed, 2 insertions(+)

<formletter>

This is not the correct way to submit patches for inclusion in the
stable kernel tree.  Please read:
    https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for how to do this properly.

</formletter>

Re: [PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[Greg KH]

On Fri, Nov 26, 2021 at 03:32:26PM +0800, Yang Wei wrote:
> From: Yang Wei <yang.wei@linux.alibaba.com>
> 
> We found _set_memory_array() and set_memory_array_wb() takes more than
> 500ms in kernel space in the following scenario.
> So use this patch to trigger schedule for each page, to avoid other
> threads getting stuck.
> 
>     0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
>     0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
>     0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
>     0xffffffff8107796f reserve_memtype  ([kernel.kallsyms])
>     0xffffffff81075e98 _set_memory_array  ([kernel.kallsyms])
>     0xffffffffc0ef6083 nv_alloc_system_pages  [nvidia] ([kernel.kallsyms])
> 
>     0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
>     0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
>     0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
>     0xffffffff8107745a free_memtype.part.7  ([kernel.kallsyms])
>     0xffffffff8107606e set_memory_array_wb  ([kernel.kallsyms])
>     0xffffffffc0ef6291 nv_free_system_pages  [nvidia]([kernel.kallsyms])
> 
> Signed-off-by: Yang Wei <yang.wei@linux.alibaba.com>
> Tested-by: Yang Wei <yang.wei@linux.alibaba.com>
> ---
>  arch/x86/mm/pageattr.c | 2 ++
>  1 file changed, 2 insertions(+)

Why is this 4.19-only?

What commit in Linus's tree resolved this issue?

confused,

greg k-h

Re: [PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[YangWei]

在 2021/11/27 00:36, Greg KH 写道:
> On Fri, Nov 26, 2021 at 03:32:26PM +0800, Yang Wei wrote:
>> From: Yang Wei <yang.wei@linux.alibaba.com>
>>
>> We found _set_memory_array() and set_memory_array_wb() takes more than
>> 500ms in kernel space in the following scenario.
>> So use this patch to trigger schedule for each page, to avoid other
>> threads getting stuck.
>>
>>      0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
>>      0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
>>      0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
>>      0xffffffff8107796f reserve_memtype  ([kernel.kallsyms])
>>      0xffffffff81075e98 _set_memory_array  ([kernel.kallsyms])
>>      0xffffffffc0ef6083 nv_alloc_system_pages  [nvidia] ([kernel.kallsyms])
>>
>>      0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
>>      0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
>>      0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
>>      0xffffffff8107745a free_memtype.part.7  ([kernel.kallsyms])
>>      0xffffffff8107606e set_memory_array_wb  ([kernel.kallsyms])
>>      0xffffffffc0ef6291 nv_free_system_pages  [nvidia]([kernel.kallsyms])
>>
>> Signed-off-by: Yang Wei <yang.wei@linux.alibaba.com>
>> Tested-by: Yang Wei <yang.wei@linux.alibaba.com>
>> ---
>>   arch/x86/mm/pageattr.c | 2 ++
>>   1 file changed, 2 insertions(+)
> Why is this 4.19-only?
>
> What commit in Linus's tree resolved this issue?
>
> confused,
>
> greg k-h

We found that the nvidia driver calling 
nv_alloc_system_pages()/nv_free_system_pages()

takes long-time in kernel space, which makes other threads getting stuck 
on non-preemptible

kernel. And it causes the GPU to drop frames on our 4.19 host environment.

This patch can significantly reduce dropped frames. We are currently 
only testing on

4.19, but it should also workson 4.4, 4.9 and 4.14.

The set_memory_array_wb() and set_memory_array_xx() have been removed 
since 5.4, so

Linus's tree does not including the fix of this issue.

Re: [PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[Dave Hansen]

On 11/27/21 8:25 AM, YangWei wrote:
> We found that the nvidia driver calling
> nv_alloc_system_pages()/nv_free_system_pages()

Is that the proprietary nvidia driver?

Re: [PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[YangWei]

在 2021/11/28 02:09, Dave Hansen 写道:
> On 11/27/21 8:25 AM, YangWei wrote:
>> We found that the nvidia driver calling
>> nv_alloc_system_pages()/nv_free_system_pages()
> Is that the proprietary nvidia driver?

Nvidia display driver 460.73.01. Since there is no source code, we 
cannot modify and recompile the driver. I think the kernel interface 
set_memory_array_wb()/set_memory_array_xx() should be adapted to any 
parameter input.

Re: [PATCH 4.19] x86/mm: add cond_resched() in _set_memory_array() and set_memory_array_wb()

[Greg KH]

On Sun, Nov 28, 2021 at 12:25:23AM +0800, YangWei wrote:
> 在 2021/11/27 00:36, Greg KH 写道:
> > On Fri, Nov 26, 2021 at 03:32:26PM +0800, Yang Wei wrote:
> > > From: Yang Wei <yang.wei@linux.alibaba.com>
> > > 
> > > We found _set_memory_array() and set_memory_array_wb() takes more than
> > > 500ms in kernel space in the following scenario.
> > > So use this patch to trigger schedule for each page, to avoid other
> > > threads getting stuck.
> > > 
> > >      0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
> > >      0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
> > >      0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
> > >      0xffffffff8107796f reserve_memtype  ([kernel.kallsyms])
> > >      0xffffffff81075e98 _set_memory_array  ([kernel.kallsyms])
> > >      0xffffffffc0ef6083 nv_alloc_system_pages  [nvidia] ([kernel.kallsyms])
> > > 
> > >      0xffffffff810a34d2 find_next_iomem_res  ([kernel.kallsyms])
> > >      0xffffffff810a3d40 walk_system_ram_range  ([kernel.kallsyms])
> > >      0xffffffff810772ca pat_pagerange_is_ram  ([kernel.kallsyms])
> > >      0xffffffff8107745a free_memtype.part.7  ([kernel.kallsyms])
> > >      0xffffffff8107606e set_memory_array_wb  ([kernel.kallsyms])
> > >      0xffffffffc0ef6291 nv_free_system_pages  [nvidia]([kernel.kallsyms])
> > > 
> > > Signed-off-by: Yang Wei <yang.wei@linux.alibaba.com>
> > > Tested-by: Yang Wei <yang.wei@linux.alibaba.com>
> > > ---
> > >   arch/x86/mm/pageattr.c | 2 ++
> > >   1 file changed, 2 insertions(+)
> > Why is this 4.19-only?
> > 
> > What commit in Linus's tree resolved this issue?
> > 
> > confused,
> > 
> > greg k-h
> 
> We found that the nvidia driver calling
> nv_alloc_system_pages()/nv_free_system_pages()

For obvious reasons, we do not care about this type of problem at all.
Please contact the vendor responsible for this as they are the only ones
who can do anything about it.

good luck!

greg k-h