[Bug 211287] New: Softdirty bit does not work with hugetlb

[Bug 211287] New: Softdirty bit does not work with hugetlb

[Mike Kravetz]

>> Start Bug Report <<
Bug ID: 211287
https://bugzilla.kernel.org/show_bug.cgi?id=211287
Summary: Softdirty bit does not work with hugetlb

When a memory region is mapped with huge pages, the softdirty bit is set only
right after the huge pages is mapped. After the memory mapping, if the
softdirty bit is cleared and the memory is written again, the softdirty bit is
not set when reading from the process's pagemap.
>> End Bug Report <<

I am not surprised with this reported bug.  The page fault code diverges
pretty quickly for hugetlb pages.  I have not looked closely at the details
of soft dirty, and do not immediately see where in the fault path for normal
pages it gets reset.  But, I only took a quick glance.

I can work on adding support for hugetlb.

Can you provide some details about your use case?
-- 
Mike Kravetz

RE: [Bug 211287] New: Softdirty bit does not work with hugetlb

[shu wang]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]

Our use case is:
We maps both PMEM and hugetlb DRAM to process’s address space to provide large amount of memory to the application. Periodically, we clear the soft dirty bit by writing ‘4’ to the clear_refs. Then, we check the page’s soft dirty bit from pagemap and use this information to track write activity to memory. We migrate the data between DRAM and PMEM based the write activity for better performance.

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Mike Kravetz<mailto:mike.kravetz@oracle.com>
Sent: 2021年1月21日 16:58
To: malate_wangshu@hotmail.com<mailto:malate_wangshu@hotmail.com>
Cc: Linux Memory Management List<mailto:linux-mm@kvack.org>
Subject: [Bug 211287] New: Softdirty bit does not work with hugetlb

>> Start Bug Report <<
Bug ID: 211287
https://bugzilla.kernel.org/show_bug.cgi?id=211287
Summary: Softdirty bit does not work with hugetlb

When a memory region is mapped with huge pages, the softdirty bit is set only
right after the huge pages is mapped. After the memory mapping, if the
softdirty bit is cleared and the memory is written again, the softdirty bit is
not set when reading from the process's pagemap.
>> End Bug Report <<

I am not surprised with this reported bug.  The page fault code diverges
pretty quickly for hugetlb pages.  I have not looked closely at the details
of soft dirty, and do not immediately see where in the fault path for normal
pages it gets reset.  But, I only took a quick glance.

I can work on adding support for hugetlb.

Can you provide some details about your use case?
--
Mike Kravetz


[-- Attachment #2: Type: text/html, Size: 4192 bytes --]

Re: [Bug 211287] New: Softdirty bit does not work with hugetlb

[Mike Kravetz]

On 1/22/21 10:42 AM, shu wang wrote:
> Our use case is:
> We maps both PMEM and hugetlb DRAM to process’s address space to provide large amount of memory to the application. Periodically, we clear the soft dirty bit by writing ‘4’ to the clear_refs. Then, we check the page’s soft dirty bit from pagemap and use this information to track write activity to memory. We migrate the data between DRAM and PMEM based the write activity for better performance.
> 

Thank you for the information!

If someone does not beat me to it, I should be able to look at this next week.

-- 
Mike Kravetz

> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
> 
> From: Mike Kravetz<mailto:mike.kravetz@oracle.com>
> Sent: 2021年1月21日 16:58
> To: malate_wangshu@hotmail.com<mailto:malate_wangshu@hotmail.com>
> Cc: Linux Memory Management List<mailto:linux-mm@kvack.org>
> Subject: [Bug 211287] New: Softdirty bit does not work with hugetlb
> 
>>> Start Bug Report <<
> Bug ID: 211287
> https://bugzilla.kernel.org/show_bug.cgi?id=211287
> Summary: Softdirty bit does not work with hugetlb
> 
> When a memory region is mapped with huge pages, the softdirty bit is set only
> right after the huge pages is mapped. After the memory mapping, if the
> softdirty bit is cleared and the memory is written again, the softdirty bit is
> not set when reading from the process's pagemap.
>>> End Bug Report <<
> 
> I am not surprised with this reported bug.  The page fault code diverges
> pretty quickly for hugetlb pages.  I have not looked closely at the details
> of soft dirty, and do not immediately see where in the fault path for normal
> pages it gets reset.  But, I only took a quick glance.
> 
> I can work on adding support for hugetlb.
> 
> Can you provide some details about your use case?
> --
> Mike Kravetz

Re: [Bug 211287] New: Softdirty bit does not work with hugetlb

[Mike Kravetz]

On 1/22/21 1:50 PM, Mike Kravetz wrote:
> On 1/22/21 10:42 AM, shu wang wrote:
>> Our use case is:
>> We maps both PMEM and hugetlb DRAM to process’s address space to provide large amount of memory to the application. Periodically, we clear the soft dirty bit by writing ‘4’ to the clear_refs. Then, we check the page’s soft dirty bit from pagemap and use this information to track write activity to memory. We migrate the data between DRAM and PMEM based the write activity for better performance.
>>
> 
> Thank you for the information!
> 
> If someone does not beat me to it, I should be able to look at this next week.

Just a quick update.

The /proc clear_refs code handles THP pages and 'normal' pages.  There is
no check/processing for hugetlb pages.  All the code will do is clear the
VM_SOFTDIRTY vma flag and vma_set_page_prot.  None of the individual ptes
or page flags are touched.

The only soft dirty checking done by the /proc pagemap code is at the vma
level.  It checks the VM_SOFTDIRTY flag.   It does not look for soft dirty
on the individual ptes.

That explains the behavior described in the bug report.

With a quick and dirty hack I could make the code work in limited testing.
Much work remains.

If anyone wants to jump in and help, feel free.
-- 
Mike Kravetz