[PATCH] After swapout/swapin private dirty mappings become clean

[PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

/proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
clean.

When a page with private file mapping becomes dirty, the vma will be in both
i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
as dirty and backed by the file.

But when those dirty pages gets swapped out, and when they are read back from
swap, they would be marked as clean, as it should be, as they are part of swap
cache now.

But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
clean. The pages are actually in same state i.e., dirty with respect to file
still, but which was once reported as dirty is now being reported as clean to
user-space.

This confuses tools like gdb which uses this information. Those tools think
that those pages were never modified and it creates problem when they create
dumps.

The file mapping of the vma also cannot be broken as pages never read earlier,
will still have to come from the file. Just that those dirty pages have become
clean anonymous pages.

During swaping in, restoring the exact state as dirty file-backed pages before
swapout would be useless, as there in no real bug. Breaking the vma with only
anonymous pages as seperate vmas unnecessary may not be a good thing as well.
So let us just export the information that a file-backed vma has anonymous
dirty pages.

Export this information in smaps by prepending file-names with "[anon]+", when
some of the pages in a file backed vma become anonymous.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..68f9806 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
 	 */
 	if (file) {
 		pad_len_spaces(m, len);
+		if (vma->anon_vma)
+			seq_puts(m, "[anon]+");
 		seq_path(m, &file->f_path, "\n");
 	} else {
 		const char *name = arch_vma_name(vma);

[PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

/proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
clean.

When a page with private file mapping becomes dirty, the vma will be in both
i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
as dirty and backed by the file.

But when those dirty pages gets swapped out, and when they are read back from
swap, they would be marked as clean, as it should be, as they are part of swap
cache now.

But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
clean. The pages are actually in same state i.e., dirty with respect to file
still, but which was once reported as dirty is now being reported as clean to
user-space.

This confuses tools like gdb which uses this information. Those tools think
that those pages were never modified and it creates problem when they create
dumps.

The file mapping of the vma also cannot be broken as pages never read earlier,
will still have to come from the file. Just that those dirty pages have become
clean anonymous pages.

During swaping in, restoring the exact state as dirty file-backed pages before
swapout would be useless, as there in no real bug. Breaking the vma with only
anonymous pages as seperate vmas unnecessary may not be a good thing as well.
So let us just export the information that a file-backed vma has anonymous
dirty pages.

Export this information in smaps by prepending file-names with "[anon]+", when
some of the pages in a file backed vma become anonymous.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..68f9806 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
 	 */
 	if (file) {
 		pad_len_spaces(m, len);
+		if (vma->anon_vma)
+			seq_puts(m, "[anon]+");
 		seq_path(m, &file->f_path, "\n");
 	} else {
 		const char *name = arch_vma_name(vma);


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Richard Guenther]

On Tue, 14 Sep 2010, Nikanth Karthikesan wrote:

> /proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
> clean.
> 
> When a page with private file mapping becomes dirty, the vma will be in both
> i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
> as dirty and backed by the file.
> 
> But when those dirty pages gets swapped out, and when they are read back from
> swap, they would be marked as clean, as it should be, as they are part of swap
> cache now.
> 
> But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
> clean. The pages are actually in same state i.e., dirty with respect to file
> still, but which was once reported as dirty is now being reported as clean to
> user-space.
> 
> This confuses tools like gdb which uses this information. Those tools think
> that those pages were never modified and it creates problem when they create
> dumps.
> 
> The file mapping of the vma also cannot be broken as pages never read earlier,
> will still have to come from the file. Just that those dirty pages have become
> clean anonymous pages.
> 
> During swaping in, restoring the exact state as dirty file-backed pages before
> swapout would be useless, as there in no real bug. Breaking the vma with only
> anonymous pages as seperate vmas unnecessary may not be a good thing as well.
> So let us just export the information that a file-backed vma has anonymous
> dirty pages.
> 
> Export this information in smaps by prepending file-names with "[anon]+", when
> some of the pages in a file backed vma become anonymous.

For the sake of not breaking existing tools I'd prefer appending
" [anon]" instead.

Though a much simpler thing would be to account
the clean anon pages as Private_Dirty (with respect to the backing
file displayed).  Anonymous vmas in /proc/smaps seem to contain
Private_Dirty pages as well.  So I still don't understand why this
isn't just an accounting bug.

Thanks,
Richard.

> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> 
> ---
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..68f9806 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
>  	 */
>  	if (file) {
>  		pad_len_spaces(m, len);
> +		if (vma->anon_vma)
> +			seq_puts(m, "[anon]+");
>  		seq_path(m, &file->f_path, "\n");
>  	} else {
>  		const char *name = arch_vma_name(vma);
> 
> 
> 

-- 
Richard Guenther <rguenther@suse.de>
Novell / SUSE Labs
SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 - GF: Markus Rex

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Richard Guenther]

On Tue, 14 Sep 2010, Nikanth Karthikesan wrote:

> /proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
> clean.
> 
> When a page with private file mapping becomes dirty, the vma will be in both
> i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
> as dirty and backed by the file.
> 
> But when those dirty pages gets swapped out, and when they are read back from
> swap, they would be marked as clean, as it should be, as they are part of swap
> cache now.
> 
> But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
> clean. The pages are actually in same state i.e., dirty with respect to file
> still, but which was once reported as dirty is now being reported as clean to
> user-space.
> 
> This confuses tools like gdb which uses this information. Those tools think
> that those pages were never modified and it creates problem when they create
> dumps.
> 
> The file mapping of the vma also cannot be broken as pages never read earlier,
> will still have to come from the file. Just that those dirty pages have become
> clean anonymous pages.
> 
> During swaping in, restoring the exact state as dirty file-backed pages before
> swapout would be useless, as there in no real bug. Breaking the vma with only
> anonymous pages as seperate vmas unnecessary may not be a good thing as well.
> So let us just export the information that a file-backed vma has anonymous
> dirty pages.
> 
> Export this information in smaps by prepending file-names with "[anon]+", when
> some of the pages in a file backed vma become anonymous.

For the sake of not breaking existing tools I'd prefer appending
" [anon]" instead.

Though a much simpler thing would be to account
the clean anon pages as Private_Dirty (with respect to the backing
file displayed).  Anonymous vmas in /proc/smaps seem to contain
Private_Dirty pages as well.  So I still don't understand why this
isn't just an accounting bug.

Thanks,
Richard.

> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> 
> ---
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..68f9806 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
>  	 */
>  	if (file) {
>  		pad_len_spaces(m, len);
> +		if (vma->anon_vma)
> +			seq_puts(m, "[anon]+");
>  		seq_path(m, &file->f_path, "\n");
>  	} else {
>  		const char *name = arch_vma_name(vma);
> 
> 
> 

-- 
Richard Guenther <rguenther@suse.de>
Novell / SUSE Labs
SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 - GF: Markus Rex

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Tuesday 14 September 2010 17:03:59 Richard Guenther wrote:
> On Tue, 14 Sep 2010, Nikanth Karthikesan wrote:
> > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > become clean.
> >
> > When a page with private file mapping becomes dirty, the vma will be in
> > both i_mmap tree and anon_vma list. The /proc/$pid/smaps will account
> > these pages as dirty and backed by the file.
> >
> > But when those dirty pages gets swapped out, and when they are read back
> > from swap, they would be marked as clean, as it should be, as they are
> > part of swap cache now.
> >
> > But the /proc/$pid/smaps would report the vma as a mapping of a file and
> > it is clean. The pages are actually in same state i.e., dirty with
> > respect to file still, but which was once reported as dirty is now being
> > reported as clean to user-space.
> >
> > This confuses tools like gdb which uses this information. Those tools
> > think that those pages were never modified and it creates problem when
> > they create dumps.
> >
> > The file mapping of the vma also cannot be broken as pages never read
> > earlier, will still have to come from the file. Just that those dirty
> > pages have become clean anonymous pages.
> >
> > During swaping in, restoring the exact state as dirty file-backed pages
> > before swapout would be useless, as there in no real bug. Breaking the
> > vma with only anonymous pages as seperate vmas unnecessary may not be a
> > good thing as well. So let us just export the information that a
> > file-backed vma has anonymous dirty pages.
> >
> > Export this information in smaps by prepending file-names with "[anon]+",
> > when some of the pages in a file backed vma become anonymous.
> 
> For the sake of not breaking existing tools I'd prefer appending
> " [anon]" instead.
> 

A filename might have it as suffix!

> Though a much simpler thing would be to account
> the clean anon pages as Private_Dirty (with respect to the backing
> file displayed).

Agreed.

> Anonymous vmas in /proc/smaps seem to contain
> Private_Dirty pages as well.  So I still don't understand why this
> isn't just an accounting bug.
> 

anonymous pages can be dirty as well. Even these pages now marked as clean 
would be marked as dirty again, if they are re-dirtied.

Thanks
Nikanth

> Thanks,
> Richard.
> 
> > Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> >
> > ---
> >
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 439fc1f..68f9806 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct
> > vm_area_struct *vma) */
> >  	if (file) {
> >  		pad_len_spaces(m, len);
> > +		if (vma->anon_vma)
> > +			seq_puts(m, "[anon]+");
> >  		seq_path(m, &file->f_path, "\n");
> >  	} else {
> >  		const char *name = arch_vma_name(vma);
> 

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Tuesday 14 September 2010 17:03:59 Richard Guenther wrote:
> On Tue, 14 Sep 2010, Nikanth Karthikesan wrote:
> > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > become clean.
> >
> > When a page with private file mapping becomes dirty, the vma will be in
> > both i_mmap tree and anon_vma list. The /proc/$pid/smaps will account
> > these pages as dirty and backed by the file.
> >
> > But when those dirty pages gets swapped out, and when they are read back
> > from swap, they would be marked as clean, as it should be, as they are
> > part of swap cache now.
> >
> > But the /proc/$pid/smaps would report the vma as a mapping of a file and
> > it is clean. The pages are actually in same state i.e., dirty with
> > respect to file still, but which was once reported as dirty is now being
> > reported as clean to user-space.
> >
> > This confuses tools like gdb which uses this information. Those tools
> > think that those pages were never modified and it creates problem when
> > they create dumps.
> >
> > The file mapping of the vma also cannot be broken as pages never read
> > earlier, will still have to come from the file. Just that those dirty
> > pages have become clean anonymous pages.
> >
> > During swaping in, restoring the exact state as dirty file-backed pages
> > before swapout would be useless, as there in no real bug. Breaking the
> > vma with only anonymous pages as seperate vmas unnecessary may not be a
> > good thing as well. So let us just export the information that a
> > file-backed vma has anonymous dirty pages.
> >
> > Export this information in smaps by prepending file-names with "[anon]+",
> > when some of the pages in a file backed vma become anonymous.
> 
> For the sake of not breaking existing tools I'd prefer appending
> " [anon]" instead.
> 

A filename might have it as suffix!

> Though a much simpler thing would be to account
> the clean anon pages as Private_Dirty (with respect to the backing
> file displayed).

Agreed.

> Anonymous vmas in /proc/smaps seem to contain
> Private_Dirty pages as well.  So I still don't understand why this
> isn't just an accounting bug.
> 

anonymous pages can be dirty as well. Even these pages now marked as clean 
would be marked as dirty again, if they are re-dirtied.

Thanks
Nikanth

> Thanks,
> Richard.
> 
> > Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> >
> > ---
> >
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 439fc1f..68f9806 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct
> > vm_area_struct *vma) */
> >  	if (file) {
> >  		pad_len_spaces(m, len);
> > +		if (vma->anon_vma)
> > +			seq_puts(m, "[anon]+");
> >  		seq_path(m, &file->f_path, "\n");
> >  	} else {
> >  		const char *name = arch_vma_name(vma);
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

/proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
clean.

When a page with private file mapping becomes dirty, the vma will be in both
i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
as dirty and backed by the file.

But when those dirty pages gets swapped out, and when they are read back from
swap, they would be marked as clean, as it should be, as they are part of swap
cache now.

But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
clean. The pages are actually in same state i.e., dirty with respect to file
still, but which was once reported as dirty is now being reported as clean to
user-space.

This confuses tools like gdb which uses this information. Those tools think
that those pages were never modified and it creates problem when they create
dumps.

The file mapping of the vma also cannot be broken as pages never read earlier,
will still have to come from the file. Just that those dirty pages have become
clean anonymous pages.

So instead when a file backed vma has anonymous pages report them as dirty
pages. As those pages are dirty with respect to the backing file.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..06fc468 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 				mss->shared_clean += PAGE_SIZE;
 			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
 		} else {
-			if (pte_dirty(ptent))
+			/*
+			 * File-backed pages, now anonymous are dirty
+			 * with respect to the file.
+			 */
+			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
 				mss->private_dirty += PAGE_SIZE;
 			else
 				mss->private_clean += PAGE_SIZE;

[PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

/proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
clean.

When a page with private file mapping becomes dirty, the vma will be in both
i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
as dirty and backed by the file.

But when those dirty pages gets swapped out, and when they are read back from
swap, they would be marked as clean, as it should be, as they are part of swap
cache now.

But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
clean. The pages are actually in same state i.e., dirty with respect to file
still, but which was once reported as dirty is now being reported as clean to
user-space.

This confuses tools like gdb which uses this information. Those tools think
that those pages were never modified and it creates problem when they create
dumps.

The file mapping of the vma also cannot be broken as pages never read earlier,
will still have to come from the file. Just that those dirty pages have become
clean anonymous pages.

So instead when a file backed vma has anonymous pages report them as dirty
pages. As those pages are dirty with respect to the backing file.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..06fc468 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 				mss->shared_clean += PAGE_SIZE;
 			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
 		} else {
-			if (pte_dirty(ptent))
+			/*
+			 * File-backed pages, now anonymous are dirty
+			 * with respect to the file.
+			 */
+			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
 				mss->private_dirty += PAGE_SIZE;
 			else
 				mss->private_clean += PAGE_SIZE;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[KOSAKI Motohiro]

> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..06fc468 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>  				mss->shared_clean += PAGE_SIZE;
>  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
>  		} else {
> -			if (pte_dirty(ptent))
> +			/*
> +			 * File-backed pages, now anonymous are dirty
> +			 * with respect to the file.
> +			 */
> +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
>  				mss->private_dirty += PAGE_SIZE;
>  			else
>  				mss->private_clean += PAGE_SIZE;

This is risky than v1. number of dirties are used a lot of application. 

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[KOSAKI Motohiro]

> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..06fc468 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>  				mss->shared_clean += PAGE_SIZE;
>  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
>  		} else {
> -			if (pte_dirty(ptent))
> +			/*
> +			 * File-backed pages, now anonymous are dirty
> +			 * with respect to the file.
> +			 */
> +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
>  				mss->private_dirty += PAGE_SIZE;
>  			else
>  				mss->private_clean += PAGE_SIZE;

This is risky than v1. number of dirties are used a lot of application. 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 439fc1f..06fc468 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long
> > addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> >  		} else {
> > -			if (pte_dirty(ptent))
> > +			/*
> > +			 * File-backed pages, now anonymous are dirty
> > +			 * with respect to the file.
> > +			 */
> > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> >  				mss->private_dirty += PAGE_SIZE;
> >  			else
> >  				mss->private_clean += PAGE_SIZE;
> 
> This is risky than v1. number of dirties are used a lot of application.
> 

This is exactly to help those applications, as currently after swap-out and 
swap-in, the same pages are accounted as "Private_Clean:" instead of 
"Private_Dirty:".

Thanks
Nikanth

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 439fc1f..06fc468 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long
> > addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> >  		} else {
> > -			if (pte_dirty(ptent))
> > +			/*
> > +			 * File-backed pages, now anonymous are dirty
> > +			 * with respect to the file.
> > +			 */
> > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> >  				mss->private_dirty += PAGE_SIZE;
> >  			else
> >  				mss->private_clean += PAGE_SIZE;
> 
> This is risky than v1. number of dirties are used a lot of application.
> 

This is exactly to help those applications, as currently after swap-out and 
swap-in, the same pages are accounted as "Private_Clean:" instead of 
"Private_Dirty:".

Thanks
Nikanth

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > index 439fc1f..06fc468 100644
> > > --- a/fs/proc/task_mmu.c
> > > +++ b/fs/proc/task_mmu.c
> > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long
> > > addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> > >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > >  		} else {
> > > -			if (pte_dirty(ptent))
> > > +			/*
> > > +			 * File-backed pages, now anonymous are dirty
> > > +			 * with respect to the file.
> > > +			 */
> > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > >  				mss->private_dirty += PAGE_SIZE;
> > >  			else
> > >  				mss->private_clean += PAGE_SIZE;
> > 
> > This is risky than v1. number of dirties are used a lot of application.
> > 
> 
> This is exactly to help those applications, as currently after swap-out and 
> swap-in, the same pages are accounted as "Private_Clean:" instead of 
> "Private_Dirty:".

I don't think so. incorrect infomation bring a lot of confusion rather than its worth.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > index 439fc1f..06fc468 100644
> > > --- a/fs/proc/task_mmu.c
> > > +++ b/fs/proc/task_mmu.c
> > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long
> > > addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> > >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > >  		} else {
> > > -			if (pte_dirty(ptent))
> > > +			/*
> > > +			 * File-backed pages, now anonymous are dirty
> > > +			 * with respect to the file.
> > > +			 */
> > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > >  				mss->private_dirty += PAGE_SIZE;
> > >  			else
> > >  				mss->private_clean += PAGE_SIZE;
> > 
> > This is risky than v1. number of dirties are used a lot of application.
> > 
> 
> This is exactly to help those applications, as currently after swap-out and 
> swap-in, the same pages are accounted as "Private_Clean:" instead of 
> "Private_Dirty:".

I don't think so. incorrect infomation bring a lot of confusion rather than its worth.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:18:11 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > index 439fc1f..06fc468 100644
> > > > --- a/fs/proc/task_mmu.c
> > > > +++ b/fs/proc/task_mmu.c
> > > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned
> > > > long addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> > > >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > > >  		} else {
> > > > -			if (pte_dirty(ptent))
> > > > +			/*
> > > > +			 * File-backed pages, now anonymous are dirty
> > > > +			 * with respect to the file.
> > > > +			 */
> > > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > > >  				mss->private_dirty += PAGE_SIZE;
> > > >  			else
> > > >  				mss->private_clean += PAGE_SIZE;
> > >
> > > This is risky than v1. number of dirties are used a lot of application.
> >
> > This is exactly to help those applications, as currently after swap-out
> > and swap-in, the same pages are accounted as "Private_Clean:" instead of
> > "Private_Dirty:".
> 
> I don't think so.

Actually this behaviour is observed. With a simple memhog, you can see pages 
which are "Private_Dirty:", become "Swap:" and then to "Private_Clean:". And 
that confused GDB.

Thanks
Nikanth

> incorrect infomation bring a lot of confusion rather than
>  its worth.
> 

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:18:11 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > index 439fc1f..06fc468 100644
> > > > --- a/fs/proc/task_mmu.c
> > > > +++ b/fs/proc/task_mmu.c
> > > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned
> > > > long addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> > > >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > > >  		} else {
> > > > -			if (pte_dirty(ptent))
> > > > +			/*
> > > > +			 * File-backed pages, now anonymous are dirty
> > > > +			 * with respect to the file.
> > > > +			 */
> > > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > > >  				mss->private_dirty += PAGE_SIZE;
> > > >  			else
> > > >  				mss->private_clean += PAGE_SIZE;
> > >
> > > This is risky than v1. number of dirties are used a lot of application.
> >
> > This is exactly to help those applications, as currently after swap-out
> > and swap-in, the same pages are accounted as "Private_Clean:" instead of
> > "Private_Dirty:".
> 
> I don't think so.

Actually this behaviour is observed. With a simple memhog, you can see pages 
which are "Private_Dirty:", become "Swap:" and then to "Private_Clean:". And 
that confused GDB.

Thanks
Nikanth

> incorrect infomation bring a lot of confusion rather than
>  its worth.
> 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 10:18:11 KOSAKI Motohiro wrote:
> > > On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > > index 439fc1f..06fc468 100644
> > > > > --- a/fs/proc/task_mmu.c
> > > > > +++ b/fs/proc/task_mmu.c
> > > > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned
> > > > > long addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> > > > >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > > > >  		} else {
> > > > > -			if (pte_dirty(ptent))
> > > > > +			/*
> > > > > +			 * File-backed pages, now anonymous are dirty
> > > > > +			 * with respect to the file.
> > > > > +			 */
> > > > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > > > >  				mss->private_dirty += PAGE_SIZE;
> > > > >  			else
> > > > >  				mss->private_clean += PAGE_SIZE;
> > > >
> > > > This is risky than v1. number of dirties are used a lot of application.
> > >
> > > This is exactly to help those applications, as currently after swap-out
> > > and swap-in, the same pages are accounted as "Private_Clean:" instead of
> > > "Private_Dirty:".
> > 
> > I don't think so.
> 
> Actually this behaviour is observed. With a simple memhog, you can see pages 
> which are "Private_Dirty:", become "Swap:" and then to "Private_Clean:". And 
> that confused GDB.

As I said, incorrect information is always no good solustion. We should concern
how to provide good and enough information, but not how to lie.
If currect gdb is crappy, it should fix. 


> > incorrect infomation bring a lot of confusion rather than
> >  its worth.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 10:18:11 KOSAKI Motohiro wrote:
> > > On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > > index 439fc1f..06fc468 100644
> > > > > --- a/fs/proc/task_mmu.c
> > > > > +++ b/fs/proc/task_mmu.c
> > > > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned
> > > > > long addr, unsigned long end, mss->shared_clean += PAGE_SIZE;
> > > > >  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > > > >  		} else {
> > > > > -			if (pte_dirty(ptent))
> > > > > +			/*
> > > > > +			 * File-backed pages, now anonymous are dirty
> > > > > +			 * with respect to the file.
> > > > > +			 */
> > > > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > > > >  				mss->private_dirty += PAGE_SIZE;
> > > > >  			else
> > > > >  				mss->private_clean += PAGE_SIZE;
> > > >
> > > > This is risky than v1. number of dirties are used a lot of application.
> > >
> > > This is exactly to help those applications, as currently after swap-out
> > > and swap-in, the same pages are accounted as "Private_Clean:" instead of
> > > "Private_Dirty:".
> > 
> > I don't think so.
> 
> Actually this behaviour is observed. With a simple memhog, you can see pages 
> which are "Private_Dirty:", become "Swap:" and then to "Private_Clean:". And 
> that confused GDB.

As I said, incorrect information is always no good solustion. We should concern
how to provide good and enough information, but not how to lie.
If currect gdb is crappy, it should fix. 


> > incorrect infomation bring a lot of confusion rather than
> >  its worth.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:50:25 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 10:18:11 KOSAKI Motohiro wrote:
> > > > On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > > > index 439fc1f..06fc468 100644
> > > > > > --- a/fs/proc/task_mmu.c
> > > > > > +++ b/fs/proc/task_mmu.c
> > > > > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd,
> > > > > > unsigned long addr, unsigned long end, mss->shared_clean +=
> > > > > > PAGE_SIZE; mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > > > > >  		} else {
> > > > > > -			if (pte_dirty(ptent))
> > > > > > +			/*
> > > > > > +			 * File-backed pages, now anonymous are dirty
> > > > > > +			 * with respect to the file.
> > > > > > +			 */
> > > > > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > > > > >  				mss->private_dirty += PAGE_SIZE;
> > > > > >  			else
> > > > > >  				mss->private_clean += PAGE_SIZE;
> > > > >
> > > > > This is risky than v1. number of dirties are used a lot of
> > > > > application.
> > > >
> > > > This is exactly to help those applications, as currently after
> > > > swap-out and swap-in, the same pages are accounted as
> > > > "Private_Clean:" instead of "Private_Dirty:".
> > >
> > > I don't think so.
> >
> > Actually this behaviour is observed. With a simple memhog, you can see
> > pages which are "Private_Dirty:", become "Swap:" and then to
> > "Private_Clean:". And that confused GDB.
> 
> As I said, incorrect information is always no good solustion. We should
>  concern how to provide good and enough information, but not how to lie.
> If currect gdb is crappy, it should fix.
> 

How? Current smaps information without this patch provides incorrect 
information. Just because a private dirty page became part of swap cache, it 
shown as clean and backed by a file. If it is shown as clean and backed by 
swap then it is fine.

Thanks
Nikanth

> > > incorrect infomation bring a lot of confusion rather than
> > >  its worth.
> 

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:50:25 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 10:18:11 KOSAKI Motohiro wrote:
> > > > On Wednesday 15 September 2010 05:56:36 KOSAKI Motohiro wrote:
> > > > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > > > index 439fc1f..06fc468 100644
> > > > > > --- a/fs/proc/task_mmu.c
> > > > > > +++ b/fs/proc/task_mmu.c
> > > > > > @@ -368,7 +368,11 @@ static int smaps_pte_range(pmd_t *pmd,
> > > > > > unsigned long addr, unsigned long end, mss->shared_clean +=
> > > > > > PAGE_SIZE; mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
> > > > > >  		} else {
> > > > > > -			if (pte_dirty(ptent))
> > > > > > +			/*
> > > > > > +			 * File-backed pages, now anonymous are dirty
> > > > > > +			 * with respect to the file.
> > > > > > +			 */
> > > > > > +			if (pte_dirty(ptent) || (vma->vm_file && PageAnon(page)))
> > > > > >  				mss->private_dirty += PAGE_SIZE;
> > > > > >  			else
> > > > > >  				mss->private_clean += PAGE_SIZE;
> > > > >
> > > > > This is risky than v1. number of dirties are used a lot of
> > > > > application.
> > > >
> > > > This is exactly to help those applications, as currently after
> > > > swap-out and swap-in, the same pages are accounted as
> > > > "Private_Clean:" instead of "Private_Dirty:".
> > >
> > > I don't think so.
> >
> > Actually this behaviour is observed. With a simple memhog, you can see
> > pages which are "Private_Dirty:", become "Swap:" and then to
> > "Private_Clean:". And that confused GDB.
> 
> As I said, incorrect information is always no good solustion. We should
>  concern how to provide good and enough information, but not how to lie.
> If currect gdb is crappy, it should fix.
> 

How? Current smaps information without this patch provides incorrect 
information. Just because a private dirty page became part of swap cache, it 
shown as clean and backed by a file. If it is shown as clean and backed by 
swap then it is fine.

Thanks
Nikanth

> > > incorrect infomation bring a lot of confusion rather than
> > >  its worth.
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Balbir Singh]

* Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:

> How? Current smaps information without this patch provides incorrect 
> information. Just because a private dirty page became part of swap cache, it 
> shown as clean and backed by a file. If it is shown as clean and backed by 
> swap then it is fine.
>

How is GDB using this information?  

-- 
	Three Cheers,
	Balbir

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Balbir Singh]

* Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:

> How? Current smaps information without this patch provides incorrect 
> information. Just because a private dirty page became part of swap cache, it 
> shown as clean and backed by a file. If it is shown as clean and backed by 
> swap then it is fine.
>

How is GDB using this information?  

-- 
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Balbir Singh wrote:

> * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> 
> > How? Current smaps information without this patch provides incorrect 
> > information. Just because a private dirty page became part of swap cache, it 
> > shown as clean and backed by a file. If it is shown as clean and backed by 
> > swap then it is fine.
> >
> 
> How is GDB using this information?  

GDB counts the number of dirty and swapped pages in a private mapping and
based on that decides whether it needs to dump it to a core file or not.
If there are no dirty or swapped pages gdb assumes it can reconstruct
the mapping from the original backing file.  This way for example
shared libraries do not end up in the core file.

Richard.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Balbir Singh wrote:

> * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> 
> > How? Current smaps information without this patch provides incorrect 
> > information. Just because a private dirty page became part of swap cache, it 
> > shown as clean and backed by a file. If it is shown as clean and backed by 
> > swap then it is fine.
> >
> 
> How is GDB using this information?  

GDB counts the number of dirty and swapped pages in a private mapping and
based on that decides whether it needs to dump it to a core file or not.
If there are no dirty or swapped pages gdb assumes it can reconstruct
the mapping from the original backing file.  This way for example
shared libraries do not end up in the core file.

Richard.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Matt Mackall]

On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> On Wed, 15 Sep 2010, Balbir Singh wrote:
> 
> > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > 
> > > How? Current smaps information without this patch provides incorrect 
> > > information. Just because a private dirty page became part of swap cache, it 
> > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > swap then it is fine.
> > >
> > 
> > How is GDB using this information?  
> 
> GDB counts the number of dirty and swapped pages in a private mapping and
> based on that decides whether it needs to dump it to a core file or not.
> If there are no dirty or swapped pages gdb assumes it can reconstruct
> the mapping from the original backing file.  This way for example
> shared libraries do not end up in the core file.

This whole discussion is a little disturbing.

The page is being reported clean as per the kernel's definition of
clean, full stop.

So either there's a latent bug/inconsistency in the kernel VM or
external tools are misinterpreting this data. But smaps is just
reporting what's there, the fault doesn't lie in smaps. So fixing smaps
just hides the problem, wherever it is.

Richard's report that the page is still clean after swapoff suggests the
inconsistency lies in the VM.

-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Matt Mackall]

On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> On Wed, 15 Sep 2010, Balbir Singh wrote:
> 
> > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > 
> > > How? Current smaps information without this patch provides incorrect 
> > > information. Just because a private dirty page became part of swap cache, it 
> > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > swap then it is fine.
> > >
> > 
> > How is GDB using this information?  
> 
> GDB counts the number of dirty and swapped pages in a private mapping and
> based on that decides whether it needs to dump it to a core file or not.
> If there are no dirty or swapped pages gdb assumes it can reconstruct
> the mapping from the original backing file.  This way for example
> shared libraries do not end up in the core file.

This whole discussion is a little disturbing.

The page is being reported clean as per the kernel's definition of
clean, full stop.

So either there's a latent bug/inconsistency in the kernel VM or
external tools are misinterpreting this data. But smaps is just
reporting what's there, the fault doesn't lie in smaps. So fixing smaps
just hides the problem, wherever it is.

Richard's report that the page is still clean after swapoff suggests the
inconsistency lies in the VM.

-- 
Mathematics is the supreme nostalgia of our time.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Matt Mackall wrote:

> On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > 
> > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > 
> > > > How? Current smaps information without this patch provides incorrect 
> > > > information. Just because a private dirty page became part of swap cache, it 
> > > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > > swap then it is fine.
> > > >
> > > 
> > > How is GDB using this information?  
> > 
> > GDB counts the number of dirty and swapped pages in a private mapping and
> > based on that decides whether it needs to dump it to a core file or not.
> > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > the mapping from the original backing file.  This way for example
> > shared libraries do not end up in the core file.
> 
> This whole discussion is a little disturbing.
>
> The page is being reported clean as per the kernel's definition of
> clean, full stop.
> 
> So either there's a latent bug/inconsistency in the kernel VM or
> external tools are misinterpreting this data. But smaps is just
> reporting what's there, the fault doesn't lie in smaps. So fixing smaps
> just hides the problem, wherever it is.
> 
> Richard's report that the page is still clean after swapoff suggests the
> inconsistency lies in the VM.

Well - the discussion is about the /proc/smaps interface and
inconsistencies in what it reports.  In particular the interface
does not have the capability of reporting all details the kernel
has, so it might make sense to not "report a page clean as per
the kernel's definition of clean", but only in a /proc/smaps
context definition of clean that makes sense.

So, for

7ffff81ff000-7ffff8201000 r--p 000a8000 08:01 16376 /bin/bash
Size:                  8 kB
Rss:                   8 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         8 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Swap:                  0 kB

I expect both pages of that mapping to be file-backed by /bin/bash.
But surprisingly one page is actually backed by anonymous memory
(it was changed, then mapped readonly, swapped out and swapped in
again).

Thus, the bug is the above inconsistency in /proc/smaps.  Whether
there are internal kernel inconsistencies as well doesn't really
matter to this problem (as there is no way to distinguish
pages that are now backed by anonymous memory in that interface).

Richard.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Matt Mackall wrote:

> On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > 
> > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > 
> > > > How? Current smaps information without this patch provides incorrect 
> > > > information. Just because a private dirty page became part of swap cache, it 
> > > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > > swap then it is fine.
> > > >
> > > 
> > > How is GDB using this information?  
> > 
> > GDB counts the number of dirty and swapped pages in a private mapping and
> > based on that decides whether it needs to dump it to a core file or not.
> > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > the mapping from the original backing file.  This way for example
> > shared libraries do not end up in the core file.
> 
> This whole discussion is a little disturbing.
>
> The page is being reported clean as per the kernel's definition of
> clean, full stop.
> 
> So either there's a latent bug/inconsistency in the kernel VM or
> external tools are misinterpreting this data. But smaps is just
> reporting what's there, the fault doesn't lie in smaps. So fixing smaps
> just hides the problem, wherever it is.
> 
> Richard's report that the page is still clean after swapoff suggests the
> inconsistency lies in the VM.

Well - the discussion is about the /proc/smaps interface and
inconsistencies in what it reports.  In particular the interface
does not have the capability of reporting all details the kernel
has, so it might make sense to not "report a page clean as per
the kernel's definition of clean", but only in a /proc/smaps
context definition of clean that makes sense.

So, for

7ffff81ff000-7ffff8201000 r--p 000a8000 08:01 16376 /bin/bash
Size:                  8 kB
Rss:                   8 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         8 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Swap:                  0 kB

I expect both pages of that mapping to be file-backed by /bin/bash.
But surprisingly one page is actually backed by anonymous memory
(it was changed, then mapped readonly, swapped out and swapped in
again).

Thus, the bug is the above inconsistency in /proc/smaps.  Whether
there are internal kernel inconsistencies as well doesn't really
matter to this problem (as there is no way to distinguish
pages that are now backed by anonymous memory in that interface).

Richard.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Matt Mackall]

[adding Hugh]

On Wed, 2010-09-15 at 16:53 +0200, Richard Guenther wrote:
> On Wed, 15 Sep 2010, Matt Mackall wrote:
> 
> > On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> > > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > 
> > > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > 
> > > > > How? Current smaps information without this patch provides incorrect 
> > > > > information. Just because a private dirty page became part of swap cache, it 
> > > > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > > > swap then it is fine.
> > > > >
> > > > 
> > > > How is GDB using this information?  
> > > 
> > > GDB counts the number of dirty and swapped pages in a private mapping and
> > > based on that decides whether it needs to dump it to a core file or not.
> > > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > > the mapping from the original backing file.  This way for example
> > > shared libraries do not end up in the core file.
> > 
> > This whole discussion is a little disturbing.
> >
> > The page is being reported clean as per the kernel's definition of
> > clean, full stop.
> > 
> > So either there's a latent bug/inconsistency in the kernel VM or
> > external tools are misinterpreting this data. But smaps is just
> > reporting what's there, the fault doesn't lie in smaps. So fixing smaps
> > just hides the problem, wherever it is.
> > 
> > Richard's report that the page is still clean after swapoff suggests the
> > inconsistency lies in the VM.
> 
> Well - the discussion is about the /proc/smaps interface and
> inconsistencies in what it reports.  In particular the interface
> does not have the capability of reporting all details the kernel
> has, so it might make sense to not "report a page clean as per
> the kernel's definition of clean", but only in a /proc/smaps
> context definition of clean that makes sense.
> 
> So, for
> 
> 7ffff81ff000-7ffff8201000 r--p 000a8000 08:01 16376 /bin/bash
> Size:                  8 kB
> Rss:                   8 kB
> Pss:                   8 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         8 kB
> Private_Dirty:         0 kB
> Referenced:            4 kB
> Swap:                  0 kB
> 
> I expect both pages of that mapping to be file-backed by /bin/bash.
> But surprisingly one page is actually backed by anonymous memory
> (it was changed, then mapped readonly, swapped out and swapped in
> again).
> 
> Thus, the bug is the above inconsistency in /proc/smaps.

But that's my point: the consistency problem is NOT in smaps. The page
is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
not there is MORE information smaps could be reporting is irrelevant,
the information it IS reporting is consistent with the underlying VM
data. If there's an inconsistency about what it means to be clean, it's
either in the VM or in your head.

And I frankly think it's in the VM.

In any case, I don't think Nikanth's fix is the right fix, as it
basically says "you can't trust any of this". Either swap should return
the pages to their pre-swap dirty state in the VM, or we should add
another field here:

Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB 

See?

-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Matt Mackall]

[adding Hugh]

On Wed, 2010-09-15 at 16:53 +0200, Richard Guenther wrote:
> On Wed, 15 Sep 2010, Matt Mackall wrote:
> 
> > On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> > > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > 
> > > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > 
> > > > > How? Current smaps information without this patch provides incorrect 
> > > > > information. Just because a private dirty page became part of swap cache, it 
> > > > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > > > swap then it is fine.
> > > > >
> > > > 
> > > > How is GDB using this information?  
> > > 
> > > GDB counts the number of dirty and swapped pages in a private mapping and
> > > based on that decides whether it needs to dump it to a core file or not.
> > > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > > the mapping from the original backing file.  This way for example
> > > shared libraries do not end up in the core file.
> > 
> > This whole discussion is a little disturbing.
> >
> > The page is being reported clean as per the kernel's definition of
> > clean, full stop.
> > 
> > So either there's a latent bug/inconsistency in the kernel VM or
> > external tools are misinterpreting this data. But smaps is just
> > reporting what's there, the fault doesn't lie in smaps. So fixing smaps
> > just hides the problem, wherever it is.
> > 
> > Richard's report that the page is still clean after swapoff suggests the
> > inconsistency lies in the VM.
> 
> Well - the discussion is about the /proc/smaps interface and
> inconsistencies in what it reports.  In particular the interface
> does not have the capability of reporting all details the kernel
> has, so it might make sense to not "report a page clean as per
> the kernel's definition of clean", but only in a /proc/smaps
> context definition of clean that makes sense.
> 
> So, for
> 
> 7ffff81ff000-7ffff8201000 r--p 000a8000 08:01 16376 /bin/bash
> Size:                  8 kB
> Rss:                   8 kB
> Pss:                   8 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         8 kB
> Private_Dirty:         0 kB
> Referenced:            4 kB
> Swap:                  0 kB
> 
> I expect both pages of that mapping to be file-backed by /bin/bash.
> But surprisingly one page is actually backed by anonymous memory
> (it was changed, then mapped readonly, swapped out and swapped in
> again).
> 
> Thus, the bug is the above inconsistency in /proc/smaps.

But that's my point: the consistency problem is NOT in smaps. The page
is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
not there is MORE information smaps could be reporting is irrelevant,
the information it IS reporting is consistent with the underlying VM
data. If there's an inconsistency about what it means to be clean, it's
either in the VM or in your head.

And I frankly think it's in the VM.

In any case, I don't think Nikanth's fix is the right fix, as it
basically says "you can't trust any of this". Either swap should return
the pages to their pre-swap dirty state in the VM, or we should add
another field here:

Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB 

See?

-- 
Mathematics is the supreme nostalgia of our time.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Matt Mackall wrote:

> [adding Hugh]
> 
> On Wed, 2010-09-15 at 16:53 +0200, Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Matt Mackall wrote:
> > 
> > > On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> > > > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > > 
> > > > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > > 
> > > > > > How? Current smaps information without this patch provides incorrect 
> > > > > > information. Just because a private dirty page became part of swap cache, it 
> > > > > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > > > > swap then it is fine.
> > > > > >
> > > > > 
> > > > > How is GDB using this information?  
> > > > 
> > > > GDB counts the number of dirty and swapped pages in a private mapping and
> > > > based on that decides whether it needs to dump it to a core file or not.
> > > > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > > > the mapping from the original backing file.  This way for example
> > > > shared libraries do not end up in the core file.
> > > 
> > > This whole discussion is a little disturbing.
> > >
> > > The page is being reported clean as per the kernel's definition of
> > > clean, full stop.
> > > 
> > > So either there's a latent bug/inconsistency in the kernel VM or
> > > external tools are misinterpreting this data. But smaps is just
> > > reporting what's there, the fault doesn't lie in smaps. So fixing smaps
> > > just hides the problem, wherever it is.
> > > 
> > > Richard's report that the page is still clean after swapoff suggests the
> > > inconsistency lies in the VM.
> > 
> > Well - the discussion is about the /proc/smaps interface and
> > inconsistencies in what it reports.  In particular the interface
> > does not have the capability of reporting all details the kernel
> > has, so it might make sense to not "report a page clean as per
> > the kernel's definition of clean", but only in a /proc/smaps
> > context definition of clean that makes sense.
> > 
> > So, for
> > 
> > 7ffff81ff000-7ffff8201000 r--p 000a8000 08:01 16376 /bin/bash
> > Size:                  8 kB
> > Rss:                   8 kB
> > Pss:                   8 kB
> > Shared_Clean:          0 kB
> > Shared_Dirty:          0 kB
> > Private_Clean:         8 kB
> > Private_Dirty:         0 kB
> > Referenced:            4 kB
> > Swap:                  0 kB
> > 
> > I expect both pages of that mapping to be file-backed by /bin/bash.
> > But surprisingly one page is actually backed by anonymous memory
> > (it was changed, then mapped readonly, swapped out and swapped in
> > again).
> > 
> > Thus, the bug is the above inconsistency in /proc/smaps.
> 
> But that's my point: the consistency problem is NOT in smaps. The page
> is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> not there is MORE information smaps could be reporting is irrelevant,
> the information it IS reporting is consistent with the underlying VM
> data. If there's an inconsistency about what it means to be clean, it's
> either in the VM or in your head.
> 
> And I frankly think it's in the VM.
> 
> In any case, I don't think Nikanth's fix is the right fix, as it
> basically says "you can't trust any of this". Either swap should return
> the pages to their pre-swap dirty state in the VM, or we should add
> another field here:
> 
> Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB 
> 
> See?

Well.  There is also the case where the page is swapped in again
but still allocated in the swap cache.  So it's swap-backed,
private and clean (because the copy in swap is still valid).  But
in that case it's not accounted to "Swap:" (presumably because
Rss + Swap wouldn't add to the mappings size).

I only care about consistency in /proc/smaps, but agree that
an anonymous page that is not backed by swap-cache should always
be dirty (in case it was cowed from the zero page at any point
of course).  Probably that inconsistency doesn't matter, as if it
isn't swap-backed even a clean anonmous page can't be simply thrown 
away (in fact, "clean" or "dirty" doesn't have a meaningful
semantics for anonymous memory IMHO).

Richard.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Matt Mackall wrote:

> [adding Hugh]
> 
> On Wed, 2010-09-15 at 16:53 +0200, Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Matt Mackall wrote:
> > 
> > > On Wed, 2010-09-15 at 16:14 +0200, Richard Guenther wrote:
> > > > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > > 
> > > > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > > 
> > > > > > How? Current smaps information without this patch provides incorrect 
> > > > > > information. Just because a private dirty page became part of swap cache, it 
> > > > > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > > > > swap then it is fine.
> > > > > >
> > > > > 
> > > > > How is GDB using this information?  
> > > > 
> > > > GDB counts the number of dirty and swapped pages in a private mapping and
> > > > based on that decides whether it needs to dump it to a core file or not.
> > > > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > > > the mapping from the original backing file.  This way for example
> > > > shared libraries do not end up in the core file.
> > > 
> > > This whole discussion is a little disturbing.
> > >
> > > The page is being reported clean as per the kernel's definition of
> > > clean, full stop.
> > > 
> > > So either there's a latent bug/inconsistency in the kernel VM or
> > > external tools are misinterpreting this data. But smaps is just
> > > reporting what's there, the fault doesn't lie in smaps. So fixing smaps
> > > just hides the problem, wherever it is.
> > > 
> > > Richard's report that the page is still clean after swapoff suggests the
> > > inconsistency lies in the VM.
> > 
> > Well - the discussion is about the /proc/smaps interface and
> > inconsistencies in what it reports.  In particular the interface
> > does not have the capability of reporting all details the kernel
> > has, so it might make sense to not "report a page clean as per
> > the kernel's definition of clean", but only in a /proc/smaps
> > context definition of clean that makes sense.
> > 
> > So, for
> > 
> > 7ffff81ff000-7ffff8201000 r--p 000a8000 08:01 16376 /bin/bash
> > Size:                  8 kB
> > Rss:                   8 kB
> > Pss:                   8 kB
> > Shared_Clean:          0 kB
> > Shared_Dirty:          0 kB
> > Private_Clean:         8 kB
> > Private_Dirty:         0 kB
> > Referenced:            4 kB
> > Swap:                  0 kB
> > 
> > I expect both pages of that mapping to be file-backed by /bin/bash.
> > But surprisingly one page is actually backed by anonymous memory
> > (it was changed, then mapped readonly, swapped out and swapped in
> > again).
> > 
> > Thus, the bug is the above inconsistency in /proc/smaps.
> 
> But that's my point: the consistency problem is NOT in smaps. The page
> is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> not there is MORE information smaps could be reporting is irrelevant,
> the information it IS reporting is consistent with the underlying VM
> data. If there's an inconsistency about what it means to be clean, it's
> either in the VM or in your head.
> 
> And I frankly think it's in the VM.
> 
> In any case, I don't think Nikanth's fix is the right fix, as it
> basically says "you can't trust any of this". Either swap should return
> the pages to their pre-swap dirty state in the VM, or we should add
> another field here:
> 
> Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB 
> 
> See?

Well.  There is also the case where the page is swapped in again
but still allocated in the swap cache.  So it's swap-backed,
private and clean (because the copy in swap is still valid).  But
in that case it's not accounted to "Swap:" (presumably because
Rss + Swap wouldn't add to the mappings size).

I only care about consistency in /proc/smaps, but agree that
an anonymous page that is not backed by swap-cache should always
be dirty (in case it was cowed from the zero page at any point
of course).  Probably that inconsistency doesn't matter, as if it
isn't swap-backed even a clean anonmous page can't be simply thrown 
away (in fact, "clean" or "dirty" doesn't have a meaningful
semantics for anonymous memory IMHO).

Richard.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Hugh Dickins]

On Wed, Sep 15, 2010 at 10:24 AM, Matt Mackall <mpm@selenic.com> wrote:

> But that's my point: the consistency problem is NOT in smaps. The page
> is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> not there is MORE information smaps could be reporting is irrelevant,
> the information it IS reporting is consistent with the underlying VM
> data. If there's an inconsistency about what it means to be clean, it's
> either in the VM or in your head.
>
> And I frankly think it's in the VM.

I don't believe there's any problem in the VM here, we'd be having
SIGSEGVs all over if there were.

The problem is that /proc/pid/smaps exports a simplified view of the
VM, and Richard and Nikanth were hoping that it gave them some info
which it has never pretended to give them,

It happens to use a pte_dirty(ptent) test: you could argue that that
should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
the issue" which Richard sees with swapoff/swapon), or you could argue
that it should be pte_dirty(ptent) || PageDirty(page) ||
PageSwapCache(page) (which would then note clean copies of swap cache
as dirty in the sense which Richard and Nikanth are interested in).

But after these years, we should probably assume that most users of
/proc/pid/smaps are used to the existing pte_dirty(ptent) test, and
would be troubled by a departure from it.

>
> In any case, I don't think Nikanth's fix is the right fix, as it
> basically says "you can't trust any of this". Either swap should return
> the pages to their pre-swap dirty state in the VM, or we should add
> another field here:
>
> Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB

I think that the most widely useful but simple extension of
/proc/pid/smaps, that would give them the info they want, would indeed
be to counts ptes pointing to PageAnon pages and report that total on
an additional line (say, just before "Swap:"); but there's no need for
the derogatory name you propose there, "Anon:" would suit fine!

The original idea of testing for vma->anon_vma was a good one, but it
doesn't fit so well into the /proc/pid/smaps layout; and showing the
actual count of Anon pages could be useful to others (though generally
it would be Anon+Swap they'd be interested in).

Hugh

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Hugh Dickins]

On Wed, Sep 15, 2010 at 10:24 AM, Matt Mackall <mpm@selenic.com> wrote:

> But that's my point: the consistency problem is NOT in smaps. The page
> is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> not there is MORE information smaps could be reporting is irrelevant,
> the information it IS reporting is consistent with the underlying VM
> data. If there's an inconsistency about what it means to be clean, it's
> either in the VM or in your head.
>
> And I frankly think it's in the VM.

I don't believe there's any problem in the VM here, we'd be having
SIGSEGVs all over if there were.

The problem is that /proc/pid/smaps exports a simplified view of the
VM, and Richard and Nikanth were hoping that it gave them some info
which it has never pretended to give them,

It happens to use a pte_dirty(ptent) test: you could argue that that
should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
the issue" which Richard sees with swapoff/swapon), or you could argue
that it should be pte_dirty(ptent) || PageDirty(page) ||
PageSwapCache(page) (which would then note clean copies of swap cache
as dirty in the sense which Richard and Nikanth are interested in).

But after these years, we should probably assume that most users of
/proc/pid/smaps are used to the existing pte_dirty(ptent) test, and
would be troubled by a departure from it.

>
> In any case, I don't think Nikanth's fix is the right fix, as it
> basically says "you can't trust any of this". Either swap should return
> the pages to their pre-swap dirty state in the VM, or we should add
> another field here:
>
> Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB

I think that the most widely useful but simple extension of
/proc/pid/smaps, that would give them the info they want, would indeed
be to counts ptes pointing to PageAnon pages and report that total on
an additional line (say, just before "Swap:"); but there's no need for
the derogatory name you propose there, "Anon:" would suit fine!

The original idea of testing for vma->anon_vma was a good one, but it
doesn't fit so well into the /proc/pid/smaps layout; and showing the
actual count of Anon pages could be useful to others (though generally
it would be Anon+Swap they'd be interested in).

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Matt Mackall]

On Wed, 2010-09-15 at 12:18 -0700, Hugh Dickins wrote:
> On Wed, Sep 15, 2010 at 10:24 AM, Matt Mackall <mpm@selenic.com> wrote:
> 
> > But that's my point: the consistency problem is NOT in smaps. The page
> > is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> > not there is MORE information smaps could be reporting is irrelevant,
> > the information it IS reporting is consistent with the underlying VM
> > data. If there's an inconsistency about what it means to be clean, it's
> > either in the VM or in your head.
> >
> > And I frankly think it's in the VM.
> 
> I don't believe there's any problem in the VM here, we'd be having
> SIGSEGVs all over if there were.

Of course it works. It's just not as orthogonal (aka consistent) as it
could be in this case: it's not actually reflecting any of the usual
meanings of dirtiness here.

> The problem is that /proc/pid/smaps exports a simplified view of the
> VM, and Richard and Nikanth were hoping that it gave them some info
> which it has never pretended to give them,
> 
> It happens to use a pte_dirty(ptent) test: you could argue that that
> should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
> the issue" which Richard sees with swapoff/swapon),

That might be interesting. Are there any other notable cases where
pte_dirty() differs from PageDirty()?

>  or you could argue
> that it should be pte_dirty(ptent) || PageDirty(page) ||
> PageSwapCache(page) (which would then note clean copies of swap cache
> as dirty in the sense which Richard and Nikanth are interested in).
> 
> But after these years, we should probably assume that most users of
> /proc/pid/smaps are used to the existing pte_dirty(ptent) test, and
> would be troubled by a departure from it.

Dunno, my smem tool[1] is probably also expecting too much here and I
should know better!

> > In any case, I don't think Nikanth's fix is the right fix, as it
> > basically says "you can't trust any of this". Either swap should return
> > the pages to their pre-swap dirty state in the VM, or we should add
> > another field here:
> >
> > Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB
> 
> I think that the most widely useful but simple extension of
> /proc/pid/smaps, that would give them the info they want, would indeed
> be to counts ptes pointing to PageAnon pages and report that total on
> an additional line (say, just before "Swap:"); but there's no need for
> the derogatory name you propose there, "Anon:" would suit fine!

Yes, that wasn't a serious suggestion.

[1] http://www.selenic.com/smem/
-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Matt Mackall]

On Wed, 2010-09-15 at 12:18 -0700, Hugh Dickins wrote:
> On Wed, Sep 15, 2010 at 10:24 AM, Matt Mackall <mpm@selenic.com> wrote:
> 
> > But that's my point: the consistency problem is NOT in smaps. The page
> > is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> > not there is MORE information smaps could be reporting is irrelevant,
> > the information it IS reporting is consistent with the underlying VM
> > data. If there's an inconsistency about what it means to be clean, it's
> > either in the VM or in your head.
> >
> > And I frankly think it's in the VM.
> 
> I don't believe there's any problem in the VM here, we'd be having
> SIGSEGVs all over if there were.

Of course it works. It's just not as orthogonal (aka consistent) as it
could be in this case: it's not actually reflecting any of the usual
meanings of dirtiness here.

> The problem is that /proc/pid/smaps exports a simplified view of the
> VM, and Richard and Nikanth were hoping that it gave them some info
> which it has never pretended to give them,
> 
> It happens to use a pte_dirty(ptent) test: you could argue that that
> should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
> the issue" which Richard sees with swapoff/swapon),

That might be interesting. Are there any other notable cases where
pte_dirty() differs from PageDirty()?

>  or you could argue
> that it should be pte_dirty(ptent) || PageDirty(page) ||
> PageSwapCache(page) (which would then note clean copies of swap cache
> as dirty in the sense which Richard and Nikanth are interested in).
> 
> But after these years, we should probably assume that most users of
> /proc/pid/smaps are used to the existing pte_dirty(ptent) test, and
> would be troubled by a departure from it.

Dunno, my smem tool[1] is probably also expecting too much here and I
should know better!

> > In any case, I don't think Nikanth's fix is the right fix, as it
> > basically says "you can't trust any of this". Either swap should return
> > the pages to their pre-swap dirty state in the VM, or we should add
> > another field here:
> >
> > Weird_Anon_Page_You_Should_Pretend_Is_Private_Dirty: 8 kB
> 
> I think that the most widely useful but simple extension of
> /proc/pid/smaps, that would give them the info they want, would indeed
> be to counts ptes pointing to PageAnon pages and report that total on
> an additional line (say, just before "Swap:"); but there's no need for
> the derogatory name you propose there, "Anon:" would suit fine!

Yes, that wasn't a serious suggestion.

[1] http://www.selenic.com/smem/
-- 
Mathematics is the supreme nostalgia of our time.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Matt Mackall wrote:

> On Wed, 2010-09-15 at 12:18 -0700, Hugh Dickins wrote:
> > On Wed, Sep 15, 2010 at 10:24 AM, Matt Mackall <mpm@selenic.com> wrote:
> > 
> > > But that's my point: the consistency problem is NOT in smaps. The page
> > > is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> > > not there is MORE information smaps could be reporting is irrelevant,
> > > the information it IS reporting is consistent with the underlying VM
> > > data. If there's an inconsistency about what it means to be clean, it's
> > > either in the VM or in your head.
> > >
> > > And I frankly think it's in the VM.
> > 
> > I don't believe there's any problem in the VM here, we'd be having
> > SIGSEGVs all over if there were.
> 
> Of course it works. It's just not as orthogonal (aka consistent) as it
> could be in this case: it's not actually reflecting any of the usual
> meanings of dirtiness here.
> 
> > The problem is that /proc/pid/smaps exports a simplified view of the
> > VM, and Richard and Nikanth were hoping that it gave them some info
> > which it has never pretended to give them,
> > 
> > It happens to use a pte_dirty(ptent) test: you could argue that that
> > should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
> > the issue" which Richard sees with swapoff/swapon),
> 
> That might be interesting. Are there any other notable cases where
> pte_dirty() differs from PageDirty()?
> 
> >  or you could argue
> > that it should be pte_dirty(ptent) || PageDirty(page) ||
> > PageSwapCache(page) (which would then note clean copies of swap cache
> > as dirty in the sense which Richard and Nikanth are interested in).
> > 
> > But after these years, we should probably assume that most users of
> > /proc/pid/smaps are used to the existing pte_dirty(ptent) test, and
> > would be troubled by a departure from it.
> 
> Dunno, my smem tool[1] is probably also expecting too much here and I
> should know better!

The question is also what smaps is supposed to deliver (I guess the
only documentation as of that respect is the kernel source ...).

The appearantly incomplete information isn't too useful at the
moment (apart from displaying the numbers in some form).  So
changing the semantics of the numbers in a slight way shouldn't
cause regressions in tools that formerly worked correct in all
cases.

Unfortunately gdb needs some reliable way of checking whether
the mappings content in memory is still equal to that of
the backing file (best without actually carrying out that
comparison ...).  /proc/pid/smaps looked like the most precise
interface for that - now unfortunately it's not usable for this.
Is there any other way?

Richard.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Wed, 15 Sep 2010, Matt Mackall wrote:

> On Wed, 2010-09-15 at 12:18 -0700, Hugh Dickins wrote:
> > On Wed, Sep 15, 2010 at 10:24 AM, Matt Mackall <mpm@selenic.com> wrote:
> > 
> > > But that's my point: the consistency problem is NOT in smaps. The page
> > > is NOT marked dirty, ergo smaps doesn't report it as dirty. Whether or
> > > not there is MORE information smaps could be reporting is irrelevant,
> > > the information it IS reporting is consistent with the underlying VM
> > > data. If there's an inconsistency about what it means to be clean, it's
> > > either in the VM or in your head.
> > >
> > > And I frankly think it's in the VM.
> > 
> > I don't believe there's any problem in the VM here, we'd be having
> > SIGSEGVs all over if there were.
> 
> Of course it works. It's just not as orthogonal (aka consistent) as it
> could be in this case: it's not actually reflecting any of the usual
> meanings of dirtiness here.
> 
> > The problem is that /proc/pid/smaps exports a simplified view of the
> > VM, and Richard and Nikanth were hoping that it gave them some info
> > which it has never pretended to give them,
> > 
> > It happens to use a pte_dirty(ptent) test: you could argue that that
> > should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
> > the issue" which Richard sees with swapoff/swapon),
> 
> That might be interesting. Are there any other notable cases where
> pte_dirty() differs from PageDirty()?
> 
> >  or you could argue
> > that it should be pte_dirty(ptent) || PageDirty(page) ||
> > PageSwapCache(page) (which would then note clean copies of swap cache
> > as dirty in the sense which Richard and Nikanth are interested in).
> > 
> > But after these years, we should probably assume that most users of
> > /proc/pid/smaps are used to the existing pte_dirty(ptent) test, and
> > would be troubled by a departure from it.
> 
> Dunno, my smem tool[1] is probably also expecting too much here and I
> should know better!

The question is also what smaps is supposed to deliver (I guess the
only documentation as of that respect is the kernel source ...).

The appearantly incomplete information isn't too useful at the
moment (apart from displaying the numbers in some form).  So
changing the semantics of the numbers in a slight way shouldn't
cause regressions in tools that formerly worked correct in all
cases.

Unfortunately gdb needs some reliable way of checking whether
the mappings content in memory is still equal to that of
the backing file (best without actually carrying out that
comparison ...).  /proc/pid/smaps looked like the most precise
interface for that - now unfortunately it's not usable for this.
Is there any other way?

Richard.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Hugh Dickins]

On Wed, Sep 15, 2010 at 12:46 PM, Matt Mackall <mpm@selenic.com> wrote:
> On Wed, 2010-09-15 at 12:18 -0700, Hugh Dickins wrote:
>> The problem is that /proc/pid/smaps exports a simplified view of the
>> VM, and Richard and Nikanth were hoping that it gave them some info
>> which it has never pretended to give them,
>>
>> It happens to use a pte_dirty(ptent) test: you could argue that that
>> should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
>> the issue" which Richard sees with swapoff/swapon),
>
> That might be interesting. Are there any other notable cases where
> pte_dirty() differs from PageDirty()?

I don't know about "other notable".  A page may very well be PageDirty
(e.g. modified by a write system call) without any of the ptes
pointing to it (if there even are any) marked as pte_dirty.  A page
may very well not be marked PageDirty yet, though one or more of the
ptes pointing to it have been marked pte_dirty when userspace made a
write access to the page via that pte.   Traditionally (when
/proc/pid/smaps was first reporting dirty versus clean ptes) the pte
dirtiness would later be found and propagated through to PageDirtiness
(clearing the pte dirtiness), which would later be cleaned when the
page was written out to backing store (file or swap).

PeterZ's writeback work in 2.6.19 (set_page_dirty_balance,
clear_page_dirty_for_io etc.) tightened up writeback from (most: tmpfs
is one exception) shared file mmaps, synchronizing the PageDirty more
carefully with the pte_dirty; and perhaps there is some inconsistency
there, that we never felt compelled to keep pte and Page so tightly in
synch in the anonymous/Swap case - it would have been unnecessary
overhead (though I repeatedly forget the essence of why not - file
syncing, and pdflush activity,  were relevant considerations; but I
cannot now put my finger on precisely why shared file writing needed
to be fixed, but anonymous dirtying could be left unchanged).

But even if you replace smaps's pte_dirty(ptent) tests by
pte_dirty(ptent) || PageDirty(page) tests, it wouldn't be doing what
Richard and Nikanth want - they want clean ptes of clean PageSwapCache
to be reported as dirty, despite being clean copies of backing store;
and I can understand your reluctance to go that far.  I think
reporting "Anon:" pages is more useful - in part because we have no
counts of Anon+Swap, yet that's the quantity which vm_enough_memory
may place a limit upon (but beware, it does of course get more
complicated: tmpfs files come out of that tally too, but would not
show up in this way).

Hugh

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Hugh Dickins]

On Wed, Sep 15, 2010 at 12:46 PM, Matt Mackall <mpm@selenic.com> wrote:
> On Wed, 2010-09-15 at 12:18 -0700, Hugh Dickins wrote:
>> The problem is that /proc/pid/smaps exports a simplified view of the
>> VM, and Richard and Nikanth were hoping that it gave them some info
>> which it has never pretended to give them,
>>
>> It happens to use a pte_dirty(ptent) test: you could argue that that
>> should be pte_dirty(ptent) || PageDirty(page) (which would then "fix
>> the issue" which Richard sees with swapoff/swapon),
>
> That might be interesting. Are there any other notable cases where
> pte_dirty() differs from PageDirty()?

I don't know about "other notable".  A page may very well be PageDirty
(e.g. modified by a write system call) without any of the ptes
pointing to it (if there even are any) marked as pte_dirty.  A page
may very well not be marked PageDirty yet, though one or more of the
ptes pointing to it have been marked pte_dirty when userspace made a
write access to the page via that pte.   Traditionally (when
/proc/pid/smaps was first reporting dirty versus clean ptes) the pte
dirtiness would later be found and propagated through to PageDirtiness
(clearing the pte dirtiness), which would later be cleaned when the
page was written out to backing store (file or swap).

PeterZ's writeback work in 2.6.19 (set_page_dirty_balance,
clear_page_dirty_for_io etc.) tightened up writeback from (most: tmpfs
is one exception) shared file mmaps, synchronizing the PageDirty more
carefully with the pte_dirty; and perhaps there is some inconsistency
there, that we never felt compelled to keep pte and Page so tightly in
synch in the anonymous/Swap case - it would have been unnecessary
overhead (though I repeatedly forget the essence of why not - file
syncing, and pdflush activity,  were relevant considerations; but I
cannot now put my finger on precisely why shared file writing needed
to be fixed, but anonymous dirtying could be left unchanged).

But even if you replace smaps's pte_dirty(ptent) tests by
pte_dirty(ptent) || PageDirty(page) tests, it wouldn't be doing what
Richard and Nikanth want - they want clean ptes of clean PageSwapCache
to be reported as dirty, despite being clean copies of backing store;
and I can understand your reluctance to go that far.  I think
reporting "Anon:" pages is more useful - in part because we have no
counts of Anon+Swap, yet that's the quantity which vm_enough_memory
may place a limit upon (but beware, it does of course get more
complicated: tmpfs files come out of that tally too, but would not
show up in this way).

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] Export amount of anonymous memory in a mapping via smaps

[Nikanth Karthikesan]

Export the number of anonymous pages in a mapping via smaps.

Even the private pages in a mapping backed by a file, would be marked as
anonymous, when they are modified. Export this information to user-space via
smaps.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..3c18fc8 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -326,6 +326,7 @@ struct mem_size_stats {
 	unsigned long private_clean;
 	unsigned long private_dirty;
 	unsigned long referenced;
+	unsigned long anonymous;
 	unsigned long swap;
 	u64 pss;
 };
@@ -356,6 +357,9 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 		if (!page)
 			continue;
 
+		if (PageAnon(page))
+			mss->anonymous += PAGE_SIZE;
+
 		mss->resident += PAGE_SIZE;
 		/* Accumulate the size in pages that have been accessed. */
 		if (pte_young(ptent) || PageReferenced(page))
@@ -409,6 +413,7 @@ static int show_smap(struct seq_file *m, void *v)
 		   "Private_Clean:  %8lu kB\n"
 		   "Private_Dirty:  %8lu kB\n"
 		   "Referenced:     %8lu kB\n"
+		   "Anonymous:      %8lu kB\n"
 		   "Swap:           %8lu kB\n"
 		   "KernelPageSize: %8lu kB\n"
 		   "MMUPageSize:    %8lu kB\n",
@@ -420,6 +425,7 @@ static int show_smap(struct seq_file *m, void *v)
 		   mss.private_clean >> 10,
 		   mss.private_dirty >> 10,
 		   mss.referenced >> 10,
+		   mss.anonymous >> 10,
 		   mss.swap >> 10,
 		   vma_kernel_pagesize(vma) >> 10,
 		   vma_mmu_pagesize(vma) >> 10);

[PATCH] Export amount of anonymous memory in a mapping via smaps

[Nikanth Karthikesan]

Export the number of anonymous pages in a mapping via smaps.

Even the private pages in a mapping backed by a file, would be marked as
anonymous, when they are modified. Export this information to user-space via
smaps.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..3c18fc8 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -326,6 +326,7 @@ struct mem_size_stats {
 	unsigned long private_clean;
 	unsigned long private_dirty;
 	unsigned long referenced;
+	unsigned long anonymous;
 	unsigned long swap;
 	u64 pss;
 };
@@ -356,6 +357,9 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 		if (!page)
 			continue;
 
+		if (PageAnon(page))
+			mss->anonymous += PAGE_SIZE;
+
 		mss->resident += PAGE_SIZE;
 		/* Accumulate the size in pages that have been accessed. */
 		if (pte_young(ptent) || PageReferenced(page))
@@ -409,6 +413,7 @@ static int show_smap(struct seq_file *m, void *v)
 		   "Private_Clean:  %8lu kB\n"
 		   "Private_Dirty:  %8lu kB\n"
 		   "Referenced:     %8lu kB\n"
+		   "Anonymous:      %8lu kB\n"
 		   "Swap:           %8lu kB\n"
 		   "KernelPageSize: %8lu kB\n"
 		   "MMUPageSize:    %8lu kB\n",
@@ -420,6 +425,7 @@ static int show_smap(struct seq_file *m, void *v)
 		   mss.private_clean >> 10,
 		   mss.private_dirty >> 10,
 		   mss.referenced >> 10,
+		   mss.anonymous >> 10,
 		   mss.swap >> 10,
 		   vma_kernel_pagesize(vma) >> 10,
 		   vma_mmu_pagesize(vma) >> 10);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Export amount of anonymous memory in a mapping via smaps

[KOSAKI Motohiro]

> Export the number of anonymous pages in a mapping via smaps.
> 
> Even the private pages in a mapping backed by a file, would be marked as
> anonymous, when they are modified. Export this information to user-space via
> smaps.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

Looks good.
	Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>

Re: [PATCH] Export amount of anonymous memory in a mapping via smaps

[KOSAKI Motohiro]

> Export the number of anonymous pages in a mapping via smaps.
> 
> Even the private pages in a mapping backed by a file, would be marked as
> anonymous, when they are modified. Export this information to user-space via
> smaps.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

Looks good.
	Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] Document the new Anonymous field in smaps.

[Nikanth Karthikesan]

Sorry, I missed to add documentation, when I sent the patch. This depends on
the patch titled, "[PATCH] Export amount of anonymous memory in a mapping via
smaps".

Thanks
Nikanth


Document the new Anonymous field in smaps.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index a6aca87..75c7368 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -370,6 +370,7 @@ Shared_Dirty:          0 kB
 Private_Clean:         0 kB
 Private_Dirty:         0 kB
 Referenced:          892 kB
+Anonymous:             0 kB
 Swap:                  0 kB
 KernelPageSize:        4 kB
 MMUPageSize:           4 kB
@@ -380,7 +381,10 @@ the amount of the mapping that is currently resident in RAM, the "proportional
 set size” (divide each shared page by the number of processes sharing it), the
 number of clean and dirty shared pages in the mapping, and the number of clean
 and dirty private pages in the mapping.  The "Referenced" indicates the amount
-of memory currently marked as referenced or accessed.
+of memory currently marked as referenced or accessed. The "Anonymous" shows
+the amount of mapping that is not associated with a file. Even the private
+pages in a mapping associated with a file, would become anonymous, when they
+are modified.
 
 This file is only present if the CONFIG_MMU kernel configuration option is
 enabled.

[PATCH] Document the new Anonymous field in smaps.

[Nikanth Karthikesan]

Sorry, I missed to add documentation, when I sent the patch. This depends on
the patch titled, "[PATCH] Export amount of anonymous memory in a mapping via
smaps".

Thanks
Nikanth


Document the new Anonymous field in smaps.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index a6aca87..75c7368 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -370,6 +370,7 @@ Shared_Dirty:          0 kB
 Private_Clean:         0 kB
 Private_Dirty:         0 kB
 Referenced:          892 kB
+Anonymous:             0 kB
 Swap:                  0 kB
 KernelPageSize:        4 kB
 MMUPageSize:           4 kB
@@ -380,7 +381,10 @@ the amount of the mapping that is currently resident in RAM, the "proportional
 set size” (divide each shared page by the number of processes sharing it), the
 number of clean and dirty shared pages in the mapping, and the number of clean
 and dirty private pages in the mapping.  The "Referenced" indicates the amount
-of memory currently marked as referenced or accessed.
+of memory currently marked as referenced or accessed. The "Anonymous" shows
+the amount of mapping that is not associated with a file. Even the private
+pages in a mapping associated with a file, would become anonymous, when they
+are modified.
 
 This file is only present if the CONFIG_MMU kernel configuration option is
 enabled.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] smaps: fix dirty pages accounting

[KOSAKI Motohiro]

Currently, /proc/<pid>/smaps have wrong dirty pages accounting.
Shared_Dirty and Private_Dirty output only pte dirty pages and
ignore PG_dirty page flag. It is difference against documentation,
but also inconsistent against Referenced field. (Referenced checks
both pte and page flags)

This patch fixes it.

Test program:

 large-array.c
 ---------------------------------------------------
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>

 char array[1*1024*1024*1024L];

 int main(void)
 {
         memset(array, 1, sizeof(array));
         pause();

         return 0;
 }
 ---------------------------------------------------

Test case:
 1. run ./large-array
 2. cat /proc/`pidof large-array`/smaps
 3. swapoff -a
 4. cat /proc/`pidof large-array`/smaps again

Test result:
 <before patch>

00601000-40601000 rw-p 00000000 00:00 0
Size:            1048576 kB
Rss:             1048576 kB
Pss:             1048576 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:    218992 kB   <-- showed pages as clean incorrectly
Private_Dirty:    829584 kB
Referenced:       388364 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB

 <after patch>

00601000-40601000 rw-p 00000000 00:00 0
Size:            1048576 kB
Rss:             1048576 kB
Pss:             1048576 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1048576 kB  <-- fixed
Referenced:       388480 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB

Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
---
 fs/proc/task_mmu.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..7415f13 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -362,13 +362,13 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 			mss->referenced += PAGE_SIZE;
 		mapcount = page_mapcount(page);
 		if (mapcount >= 2) {
-			if (pte_dirty(ptent))
+			if (pte_dirty(ptent) || PageDirty(page))
 				mss->shared_dirty += PAGE_SIZE;
 			else
 				mss->shared_clean += PAGE_SIZE;
 			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
 		} else {
-			if (pte_dirty(ptent))
+			if (pte_dirty(ptent) || PageDirty(page))
 				mss->private_dirty += PAGE_SIZE;
 			else
 				mss->private_clean += PAGE_SIZE;
-- 
1.6.5.2

[PATCH] smaps: fix dirty pages accounting

[KOSAKI Motohiro]

Currently, /proc/<pid>/smaps have wrong dirty pages accounting.
Shared_Dirty and Private_Dirty output only pte dirty pages and
ignore PG_dirty page flag. It is difference against documentation,
but also inconsistent against Referenced field. (Referenced checks
both pte and page flags)

This patch fixes it.

Test program:

 large-array.c
 ---------------------------------------------------
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>

 char array[1*1024*1024*1024L];

 int main(void)
 {
         memset(array, 1, sizeof(array));
         pause();

         return 0;
 }
 ---------------------------------------------------

Test case:
 1. run ./large-array
 2. cat /proc/`pidof large-array`/smaps
 3. swapoff -a
 4. cat /proc/`pidof large-array`/smaps again

Test result:
 <before patch>

00601000-40601000 rw-p 00000000 00:00 0
Size:            1048576 kB
Rss:             1048576 kB
Pss:             1048576 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:    218992 kB   <-- showed pages as clean incorrectly
Private_Dirty:    829584 kB
Referenced:       388364 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB

 <after patch>

00601000-40601000 rw-p 00000000 00:00 0
Size:            1048576 kB
Rss:             1048576 kB
Pss:             1048576 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1048576 kB  <-- fixed
Referenced:       388480 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB

Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
---
 fs/proc/task_mmu.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 439fc1f..7415f13 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -362,13 +362,13 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 			mss->referenced += PAGE_SIZE;
 		mapcount = page_mapcount(page);
 		if (mapcount >= 2) {
-			if (pte_dirty(ptent))
+			if (pte_dirty(ptent) || PageDirty(page))
 				mss->shared_dirty += PAGE_SIZE;
 			else
 				mss->shared_clean += PAGE_SIZE;
 			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
 		} else {
-			if (pte_dirty(ptent))
+			if (pte_dirty(ptent) || PageDirty(page))
 				mss->private_dirty += PAGE_SIZE;
 			else
 				mss->private_clean += PAGE_SIZE;
-- 
1.6.5.2



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] smaps: fix dirty pages accounting

[Hugh Dickins]

On Thu, 16 Sep 2010, KOSAKI Motohiro wrote:

> Currently, /proc/<pid>/smaps have wrong dirty pages accounting.
> Shared_Dirty and Private_Dirty output only pte dirty pages and
> ignore PG_dirty page flag. It is difference against documentation,
> but also inconsistent against Referenced field. (Referenced checks
> both pte and page flags)
> 
> This patch fixes it.
> 
> Test program:
> 
>  large-array.c
>  ---------------------------------------------------
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <string.h>
>  #include <unistd.h>
> 
>  char array[1*1024*1024*1024L];
> 
>  int main(void)
>  {
>          memset(array, 1, sizeof(array));
>          pause();
> 
>          return 0;
>  }
>  ---------------------------------------------------
> 
> Test case:
>  1. run ./large-array
>  2. cat /proc/`pidof large-array`/smaps
>  3. swapoff -a
>  4. cat /proc/`pidof large-array`/smaps again
> 
> Test result:
>  <before patch>
> 
> 00601000-40601000 rw-p 00000000 00:00 0
> Size:            1048576 kB
> Rss:             1048576 kB
> Pss:             1048576 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:    218992 kB   <-- showed pages as clean incorrectly
> Private_Dirty:    829584 kB
> Referenced:       388364 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 
>  <after patch>
> 
> 00601000-40601000 rw-p 00000000 00:00 0
> Size:            1048576 kB
> Rss:             1048576 kB
> Pss:             1048576 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         0 kB
> Private_Dirty:   1048576 kB  <-- fixed
> Referenced:       388480 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 
> Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>

Certainly you can have my

Acked-by: Hugh Dickins <hughd@google.com>

but I think it's for Matt to decide what he's wanting to show there,
and whether it's safe to change after all this time.  I hadn't noticed
the descrepancy between "dirty" and "referenced", that certainly argues
for your patch.

> ---
>  fs/proc/task_mmu.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..7415f13 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -362,13 +362,13 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>  			mss->referenced += PAGE_SIZE;
>  		mapcount = page_mapcount(page);
>  		if (mapcount >= 2) {
> -			if (pte_dirty(ptent))
> +			if (pte_dirty(ptent) || PageDirty(page))
>  				mss->shared_dirty += PAGE_SIZE;
>  			else
>  				mss->shared_clean += PAGE_SIZE;
>  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
>  		} else {
> -			if (pte_dirty(ptent))
> +			if (pte_dirty(ptent) || PageDirty(page))
>  				mss->private_dirty += PAGE_SIZE;
>  			else
>  				mss->private_clean += PAGE_SIZE;
> -- 
> 1.6.5.2

Re: [PATCH] smaps: fix dirty pages accounting

[Hugh Dickins]

On Thu, 16 Sep 2010, KOSAKI Motohiro wrote:

> Currently, /proc/<pid>/smaps have wrong dirty pages accounting.
> Shared_Dirty and Private_Dirty output only pte dirty pages and
> ignore PG_dirty page flag. It is difference against documentation,
> but also inconsistent against Referenced field. (Referenced checks
> both pte and page flags)
> 
> This patch fixes it.
> 
> Test program:
> 
>  large-array.c
>  ---------------------------------------------------
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <string.h>
>  #include <unistd.h>
> 
>  char array[1*1024*1024*1024L];
> 
>  int main(void)
>  {
>          memset(array, 1, sizeof(array));
>          pause();
> 
>          return 0;
>  }
>  ---------------------------------------------------
> 
> Test case:
>  1. run ./large-array
>  2. cat /proc/`pidof large-array`/smaps
>  3. swapoff -a
>  4. cat /proc/`pidof large-array`/smaps again
> 
> Test result:
>  <before patch>
> 
> 00601000-40601000 rw-p 00000000 00:00 0
> Size:            1048576 kB
> Rss:             1048576 kB
> Pss:             1048576 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:    218992 kB   <-- showed pages as clean incorrectly
> Private_Dirty:    829584 kB
> Referenced:       388364 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 
>  <after patch>
> 
> 00601000-40601000 rw-p 00000000 00:00 0
> Size:            1048576 kB
> Rss:             1048576 kB
> Pss:             1048576 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         0 kB
> Private_Dirty:   1048576 kB  <-- fixed
> Referenced:       388480 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 
> Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>

Certainly you can have my

Acked-by: Hugh Dickins <hughd@google.com>

but I think it's for Matt to decide what he's wanting to show there,
and whether it's safe to change after all this time.  I hadn't noticed
the descrepancy between "dirty" and "referenced", that certainly argues
for your patch.

> ---
>  fs/proc/task_mmu.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..7415f13 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -362,13 +362,13 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>  			mss->referenced += PAGE_SIZE;
>  		mapcount = page_mapcount(page);
>  		if (mapcount >= 2) {
> -			if (pte_dirty(ptent))
> +			if (pte_dirty(ptent) || PageDirty(page))
>  				mss->shared_dirty += PAGE_SIZE;
>  			else
>  				mss->shared_clean += PAGE_SIZE;
>  			mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
>  		} else {
> -			if (pte_dirty(ptent))
> +			if (pte_dirty(ptent) || PageDirty(page))
>  				mss->private_dirty += PAGE_SIZE;
>  			else
>  				mss->private_clean += PAGE_SIZE;
> -- 
> 1.6.5.2

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Document the new Anonymous field in smaps.

[Hugh Dickins]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2000 bytes --]

On Thu, 16 Sep 2010, Nikanth Karthikesan wrote:

> Sorry, I missed to add documentation, when I sent the patch. This depends on
> the patch titled, "[PATCH] Export amount of anonymous memory in a mapping via
> smaps".
> 
> Thanks
> Nikanth
> 
> 
> Document the new Anonymous field in smaps.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

Thanks for adding this, but...

> 
> ---
> 
> diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
> index a6aca87..75c7368 100644
> --- a/Documentation/filesystems/proc.txt
> +++ b/Documentation/filesystems/proc.txt
> @@ -370,6 +370,7 @@ Shared_Dirty:          0 kB
>  Private_Clean:         0 kB
>  Private_Dirty:         0 kB
>  Referenced:          892 kB
> +Anonymous:             0 kB
>  Swap:                  0 kB
>  KernelPageSize:        4 kB
>  MMUPageSize:           4 kB
> @@ -380,7 +381,10 @@ the amount of the mapping that is currently resident in RAM, the "proportional
>  set size” (divide each shared page by the number of processes sharing it), the
>  number of clean and dirty shared pages in the mapping, and the number of clean
>  and dirty private pages in the mapping.  The "Referenced" indicates the amount
> -of memory currently marked as referenced or accessed.
> +of memory currently marked as referenced or accessed. The "Anonymous" shows
> +the amount of mapping that is not associated with a file. Even the private
> +pages in a mapping associated with a file, would become anonymous, when they
> +are modified.

... unfortunately it needs rewording, because you're describing it well
with respect to MAP_SHARED versus MAP_PRIVATE, but /proc/pid/smaps has a
different definition of whether pages are shared or private, just depending
on their mapcount at the instant.

Indeed, adding a comment on that would be helpful too, if you've time.

>  
>  This file is only present if the CONFIG_MMU kernel configuration option is
>  enabled.

[PATCH v2] Document the new Anonymous field in smaps.

[Nikanth Karthikesan]

Document the new Anonymous field in smaps.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index a6aca87..b430576 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -370,17 +370,24 @@ Shared_Dirty:          0 kB
 Private_Clean:         0 kB
 Private_Dirty:         0 kB
 Referenced:          892 kB
+Anonymous:             0 kB
 Swap:                  0 kB
 KernelPageSize:        4 kB
 MMUPageSize:           4 kB
 
-The first  of these lines shows  the same information  as is displayed for the
-mapping in /proc/PID/maps.  The remaining lines show  the size of the mapping,
+The first of these lines shows the same information as is displayed for the
+mapping in /proc/PID/maps. The remaining lines show the size of the mapping,
 the amount of the mapping that is currently resident in RAM, the "proportional
 set size” (divide each shared page by the number of processes sharing it), the
 number of clean and dirty shared pages in the mapping, and the number of clean
-and dirty private pages in the mapping.  The "Referenced" indicates the amount
-of memory currently marked as referenced or accessed.
+and dirty private pages in the mapping. Even pages which are part of
+MAP_SHARED mappings, but has only a single pte mapped i.e., used exclusively
+by a process is accounted as private and not as shared. The "Referenced"
+indicates the amount of memory currently marked as referenced or accessed. The
+"Anonymous" shows the number of pages that is not associated with a file. Even
+mappings associated with a file can have anonymous pages. When the mapping is
+MAP_PRIVATE and its pages are modified, the pages are COWed and marked as
+anonymous.
 
 This file is only present if the CONFIG_MMU kernel configuration option is
 enabled.

[PATCH v2] Document the new Anonymous field in smaps.

[Nikanth Karthikesan]

Document the new Anonymous field in smaps.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---

diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index a6aca87..b430576 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -370,17 +370,24 @@ Shared_Dirty:          0 kB
 Private_Clean:         0 kB
 Private_Dirty:         0 kB
 Referenced:          892 kB
+Anonymous:             0 kB
 Swap:                  0 kB
 KernelPageSize:        4 kB
 MMUPageSize:           4 kB
 
-The first  of these lines shows  the same information  as is displayed for the
-mapping in /proc/PID/maps.  The remaining lines show  the size of the mapping,
+The first of these lines shows the same information as is displayed for the
+mapping in /proc/PID/maps. The remaining lines show the size of the mapping,
 the amount of the mapping that is currently resident in RAM, the "proportional
 set size” (divide each shared page by the number of processes sharing it), the
 number of clean and dirty shared pages in the mapping, and the number of clean
-and dirty private pages in the mapping.  The "Referenced" indicates the amount
-of memory currently marked as referenced or accessed.
+and dirty private pages in the mapping. Even pages which are part of
+MAP_SHARED mappings, but has only a single pte mapped i.e., used exclusively
+by a process is accounted as private and not as shared. The "Referenced"
+indicates the amount of memory currently marked as referenced or accessed. The
+"Anonymous" shows the number of pages that is not associated with a file. Even
+mappings associated with a file can have anonymous pages. When the mapping is
+MAP_PRIVATE and its pages are modified, the pages are COWed and marked as
+anonymous.
 
 This file is only present if the CONFIG_MMU kernel configuration option is
 enabled.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] Document the new Anonymous field in smaps.

[Hugh Dickins]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2653 bytes --]

On Fri, 17 Sep 2010, Nikanth Karthikesan wrote:
> Document the new Anonymous field in smaps.

Thanks for doing that, good effort, but your shifts between singular
and plural rather jarred on my ear, so I've rewritten it a little below.
Also added a sentence on "Swap"; but gave up when it came to KernelPageSize
and MMUPageSize, let someone else clarify those later.


[PATCH v3] Document the new Anonymous field in smaps.

From: Nikanth Karthikesan <knikanth@suse.de>

Document the new Anonymous field in smaps, and also the Swap field.
Explain what smaps means by shared and private, which differs from
MAP_SHARED and MAP_PRIVATE.

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
Signed-off-by: Hugh Dickins <hughd@google.com>
---

 Documentation/filesystems/proc.txt |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -370,17 +370,24 @@ Shared_Dirty:          0 kB
 Private_Clean:         0 kB
 Private_Dirty:         0 kB
 Referenced:          892 kB
+Anonymous:             0 kB
 Swap:                  0 kB
 KernelPageSize:        4 kB
 MMUPageSize:           4 kB
 
-The first  of these lines shows  the same information  as is displayed for the
-mapping in /proc/PID/maps.  The remaining lines show  the size of the mapping,
+The first of these lines shows the same information as is displayed for the
+mapping in /proc/PID/maps. The remaining lines show the size of the mapping,
 the amount of the mapping that is currently resident in RAM, the "proportional
 set size” (divide each shared page by the number of processes sharing it), the
 number of clean and dirty shared pages in the mapping, and the number of clean
-and dirty private pages in the mapping.  The "Referenced" indicates the amount
-of memory currently marked as referenced or accessed.
+and dirty private pages in the mapping. Note that even a page which is part of
+a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used
+by only one process, is accounted as private and not as shared. "Referenced"
+indicates the amount of memory currently marked as referenced or accessed.
+"Anonymous" shows the amount of memory that does not belong to any file. Even
+a mapping associated with a file may contain anonymous pages: when MAP_PRIVATE
+and a page is modified, the file page is replaced by a private anonymous copy.
+"Swap" shows how much would-be-anonymous memory is also used, but out on swap.
 
 This file is only present if the CONFIG_MMU kernel configuration option is
 enabled.

Re: [PATCH v2] Document the new Anonymous field in smaps.

[Matt Mackall]

On Mon, 2010-09-20 at 00:11 -0700, Hugh Dickins wrote:
> On Fri, 17 Sep 2010, Nikanth Karthikesan wrote:
> > Document the new Anonymous field in smaps.
> 
> Thanks for doing that, good effort, but your shifts between singular
> and plural rather jarred on my ear, so I've rewritten it a little below.
> Also added a sentence on "Swap"; but gave up when it came to KernelPageSize
> and MMUPageSize, let someone else clarify those later.
> 
> 
> [PATCH v3] Document the new Anonymous field in smaps.
> 
> From: Nikanth Karthikesan <knikanth@suse.de>
> 
> Document the new Anonymous field in smaps, and also the Swap field.
> Explain what smaps means by shared and private, which differs from
> MAP_SHARED and MAP_PRIVATE.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> Signed-off-by: Hugh Dickins <hughd@google.com>

Looks good, 

Acked-by: Matt Mackall <mpm@selenic.com>

> ---
> 
>  Documentation/filesystems/proc.txt |   15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
> 
> --- a/Documentation/filesystems/proc.txt
> +++ b/Documentation/filesystems/proc.txt
> @@ -370,17 +370,24 @@ Shared_Dirty:          0 kB
>  Private_Clean:         0 kB
>  Private_Dirty:         0 kB
>  Referenced:          892 kB
> +Anonymous:             0 kB
>  Swap:                  0 kB
>  KernelPageSize:        4 kB
>  MMUPageSize:           4 kB
>  
> -The first  of these lines shows  the same information  as is displayed for the
> -mapping in /proc/PID/maps.  The remaining lines show  the size of the mapping,
> +The first of these lines shows the same information as is displayed for the
> +mapping in /proc/PID/maps. The remaining lines show the size of the mapping,
>  the amount of the mapping that is currently resident in RAM, the "proportional
>  set size” (divide each shared page by the number of processes sharing it), the
>  number of clean and dirty shared pages in the mapping, and the number of clean
> -and dirty private pages in the mapping.  The "Referenced" indicates the amount
> -of memory currently marked as referenced or accessed.
> +and dirty private pages in the mapping. Note that even a page which is part of
> +a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used
> +by only one process, is accounted as private and not as shared. "Referenced"
> +indicates the amount of memory currently marked as referenced or accessed.
> +"Anonymous" shows the amount of memory that does not belong to any file. Even
> +a mapping associated with a file may contain anonymous pages: when MAP_PRIVATE
> +and a page is modified, the file page is replaced by a private anonymous copy.
> +"Swap" shows how much would-be-anonymous memory is also used, but out on swap.
>  
>  This file is only present if the CONFIG_MMU kernel configuration option is
>  enabled.


-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH v2] Document the new Anonymous field in smaps.

[Matt Mackall]

On Mon, 2010-09-20 at 00:11 -0700, Hugh Dickins wrote:
> On Fri, 17 Sep 2010, Nikanth Karthikesan wrote:
> > Document the new Anonymous field in smaps.
> 
> Thanks for doing that, good effort, but your shifts between singular
> and plural rather jarred on my ear, so I've rewritten it a little below.
> Also added a sentence on "Swap"; but gave up when it came to KernelPageSize
> and MMUPageSize, let someone else clarify those later.
> 
> 
> [PATCH v3] Document the new Anonymous field in smaps.
> 
> From: Nikanth Karthikesan <knikanth@suse.de>
> 
> Document the new Anonymous field in smaps, and also the Swap field.
> Explain what smaps means by shared and private, which differs from
> MAP_SHARED and MAP_PRIVATE.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> Signed-off-by: Hugh Dickins <hughd@google.com>

Looks good, 

Acked-by: Matt Mackall <mpm@selenic.com>

> ---
> 
>  Documentation/filesystems/proc.txt |   15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
> 
> --- a/Documentation/filesystems/proc.txt
> +++ b/Documentation/filesystems/proc.txt
> @@ -370,17 +370,24 @@ Shared_Dirty:          0 kB
>  Private_Clean:         0 kB
>  Private_Dirty:         0 kB
>  Referenced:          892 kB
> +Anonymous:             0 kB
>  Swap:                  0 kB
>  KernelPageSize:        4 kB
>  MMUPageSize:           4 kB
>  
> -The first  of these lines shows  the same information  as is displayed for the
> -mapping in /proc/PID/maps.  The remaining lines show  the size of the mapping,
> +The first of these lines shows the same information as is displayed for the
> +mapping in /proc/PID/maps. The remaining lines show the size of the mapping,
>  the amount of the mapping that is currently resident in RAM, the "proportional
>  set sizea?? (divide each shared page by the number of processes sharing it), the
>  number of clean and dirty shared pages in the mapping, and the number of clean
> -and dirty private pages in the mapping.  The "Referenced" indicates the amount
> -of memory currently marked as referenced or accessed.
> +and dirty private pages in the mapping. Note that even a page which is part of
> +a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used
> +by only one process, is accounted as private and not as shared. "Referenced"
> +indicates the amount of memory currently marked as referenced or accessed.
> +"Anonymous" shows the amount of memory that does not belong to any file. Even
> +a mapping associated with a file may contain anonymous pages: when MAP_PRIVATE
> +and a page is modified, the file page is replaced by a private anonymous copy.
> +"Swap" shows how much would-be-anonymous memory is also used, but out on swap.
>  
>  This file is only present if the CONFIG_MMU kernel configuration option is
>  enabled.


-- 
Mathematics is the supreme nostalgia of our time.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Export amount of anonymous memory in a mapping via smaps

[Hugh Dickins]

On Thu, 16 Sep 2010, Nikanth Karthikesan wrote:

> Export the number of anonymous pages in a mapping via smaps.
> 
> Even the private pages in a mapping backed by a file, would be marked as
> anonymous, when they are modified. Export this information to user-space via
> smaps.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

Acked-by: Hugh Dickins <hughd@google.com>

but I'd prefer if we added a little more justification, such as:

Exporting this count will help gdb to make a better decision on which
areas need to be dumped in its coredump; and should be useful to others
studying the memory usage of a process.

> 
> ---
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..3c18fc8 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -326,6 +326,7 @@ struct mem_size_stats {
>  	unsigned long private_clean;
>  	unsigned long private_dirty;
>  	unsigned long referenced;
> +	unsigned long anonymous;
>  	unsigned long swap;
>  	u64 pss;
>  };
> @@ -356,6 +357,9 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>  		if (!page)
>  			continue;
>  
> +		if (PageAnon(page))
> +			mss->anonymous += PAGE_SIZE;
> +
>  		mss->resident += PAGE_SIZE;
>  		/* Accumulate the size in pages that have been accessed. */
>  		if (pte_young(ptent) || PageReferenced(page))
> @@ -409,6 +413,7 @@ static int show_smap(struct seq_file *m, void *v)
>  		   "Private_Clean:  %8lu kB\n"
>  		   "Private_Dirty:  %8lu kB\n"
>  		   "Referenced:     %8lu kB\n"
> +		   "Anonymous:      %8lu kB\n"
>  		   "Swap:           %8lu kB\n"
>  		   "KernelPageSize: %8lu kB\n"
>  		   "MMUPageSize:    %8lu kB\n",
> @@ -420,6 +425,7 @@ static int show_smap(struct seq_file *m, void *v)
>  		   mss.private_clean >> 10,
>  		   mss.private_dirty >> 10,
>  		   mss.referenced >> 10,
> +		   mss.anonymous >> 10,
>  		   mss.swap >> 10,
>  		   vma_kernel_pagesize(vma) >> 10,
>  		   vma_mmu_pagesize(vma) >> 10);
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 
> 

Re: [PATCH] Export amount of anonymous memory in a mapping via smaps

[Hugh Dickins]

On Thu, 16 Sep 2010, Nikanth Karthikesan wrote:

> Export the number of anonymous pages in a mapping via smaps.
> 
> Even the private pages in a mapping backed by a file, would be marked as
> anonymous, when they are modified. Export this information to user-space via
> smaps.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

Acked-by: Hugh Dickins <hughd@google.com>

but I'd prefer if we added a little more justification, such as:

Exporting this count will help gdb to make a better decision on which
areas need to be dumped in its coredump; and should be useful to others
studying the memory usage of a process.

> 
> ---
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..3c18fc8 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -326,6 +326,7 @@ struct mem_size_stats {
>  	unsigned long private_clean;
>  	unsigned long private_dirty;
>  	unsigned long referenced;
> +	unsigned long anonymous;
>  	unsigned long swap;
>  	u64 pss;
>  };
> @@ -356,6 +357,9 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>  		if (!page)
>  			continue;
>  
> +		if (PageAnon(page))
> +			mss->anonymous += PAGE_SIZE;
> +
>  		mss->resident += PAGE_SIZE;
>  		/* Accumulate the size in pages that have been accessed. */
>  		if (pte_young(ptent) || PageReferenced(page))
> @@ -409,6 +413,7 @@ static int show_smap(struct seq_file *m, void *v)
>  		   "Private_Clean:  %8lu kB\n"
>  		   "Private_Dirty:  %8lu kB\n"
>  		   "Referenced:     %8lu kB\n"
> +		   "Anonymous:      %8lu kB\n"
>  		   "Swap:           %8lu kB\n"
>  		   "KernelPageSize: %8lu kB\n"
>  		   "MMUPageSize:    %8lu kB\n",
> @@ -420,6 +425,7 @@ static int show_smap(struct seq_file *m, void *v)
>  		   mss.private_clean >> 10,
>  		   mss.private_dirty >> 10,
>  		   mss.referenced >> 10,
> +		   mss.anonymous >> 10,
>  		   mss.swap >> 10,
>  		   vma_kernel_pagesize(vma) >> 10,
>  		   vma_mmu_pagesize(vma) >> 10);
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Balbir Singh]

* Richard Guenther <rguenther@suse.de> [2010-09-15 16:14:17]:

> On Wed, 15 Sep 2010, Balbir Singh wrote:
> 
> > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > 
> > > How? Current smaps information without this patch provides incorrect 
> > > information. Just because a private dirty page became part of swap cache, it 
> > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > swap then it is fine.
> > >
> > 
> > How is GDB using this information?  
> 
> GDB counts the number of dirty and swapped pages in a private mapping and
> based on that decides whether it needs to dump it to a core file or not.
> If there are no dirty or swapped pages gdb assumes it can reconstruct
> the mapping from the original backing file.  This way for example
> shared libraries do not end up in the core file.
>

Thanks for clarifying 

-- 
	Three Cheers,
	Balbir

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Balbir Singh]

* Richard Guenther <rguenther@suse.de> [2010-09-15 16:14:17]:

> On Wed, 15 Sep 2010, Balbir Singh wrote:
> 
> > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > 
> > > How? Current smaps information without this patch provides incorrect 
> > > information. Just because a private dirty page became part of swap cache, it 
> > > shown as clean and backed by a file. If it is shown as clean and backed by 
> > > swap then it is fine.
> > >
> > 
> > How is GDB using this information?  
> 
> GDB counts the number of dirty and swapped pages in a private mapping and
> based on that decides whether it needs to dump it to a core file or not.
> If there are no dirty or swapped pages gdb assumes it can reconstruct
> the mapping from the original backing file.  This way for example
> shared libraries do not end up in the core file.
>

Thanks for clarifying 

-- 
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 19:44:17 Richard Guenther wrote:
> On Wed, 15 Sep 2010, Balbir Singh wrote:
> > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > How? Current smaps information without this patch provides incorrect
> > > information. Just because a private dirty page became part of swap
> > > cache, it shown as clean and backed by a file. If it is shown as clean
> > > and backed by swap then it is fine.
> >
> > How is GDB using this information?
> 
> GDB counts the number of dirty and swapped pages in a private mapping and
> based on that decides whether it needs to dump it to a core file or not.
> If there are no dirty or swapped pages gdb assumes it can reconstruct
> the mapping from the original backing file.  This way for example
> shared libraries do not end up in the core file.
> 

Well, may be /proc/pid/pagemap + /proc/kpageflags is enough for this! One can 
get the pageflags using these interfaces. See Documentation/vm/pagemap.txt for 
the explanation on how to do it. There is also a sample program that prints 
page flags using this interface in Documentation/vm/page-types.c.

It is bad that /proc/pid/pagemap is never mentioned in 
Documentation/filesystems/proc.txt. I will send a patch to rectify this.

Thanks
Nikanth

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Wednesday 15 September 2010 19:44:17 Richard Guenther wrote:
> On Wed, 15 Sep 2010, Balbir Singh wrote:
> > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > How? Current smaps information without this patch provides incorrect
> > > information. Just because a private dirty page became part of swap
> > > cache, it shown as clean and backed by a file. If it is shown as clean
> > > and backed by swap then it is fine.
> >
> > How is GDB using this information?
> 
> GDB counts the number of dirty and swapped pages in a private mapping and
> based on that decides whether it needs to dump it to a core file or not.
> If there are no dirty or swapped pages gdb assumes it can reconstruct
> the mapping from the original backing file.  This way for example
> shared libraries do not end up in the core file.
> 

Well, may be /proc/pid/pagemap + /proc/kpageflags is enough for this! One can 
get the pageflags using these interfaces. See Documentation/vm/pagemap.txt for 
the explanation on how to do it. There is also a sample program that prints 
page flags using this interface in Documentation/vm/page-types.c.

It is bad that /proc/pid/pagemap is never mentioned in 
Documentation/filesystems/proc.txt. I will send a patch to rectify this.

Thanks
Nikanth

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] Document /proc/pid/pagemap in Documentation/filesystems/proc.txt

[Nikanth Karthikesan]

Document /proc/pid/pagemap in Documentation/filesystems/proc.txt

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---


diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index a6aca87..a9c85a6 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -136,6 +136,7 @@ Table 1-1: Process specific entries in /proc
  statm		Process memory status information
  status		Process status in human readable form
  wchan		If CONFIG_KALLSYMS is set, a pre-decoded wchan
+ pagemap	Page table
  stack		Report full stack trace, enable via CONFIG_STACKTRACE
  smaps		a extension based on maps, showing the memory consumption of
 		each mapping
@@ -397,6 +398,9 @@ To clear the bits for the file mapped pages associated with the process
     > echo 3 > /proc/PID/clear_refs
 Any other value written to /proc/PID/clear_refs will have no effect.
 
+The /proc/pid/pagemap gives the PFN, which can be used to find the pageflags
+using /proc/kpageflags and number of times a page is mapped using
+/proc/kpagecount. For detailed explanation, see Documentation/vm/pagemap.txt.
 
 1.2 Kernel data
 ---------------

[PATCH] Document /proc/pid/pagemap in Documentation/filesystems/proc.txt

[Nikanth Karthikesan]

Document /proc/pid/pagemap in Documentation/filesystems/proc.txt

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---


diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index a6aca87..a9c85a6 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -136,6 +136,7 @@ Table 1-1: Process specific entries in /proc
  statm		Process memory status information
  status		Process status in human readable form
  wchan		If CONFIG_KALLSYMS is set, a pre-decoded wchan
+ pagemap	Page table
  stack		Report full stack trace, enable via CONFIG_STACKTRACE
  smaps		a extension based on maps, showing the memory consumption of
 		each mapping
@@ -397,6 +398,9 @@ To clear the bits for the file mapped pages associated with the process
     > echo 3 > /proc/PID/clear_refs
 Any other value written to /proc/PID/clear_refs will have no effect.
 
+The /proc/pid/pagemap gives the PFN, which can be used to find the pageflags
+using /proc/kpageflags and number of times a page is mapped using
+/proc/kpagecount. For detailed explanation, see Documentation/vm/pagemap.txt.
 
 1.2 Kernel data
 ---------------

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Document /proc/pid/pagemap in Documentation/filesystems/proc.txt

[Matt Mackall]

On Sun, 2010-09-19 at 23:08 +0530, Nikanth Karthikesan wrote:
> Document /proc/pid/pagemap in Documentation/filesystems/proc.txt
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

Acked-by: Matt Mackall <mpm@selenic.com>

-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH] Document /proc/pid/pagemap in Documentation/filesystems/proc.txt

[Matt Mackall]

On Sun, 2010-09-19 at 23:08 +0530, Nikanth Karthikesan wrote:
> Document /proc/pid/pagemap in Documentation/filesystems/proc.txt
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

Acked-by: Matt Mackall <mpm@selenic.com>

-- 
Mathematics is the supreme nostalgia of our time.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Sunday 19 September 2010 23:07:09 Nikanth Karthikesan wrote:
> On Wednesday 15 September 2010 19:44:17 Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > How? Current smaps information without this patch provides incorrect
> > > > information. Just because a private dirty page became part of swap
> > > > cache, it shown as clean and backed by a file. If it is shown as
> > > > clean and backed by swap then it is fine.
> > >
> > > How is GDB using this information?
> >
> > GDB counts the number of dirty and swapped pages in a private mapping and
> > based on that decides whether it needs to dump it to a core file or not.
> > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > the mapping from the original backing file.  This way for example
> > shared libraries do not end up in the core file.
> 
> Well, may be /proc/pid/pagemap + /proc/kpageflags is enough for this! One
>  can get the pageflags using these interfaces. See
>  Documentation/vm/pagemap.txt for the explanation on how to do it. There is
>  also a sample program that prints page flags using this interface in
>  Documentation/vm/page-types.c.
> 
> It is bad that /proc/pid/pagemap is never mentioned in
> Documentation/filesystems/proc.txt. I will send a patch to rectify this.

Or even simpler, /proc/pid/numa_maps already exports the number of anonymous 
pages in a mapping, if you have CONFIG_NUMA=y! Again not documented in 
Documentation/filesystems/proc.txt

Thanks
Nikanth

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Nikanth Karthikesan]

On Sunday 19 September 2010 23:07:09 Nikanth Karthikesan wrote:
> On Wednesday 15 September 2010 19:44:17 Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > How? Current smaps information without this patch provides incorrect
> > > > information. Just because a private dirty page became part of swap
> > > > cache, it shown as clean and backed by a file. If it is shown as
> > > > clean and backed by swap then it is fine.
> > >
> > > How is GDB using this information?
> >
> > GDB counts the number of dirty and swapped pages in a private mapping and
> > based on that decides whether it needs to dump it to a core file or not.
> > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > the mapping from the original backing file.  This way for example
> > shared libraries do not end up in the core file.
> 
> Well, may be /proc/pid/pagemap + /proc/kpageflags is enough for this! One
>  can get the pageflags using these interfaces. See
>  Documentation/vm/pagemap.txt for the explanation on how to do it. There is
>  also a sample program that prints page flags using this interface in
>  Documentation/vm/page-types.c.
> 
> It is bad that /proc/pid/pagemap is never mentioned in
> Documentation/filesystems/proc.txt. I will send a patch to rectify this.

Or even simpler, /proc/pid/numa_maps already exports the number of anonymous 
pages in a mapping, if you have CONFIG_NUMA=y! Again not documented in 
Documentation/filesystems/proc.txt

Thanks
Nikanth

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Sun, 19 Sep 2010, Nikanth Karthikesan wrote:

> On Wednesday 15 September 2010 19:44:17 Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > How? Current smaps information without this patch provides incorrect
> > > > information. Just because a private dirty page became part of swap
> > > > cache, it shown as clean and backed by a file. If it is shown as clean
> > > > and backed by swap then it is fine.
> > >
> > > How is GDB using this information?
> > 
> > GDB counts the number of dirty and swapped pages in a private mapping and
> > based on that decides whether it needs to dump it to a core file or not.
> > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > the mapping from the original backing file.  This way for example
> > shared libraries do not end up in the core file.
> > 
> 
> Well, may be /proc/pid/pagemap + /proc/kpageflags is enough for this! One can 
> get the pageflags using these interfaces. See Documentation/vm/pagemap.txt for 
> the explanation on how to do it. There is also a sample program that prints 
> page flags using this interface in Documentation/vm/page-types.c.
> 
> It is bad that /proc/pid/pagemap is never mentioned in 
> Documentation/filesystems/proc.txt. I will send a patch to rectify this.

Looks like /proc/kpageflags is root-only, so not a solution for gdb.

Richard.

Re: [PATCH v2] After swapout/swapin private dirty mappings are reported clean in smaps

[Richard Guenther]

On Sun, 19 Sep 2010, Nikanth Karthikesan wrote:

> On Wednesday 15 September 2010 19:44:17 Richard Guenther wrote:
> > On Wed, 15 Sep 2010, Balbir Singh wrote:
> > > * Nikanth Karthikesan <knikanth@suse.de> [2010-09-15 12:01:11]:
> > > > How? Current smaps information without this patch provides incorrect
> > > > information. Just because a private dirty page became part of swap
> > > > cache, it shown as clean and backed by a file. If it is shown as clean
> > > > and backed by swap then it is fine.
> > >
> > > How is GDB using this information?
> > 
> > GDB counts the number of dirty and swapped pages in a private mapping and
> > based on that decides whether it needs to dump it to a core file or not.
> > If there are no dirty or swapped pages gdb assumes it can reconstruct
> > the mapping from the original backing file.  This way for example
> > shared libraries do not end up in the core file.
> > 
> 
> Well, may be /proc/pid/pagemap + /proc/kpageflags is enough for this! One can 
> get the pageflags using these interfaces. See Documentation/vm/pagemap.txt for 
> the explanation on how to do it. There is also a sample program that prints 
> page flags using this interface in Documentation/vm/page-types.c.
> 
> It is bad that /proc/pid/pagemap is never mentioned in 
> Documentation/filesystems/proc.txt. I will send a patch to rectify this.

Looks like /proc/kpageflags is root-only, so not a solution for gdb.

Richard.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> /proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
> clean.
> 
> When a page with private file mapping becomes dirty, the vma will be in both
> i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
> as dirty and backed by the file.
> 
> But when those dirty pages gets swapped out, and when they are read back from
> swap, they would be marked as clean, as it should be, as they are part of swap
> cache now.
> 
> But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
> clean. The pages are actually in same state i.e., dirty with respect to file
> still, but which was once reported as dirty is now being reported as clean to
> user-space.
> 
> This confuses tools like gdb which uses this information. Those tools think
> that those pages were never modified and it creates problem when they create
> dumps.
> 
> The file mapping of the vma also cannot be broken as pages never read earlier,
> will still have to come from the file. Just that those dirty pages have become
> clean anonymous pages.
> 
> During swaping in, restoring the exact state as dirty file-backed pages before
> swapout would be useless, as there in no real bug. Breaking the vma with only
> anonymous pages as seperate vmas unnecessary may not be a good thing as well.
> So let us just export the information that a file-backed vma has anonymous
> dirty pages.

Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need dump out.

Am I missing anything?

 - kosaki

> 
> Export this information in smaps by prepending file-names with "[anon]+", when
> some of the pages in a file backed vma become anonymous.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> 
> ---
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..68f9806 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
>  	 */
>  	if (file) {
>  		pad_len_spaces(m, len);
> +		if (vma->anon_vma)
> +			seq_puts(m, "[anon]+");
>  		seq_path(m, &file->f_path, "\n");
>  	} else {
>  		const char *name = arch_vma_name(vma);
> 
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> /proc/$pid/smaps broken: After swapout/swapin private dirty mappings become
> clean.
> 
> When a page with private file mapping becomes dirty, the vma will be in both
> i_mmap tree and anon_vma list. The /proc/$pid/smaps will account these pages
> as dirty and backed by the file.
> 
> But when those dirty pages gets swapped out, and when they are read back from
> swap, they would be marked as clean, as it should be, as they are part of swap
> cache now.
> 
> But the /proc/$pid/smaps would report the vma as a mapping of a file and it is
> clean. The pages are actually in same state i.e., dirty with respect to file
> still, but which was once reported as dirty is now being reported as clean to
> user-space.
> 
> This confuses tools like gdb which uses this information. Those tools think
> that those pages were never modified and it creates problem when they create
> dumps.
> 
> The file mapping of the vma also cannot be broken as pages never read earlier,
> will still have to come from the file. Just that those dirty pages have become
> clean anonymous pages.
> 
> During swaping in, restoring the exact state as dirty file-backed pages before
> swapout would be useless, as there in no real bug. Breaking the vma with only
> anonymous pages as seperate vmas unnecessary may not be a good thing as well.
> So let us just export the information that a file-backed vma has anonymous
> dirty pages.

Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need dump out.

Am I missing anything?

 - kosaki

> 
> Export this information in smaps by prepending file-names with "[anon]+", when
> some of the pages in a file backed vma become anonymous.
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> 
> ---
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 439fc1f..68f9806 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
>  	 */
>  	if (file) {
>  		pad_len_spaces(m, len);
> +		if (vma->anon_vma)
> +			seq_puts(m, "[anon]+");
>  		seq_path(m, &file->f_path, "\n");
>  	} else {
>  		const char *name = arch_vma_name(vma);
> 
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > become clean.
> >
> > When a page with private file mapping becomes dirty, the vma will be in
> > both i_mmap tree and anon_vma list. The /proc/$pid/smaps will account
> > these pages as dirty and backed by the file.
> >
> > But when those dirty pages gets swapped out, and when they are read back
> > from swap, they would be marked as clean, as it should be, as they are
> > part of swap cache now.
> >
> > But the /proc/$pid/smaps would report the vma as a mapping of a file and
> > it is clean. The pages are actually in same state i.e., dirty with
> > respect to file still, but which was once reported as dirty is now being
> > reported as clean to user-space.
> >
> > This confuses tools like gdb which uses this information. Those tools
> > think that those pages were never modified and it creates problem when
> > they create dumps.
> >
> > The file mapping of the vma also cannot be broken as pages never read
> > earlier, will still have to come from the file. Just that those dirty
> > pages have become clean anonymous pages.
> >
> > During swaping in, restoring the exact state as dirty file-backed pages
> > before swapout would be useless, as there in no real bug. Breaking the
> > vma with only anonymous pages as seperate vmas unnecessary may not be a
> > good thing as well. So let us just export the information that a
> > file-backed vma has anonymous dirty pages.
> 
> Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need dump
>  out.
> 

Yes. When the page is swapped out it is accounted in "Swap:".

> Am I missing anything?
> 

But when it gets swapped in back to memory, it is removed from "Swap:" and 
added to "Private_Clean:" instead of "Private_Dirty:".

Thanks
Nikanth

>  - kosaki
> 
> > Export this information in smaps by prepending file-names with "[anon]+",
> > when some of the pages in a file backed vma become anonymous.
> >
> > Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> >
> > ---
> >
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 439fc1f..68f9806 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct
> > vm_area_struct *vma) */
> >  	if (file) {
> >  		pad_len_spaces(m, len);
> > +		if (vma->anon_vma)
> > +			seq_puts(m, "[anon]+");
> >  		seq_path(m, &file->f_path, "\n");
> >  	} else {
> >  		const char *name = arch_vma_name(vma);
> >
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org.  For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > become clean.
> >
> > When a page with private file mapping becomes dirty, the vma will be in
> > both i_mmap tree and anon_vma list. The /proc/$pid/smaps will account
> > these pages as dirty and backed by the file.
> >
> > But when those dirty pages gets swapped out, and when they are read back
> > from swap, they would be marked as clean, as it should be, as they are
> > part of swap cache now.
> >
> > But the /proc/$pid/smaps would report the vma as a mapping of a file and
> > it is clean. The pages are actually in same state i.e., dirty with
> > respect to file still, but which was once reported as dirty is now being
> > reported as clean to user-space.
> >
> > This confuses tools like gdb which uses this information. Those tools
> > think that those pages were never modified and it creates problem when
> > they create dumps.
> >
> > The file mapping of the vma also cannot be broken as pages never read
> > earlier, will still have to come from the file. Just that those dirty
> > pages have become clean anonymous pages.
> >
> > During swaping in, restoring the exact state as dirty file-backed pages
> > before swapout would be useless, as there in no real bug. Breaking the
> > vma with only anonymous pages as seperate vmas unnecessary may not be a
> > good thing as well. So let us just export the information that a
> > file-backed vma has anonymous dirty pages.
> 
> Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need dump
>  out.
> 

Yes. When the page is swapped out it is accounted in "Swap:".

> Am I missing anything?
> 

But when it gets swapped in back to memory, it is removed from "Swap:" and 
added to "Private_Clean:" instead of "Private_Dirty:".

Thanks
Nikanth

>  - kosaki
> 
> > Export this information in smaps by prepending file-names with "[anon]+",
> > when some of the pages in a file backed vma become anonymous.
> >
> > Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> >
> > ---
> >
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 439fc1f..68f9806 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -242,6 +242,8 @@ static void show_map_vma(struct seq_file *m, struct
> > vm_area_struct *vma) */
> >  	if (file) {
> >  		pad_len_spaces(m, len);
> > +		if (vma->anon_vma)
> > +			seq_puts(m, "[anon]+");
> >  		seq_path(m, &file->f_path, "\n");
> >  	} else {
> >  		const char *name = arch_vma_name(vma);
> >
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org.  For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > become clean.
> > >
> > > When a page with private file mapping becomes dirty, the vma will be in
> > > both i_mmap tree and anon_vma list. The /proc/$pid/smaps will account
> > > these pages as dirty and backed by the file.
> > >
> > > But when those dirty pages gets swapped out, and when they are read back
> > > from swap, they would be marked as clean, as it should be, as they are
> > > part of swap cache now.
> > >
> > > But the /proc/$pid/smaps would report the vma as a mapping of a file and
> > > it is clean. The pages are actually in same state i.e., dirty with
> > > respect to file still, but which was once reported as dirty is now being
> > > reported as clean to user-space.
> > >
> > > This confuses tools like gdb which uses this information. Those tools
> > > think that those pages were never modified and it creates problem when
> > > they create dumps.
> > >
> > > The file mapping of the vma also cannot be broken as pages never read
> > > earlier, will still have to come from the file. Just that those dirty
> > > pages have become clean anonymous pages.
> > >
> > > During swaping in, restoring the exact state as dirty file-backed pages
> > > before swapout would be useless, as there in no real bug. Breaking the
> > > vma with only anonymous pages as seperate vmas unnecessary may not be a
> > > good thing as well. So let us just export the information that a
> > > file-backed vma has anonymous dirty pages.
> > 
> > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need dump
> >  out.
> > 
> 
> Yes. When the page is swapped out it is accounted in "Swap:".
> 
> > Am I missing anything?
> > 
> 
> But when it gets swapped in back to memory, it is removed from "Swap:" and 
> added to "Private_Clean:" instead of "Private_Dirty:".

Here is the code.
I think the page will become dirty, again.

--------------------------------------------------------------
int try_to_free_swap(struct page *page)
{
        VM_BUG_ON(!PageLocked(page));

        if (!PageSwapCache(page))
                return 0;
        if (PageWriteback(page))
                return 0;
        if (page_swapcount(page))
                return 0;

        delete_from_swap_cache(page);
        SetPageDirty(page);
        return 1;
}

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > become clean.
> > >
> > > When a page with private file mapping becomes dirty, the vma will be in
> > > both i_mmap tree and anon_vma list. The /proc/$pid/smaps will account
> > > these pages as dirty and backed by the file.
> > >
> > > But when those dirty pages gets swapped out, and when they are read back
> > > from swap, they would be marked as clean, as it should be, as they are
> > > part of swap cache now.
> > >
> > > But the /proc/$pid/smaps would report the vma as a mapping of a file and
> > > it is clean. The pages are actually in same state i.e., dirty with
> > > respect to file still, but which was once reported as dirty is now being
> > > reported as clean to user-space.
> > >
> > > This confuses tools like gdb which uses this information. Those tools
> > > think that those pages were never modified and it creates problem when
> > > they create dumps.
> > >
> > > The file mapping of the vma also cannot be broken as pages never read
> > > earlier, will still have to come from the file. Just that those dirty
> > > pages have become clean anonymous pages.
> > >
> > > During swaping in, restoring the exact state as dirty file-backed pages
> > > before swapout would be useless, as there in no real bug. Breaking the
> > > vma with only anonymous pages as seperate vmas unnecessary may not be a
> > > good thing as well. So let us just export the information that a
> > > file-backed vma has anonymous dirty pages.
> > 
> > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need dump
> >  out.
> > 
> 
> Yes. When the page is swapped out it is accounted in "Swap:".
> 
> > Am I missing anything?
> > 
> 
> But when it gets swapped in back to memory, it is removed from "Swap:" and 
> added to "Private_Clean:" instead of "Private_Dirty:".

Here is the code.
I think the page will become dirty, again.

--------------------------------------------------------------
int try_to_free_swap(struct page *page)
{
        VM_BUG_ON(!PageLocked(page));

        if (!PageSwapCache(page))
                return 0;
        if (PageWriteback(page))
                return 0;
        if (page_swapcount(page))
                return 0;

        delete_from_swap_cache(page);
        SetPageDirty(page);
        return 1;
}



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > become clean.
> > > >
> > > > When a page with private file mapping becomes dirty, the vma will be
> > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > account these pages as dirty and backed by the file.
> > > >
> > > > But when those dirty pages gets swapped out, and when they are read
> > > > back from swap, they would be marked as clean, as it should be, as
> > > > they are part of swap cache now.
> > > >
> > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > with respect to file still, but which was once reported as dirty is
> > > > now being reported as clean to user-space.
> > > >
> > > > This confuses tools like gdb which uses this information. Those tools
> > > > think that those pages were never modified and it creates problem
> > > > when they create dumps.
> > > >
> > > > The file mapping of the vma also cannot be broken as pages never read
> > > > earlier, will still have to come from the file. Just that those dirty
> > > > pages have become clean anonymous pages.
> > > >
> > > > During swaping in, restoring the exact state as dirty file-backed
> > > > pages before swapout would be useless, as there in no real bug.
> > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > unnecessary may not be a good thing as well. So let us just export
> > > > the information that a file-backed vma has anonymous dirty pages.
> > >
> > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > dump out.
> >
> > Yes. When the page is swapped out it is accounted in "Swap:".
> >
> > > Am I missing anything?
> >
> > But when it gets swapped in back to memory, it is removed from "Swap:"
> > and added to "Private_Clean:" instead of "Private_Dirty:".
> 
> Here is the code.
> I think the page will become dirty, again.
> 
> --------------------------------------------------------------
> int try_to_free_swap(struct page *page)
> {
>         VM_BUG_ON(!PageLocked(page));
> 
>         if (!PageSwapCache(page))
>                 return 0;
>         if (PageWriteback(page))
>                 return 0;
>         if (page_swapcount(page))
>                 return 0;
> 
>         delete_from_swap_cache(page);
>         SetPageDirty(page);
>         return 1;
> }
> 

I think this gets called only when the swap space gets freed. But when the 
page is just swapped out and swapped in, and the page is still part of 
SwapCache, it will be marked as clean, when the I/O read from swap completes.

Thanks
Nikanth

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > become clean.
> > > >
> > > > When a page with private file mapping becomes dirty, the vma will be
> > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > account these pages as dirty and backed by the file.
> > > >
> > > > But when those dirty pages gets swapped out, and when they are read
> > > > back from swap, they would be marked as clean, as it should be, as
> > > > they are part of swap cache now.
> > > >
> > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > with respect to file still, but which was once reported as dirty is
> > > > now being reported as clean to user-space.
> > > >
> > > > This confuses tools like gdb which uses this information. Those tools
> > > > think that those pages were never modified and it creates problem
> > > > when they create dumps.
> > > >
> > > > The file mapping of the vma also cannot be broken as pages never read
> > > > earlier, will still have to come from the file. Just that those dirty
> > > > pages have become clean anonymous pages.
> > > >
> > > > During swaping in, restoring the exact state as dirty file-backed
> > > > pages before swapout would be useless, as there in no real bug.
> > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > unnecessary may not be a good thing as well. So let us just export
> > > > the information that a file-backed vma has anonymous dirty pages.
> > >
> > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > dump out.
> >
> > Yes. When the page is swapped out it is accounted in "Swap:".
> >
> > > Am I missing anything?
> >
> > But when it gets swapped in back to memory, it is removed from "Swap:"
> > and added to "Private_Clean:" instead of "Private_Dirty:".
> 
> Here is the code.
> I think the page will become dirty, again.
> 
> --------------------------------------------------------------
> int try_to_free_swap(struct page *page)
> {
>         VM_BUG_ON(!PageLocked(page));
> 
>         if (!PageSwapCache(page))
>                 return 0;
>         if (PageWriteback(page))
>                 return 0;
>         if (page_swapcount(page))
>                 return 0;
> 
>         delete_from_swap_cache(page);
>         SetPageDirty(page);
>         return 1;
> }
> 

I think this gets called only when the swap space gets freed. But when the 
page is just swapped out and swapped in, and the page is still part of 
SwapCache, it will be marked as clean, when the I/O read from swap completes.

Thanks
Nikanth

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > > become clean.
> > > > >
> > > > > When a page with private file mapping becomes dirty, the vma will be
> > > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > > account these pages as dirty and backed by the file.
> > > > >
> > > > > But when those dirty pages gets swapped out, and when they are read
> > > > > back from swap, they would be marked as clean, as it should be, as
> > > > > they are part of swap cache now.
> > > > >
> > > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > > with respect to file still, but which was once reported as dirty is
> > > > > now being reported as clean to user-space.
> > > > >
> > > > > This confuses tools like gdb which uses this information. Those tools
> > > > > think that those pages were never modified and it creates problem
> > > > > when they create dumps.
> > > > >
> > > > > The file mapping of the vma also cannot be broken as pages never read
> > > > > earlier, will still have to come from the file. Just that those dirty
> > > > > pages have become clean anonymous pages.
> > > > >
> > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > pages before swapout would be useless, as there in no real bug.
> > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > unnecessary may not be a good thing as well. So let us just export
> > > > > the information that a file-backed vma has anonymous dirty pages.
> > > >
> > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > > dump out.
> > >
> > > Yes. When the page is swapped out it is accounted in "Swap:".
> > >
> > > > Am I missing anything?
> > >
> > > But when it gets swapped in back to memory, it is removed from "Swap:"
> > > and added to "Private_Clean:" instead of "Private_Dirty:".
> > 
> > Here is the code.
> > I think the page will become dirty, again.
> > 
> > --------------------------------------------------------------
> > int try_to_free_swap(struct page *page)
> > {
> >         VM_BUG_ON(!PageLocked(page));
> > 
> >         if (!PageSwapCache(page))
> >                 return 0;
> >         if (PageWriteback(page))
> >                 return 0;
> >         if (page_swapcount(page))
> >                 return 0;
> > 
> >         delete_from_swap_cache(page);
> >         SetPageDirty(page);
> >         return 1;
> > }
> > 
> 
> I think this gets called only when the swap space gets freed. 

this is try-to-free-swap-space.
delete_from_swap_cache() does actual free.


> But when the 
> page is just swapped out and swapped in, and the page is still part of 
> SwapCache, it will be marked as clean, when the I/O read from swap completes.

Because in this case, the swap entry is not freed yet. Then the page is still clean
and swap field is still !0.

PageSwapCache == the page has backend swap entry == the page may be clean.
But, When the swap entry is removed, page will become dirty again.

As I said, following is incorrect.  In almost case, swap entry is not removed at
swap-in. Please grep try_to_free_swap() callers and 

> > > But when it gets swapped in back to memory, it is removed from "Swap:"

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > > become clean.
> > > > >
> > > > > When a page with private file mapping becomes dirty, the vma will be
> > > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > > account these pages as dirty and backed by the file.
> > > > >
> > > > > But when those dirty pages gets swapped out, and when they are read
> > > > > back from swap, they would be marked as clean, as it should be, as
> > > > > they are part of swap cache now.
> > > > >
> > > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > > with respect to file still, but which was once reported as dirty is
> > > > > now being reported as clean to user-space.
> > > > >
> > > > > This confuses tools like gdb which uses this information. Those tools
> > > > > think that those pages were never modified and it creates problem
> > > > > when they create dumps.
> > > > >
> > > > > The file mapping of the vma also cannot be broken as pages never read
> > > > > earlier, will still have to come from the file. Just that those dirty
> > > > > pages have become clean anonymous pages.
> > > > >
> > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > pages before swapout would be useless, as there in no real bug.
> > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > unnecessary may not be a good thing as well. So let us just export
> > > > > the information that a file-backed vma has anonymous dirty pages.
> > > >
> > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > > dump out.
> > >
> > > Yes. When the page is swapped out it is accounted in "Swap:".
> > >
> > > > Am I missing anything?
> > >
> > > But when it gets swapped in back to memory, it is removed from "Swap:"
> > > and added to "Private_Clean:" instead of "Private_Dirty:".
> > 
> > Here is the code.
> > I think the page will become dirty, again.
> > 
> > --------------------------------------------------------------
> > int try_to_free_swap(struct page *page)
> > {
> >         VM_BUG_ON(!PageLocked(page));
> > 
> >         if (!PageSwapCache(page))
> >                 return 0;
> >         if (PageWriteback(page))
> >                 return 0;
> >         if (page_swapcount(page))
> >                 return 0;
> > 
> >         delete_from_swap_cache(page);
> >         SetPageDirty(page);
> >         return 1;
> > }
> > 
> 
> I think this gets called only when the swap space gets freed. 

this is try-to-free-swap-space.
delete_from_swap_cache() does actual free.


> But when the 
> page is just swapped out and swapped in, and the page is still part of 
> SwapCache, it will be marked as clean, when the I/O read from swap completes.

Because in this case, the swap entry is not freed yet. Then the page is still clean
and swap field is still !0.

PageSwapCache == the page has backend swap entry == the page may be clean.
But, When the swap entry is removed, page will become dirty again.

As I said, following is incorrect.  In almost case, swap entry is not removed at
swap-in. Please grep try_to_free_swap() callers and 

> > > But when it gets swapped in back to memory, it is removed from "Swap:"



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:45:28 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty
> > > > > > mappings become clean.
> > > > > >
> > > > > > When a page with private file mapping becomes dirty, the vma will
> > > > > > be in both i_mmap tree and anon_vma list. The /proc/$pid/smaps
> > > > > > will account these pages as dirty and backed by the file.
> > > > > >
> > > > > > But when those dirty pages gets swapped out, and when they are
> > > > > > read back from swap, they would be marked as clean, as it should
> > > > > > be, as they are part of swap cache now.
> > > > > >
> > > > > > But the /proc/$pid/smaps would report the vma as a mapping of a
> > > > > > file and it is clean. The pages are actually in same state i.e.,
> > > > > > dirty with respect to file still, but which was once reported as
> > > > > > dirty is now being reported as clean to user-space.
> > > > > >
> > > > > > This confuses tools like gdb which uses this information. Those
> > > > > > tools think that those pages were never modified and it creates
> > > > > > problem when they create dumps.
> > > > > >
> > > > > > The file mapping of the vma also cannot be broken as pages never
> > > > > > read earlier, will still have to come from the file. Just that
> > > > > > those dirty pages have become clean anonymous pages.
> > > > > >
> > > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > > pages before swapout would be useless, as there in no real bug.
> > > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > > unnecessary may not be a good thing as well. So let us just
> > > > > > export the information that a file-backed vma has anonymous dirty
> > > > > > pages.
> > > > >
> > > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we
> > > > > need dump out.
> > > >
> > > > Yes. When the page is swapped out it is accounted in "Swap:".
> > > >
> > > > > Am I missing anything?
> > > >
> > > > But when it gets swapped in back to memory, it is removed from
> > > > "Swap:" and added to "Private_Clean:" instead of "Private_Dirty:".
> > >
> > > Here is the code.
> > > I think the page will become dirty, again.
> > >
> > > --------------------------------------------------------------
> > > int try_to_free_swap(struct page *page)
> > > {
> > >         VM_BUG_ON(!PageLocked(page));
> > >
> > >         if (!PageSwapCache(page))
> > >                 return 0;
> > >         if (PageWriteback(page))
> > >                 return 0;
> > >         if (page_swapcount(page))
> > >                 return 0;
> > >
> > >         delete_from_swap_cache(page);
> > >         SetPageDirty(page);
> > >         return 1;
> > > }
> >
> > I think this gets called only when the swap space gets freed.
> 
> this is try-to-free-swap-space.
> delete_from_swap_cache() does actual free.
> 
> > But when the
> > page is just swapped out and swapped in, and the page is still part of
> > SwapCache, it will be marked as clean, when the I/O read from swap
> > completes.
> 
> Because in this case, the swap entry is not freed yet. Then the page is
>  still clean and swap field is still !0.
> 
> PageSwapCache == the page has backend swap entry == the page may be clean.
> But, When the swap entry is removed, page will become dirty again.
> 

Correct.

> As I said, following is incorrect.

No.

> In almost case, swap entry is not
>  removed at swap-in. Please grep try_to_free_swap() callers and
> 

Correct

> > > > But when it gets swapped in back to memory, it is removed from
> > > > "Swap:"
> 

I mean the "Swap:" field in smaps file here, not the swapcache.

Thanks
Nikanth

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Nikanth Karthikesan]

On Wednesday 15 September 2010 10:45:28 KOSAKI Motohiro wrote:
> > On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty
> > > > > > mappings become clean.
> > > > > >
> > > > > > When a page with private file mapping becomes dirty, the vma will
> > > > > > be in both i_mmap tree and anon_vma list. The /proc/$pid/smaps
> > > > > > will account these pages as dirty and backed by the file.
> > > > > >
> > > > > > But when those dirty pages gets swapped out, and when they are
> > > > > > read back from swap, they would be marked as clean, as it should
> > > > > > be, as they are part of swap cache now.
> > > > > >
> > > > > > But the /proc/$pid/smaps would report the vma as a mapping of a
> > > > > > file and it is clean. The pages are actually in same state i.e.,
> > > > > > dirty with respect to file still, but which was once reported as
> > > > > > dirty is now being reported as clean to user-space.
> > > > > >
> > > > > > This confuses tools like gdb which uses this information. Those
> > > > > > tools think that those pages were never modified and it creates
> > > > > > problem when they create dumps.
> > > > > >
> > > > > > The file mapping of the vma also cannot be broken as pages never
> > > > > > read earlier, will still have to come from the file. Just that
> > > > > > those dirty pages have become clean anonymous pages.
> > > > > >
> > > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > > pages before swapout would be useless, as there in no real bug.
> > > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > > unnecessary may not be a good thing as well. So let us just
> > > > > > export the information that a file-backed vma has anonymous dirty
> > > > > > pages.
> > > > >
> > > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we
> > > > > need dump out.
> > > >
> > > > Yes. When the page is swapped out it is accounted in "Swap:".
> > > >
> > > > > Am I missing anything?
> > > >
> > > > But when it gets swapped in back to memory, it is removed from
> > > > "Swap:" and added to "Private_Clean:" instead of "Private_Dirty:".
> > >
> > > Here is the code.
> > > I think the page will become dirty, again.
> > >
> > > --------------------------------------------------------------
> > > int try_to_free_swap(struct page *page)
> > > {
> > >         VM_BUG_ON(!PageLocked(page));
> > >
> > >         if (!PageSwapCache(page))
> > >                 return 0;
> > >         if (PageWriteback(page))
> > >                 return 0;
> > >         if (page_swapcount(page))
> > >                 return 0;
> > >
> > >         delete_from_swap_cache(page);
> > >         SetPageDirty(page);
> > >         return 1;
> > > }
> >
> > I think this gets called only when the swap space gets freed.
> 
> this is try-to-free-swap-space.
> delete_from_swap_cache() does actual free.
> 
> > But when the
> > page is just swapped out and swapped in, and the page is still part of
> > SwapCache, it will be marked as clean, when the I/O read from swap
> > completes.
> 
> Because in this case, the swap entry is not freed yet. Then the page is
>  still clean and swap field is still !0.
> 
> PageSwapCache == the page has backend swap entry == the page may be clean.
> But, When the swap entry is removed, page will become dirty again.
> 

Correct.

> As I said, following is incorrect.

No.

> In almost case, swap entry is not
>  removed at swap-in. Please grep try_to_free_swap() callers and
> 

Correct

> > > > But when it gets swapped in back to memory, it is removed from
> > > > "Swap:"
> 

I mean the "Swap:" field in smaps file here, not the swapcache.

Thanks
Nikanth

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Richard Guenther]

On Wed, 15 Sep 2010, Nikanth Karthikesan wrote:

> On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > > become clean.
> > > > >
> > > > > When a page with private file mapping becomes dirty, the vma will be
> > > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > > account these pages as dirty and backed by the file.
> > > > >
> > > > > But when those dirty pages gets swapped out, and when they are read
> > > > > back from swap, they would be marked as clean, as it should be, as
> > > > > they are part of swap cache now.
> > > > >
> > > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > > with respect to file still, but which was once reported as dirty is
> > > > > now being reported as clean to user-space.
> > > > >
> > > > > This confuses tools like gdb which uses this information. Those tools
> > > > > think that those pages were never modified and it creates problem
> > > > > when they create dumps.
> > > > >
> > > > > The file mapping of the vma also cannot be broken as pages never read
> > > > > earlier, will still have to come from the file. Just that those dirty
> > > > > pages have become clean anonymous pages.
> > > > >
> > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > pages before swapout would be useless, as there in no real bug.
> > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > unnecessary may not be a good thing as well. So let us just export
> > > > > the information that a file-backed vma has anonymous dirty pages.
> > > >
> > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > > dump out.
> > >
> > > Yes. When the page is swapped out it is accounted in "Swap:".
> > >
> > > > Am I missing anything?
> > >
> > > But when it gets swapped in back to memory, it is removed from "Swap:"
> > > and added to "Private_Clean:" instead of "Private_Dirty:".
> > 
> > Here is the code.
> > I think the page will become dirty, again.
> > 
> > --------------------------------------------------------------
> > int try_to_free_swap(struct page *page)
> > {
> >         VM_BUG_ON(!PageLocked(page));
> > 
> >         if (!PageSwapCache(page))
> >                 return 0;
> >         if (PageWriteback(page))
> >                 return 0;
> >         if (page_swapcount(page))
> >                 return 0;
> > 
> >         delete_from_swap_cache(page);
> >         SetPageDirty(page);
> >         return 1;
> > }
> > 
> 
> I think this gets called only when the swap space gets freed. But when the 
> page is just swapped out and swapped in, and the page is still part of 
> SwapCache, it will be marked as clean, when the I/O read from swap completes.

And it will be still marked clean if I do a swapoff -a after it has
been swapped in again.  Thus /proc/smaps shows it as file-backed,
private, clean and not swapped.  Which is wrong.

Richard.

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[Richard Guenther]

On Wed, 15 Sep 2010, Nikanth Karthikesan wrote:

> On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > > become clean.
> > > > >
> > > > > When a page with private file mapping becomes dirty, the vma will be
> > > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > > account these pages as dirty and backed by the file.
> > > > >
> > > > > But when those dirty pages gets swapped out, and when they are read
> > > > > back from swap, they would be marked as clean, as it should be, as
> > > > > they are part of swap cache now.
> > > > >
> > > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > > with respect to file still, but which was once reported as dirty is
> > > > > now being reported as clean to user-space.
> > > > >
> > > > > This confuses tools like gdb which uses this information. Those tools
> > > > > think that those pages were never modified and it creates problem
> > > > > when they create dumps.
> > > > >
> > > > > The file mapping of the vma also cannot be broken as pages never read
> > > > > earlier, will still have to come from the file. Just that those dirty
> > > > > pages have become clean anonymous pages.
> > > > >
> > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > pages before swapout would be useless, as there in no real bug.
> > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > unnecessary may not be a good thing as well. So let us just export
> > > > > the information that a file-backed vma has anonymous dirty pages.
> > > >
> > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > > dump out.
> > >
> > > Yes. When the page is swapped out it is accounted in "Swap:".
> > >
> > > > Am I missing anything?
> > >
> > > But when it gets swapped in back to memory, it is removed from "Swap:"
> > > and added to "Private_Clean:" instead of "Private_Dirty:".
> > 
> > Here is the code.
> > I think the page will become dirty, again.
> > 
> > --------------------------------------------------------------
> > int try_to_free_swap(struct page *page)
> > {
> >         VM_BUG_ON(!PageLocked(page));
> > 
> >         if (!PageSwapCache(page))
> >                 return 0;
> >         if (PageWriteback(page))
> >                 return 0;
> >         if (page_swapcount(page))
> >                 return 0;
> > 
> >         delete_from_swap_cache(page);
> >         SetPageDirty(page);
> >         return 1;
> > }
> > 
> 
> I think this gets called only when the swap space gets freed. But when the 
> page is just swapped out and swapped in, and the page is still part of 
> SwapCache, it will be marked as clean, when the I/O read from swap completes.

And it will be still marked clean if I do a swapoff -a after it has
been swapped in again.  Thus /proc/smaps shows it as file-backed,
private, clean and not swapped.  Which is wrong.

Richard.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> On Wed, 15 Sep 2010, Nikanth Karthikesan wrote:
> 
> > On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > > > become clean.
> > > > > >
> > > > > > When a page with private file mapping becomes dirty, the vma will be
> > > > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > > > account these pages as dirty and backed by the file.
> > > > > >
> > > > > > But when those dirty pages gets swapped out, and when they are read
> > > > > > back from swap, they would be marked as clean, as it should be, as
> > > > > > they are part of swap cache now.
> > > > > >
> > > > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > > > with respect to file still, but which was once reported as dirty is
> > > > > > now being reported as clean to user-space.
> > > > > >
> > > > > > This confuses tools like gdb which uses this information. Those tools
> > > > > > think that those pages were never modified and it creates problem
> > > > > > when they create dumps.
> > > > > >
> > > > > > The file mapping of the vma also cannot be broken as pages never read
> > > > > > earlier, will still have to come from the file. Just that those dirty
> > > > > > pages have become clean anonymous pages.
> > > > > >
> > > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > > pages before swapout would be useless, as there in no real bug.
> > > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > > unnecessary may not be a good thing as well. So let us just export
> > > > > > the information that a file-backed vma has anonymous dirty pages.
> > > > >
> > > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > > > dump out.
> > > >
> > > > Yes. When the page is swapped out it is accounted in "Swap:".
> > > >
> > > > > Am I missing anything?
> > > >
> > > > But when it gets swapped in back to memory, it is removed from "Swap:"
> > > > and added to "Private_Clean:" instead of "Private_Dirty:".
> > > 
> > > Here is the code.
> > > I think the page will become dirty, again.
> > > 
> > > --------------------------------------------------------------
> > > int try_to_free_swap(struct page *page)
> > > {
> > >         VM_BUG_ON(!PageLocked(page));
> > > 
> > >         if (!PageSwapCache(page))
> > >                 return 0;
> > >         if (PageWriteback(page))
> > >                 return 0;
> > >         if (page_swapcount(page))
> > >                 return 0;
> > > 
> > >         delete_from_swap_cache(page);
> > >         SetPageDirty(page);
> > >         return 1;
> > > }
> > > 
> > 
> > I think this gets called only when the swap space gets freed. But when the 
> > page is just swapped out and swapped in, and the page is still part of 
> > SwapCache, it will be marked as clean, when the I/O read from swap completes.
> 
> And it will be still marked clean if I do a swapoff -a after it has
> been swapped in again.  Thus /proc/smaps shows it as file-backed,
> private, clean and not swapped.  Which is wrong.

Ahh, my fault.

As Hugh explained, current smaps's Private_Dirty is buggy. If it's not bug,
at minimum it's not straight implement. the VM has two dirty flags 
1) pte dirty 2) page dirty, but smaps only show (1). then, gdb was confused.

Side note: SetPageDirty() turn on (2).

As I said, the VM doesn't have for file and for anon dirty. but it has
another different dirty flags.


And, Dirty is not suitable gdb at all (this also was pointed out by hugh).
Because Dirty mean "The page need writeback", but swapcache doesn't need.
actual data is in swap already.

So, I vote Hugh's Anon field idea.


Thanks.

Re: [PATCH] After swapout/swapin private dirty mappings become clean

[KOSAKI Motohiro]

> On Wed, 15 Sep 2010, Nikanth Karthikesan wrote:
> 
> > On Wednesday 15 September 2010 10:16:36 KOSAKI Motohiro wrote:
> > > > On Wednesday 15 September 2010 05:54:31 KOSAKI Motohiro wrote:
> > > > > > /proc/$pid/smaps broken: After swapout/swapin private dirty mappings
> > > > > > become clean.
> > > > > >
> > > > > > When a page with private file mapping becomes dirty, the vma will be
> > > > > > in both i_mmap tree and anon_vma list. The /proc/$pid/smaps will
> > > > > > account these pages as dirty and backed by the file.
> > > > > >
> > > > > > But when those dirty pages gets swapped out, and when they are read
> > > > > > back from swap, they would be marked as clean, as it should be, as
> > > > > > they are part of swap cache now.
> > > > > >
> > > > > > But the /proc/$pid/smaps would report the vma as a mapping of a file
> > > > > > and it is clean. The pages are actually in same state i.e., dirty
> > > > > > with respect to file still, but which was once reported as dirty is
> > > > > > now being reported as clean to user-space.
> > > > > >
> > > > > > This confuses tools like gdb which uses this information. Those tools
> > > > > > think that those pages were never modified and it creates problem
> > > > > > when they create dumps.
> > > > > >
> > > > > > The file mapping of the vma also cannot be broken as pages never read
> > > > > > earlier, will still have to come from the file. Just that those dirty
> > > > > > pages have become clean anonymous pages.
> > > > > >
> > > > > > During swaping in, restoring the exact state as dirty file-backed
> > > > > > pages before swapout would be useless, as there in no real bug.
> > > > > > Breaking the vma with only anonymous pages as seperate vmas
> > > > > > unnecessary may not be a good thing as well. So let us just export
> > > > > > the information that a file-backed vma has anonymous dirty pages.
> > > > >
> > > > > Why can't gdb check Swap: field in smaps? I think Swap!=0 mean we need
> > > > > dump out.
> > > >
> > > > Yes. When the page is swapped out it is accounted in "Swap:".
> > > >
> > > > > Am I missing anything?
> > > >
> > > > But when it gets swapped in back to memory, it is removed from "Swap:"
> > > > and added to "Private_Clean:" instead of "Private_Dirty:".
> > > 
> > > Here is the code.
> > > I think the page will become dirty, again.
> > > 
> > > --------------------------------------------------------------
> > > int try_to_free_swap(struct page *page)
> > > {
> > >         VM_BUG_ON(!PageLocked(page));
> > > 
> > >         if (!PageSwapCache(page))
> > >                 return 0;
> > >         if (PageWriteback(page))
> > >                 return 0;
> > >         if (page_swapcount(page))
> > >                 return 0;
> > > 
> > >         delete_from_swap_cache(page);
> > >         SetPageDirty(page);
> > >         return 1;
> > > }
> > > 
> > 
> > I think this gets called only when the swap space gets freed. But when the 
> > page is just swapped out and swapped in, and the page is still part of 
> > SwapCache, it will be marked as clean, when the I/O read from swap completes.
> 
> And it will be still marked clean if I do a swapoff -a after it has
> been swapped in again.  Thus /proc/smaps shows it as file-backed,
> private, clean and not swapped.  Which is wrong.

Ahh, my fault.

As Hugh explained, current smaps's Private_Dirty is buggy. If it's not bug,
at minimum it's not straight implement. the VM has two dirty flags 
1) pte dirty 2) page dirty, but smaps only show (1). then, gdb was confused.

Side note: SetPageDirty() turn on (2).

As I said, the VM doesn't have for file and for anon dirty. but it has
another different dirty flags.


And, Dirty is not suitable gdb at all (this also was pointed out by hugh).
Because Dirty mean "The page need writeback", but swapcache doesn't need.
actual data is in swap already.

So, I vote Hugh's Anon field idea.


Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>