Mlocked pages statistics shows bogus value.

Mlocked pages statistics shows bogus value.

[Tetsuo Handa]

While reading OOM report from Jan Stancek, I noticed that
NR_MLOCK statistics shows bogus values.


Steps to reproduce:

(1) Check Mlocked: field of /proc/meminfo or mlocked: field of SysRq-m.

(2) Compile and run below program with appropriate size as argument.
    There is no need to invoke the OOM killer.

----------
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>

int main(int argc, char *argv[])
{
	unsigned long length = atoi(argv[1]);
	void *addr = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
	if (addr == MAP_FAILED)
		printf("mmap() failed\n");
	else if (mlock(addr, length) == -1)
		printf("mlock() failed\n");
	else
		printf("MLocked %lu bytes\n", length);
	return 0;
}
----------

(3) Check Mlocked: field or mlocked: field again.
    You can see the value became very large due to
    NR_MLOCK counter going negative.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Mlocked pages statistics shows bogus value.

[Kirill A. Shutemov]

On Tue, Jan 19, 2016 at 07:36:37PM +0900, Tetsuo Handa wrote:
> While reading OOM report from Jan Stancek, I noticed that
> NR_MLOCK statistics shows bogus values.
> 
> 
> Steps to reproduce:
> 
> (1) Check Mlocked: field of /proc/meminfo or mlocked: field of SysRq-m.
> 
> (2) Compile and run below program with appropriate size as argument.
>     There is no need to invoke the OOM killer.
> 
> ----------
> #include <stdio.h>
> #include <stdlib.h>
> #include <sys/mman.h>
> 
> int main(int argc, char *argv[])
> {
> 	unsigned long length = atoi(argv[1]);
> 	void *addr = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
> 	if (addr == MAP_FAILED)
> 		printf("mmap() failed\n");
> 	else if (mlock(addr, length) == -1)
> 		printf("mlock() failed\n");
> 	else
> 		printf("MLocked %lu bytes\n", length);
> 	return 0;
> }
> ----------
> 
> (3) Check Mlocked: field or mlocked: field again.
>     You can see the value became very large due to
>     NR_MLOCK counter going negative.

Oh. Looks like a bug from 2013...

Thanks for report.

Re: Mlocked pages statistics shows bogus value.

[Tetsuo Handa]

Kirill A. Shutemov wrote:
> Oh. Looks like a bug from 2013...
> 
> Thanks for report.
> 
> From 6f80a79dc5f65f29899e396942d40f727cd36480 Mon Sep 17 00:00:00 2001
> From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
> Date: Tue, 19 Jan 2016 14:59:19 +0300
> Subject: [PATCH] mm: fix mlock accouting
> 
> Tetsuo Handa reported underflow of NR_MLOCK on munlock.
> 
> Testcase:
> 	#include <stdio.h>
> 	#include <stdlib.h>
> 	#include <sys/mman.h>
> 
> 	#define BASE ((void *)0x400000000000)
> 	#define SIZE (1UL << 21)
> 
> 	int main(int argc, char *argv[])
> 	{
> 		void *addr;
> 
> 		system("grep Mlocked /proc/meminfo");
> 		addr = mmap(BASE, SIZE, PROT_READ | PROT_WRITE,
> 				MAP_ANONYMOUS | MAP_PRIVATE | MAP_LOCKED | MAP_FIXED,
> 				-1, 0);
> 		if (addr == MAP_FAILED)
> 			printf("mmap() failed\n"), exit(1);
> 		munmap(addr, SIZE);
> 		system("grep Mlocked /proc/meminfo");
> 		return 0;
> 	}
> 
> It happens on munlock_vma_page() due to unfortunate choice of nr_pages
> data type:
> 
> 	__mod_zone_page_state(zone, NR_MLOCK, -nr_pages);
> 
> For unsigned int nr_pages, implicitly casted to long in
> __mod_zone_page_state(), it becomes something around UINT_MAX.
> 
> munlock_vma_page() usually called for THP as small pages go though
> pagevec.
> 
> Let's make nr_pages singed int.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> Fixes: ff6a6da60b89 ("mm: accelerate munlock() treatment of THP pages")
> Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Cc: Michel Lespinasse <walken@google.com>
> ---
>  mm/mlock.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/mlock.c b/mm/mlock.c
> index e1e2b1207bf2..96f001041928 100644
> --- a/mm/mlock.c
> +++ b/mm/mlock.c
> @@ -175,7 +175,7 @@ static void __munlock_isolation_failed(struct page *page)
>   */
>  unsigned int munlock_vma_page(struct page *page)
>  {
> -	unsigned int nr_pages;
> +	int nr_pages;
>  	struct zone *zone = page_zone(page);
>  
>  	/* For try_to_munlock() and to serialize with page migration */
> -- 
>  Kirill A. Shutemov
> 

Don't we want to use "long" than "int" for all variables that count number
of pages, for recently commit 6cdb18ad98a49f7e9b95d538a0614cde827404b8
"mm/vmstat: fix overflow in mod_zone_page_state()" changed to use "long" ?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Mlocked pages statistics shows bogus value.

[Kirill A. Shutemov]

On Tue, Jan 19, 2016 at 09:46:21PM +0900, Tetsuo Handa wrote:
> Kirill A. Shutemov wrote:
> > Oh. Looks like a bug from 2013...
> > 
> > Thanks for report.
> > 
> > From 6f80a79dc5f65f29899e396942d40f727cd36480 Mon Sep 17 00:00:00 2001
> > From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
> > Date: Tue, 19 Jan 2016 14:59:19 +0300
> > Subject: [PATCH] mm: fix mlock accouting
> > 
> > Tetsuo Handa reported underflow of NR_MLOCK on munlock.
> > 
> > Testcase:
> > 	#include <stdio.h>
> > 	#include <stdlib.h>
> > 	#include <sys/mman.h>
> > 
> > 	#define BASE ((void *)0x400000000000)
> > 	#define SIZE (1UL << 21)
> > 
> > 	int main(int argc, char *argv[])
> > 	{
> > 		void *addr;
> > 
> > 		system("grep Mlocked /proc/meminfo");
> > 		addr = mmap(BASE, SIZE, PROT_READ | PROT_WRITE,
> > 				MAP_ANONYMOUS | MAP_PRIVATE | MAP_LOCKED | MAP_FIXED,
> > 				-1, 0);
> > 		if (addr == MAP_FAILED)
> > 			printf("mmap() failed\n"), exit(1);
> > 		munmap(addr, SIZE);
> > 		system("grep Mlocked /proc/meminfo");
> > 		return 0;
> > 	}
> > 
> > It happens on munlock_vma_page() due to unfortunate choice of nr_pages
> > data type:
> > 
> > 	__mod_zone_page_state(zone, NR_MLOCK, -nr_pages);
> > 
> > For unsigned int nr_pages, implicitly casted to long in
> > __mod_zone_page_state(), it becomes something around UINT_MAX.
> > 
> > munlock_vma_page() usually called for THP as small pages go though
> > pagevec.
> > 
> > Let's make nr_pages singed int.
> > 
> > Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> > Fixes: ff6a6da60b89 ("mm: accelerate munlock() treatment of THP pages")
> > Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> > Cc: Michel Lespinasse <walken@google.com>
> > ---
> >  mm/mlock.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/mm/mlock.c b/mm/mlock.c
> > index e1e2b1207bf2..96f001041928 100644
> > --- a/mm/mlock.c
> > +++ b/mm/mlock.c
> > @@ -175,7 +175,7 @@ static void __munlock_isolation_failed(struct page *page)
> >   */
> >  unsigned int munlock_vma_page(struct page *page)
> >  {
> > -	unsigned int nr_pages;
> > +	int nr_pages;
> >  	struct zone *zone = page_zone(page);
> >  
> >  	/* For try_to_munlock() and to serialize with page migration */
> > -- 
> >  Kirill A. Shutemov
> > 
> 
> Don't we want to use "long" than "int" for all variables that count number
> of pages, for recently commit 6cdb18ad98a49f7e9b95d538a0614cde827404b8
> "mm/vmstat: fix overflow in mod_zone_page_state()" changed to use "long" ?

Potentially, yes. But here we count number of small pages in the compound
page. We're far from being able to allocate 8 terabyte pages ;)

Anyway, it's out-of-scope for this bug fix.

My "Fixes:" is probably misleading, since we don't have bug visible until
6cdb18ad98a4.

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Mlocked pages statistics shows bogus value.

[Tetsuo Handa]

Kirill A. Shutemov wrote:
> On Tue, Jan 19, 2016 at 09:46:21PM +0900, Tetsuo Handa wrote:
> > Kirill A. Shutemov wrote:
> > > Oh. Looks like a bug from 2013...
> > > 
> > > Thanks for report.
> > > 
> > > From 6f80a79dc5f65f29899e396942d40f727cd36480 Mon Sep 17 00:00:00 2001
> > > From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
> > > Date: Tue, 19 Jan 2016 14:59:19 +0300
> > > Subject: [PATCH] mm: fix mlock accouting
> > > 
> > > Tetsuo Handa reported underflow of NR_MLOCK on munlock.
> > > 
> > > Testcase:
> > > 	#include <stdio.h>
> > > 	#include <stdlib.h>
> > > 	#include <sys/mman.h>
> > > 
> > > 	#define BASE ((void *)0x400000000000)
> > > 	#define SIZE (1UL << 21)
> > > 
> > > 	int main(int argc, char *argv[])
> > > 	{
> > > 		void *addr;
> > > 
> > > 		system("grep Mlocked /proc/meminfo");
> > > 		addr = mmap(BASE, SIZE, PROT_READ | PROT_WRITE,
> > > 				MAP_ANONYMOUS | MAP_PRIVATE | MAP_LOCKED | MAP_FIXED,
> > > 				-1, 0);
> > > 		if (addr == MAP_FAILED)
> > > 			printf("mmap() failed\n"), exit(1);
> > > 		munmap(addr, SIZE);
> > > 		system("grep Mlocked /proc/meminfo");
> > > 		return 0;
> > > 	}
> > > 
> > > It happens on munlock_vma_page() due to unfortunate choice of nr_pages
> > > data type:
> > > 
> > > 	__mod_zone_page_state(zone, NR_MLOCK, -nr_pages);
> > > 
> > > For unsigned int nr_pages, implicitly casted to long in
> > > __mod_zone_page_state(), it becomes something around UINT_MAX.
> > > 
> > > munlock_vma_page() usually called for THP as small pages go though
> > > pagevec.
> > > 
> > > Let's make nr_pages singed int.
> > > 
> > > Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> > > Fixes: ff6a6da60b89 ("mm: accelerate munlock() treatment of THP pages")
> > > Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> > > Cc: Michel Lespinasse <walken@google.com>
> > > ---
> > >  mm/mlock.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/mm/mlock.c b/mm/mlock.c
> > > index e1e2b1207bf2..96f001041928 100644
> > > --- a/mm/mlock.c
> > > +++ b/mm/mlock.c
> > > @@ -175,7 +175,7 @@ static void __munlock_isolation_failed(struct page *page)
> > >   */
> > >  unsigned int munlock_vma_page(struct page *page)
> > >  {
> > > -	unsigned int nr_pages;
> > > +	int nr_pages;
> > >  	struct zone *zone = page_zone(page);
> > >  
> > >  	/* For try_to_munlock() and to serialize with page migration */
> > > -- 
> > >  Kirill A. Shutemov
> > > 

I tested your patch on Linux 4.4 and confirmed that your patch fixed this bug.
Please also send to stable.

Cc: <stable@vger.kernel.org>  [4.4+]

> > 
> > Don't we want to use "long" than "int" for all variables that count number
> > of pages, for recently commit 6cdb18ad98a49f7e9b95d538a0614cde827404b8
> > "mm/vmstat: fix overflow in mod_zone_page_state()" changed to use "long" ?
> 
> Potentially, yes. But here we count number of small pages in the compound
> page. We're far from being able to allocate 8 terabyte pages ;)

That commit says "we have a 9TB system with only one node".
You might encounter such machines in near future. ;-)

> 
> Anyway, it's out-of-scope for this bug fix.
> 
> My "Fixes:" is probably misleading, since we don't have bug visible until
> 6cdb18ad98a4.
> 
> -- 
>  Kirill A. Shutemov
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Mlocked pages statistics shows bogus value.

[Michal Hocko]

On Tue 19-01-16 14:21:01, Kirill A. Shutemov wrote:
[...]
> >From 6f80a79dc5f65f29899e396942d40f727cd36480 Mon Sep 17 00:00:00 2001
> From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
> Date: Tue, 19 Jan 2016 14:59:19 +0300
> Subject: [PATCH] mm: fix mlock accouting
> 
> Tetsuo Handa reported underflow of NR_MLOCK on munlock.
> 
> Testcase:
> 	#include <stdio.h>
> 	#include <stdlib.h>
> 	#include <sys/mman.h>
> 
> 	#define BASE ((void *)0x400000000000)
> 	#define SIZE (1UL << 21)
> 
> 	int main(int argc, char *argv[])
> 	{
> 		void *addr;
> 
> 		system("grep Mlocked /proc/meminfo");
> 		addr = mmap(BASE, SIZE, PROT_READ | PROT_WRITE,
> 				MAP_ANONYMOUS | MAP_PRIVATE | MAP_LOCKED | MAP_FIXED,
> 				-1, 0);
> 		if (addr == MAP_FAILED)
> 			printf("mmap() failed\n"), exit(1);
> 		munmap(addr, SIZE);
> 		system("grep Mlocked /proc/meminfo");
> 		return 0;
> 	}
> 
> It happens on munlock_vma_page() due to unfortunate choice of nr_pages
> data type:
> 
> 	__mod_zone_page_state(zone, NR_MLOCK, -nr_pages);
> 
> For unsigned int nr_pages, implicitly casted to long in
> __mod_zone_page_state(), it becomes something around UINT_MAX.
> 
> munlock_vma_page() usually called for THP as small pages go though
> pagevec.
> 
> Let's make nr_pages singed int.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> Fixes: ff6a6da60b89 ("mm: accelerate munlock() treatment of THP pages")
> Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Cc: Michel Lespinasse <walken@google.com>

$ grep Mlocked /proc/meminfo 
Mlocked:        10840497455108 kB

I've started seeing the same issue recently but was too busy to track it
down. Thanks! Mentioning 6cdb18ad98a4 ("mm/vmstat: fix overflow in
mod_zone_page_state()") would be really helpful here.

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  mm/mlock.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/mlock.c b/mm/mlock.c
> index e1e2b1207bf2..96f001041928 100644
> --- a/mm/mlock.c
> +++ b/mm/mlock.c
> @@ -175,7 +175,7 @@ static void __munlock_isolation_failed(struct page *page)
>   */
>  unsigned int munlock_vma_page(struct page *page)
>  {
> -	unsigned int nr_pages;
> +	int nr_pages;
>  	struct zone *zone = page_zone(page);
>  
>  	/* For try_to_munlock() and to serialize with page migration */
> -- 
>  Kirill A. Shutemov
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Mlocked pages statistics shows bogus value.

[Heiko Carstens]

(added "Huang, Ying" <ying.huang@intel.com> to cc.)

On Tue, Jan 19, 2016 at 10:38:50PM +0900, Tetsuo Handa wrote:
> Kirill A. Shutemov wrote:
> > On Tue, Jan 19, 2016 at 09:46:21PM +0900, Tetsuo Handa wrote:
> > > Kirill A. Shutemov wrote:
> > > > Oh. Looks like a bug from 2013...
> > > > 
> > > > Thanks for report.
> > > > For unsigned int nr_pages, implicitly casted to long in
> > > > __mod_zone_page_state(), it becomes something around UINT_MAX.
> > > > 
> > > > munlock_vma_page() usually called for THP as small pages go though
> > > > pagevec.
> > > > 
> > > > Let's make nr_pages singed int.
> > > > 
> > > > Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> > > > Fixes: ff6a6da60b89 ("mm: accelerate munlock() treatment of THP pages")
> > > > Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> > > > Cc: Michel Lespinasse <walken@google.com>
> > > > ---
> > > >  mm/mlock.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/mm/mlock.c b/mm/mlock.c
> > > > index e1e2b1207bf2..96f001041928 100644
> > > > --- a/mm/mlock.c
> > > > +++ b/mm/mlock.c
> > > > @@ -175,7 +175,7 @@ static void __munlock_isolation_failed(struct page *page)
> > > >   */
> > > >  unsigned int munlock_vma_page(struct page *page)
> > > >  {
> > > > -	unsigned int nr_pages;
> > > > +	int nr_pages;
> > > >  	struct zone *zone = page_zone(page);
> > > >  
> > > >  	/* For try_to_munlock() and to serialize with page migration */
> > > > -- 
> > > >  Kirill A. Shutemov
> > > > 
> 
> I tested your patch on Linux 4.4 and confirmed that your patch fixed this bug.
> Please also send to stable.
> 
> Cc: <stable@vger.kernel.org>  [4.4+]
> 
> > > Don't we want to use "long" than "int" for all variables that count number
> > > of pages, for recently commit 6cdb18ad98a49f7e9b95d538a0614cde827404b8
> > > "mm/vmstat: fix overflow in mod_zone_page_state()" changed to use "long" ?
> > 
> > Potentially, yes. But here we count number of small pages in the compound
> > page. We're far from being able to allocate 8 terabyte pages ;)
> 
> That commit says "we have a 9TB system with only one node".
> You might encounter such machines in near future. ;-)
> 
> > 
> > Anyway, it's out-of-scope for this bug fix.
> > 
> > My "Fixes:" is probably misleading, since we don't have bug visible until
> > 6cdb18ad98a4.

Please also mention 6cdb18ad98a4 in the changelog. I didn't request to add
my "obviously correct" ;) patch to be added to -stable.
But just in case somebody backports it..

There was also a performance regression reported that was introduced with
6cdb18ad98a4. However I couldn't make any sense of it. Maybe this patch
fixes it also?

See https://lkml.org/lkml/2016/1/5/1103

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Mlocked pages statistics shows bogus value.

[Heiko Carstens]

On Tue, Jan 19, 2016 at 10:38:50PM +0900, Tetsuo Handa wrote:
> > > Don't we want to use "long" than "int" for all variables that count number
> > > of pages, for recently commit 6cdb18ad98a49f7e9b95d538a0614cde827404b8
> > > "mm/vmstat: fix overflow in mod_zone_page_state()" changed to use "long" ?
> > 
> > Potentially, yes. But here we count number of small pages in the compound
> > page. We're far from being able to allocate 8 terabyte pages ;)
> 
> That commit says "we have a 9TB system with only one node".
> You might encounter such machines in near future. ;-)

FWIW: in the above mentioned patch I wrote that I was just hoping that the
int -> long conversion could fix the boot issue with the >8TB machine.

It actually did fix it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>