[LTP] numastats updates

[LTP] numastats updates

[Stanislav Kholmanskikh]

Hello!

In LTP we have a test - numa01.sh 
(https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/numa/numa01.sh)

In general, this script does something like:
   * numastat > /tmp/before
   * starts a binary with the specified numa memory policy using numactl 
(or a like):
     numactl --interleave=all get_some_memory_with_malloc_and_write_it
   * `sleep` for few seconds
   * numastat > /tmp/after
   * compares /tmp/before and /tmp/after to check that the numa policy
was applied the right way

But the problem is that on a host with many NUMA nodes (8) the process
of updating that numastats statistics takes some time. Even 10 seconds
may be not enough. Therefore the test fails.

Is there a direct or indirect way to force the kernel to update the NUMA 
statistics?

Thank you.

------------------------------------------------------------------------------
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

numastats updates

[Stanislav Kholmanskikh]

Hello!

In LTP we have a test - numa01.sh 
(https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/numa/numa01.sh)

In general, this script does something like:
   * numastat > /tmp/before
   * starts a binary with the specified numa memory policy using numactl 
(or a like):
     numactl --interleave=all get_some_memory_with_malloc_and_write_it
   * `sleep` for few seconds
   * numastat > /tmp/after
   * compares /tmp/before and /tmp/after to check that the numa policy
was applied the right way

But the problem is that on a host with many NUMA nodes (8) the process
of updating that numastats statistics takes some time. Even 10 seconds
may be not enough. Therefore the test fails.

Is there a direct or indirect way to force the kernel to update the NUMA 
statistics?

Thank you.

Re: [LTP] numastats updates

[Andi Kleen]

On Thu, Apr 03, 2014 at 02:45:27PM +0400, Stanislav Kholmanskikh wrote:
> In LTP we have a test - numa01.sh (https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/numa/numa01.sh)

numactl has similar tests

> 
> In general, this script does something like:
>   * numastat > /tmp/before
>   * starts a binary with the specified numa memory policy using
> numactl (or a like):
>     numactl --interleave=all get_some_memory_with_malloc_and_write_it
>   * `sleep` for few seconds
>   * numastat > /tmp/after
>   * compares /tmp/before and /tmp/after to check that the numa policy
> was applied the right way
> 
> But the problem is that on a host with many NUMA nodes (8) the process
> of updating that numastats statistics takes some time. Even 10 seconds
> may be not enough. Therefore the test fails.
> 
> Is there a direct or indirect way to force the kernel to update the
> NUMA statistics?

Not currently. It depends on how much memory you have and subsequent
operations. I guess would need to add one.


-Andi

------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_APR
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Andi Kleen]

On Thu, Apr 03, 2014 at 02:45:27PM +0400, Stanislav Kholmanskikh wrote:
> In LTP we have a test - numa01.sh (https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/numa/numa01.sh)

numactl has similar tests

> 
> In general, this script does something like:
>   * numastat > /tmp/before
>   * starts a binary with the specified numa memory policy using
> numactl (or a like):
>     numactl --interleave=all get_some_memory_with_malloc_and_write_it
>   * `sleep` for few seconds
>   * numastat > /tmp/after
>   * compares /tmp/before and /tmp/after to check that the numa policy
> was applied the right way
> 
> But the problem is that on a host with many NUMA nodes (8) the process
> of updating that numastats statistics takes some time. Even 10 seconds
> may be not enough. Therefore the test fails.
> 
> Is there a direct or indirect way to force the kernel to update the
> NUMA statistics?

Not currently. It depends on how much memory you have and subsequent
operations. I guess would need to add one.


-Andi

Re: [LTP] numastats updates

[Christoph Lameter]

On Mon, 7 Apr 2014, Andi Kleen wrote:

> >   * starts a binary with the specified numa memory policy using
> > numactl (or a like):
> >     numactl --interleave=all get_some_memory_with_malloc_and_write_it
> >   * `sleep` for few seconds
> >   * numastat > /tmp/after
> >   * compares /tmp/before and /tmp/after to check that the numa policy
> > was applied the right way
> >
> > But the problem is that on a host with many NUMA nodes (8) the process
> > of updating that numastats statistics takes some time. Even 10 seconds
> > may be not enough. Therefore the test fails.
> >
> > Is there a direct or indirect way to force the kernel to update the
> > NUMA statistics?
>
> Not currently. It depends on how much memory you have and subsequent
> operations. I guess would need to add one.

The kernel vm statistics are brought up to date with the default
settings every 2 seconds.

The interval is controlled via /proc/sys/vm/stat_interval

Check the value that you have setup there.


------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_APR
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Christoph Lameter]

On Mon, 7 Apr 2014, Andi Kleen wrote:

> >   * starts a binary with the specified numa memory policy using
> > numactl (or a like):
> >     numactl --interleave=all get_some_memory_with_malloc_and_write_it
> >   * `sleep` for few seconds
> >   * numastat > /tmp/after
> >   * compares /tmp/before and /tmp/after to check that the numa policy
> > was applied the right way
> >
> > But the problem is that on a host with many NUMA nodes (8) the process
> > of updating that numastats statistics takes some time. Even 10 seconds
> > may be not enough. Therefore the test fails.
> >
> > Is there a direct or indirect way to force the kernel to update the
> > NUMA statistics?
>
> Not currently. It depends on how much memory you have and subsequent
> operations. I guess would need to add one.

The kernel vm statistics are brought up to date with the default
settings every 2 seconds.

The interval is controlled via /proc/sys/vm/stat_interval

Check the value that you have setup there.

Re: [LTP] numastats updates

[Stanislav Kholmanskikh]

On 04/07/2014 07:47 PM, Christoph Lameter wrote:
> On Mon, 7 Apr 2014, Andi Kleen wrote:
>
>>>    * starts a binary with the specified numa memory policy using
>>> numactl (or a like):
>>>      numactl --interleave=all get_some_memory_with_malloc_and_write_it
>>>    * `sleep` for few seconds
>>>    * numastat > /tmp/after
>>>    * compares /tmp/before and /tmp/after to check that the numa policy
>>> was applied the right way
>>>
>>> But the problem is that on a host with many NUMA nodes (8) the process
>>> of updating that numastats statistics takes some time. Even 10 seconds
>>> may be not enough. Therefore the test fails.
>>>
>>> Is there a direct or indirect way to force the kernel to update the
>>> NUMA statistics?
>>
>> Not currently. It depends on how much memory you have and subsequent
>> operations. I guess would need to add one.
>
> The kernel vm statistics are brought up to date with the default
> settings every 2 seconds.
>
> The interval is controlled via /proc/sys/vm/stat_interval
>
> Check the value that you have setup there.
>

Thank you, Andi, Christoph.

In my setup stat_interval is 1.

Please, look at this reproducer:
#!/bin/bash

sum_pages()
{
         local i

         ret=0
         for i in $@; do
                 ret=$(( $ret + $i ))
         done
}

ret=0

for i in `seq 20`; do
         sum_pages $( numastat | grep interleav | cut -d ' ' -f 2-)
         val_before=$ret
         numactl --interleave=all support_numa 2
         sleep 2
         sum_pages $( numastat | grep interleav | cut -d ' ' -f 2-)
         val_after=$ret

         echo "$i: $(( $val_after - $val_before))"
done

On a two-node system it prints:
1: 294
2: 294
3: 295
4: 294
5: 294
6: 295
7: 294
8: 294
9: 293
10: 293
11: 294
12: 295
13: 296
14: 293
15: 295
16: 294
17: 295
18: 294
19: 294
20: 294

i.e. everything is ok.

But on an eight-node system:
1: 173
2: 0
3: 0
4: 173
5: 173
6: 0
7: 173
8: 173
9: 0
10: 0
11: 173
12: 0
13: 173
14: 0
15: 346
16: 0
17: 0
18: 89
19: 0
20: 173

So in general we can't rely on stat_interval value. Correct?

------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Stanislav Kholmanskikh]

On 04/07/2014 07:47 PM, Christoph Lameter wrote:
> On Mon, 7 Apr 2014, Andi Kleen wrote:
>
>>>    * starts a binary with the specified numa memory policy using
>>> numactl (or a like):
>>>      numactl --interleave=all get_some_memory_with_malloc_and_write_it
>>>    * `sleep` for few seconds
>>>    * numastat > /tmp/after
>>>    * compares /tmp/before and /tmp/after to check that the numa policy
>>> was applied the right way
>>>
>>> But the problem is that on a host with many NUMA nodes (8) the process
>>> of updating that numastats statistics takes some time. Even 10 seconds
>>> may be not enough. Therefore the test fails.
>>>
>>> Is there a direct or indirect way to force the kernel to update the
>>> NUMA statistics?
>>
>> Not currently. It depends on how much memory you have and subsequent
>> operations. I guess would need to add one.
>
> The kernel vm statistics are brought up to date with the default
> settings every 2 seconds.
>
> The interval is controlled via /proc/sys/vm/stat_interval
>
> Check the value that you have setup there.
>

Thank you, Andi, Christoph.

In my setup stat_interval is 1.

Please, look at this reproducer:
#!/bin/bash

sum_pages()
{
         local i

         ret=0
         for i in $@; do
                 ret=$(( $ret + $i ))
         done
}

ret=0

for i in `seq 20`; do
         sum_pages $( numastat | grep interleav | cut -d ' ' -f 2-)
         val_before=$ret
         numactl --interleave=all support_numa 2
         sleep 2
         sum_pages $( numastat | grep interleav | cut -d ' ' -f 2-)
         val_after=$ret

         echo "$i: $(( $val_after - $val_before))"
done

On a two-node system it prints:
1: 294
2: 294
3: 295
4: 294
5: 294
6: 295
7: 294
8: 294
9: 293
10: 293
11: 294
12: 295
13: 296
14: 293
15: 295
16: 294
17: 295
18: 294
19: 294
20: 294

i.e. everything is ok.

But on an eight-node system:
1: 173
2: 0
3: 0
4: 173
5: 173
6: 0
7: 173
8: 173
9: 0
10: 0
11: 173
12: 0
13: 173
14: 0
15: 346
16: 0
17: 0
18: 89
19: 0
20: 173

So in general we can't rely on stat_interval value. Correct?

Re: numastats updates

[Christoph Lameter]

On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:

> On a two-node system it prints:
> 1: 294
..

> 20: 294
>
> i.e. everything is ok.
>
> But on an eight-node system:
> 1: 173
> 2: 0
> 3: 0
> 4: 173
> 5: 173


> So in general we can't rely on stat_interval value. Correct?

Owww... This could have something to do with the tick being disabled on
idle without regard for pending vmstat updates. The processor goes to
sleep and never updates the global counters. Once in awhile the processor
wakes up to deal with the statistics.

The idle logic needs to check that all differentials are folded before
going to slee. Frederic: Can we add such a check?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LTP] numastats updates

[Christoph Lameter]

On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:

> On a two-node system it prints:
> 1: 294
..

> 20: 294
>
> i.e. everything is ok.
>
> But on an eight-node system:
> 1: 173
> 2: 0
> 3: 0
> 4: 173
> 5: 173


> So in general we can't rely on stat_interval value. Correct?

Owww... This could have something to do with the tick being disabled on
idle without regard for pending vmstat updates. The processor goes to
sleep and never updates the global counters. Once in awhile the processor
wakes up to deal with the statistics.

The idle logic needs to check that all differentials are folded before
going to slee. Frederic: Can we add such a check?


------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] numastats updates

[Stanislav Kholmanskikh]

On 04/08/2014 08:49 PM, Stanislav Kholmanskikh wrote:

>
> On a two-node system it prints:
> 1: 294
> 2: 294
> 3: 295
> 4: 294
> 5: 294
> 6: 295
> 7: 294
> 8: 294
> 9: 293
> 10: 293
> 11: 294
> 12: 295
> 13: 296
> 14: 293
> 15: 295
> 16: 294
> 17: 295
> 18: 294
> 19: 294
> 20: 294
>
> i.e. everything is ok.
>
> But on an eight-node system:
> 1: 173
> 2: 0
> 3: 0
> 4: 173
> 5: 173
> 6: 0
> 7: 173
> 8: 173
> 9: 0
> 10: 0
> 11: 173
> 12: 0
> 13: 173
> 14: 0
> 15: 346
> 16: 0
> 17: 0
> 18: 89
> 19: 0
> 20: 173
>

Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.

------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Stanislav Kholmanskikh]

On 04/08/2014 08:49 PM, Stanislav Kholmanskikh wrote:

>
> On a two-node system it prints:
> 1: 294
> 2: 294
> 3: 295
> 4: 294
> 5: 294
> 6: 295
> 7: 294
> 8: 294
> 9: 293
> 10: 293
> 11: 294
> 12: 295
> 13: 296
> 14: 293
> 15: 295
> 16: 294
> 17: 295
> 18: 294
> 19: 294
> 20: 294
>
> i.e. everything is ok.
>
> But on an eight-node system:
> 1: 173
> 2: 0
> 3: 0
> 4: 173
> 5: 173
> 6: 0
> 7: 173
> 8: 173
> 9: 0
> 10: 0
> 11: 173
> 12: 0
> 13: 173
> 14: 0
> 15: 346
> 16: 0
> 17: 0
> 18: 89
> 19: 0
> 20: 173
>

Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.

Re: [LTP] numastats updates

[Christoph Lameter]

On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:

> Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.

So they are not x86? There are some counter races on !x86.


------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Christoph Lameter]

On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:

> Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.

So they are not x86? There are some counter races on !x86.

Re: [LTP] numastats updates

[Stanislav Kholmanskikh]

On 04/08/2014 10:03 PM, Christoph Lameter wrote:
> On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:
>
>> Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.
>
> So they are not x86? There are some counter races on !x86.

Yes. This is Linux on SPARC.

>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-numa" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Stanislav Kholmanskikh]

On 04/08/2014 10:03 PM, Christoph Lameter wrote:
> On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:
>
>> Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.
>
> So they are not x86? There are some counter races on !x86.

Yes. This is Linux on SPARC.

>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-numa" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [LTP] numastats updates

[Christoph Lameter]

On Thu, 10 Apr 2014, Stanislav Kholmanskikh wrote:

> On 04/08/2014 10:03 PM, Christoph Lameter wrote:
> > On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:
> >
> > > Forgot to mention. This eight-node system has 8k pages, two-node - 4k
> > > pages.
> >
> > So they are not x86? There are some counter races on !x86.
>
> Yes. This is Linux on SPARC.

Hmmm... I just got the infrastructure merged to define fast counter
updates. If we would define raw_cpu_ops for sparc then we could fix
the issue. Sparc has atomics that can increment a register relative to a
percpu base register right?


------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Christoph Lameter]

On Thu, 10 Apr 2014, Stanislav Kholmanskikh wrote:

> On 04/08/2014 10:03 PM, Christoph Lameter wrote:
> > On Tue, 8 Apr 2014, Stanislav Kholmanskikh wrote:
> >
> > > Forgot to mention. This eight-node system has 8k pages, two-node - 4k
> > > pages.
> >
> > So they are not x86? There are some counter races on !x86.
>
> Yes. This is Linux on SPARC.

Hmmm... I just got the infrastructure merged to define fast counter
updates. If we would define raw_cpu_ops for sparc then we could fix
the issue. Sparc has atomics that can increment a register relative to a
percpu base register right?

Re: [LTP] numastats updates

[Christoph Lameter]

Interrupts are disabled when the hit and miss statistics are
incremented. SO no problem there. Weird situation.


------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: numastats updates

[Christoph Lameter]

Interrupts are disabled when the hit and miss statistics are
incremented. SO no problem there. Weird situation.