Re: [LKP] Re: [sched] bdfcae1140: will-it-scale.per_thread_ops -37.0% regression

[Xing Zhengjun <zhengjun.xing@linux.intel.com>]

On 10/7/2020 10:50 PM, Mathieu Desnoyers wrote:
> ----- On Oct 2, 2020, at 4:33 AM, Rong Chen rong.a.chen@intel.com wrote:
> 
>> Greeting,
>>
>> FYI, we noticed a -37.0% regression of will-it-scale.per_thread_ops due to
>> commit:
>>
>>
>> commit: bdfcae11403e5099769a7c8dc3262e3c4193edef ("[RFC PATCH 2/3] sched:
>> membarrier: cover kthread_use_mm (v3)")
>> url:
>> https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/Membarrier-updates/20200925-012549
>> base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git
>> 848785df48835eefebe0c4eb5da7690690b0a8b7
>>
>> in testcase: will-it-scale
>> on test machine: 104 threads Skylake with 192G memory
>> with following parameters:
>>
>> 	nr_task: 50%
>> 	mode: thread
>> 	test: context_switch1
>> 	cpufreq_governor: performance
>> 	ucode: 0x2006906
>>
>> test-description: Will It Scale takes a testcase and runs it from 1 through to n
>> parallel copies to see if the testcase will scale. It builds both a process and
>> threads based test in order to see any differences between the two.
>> test-url: https://github.com/antonblanchard/will-it-scale
>>
> 
> Hi,
> 
> I would like to report what I suspect is a random thread placement issue in the
> context_switch1 test used by the 0day bot when running on a machine with hyperthread
> enabled.
> 
> AFAIU the test code uses hwloc for thread placement which should theoretically ensure
> that each thread is placed on same processing unit, core and numa node between runs.
> 
> We can find the test code here:
> 
> https://github.com/antonblanchard/will-it-scale/blob/master/tests/context_switch1.c
> 
> And the main file containing thread setup is here:
> 
> https://github.com/antonblanchard/will-it-scale/blob/master/main.c
> 
> AFAIU, the test is started without the "-m" switch, which therefore affinitizes
> tasks on cores rather than on processing units (SMT threads).
> 
> When testcase() creates the child thread with new_task(), it basically issues:
> 
>    pthread_create(&threads[nr_threads++], NULL, func, arg);
> 
> passing a NULL pthread_attr_t, and not executing any pre_trampoline on the child.
> The pre_trampoline would have issued hwloc_set_thread_cpubind if it were executed on
> the child, but it's not. Therefore, we expect the cpu affinity mask of the parent to
> be copied on clone and used by the child.
> 
> A quick test on a machine with hyperthreading enabled shows that the cpu affinity mask
> for the parent and child has two bits set:
> 
> taskset -p 1868607
> pid 1868607's current affinity mask: 10001
> taskset -p 1868606
> pid 1868606's current affinity mask: 10001
> 
> So AFAIU the placement of the parent and child will be random on either the same
> processing unit, or on separate processing units within the same core.
> 
> I suspect this randomness can significantly affect the performance number between
> runs, and trigger unwarranted performance regression warnings.
> 
> Thanks,
> 
> Mathieu
> 
Yes, the randomness may happen in some special cases.  But in 0-day, we 
test multi times (>=3), the report is the average number.
For this case, we test 4 times, it is stable, the wave is ±  2%.
So I don't think the -37.0% regression is caused by the randomness.

0/stats.json:  "will-it-scale.per_thread_ops": 105228,
1/stats.json:  "will-it-scale.per_thread_ops": 100443,
2/stats.json:  "will-it-scale.per_thread_ops": 98786,
3/stats.json:  "will-it-scale.per_thread_ops": 102821,

c2daff748f0ea954 bdfcae11403e5099769a7c8dc32
---------------- ---------------------------
          %stddev     %change         %stddev
              \          |                \
     161714 ±  2%     -37.0%     101819 ±  2%  will-it-scale.per_thread_ops


-- 
Zhengjun Xing