Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync

[Vikram Mulukutla <markivx@codeaurora.org>]

Hi Will,

On 2017-08-15 11:40, Will Deacon wrote:
> Hi Vikram,
> 
> On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote:
>> On 2017-07-31 06:13, Will Deacon wrote:
>> >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote:
>> >>On 2017-07-28 02:28, Will Deacon wrote:
>> >>>On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote:
>> 
>> >>>
>> >>This does seem to help. Here's some data after 5 runs with and without
>> >>the
>> >>patch.
>> >
>> >Blimey, that does seem to make a difference. Shame it's so ugly! Would you
>> >be able to experiment with other values for CPU_RELAX_WFE_THRESHOLD? I had
>> >it set to 10000 in the diff I posted, but that might be higher than
>> >optimal.
>> >It would be interested to see if it correlates with num_possible_cpus()
>> >for the highly contended case.
>> >
>> >Will
>> 
>> Sorry for the late response - I should hopefully have some more data 
>> with
>> different thresholds before the week is finished or on Monday.
> 
> Did you get anywhere with the threshold heuristic?
> 
> Will

Here's some data from experiments that I finally got to today. I decided
to recompile for every value of the threshold. Was doing a binary search
of sorts and then started reducing by orders of magnitude. There pairs 
of rows here:

Row1 is with cpu0 (little) at 300MHz and cpu4 at 1.9Ghz
Row2 is with cpu0 (little) at 1.5GHz and cpu4 at 1.9Ghz

It looks like even with the threshold set to 1, we don't hit the worst
case of a single instance of locking taking a long time, probably a 
consequence
of how the test is designed? However as we reduce the threshold, the 
fairness
in terms of how many times each CPU acquires the lock skews towards the 
big CPU,
starting with threshold=500

If I understand the code correctly, the upper 32 bits of an ARM64 
virtual
address will overflow when 1 is added to it, and so we'll keep WFE'ing 
on
every subsequent cpu_relax invoked from the same PC, until we cross the
hard-coded threshold, right?


CPU_RELAX_WFE_THRESHOLD = 5000 and 2500 (very similar results)
=====================================================
cpu0 time | cpu0 counter | cpu4 time | cpu4 counter |
==========|==============|===========|==============|
2|2763059|2|7323169
0|11477590|1|12110373

3|2762788|2|7329881
1|11557987|1|12042980

3|2765912|2|7308789
1|11470294|1|12120074

3|2761793|2|7333907
1|11431841|1|12155046

3|2762402|2|7328843
1|11495705|1|12096518

3|2764392|2|7308640
1|11479146|1|12111419
====================================================|

CPU_RELAX_WFE_THRESHOLD = 500
=====================================================
cpu0 time | cpu0 counter | cpu4 time | cpu4 counter |
==========|==============|===========|==============|
3|2338277|2|10052592
1|6963131|1|18103639

3|2337982|2|10037188
1|6979396|1|18082811

3|2337282|2|10052184
0|6954990|1|18109860

3|2338737|2|10039556
1|7185046|1|17809240

4|2338857|2|10027407
1|6958274|1|18111394

4|2340208|2|10031173
0|7097088|1|17921861
-----------------------------------------------------
=====================================================

CPU_RELAX_WFE_THRESHOLD = 50
=====================================================
cpu0 time | cpu0 counter | cpu4 time | cpu4 counter |
==========|==============|===========|==============|
4|1219792|2|18005180
0|1252767|1|25296935

4|1219312|2|18049566
1|1252625|1|25227292

4|1219884|2|18020775
1|1252363|1|25298387

4|1220862|2|18012062
1|1251827|1|25283787

4|1220489|2|18010055
0|1251729|1|25272917

3|1220088|2|18027279
0|1253264|1|25268834
-----------------------------------------------------
=====================================================

CPU_RELAX_WFE_THRESHOLD = 10
=====================================================
cpu0 time | cpu0 counter | cpu4 time | cpu4 counter |
==========|==============|===========|==============|
3|298604|1|23784805
0|293511|1|24604172

3|294707|2|23857487
0|292519|1|24564209

4|294199|1|23832180
0|293840|1|24593323

4|294314|1|23853353
0|293609|1|24635190

4|293802|1|23836764
0|293322|1|24553212

3|293658|1|23889801
0|292663|1|24552118
-----------------------------------------------------
=====================================================

CPU_RELAX_WFE_THRESHOLD = 5
=====================================================
cpu0 time | cpu0 counter | cpu4 time | cpu4 counter |
==========|==============|===========|==============|
3|173061|1|22332479
0|173759|1|23774009

3|174471|1|22342362
0|173161|1|23814466

3|173851|2|22235422
0|172734|1|23705848

2|173452|1|22255166
0|172830|1|23824301

2|173028|1|22390297
0|172336|1|23836407

3|172968|1|22285954
0|173207|1|23844900
-----------------------------------------------------
=====================================================

CPU_RELAX_WFE_THRESHOLD = 1
=====================================================
cpu0 time | cpu0 counter | cpu4 time | cpu4 counter |
==========|==============|===========|==============|
2|64245|1|6266848
0|77117|1|20917346

2|71310|1|5184106
1|77426|1|21040797

3|71335|2|5024650
0|77167|1|20934429

3|71295|1|5361696
0|77377|1|20902970

2|71357|1|5302482
0|77278|1|20967106

3|71158|1|5214564
0|77334|1|21022485
-----------------------------------------------------
=====================================================

Thanks,
Vikram

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project