* arm64 + ARM64_64K_PAGES=y
@ 2018-11-06 21:34 Grygorii Strashko
2018-11-08 12:00 ` Sebastian Andrzej Siewior
2018-11-09 4:42 ` Anshuman Khandual
0 siblings, 2 replies; 7+ messages in thread
From: Grygorii Strashko @ 2018-11-06 21:34 UTC (permalink / raw)
To: linux-rt-users; +Cc: linux-kernel, Linux ARM Mailing List
Hi All,
Do anybody tried to use ARM64 RT with 76K pages enabled?
My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y increases latencies by ~30%
cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=y
T: 0 ( 772) P:98 I:1000 C: 120000 Min: 7 Act: 13 Avg: 10 Max: 85
T: 1 ( 773) P:98 I:1500 C: 79998 Min: 7 Act: 13 Avg: 10 Max: 71
T: 2 ( 774) P:98 I:2000 C: 59997 Min: 7 Act: 11 Avg: 11 Max: 64
T: 3 ( 775) P:98 I:2500 C: 47996 Min: 7 Act: 14 Avg: 12 Max: 66
cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=n
T: 0 ( 697) P:98 I:1000 C: 120000 Min: 7 Act: 10 Avg: 9 Max: 38
T: 1 ( 698) P:98 I:1500 C: 79987 Min: 7 Act: 10 Avg: 10 Max: 32
T: 2 ( 699) P:98 I:2000 C: 59981 Min: 7 Act: 14 Avg: 11 Max: 46
T: 3 ( 700) P:98 I:2500 C: 47977 Min: 6 Act: 11 Avg: 10 Max: 45
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: arm64 + ARM64_64K_PAGES=y
2018-11-06 21:34 arm64 + ARM64_64K_PAGES=y Grygorii Strashko
@ 2018-11-08 12:00 ` Sebastian Andrzej Siewior
2018-11-08 18:14 ` Grygorii Strashko
2018-11-09 4:42 ` Anshuman Khandual
1 sibling, 1 reply; 7+ messages in thread
From: Sebastian Andrzej Siewior @ 2018-11-08 12:00 UTC (permalink / raw)
To: Grygorii Strashko; +Cc: linux-rt-users, linux-kernel, Linux ARM Mailing List
On 2018-11-06 15:34:55 [-0600], Grygorii Strashko wrote:
> Hi All,
Hi,
> Do anybody tried to use ARM64 RT with 76K pages enabled?
75 would be an off by one but this :)
> My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y increases latencies by ~30%
>
> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=y
>
>
> T: 0 ( 772) P:98 I:1000 C: 120000 Min: 7 Act: 13 Avg: 10 Max: 85
> T: 1 ( 773) P:98 I:1500 C: 79998 Min: 7 Act: 13 Avg: 10 Max: 71
> T: 2 ( 774) P:98 I:2000 C: 59997 Min: 7 Act: 11 Avg: 11 Max: 64
> T: 3 ( 775) P:98 I:2500 C: 47996 Min: 7 Act: 14 Avg: 12 Max: 66
>
>
> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=n
>
>
> T: 0 ( 697) P:98 I:1000 C: 120000 Min: 7 Act: 10 Avg: 9 Max: 38
> T: 1 ( 698) P:98 I:1500 C: 79987 Min: 7 Act: 10 Avg: 10 Max: 32
> T: 2 ( 699) P:98 I:2000 C: 59981 Min: 7 Act: 14 Avg: 11 Max: 46
> T: 3 ( 700) P:98 I:2500 C: 47977 Min: 6 Act: 11 Avg: 10 Max: 45
So this is an idle system?
The Kconfig help says "faster TLB lookup". Interesting.
Are the 16k pages in between (latency wise) by any chance?
> --
> regards,
> -grygorii
Sebastian
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: arm64 + ARM64_64K_PAGES=y
2018-11-08 12:00 ` Sebastian Andrzej Siewior
@ 2018-11-08 18:14 ` Grygorii Strashko
2018-11-09 19:15 ` Grygorii Strashko
0 siblings, 1 reply; 7+ messages in thread
From: Grygorii Strashko @ 2018-11-08 18:14 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: linux-rt-users, linux-kernel, Linux ARM Mailing List
On 11/8/18 6:00 AM, Sebastian Andrzej Siewior wrote:
> On 2018-11-06 15:34:55 [-0600], Grygorii Strashko wrote:
>> Hi All,
> Hi,
>
>> Do anybody tried to use ARM64 RT with 76K pages enabled?
>
> 75 would be an off by one but this :)
Ops 8-). at least subj is correct.
>
>> My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y increases latencies by ~30%
>>
>> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=y
>>
>>
>> T: 0 ( 772) P:98 I:1000 C: 120000 Min: 7 Act: 13 Avg: 10 Max: 85
>> T: 1 ( 773) P:98 I:1500 C: 79998 Min: 7 Act: 13 Avg: 10 Max: 71
>> T: 2 ( 774) P:98 I:2000 C: 59997 Min: 7 Act: 11 Avg: 11 Max: 64
>> T: 3 ( 775) P:98 I:2500 C: 47996 Min: 7 Act: 14 Avg: 12 Max: 66
>>
>>
>> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=n
>>
>>
>> T: 0 ( 697) P:98 I:1000 C: 120000 Min: 7 Act: 10 Avg: 9 Max: 38
>> T: 1 ( 698) P:98 I:1500 C: 79987 Min: 7 Act: 10 Avg: 10 Max: 32
>> T: 2 ( 699) P:98 I:2000 C: 59981 Min: 7 Act: 14 Avg: 11 Max: 46
>> T: 3 ( 700) P:98 I:2500 C: 47977 Min: 6 Act: 11 Avg: 10 Max: 45
>
> So this is an idle system?
Yes (in general) - it's collected with systemd, so some daemons are active.
> The Kconfig help says "faster TLB lookup". Interesting.
> Are the 16k pages in between (latency wise) by any chance?
I'll try it.
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: arm64 + ARM64_64K_PAGES=y
2018-11-06 21:34 arm64 + ARM64_64K_PAGES=y Grygorii Strashko
2018-11-08 12:00 ` Sebastian Andrzej Siewior
@ 2018-11-09 4:42 ` Anshuman Khandual
1 sibling, 0 replies; 7+ messages in thread
From: Anshuman Khandual @ 2018-11-09 4:42 UTC (permalink / raw)
To: Grygorii Strashko, linux-rt-users; +Cc: linux-kernel, Linux ARM Mailing List
On 11/07/2018 03:04 AM, Grygorii Strashko wrote:
> Hi All,
>
> Do anybody tried to use ARM64 RT with 76K pages enabled?
>
> My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y increases latencies by ~30%
Depends on what the workload is actually doing. 64K pages should help if
the mapping is multiple of 64K, persistent and accesses patterns are more
or less linear to be nice with the TLB. 64K can take bit more time if the
memory requirement is way smaller than 64K in which case latency probably
can increase just to zero out the single page allocated. Latency can add
up if this happens on a regular basis. perf report can help find out more
on this.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: arm64 + ARM64_64K_PAGES=y
2018-11-08 18:14 ` Grygorii Strashko
@ 2018-11-09 19:15 ` Grygorii Strashko
2018-11-12 14:27 ` Andre Przywara
0 siblings, 1 reply; 7+ messages in thread
From: Grygorii Strashko @ 2018-11-09 19:15 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: linux-rt-users, linux-kernel, Linux ARM Mailing List
On 11/8/18 12:14 PM, Grygorii Strashko wrote:
>
>
> On 11/8/18 6:00 AM, Sebastian Andrzej Siewior wrote:
>> On 2018-11-06 15:34:55 [-0600], Grygorii Strashko wrote:
>>> Hi All,
>> Hi,
>>
>>> Do anybody tried to use ARM64 RT with 76K pages enabled?
>>
>> 75 would be an off by one but this :)
>
> Ops 8-). at least subj is correct.
>
>>
>>> My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y increases latencies by ~30%
>>>
>>> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=y
>>>
>>>
>>> T: 0 ( 772) P:98 I:1000 C: 120000 Min: 7 Act: 13 Avg: 10 Max: 85
>>> T: 1 ( 773) P:98 I:1500 C: 79998 Min: 7 Act: 13 Avg: 10 Max: 71
>>> T: 2 ( 774) P:98 I:2000 C: 59997 Min: 7 Act: 11 Avg: 11 Max: 64
>>> T: 3 ( 775) P:98 I:2500 C: 47996 Min: 7 Act: 14 Avg: 12 Max: 66
>>>
>>>
>>> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=n
>>>
>>>
>>> T: 0 ( 697) P:98 I:1000 C: 120000 Min: 7 Act: 10 Avg: 9 Max: 38
>>> T: 1 ( 698) P:98 I:1500 C: 79987 Min: 7 Act: 10 Avg: 10 Max: 32
>>> T: 2 ( 699) P:98 I:2000 C: 59981 Min: 7 Act: 14 Avg: 11 Max: 46
>>> T: 3 ( 700) P:98 I:2500 C: 47977 Min: 6 Act: 11 Avg: 10 Max: 45
>>
>> So this is an idle system?
>
> Yes (in general) - it's collected with systemd, so some daemons are active.
>
>> The Kconfig help says "faster TLB lookup". Interesting.
>> Are the 16k pages in between (latency wise) by any chance?
>
> I'll try it.
no i'll not, at least not fast. with 16k pages enabled I can't boot TI 4.14 kernel
- 4.14.71-rt44.
No msg in log, just "Starting kernel ..."
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: arm64 + ARM64_64K_PAGES=y
2018-11-09 19:15 ` Grygorii Strashko
@ 2018-11-12 14:27 ` Andre Przywara
2018-11-12 21:22 ` Grygorii Strashko
0 siblings, 1 reply; 7+ messages in thread
From: Andre Przywara @ 2018-11-12 14:27 UTC (permalink / raw)
To: Grygorii Strashko
Cc: Sebastian Andrzej Siewior, linux-rt-users,
Linux ARM Mailing List, linux-kernel
On Fri, 9 Nov 2018 13:15:47 -0600
Grygorii Strashko <grygorii.strashko@ti.com> wrote:
Hi,
> On 11/8/18 12:14 PM, Grygorii Strashko wrote:
> >
> >
> > On 11/8/18 6:00 AM, Sebastian Andrzej Siewior wrote:
> >> On 2018-11-06 15:34:55 [-0600], Grygorii Strashko wrote:
> >>> Hi All,
> >> Hi,
> >>
> >>> Do anybody tried to use ARM64 RT with 76K pages enabled?
> >>
> >> 75 would be an off by one but this :)
> >
> > Ops 8-). at least subj is correct.
> >
> >>
> >>> My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y
> >>> increases latencies by ~30%
That's not really surprising. Performance on systems using a bigger page
size granules might have some trade-offs (bigger memory overhead, worse
cache utilization), so 64K pages might not be really great for your
particular workload. You would probably need a real performance
analysis (using perf, for instance) to pinpoint TLB misses as your
bottleneck.
> >>> cyclictest -n -m -Sp98 -q -D2m with =y
> >>>
> >>>
> >>> T: 0 ( 772) P:98 I:1000 C: 120000 Min: 7 Act: 13 Avg:
> >>> 10 Max: 85 T: 1 ( 773) P:98 I:1500 C: 79998 Min: 7
> >>> Act: 13 Avg: 10 Max: 71 T: 2 ( 774) P:98 I:2000 C:
> >>> 59997 Min: 7 Act: 11 Avg: 11 Max: 64 T: 3 ( 775)
> >>> P:98 I:2500 C: 47996 Min: 7 Act: 14 Avg: 12 Max: 66
> >>>
> >>>
> >>> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=n
> >>>
> >>>
> >>> T: 0 ( 697) P:98 I:1000 C: 120000 Min: 7 Act: 10 Avg:
> >>> 9 Max: 38 T: 1 ( 698) P:98 I:1500 C: 79987 Min: 7
> >>> Act: 10 Avg: 10 Max: 32 T: 2 ( 699) P:98 I:2000 C:
> >>> 59981 Min: 7 Act: 14 Avg: 11 Max: 46 T: 3 ( 700)
> >>> P:98 I:2500 C: 47977 Min: 6 Act: 11 Avg: 10 Max:
> >>> 45
> >>
> >> So this is an idle system?
> >
> > Yes (in general) - it's collected with systemd, so some daemons are
> > active.
> >> The Kconfig help says "faster TLB lookup". Interesting.
> >> Are the 16k pages in between (latency wise) by any chance?
> >
> > I'll try it.
>
> no i'll not, at least not fast. with 16k pages enabled I can't boot
> TI 4.14 kernel
> - 4.14.71-rt44.
> No msg in log, just "Starting kernel ..."
You need a core that actually supports 16K pages (supporting
certain page size granules is architecturally optional).
From the Arm Ltd. cores it's Cortex-A73, A75 or A55, possibly other
newer ones as well. Cortex-A53, A57 and A72 do not support 16k pages.
Cheers,
Andre.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: arm64 + ARM64_64K_PAGES=y
2018-11-12 14:27 ` Andre Przywara
@ 2018-11-12 21:22 ` Grygorii Strashko
0 siblings, 0 replies; 7+ messages in thread
From: Grygorii Strashko @ 2018-11-12 21:22 UTC (permalink / raw)
To: Andre Przywara
Cc: Sebastian Andrzej Siewior, linux-rt-users,
Linux ARM Mailing List, linux-kernel
On 11/12/18 8:27 AM, Andre Przywara wrote:
> On Fri, 9 Nov 2018 13:15:47 -0600
> Grygorii Strashko <grygorii.strashko@ti.com> wrote:
>
> Hi,
>
>> On 11/8/18 12:14 PM, Grygorii Strashko wrote:
>>>
>>>
>>> On 11/8/18 6:00 AM, Sebastian Andrzej Siewior wrote:
>>>> On 2018-11-06 15:34:55 [-0600], Grygorii Strashko wrote:
>>>>> Hi All,
>>>> Hi,
>>>>
>>>>> Do anybody tried to use ARM64 RT with 76K pages enabled?
>>>>
>>>> 75 would be an off by one but this :)
>>>
>>> Ops 8-). at least subj is correct.
>>>
>>>>
>>>>> My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y
>>>>> increases latencies by ~30%
>
> That's not really surprising. Performance on systems using a bigger page
> size granules might have some trade-offs (bigger memory overhead, worse
> cache utilization), so 64K pages might not be really great for your
> particular workload. You would probably need a real performance
> analysis (using perf, for instance) to pinpoint TLB misses as your
> bottleneck.
>
>>>>> cyclictest -n -m -Sp98 -q -D2m with =y
>>>>>
>>>>>
>>>>> T: 0 ( 772) P:98 I:1000 C: 120000 Min: 7 Act: 13 Avg:
>>>>> 10 Max: 85 T: 1 ( 773) P:98 I:1500 C: 79998 Min: 7
>>>>> Act: 13 Avg: 10 Max: 71 T: 2 ( 774) P:98 I:2000 C:
>>>>> 59997 Min: 7 Act: 11 Avg: 11 Max: 64 T: 3 ( 775)
>>>>> P:98 I:2500 C: 47996 Min: 7 Act: 14 Avg: 12 Max: 66
>>>>>
>>>>>
>>>>> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=n
>>>>>
>>>>>
>>>>> T: 0 ( 697) P:98 I:1000 C: 120000 Min: 7 Act: 10 Avg:
>>>>> 9 Max: 38 T: 1 ( 698) P:98 I:1500 C: 79987 Min: 7
>>>>> Act: 10 Avg: 10 Max: 32 T: 2 ( 699) P:98 I:2000 C:
>>>>> 59981 Min: 7 Act: 14 Avg: 11 Max: 46 T: 3 ( 700)
>>>>> P:98 I:2500 C: 47977 Min: 6 Act: 11 Avg: 10 Max:
>>>>> 45
>>>>
>>>> So this is an idle system?
>>>
>>> Yes (in general) - it's collected with systemd, so some daemons are
>>> active.
>>>> The Kconfig help says "faster TLB lookup". Interesting.
>>>> Are the 16k pages in between (latency wise) by any chance?
>>>
>>> I'll try it.
>>
>> no i'll not, at least not fast. with 16k pages enabled I can't boot
>> TI 4.14 kernel
>> - 4.14.71-rt44.
>> No msg in log, just "Starting kernel ..."
>
> You need a core that actually supports 16K pages (supporting
> certain page size granules is architecturally optional).
> From the Arm Ltd. cores it's Cortex-A73, A75 or A55, possibly other
> newer ones as well. Cortex-A53, A57 and A72 do not support 16k pages.
Thank a lot for you reply.
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-11-12 21:23 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-06 21:34 arm64 + ARM64_64K_PAGES=y Grygorii Strashko
2018-11-08 12:00 ` Sebastian Andrzej Siewior
2018-11-08 18:14 ` Grygorii Strashko
2018-11-09 19:15 ` Grygorii Strashko
2018-11-12 14:27 ` Andre Przywara
2018-11-12 21:22 ` Grygorii Strashko
2018-11-09 4:42 ` Anshuman Khandual
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).