poor cyclictest results with 5.0 series rt for arm64

* poor cyclictest results with 5.0 series rt for arm64
@ 2019-07-12 11:18 Paul Thomas
       [not found] ` <CAD56B7c2=95r7r-6tr0Mby8jSeuQPDNSd4LELnUw1LVRn9Rqew@mail.gmail.com>
  2019-07-16 20:24 ` Thomas Gleixner
  0 siblings, 2 replies; 28+ messages in thread
From: Paul Thomas @ 2019-07-12 11:18 UTC (permalink / raw)
  To: linux-rt-users; +Cc: dunc.mcphie

Hello,

We're in the process of implementing a CI test setup for our use case
and HW. Unfortunately this was not up and running when the first 5.0
RT releases became available in March. We're using the Arm64 based
Zynq Ultrascale+ parts from Xilinx.

We have been using 4.18 since last fall with very good results, and we
see this carry through to 4.19.25-rt16. However, for the 5.0.3-rt1
release, cyclictest hits multi thousand microsecond latenies this
continues all the way through to 5.0.21-rt15.

For the 4.19 kernels the v11 zynqmp firmware patch is applied:
https://patchwork.kernel.org/cover/10555405/
Otherwise only a small EPROBE_AGAIN change is needed in
xilinx_uartps.c on top of the rt tag. For the 5.0 kernels there are no
c file changes. Both for 4.19 and 5.0 a custom devicetree is used.

We've tried to make the configs as simple as possible, and for the 5.0
config we start with the 4.19 .config and say N to anything new. (I had
an issue with the first email I'll follow up with the configs if this goes
through).

We do have trace data for a instance where cyclictest hit 5000 uS, but
the file is large 11MB, and I wasn't sure if I should directly attach
it to the email. What is the best way to post or send this?

We would like to bisect this further, but it becomes more difficult
between 4.19.25-rt16 and 5.0.3-rt1.

Some other details: CPU frequency is 1.2 GHz, memory is 2GB, rootfs is debian 9.

For 4.19.25-rt16 we get this (histogram removed):
# ./cyclictest -S -m -n -p 99 -i 200 -h 400 -D 900
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.58 2.77 1.93 1/122 2595

T: 0 ( 2577) P:99 I:200 C:4499985 Min:      5 Act:    6 Avg:    6 Max:      24
T: 1 ( 2578) P:99 I:200 C:4499893 Min:      6 Act:    6 Avg:    6 Max:      13
T: 2 ( 2579) P:99 I:200 C:4499779 Min:      6 Act:    6 Avg:    6 Max:      33
T: 3 ( 2580) P:99 I:200 C:4499666 Min:      6 Act:    6 Avg:    6 Max:      26

And for 5.0.3-rt1 we get:
# ./cyclictest -S -m -n -p 99 -i 200 -h 400 -D 900
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 4.37 4.28 2.88 1/129 2773

T: 0 ( 2757) P:99 I:200 C:4499958 Min:      5 Act:    9 Avg:    6 Max:      31
T: 1 ( 2758) P:99 I:200 C:4499500 Min:      5 Act:    7 Avg:    6 Max:    9045
T: 2 ( 2759) P:99 I:200 C:4498926 Min:      5 Act:    6 Avg:    6 Max:     263
T: 3 ( 2760) P:99 I:200 C:4499644 Min:      5 Act:    6 Avg:    6 Max:      34

Thanks in advance for any help.

-Paul

^ permalink raw reply	[flat|nested] 28+ messages in thread