Re: Xen optimization

[Milan Boberic <milanboberic94@gmail.com>]

Hi,
>
> The device tree with everything seems to be system.dts, that was enough
> :-)  I don't need the dtsi files you used to build the final dts, I only
> need the one you use in uboot and for your guest.

 I wasn't sure so I sent everything, sorry for being bombarded with
all those files. :-)

> It looks like you set xen,passthrough correctly in system.dts for
> timer@ff110000, serial@ff010000, and gpio@ff0a0000.

Thank you for taking a look, now we are sure that passthrough works
correctly because there is no error during guest creation and there
are no prints of "DEBUG irq slow path".

> If you are not getting any errors anymore when creating your baremetal
> guest, then yes, it should be working passthrough. I would double-check
> that everything is working as expected using the DEBUG patch for Xen I
> suggested to you in the other email. You might even want to remove the
> "if" check and always print something for every interrupt of your guest
> just to get an idea of what's going on. See the attached patch.

When I apply this patch it prints forever:
(XEN) DEBUG virq=68 local=1
which is a good thing I guess because interrupts are being generated non-stop.

> Once everything is as expected I would change the frequency of the
> timer, because 1u is way too frequent. I think it should be at least
> 3us, more like 5us.

Okay, about this... I double checked my bare-metal application and
looks like interrupts weren't generated every 1 us. Maximum frequency
of interrupts is 8 us. I checked interrupt frequency with oscilloscope
just to be sure (toggling LED on/off when interrupts occur). So, when
I set:
- interrupts to be generated every 8 us I get jitter of 6 us
- interrupts to be generated every 10 us I get jitter of 3 us (after
2-3mins it jumps to 6 us)
- interrupts to be generated every 15 us jitter is the same as when
only bare-metal application runs on board (without Xen or any OS)

I want to remind you that bare-metal application that only blinks LED
with high speed gives 1 us jitter, somehow introducing frequent
interrupts causes this jitter, that's why I was unsecure about this
timer passthrough. Taking in consideration that you measured Xen
overhead of 1 us I have a feeling that I'm missing something, is there
anything else I could do to get better results except sched=null,
vwfi=native, hard vCPU pinning (1 vCPU on 1 pCPU) and passthrough (not
sure if it affects the jitter) ?
I'm forcing frequent interrupts because I'm testing to see if this
board with Xen on it could be used for real-time simulations,
real-time signal processing, etc. If I could get results like yours (1
us Xen overhead) of even better that would be great! BTW how did you
measure Xen's overhead?

> Keep in mind that jitter is about having
> deterministic IRQ latency, not about having extremely frequent
> interrupts.

Yes, but I want to see exactly where will I lose deterministic IRQ
latency which is extremely important in real-time signal processing.
So, what causes this jitter, are those Xen limits, ARM limits, etc? It
would be nice to know, I'll share all the results I get.

> I would also double check that you are not using any other devices or
> virtual interfaces in your baremetal app because that could negatively
> affect the numbers.

I checked the bare-metal app and I think there is no other devices
that bm app is using.

> Linux by default uses the virtual
> timer interface ("arm,armv8-timer", I would double check that the
> baremetal app is not doing the same -- you don't want to be using two
> timers when doing your measurements.

Hmm, I'm not sure how to check that, I could send bare-metal app if
that helps, it's created in Xilinx SDK 2017.4.
Also, should I move to Xilinx SDK 2018.2 because I'm using PetaLinux 2018.2 ?
I'm also using hardware description file for SDK that is created in
Vivado 2017.4.
Is all this could be a "not matching version" problem (I don't think
so because bm app works)?

Meng mentioned in some of his earlier posts:

> Even though the app. is the only one running on the CPU, the CPU may
> be used to handle other interrupts and its context (such as TLB and
> cache) might be flushed by other components. When these happen, the
> interrupt handling latency can vary a lot.

What do you think about this? I don't know how would I check this.

I also tried using default scheduler (removed sched=null and
vwfi=native) and jitter is 10 us when interrupt is generated every 10
us.

Thanks in advance!

Milan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel