Re: Xen optimization

[Julien Grall <julien.grall@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 6081 bytes --]

Hi,

Sorry for the formatting.

On Fri, 12 Oct 2018, 17:36 Milan Boberic, <milanboberic94@gmail.com> wrote:

> Hi Stefano, glad to have you back :D,
> this is my setup:
>         - dom0 is PetaLinux, has 1 vCPU and it's pinned for pCPU0
>         - there is only one domU and this is my bare-metal app that also
> have one vCPU and it's pinned for pCPU1
> so yeah, there is only dom0 and bare-metal app on the board.
>
> Jitter is the same with and without Dario's patch.
>
> I'm still not sure about timer's passthrough because there is no mention
> of triple timer counter is device tree so I added:
>
> &ttc0 {
>    xen,passthrough = <0x1>;
> };
>

Would you mind to explain what is the triple timer counter?



> at the end of the xen-overlay.dtsi file which I included in attachment.
>
> About patch you sent, I can't find this funcion void vgic_inject_irq in
> /xen/arch/arm/vgic.c file, this is link of git repository from where I
> build my xen so you can take a look if that printk can be put somewhere
> else.
>

There was some vGIC rework in Xen 4.11. There was also a new vGIC added
(selectable using NEW_VGIC). It might be worth to look at it.


> https://github.com/Xilinx/xen/
>

This is not the official Xen repository and look like patches have been
applied on top. I am afraid, I am not going to be able help here. Could you
do the same experiment with Xen 4.11?


>
> I ran some more testing and realized that results are the same with or
> without vwfi=native, which I think again points out that passthrough that I
> need to provide in device tree isn't valid.
>

This could also means that wfi is not used by the guest or you never go to
the idle vCPU.


>  And of course, higher the frequency of interrupts results in higher
> jitter. I'm still battling with Xilinx SDK and triple timer counter that's
> why I can't figure out what is the exact frequency set (I'm just rising it
> and lowering it), I'll give my best to solve that ASAP because we need to
> know exact value of frequency set.
>
> Thanks in advance!
>
> Milan
>
>
>
> On Fri, Oct 12, 2018 at 12:29 AM Stefano Stabellini <
> stefano.stabellini@xilinx.com> wrote:
>
>> On Thu, 11 Oct 2018, Milan Boberic wrote:
>> > On Wed, Oct 10, 2018 at 6:41 PM Meng Xu <xumengpanda@gmail.com> wrote:
>> > >
>> > > The jitter may come from Xen or the OS in dom0.
>> > > It will be useful to know what is the jitter if you run the test on
>> PetaLinux.
>> > > (It's understandable the jitter is gone without OS. It is also common
>> > > that OS introduces various interferences.)
>> >
>> > Hi Meng,
>> > well... I'm using bare-metal application and I need it exclusively to
>> > be ran on one CPU as domU (guest) without OS (and I'm not sure how
>> > would I make the same app to be ran on PetaLinux dom0 :D haha).
>> > Is there a chance that PetaLinux as dom0 is creating this jitter and
>> > how? Is there a way of decreasing it?
>> >
>> > Yes, there are no prints.
>> >
>> > I'm not sure about this timer interrupt passthrough because I didn't
>> > find any example of it, in attachment I included xen-overlay.dtsi file
>> > which I edited to add passthrough, in earlier replies there are
>> > bare-metal configuration file. It would be helpful to know if those
>> > setting are correct. If they are not correct it would explain the
>> > jitter.
>> >
>> > Thanks in advance, Milan Boberic!
>>
>> Hi Milan,
>>
>> Sorry for taking so long to go back to this thread. But I am here now :)
>>
>> First, let me ask a couple of questions to understand the scenario
>> better: is there any interference from other virtual machines while you
>> measure the jitter? Or is the baremetal app the only thing actively
>> running on the board?
>>
>> Second, it would be worth double-checking that Dario's patch to fix
>> sched=null is not having unexpected side effects. I don't think so, it
>> would be worth testing with it and without it to be sure.
>>
>> I gave a look at your VM configuration. The configuration looks correct.
>> There is no dtdev settings, but given that none of the devices you are
>> assigning to the guest does any DMA, it should be OK. You want to make
>> sure that Dom0 is not trying to use those same devices -- make sure to
>> add "xen,passthrough;" to each corresponding node on the host device
>> tree.
>>
>> The error messages "No valid vCPU found" are due to the baremetal
>> applications trying to configure as target cpu for the interrupt cpu1
>> (the second cpu in the system), while actually only 1 vcpu is assigned
>> to the VM. Hence, only cpu0 is allowed. I don't think it should cause
>> any jitter issues, because the request is simply ignored. Just to be
>> safe, you might want to double check that the physical interrupt is
>> delivered to the right physical cpu, which would be cpu1 in your
>> configuration, the one running the only vcpu of the baremetal app. You
>> can do that by adding a printk to xen/arch/arm/vgic.c:vgic_inject_irq,
>> for example:
>>
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 5a4f082..208fde7 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -591,6 +591,7 @@ void vgic_inject_irq(struct domain *d, struct vcpu
>> *v, unsigned int virq,
>>  out:
>>      spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
>>
>> +    if (v != current) printk("DEBUG irq slow path!\n");
>>      /* we have a new higher priority irq, inject it into the guest */
>>      vcpu_kick(v);
>>
>> You don't want "DEBUG irq slow path!" to get printed.
>>
>> Finally, I would try to set the timer to generate events less frequently
>> than every 1us and see what happens, maybe every 5-10us. In my tests,
>> the IRQ latency overhead caused by Xen is around 1us, so injecting 1
>> interrupt every 1us, plus 1us of latency caused by Xen, cannot lead to
>> good results.
>>
>> I hope this helps, please keep us updated with your results, they are
>> very interesting!
>>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 8482 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel