IPI broadcast latency in the guest is worse when AVIC is enabled

IPI broadcast latency in the guest is worse when AVIC is enabled

[Wanpeng Li]

Hi all,

We found that the IPI broadcast latency in the guest when AVIC=1,
exposing xapic is worse than when AVIC=0, exposing xapic. The host is
AMD ROME, 2 sockets, 96 cores, 192 threads, the VM is 180 vCPUs. The
guest boots with kvm-hint-dedicated=on, --overcommit cpu-pm=on, -smp
180,sockets=2,cores=45,threads=2, l3-cache=on qemu command-line, the
pCPU which vCPU is running on is isolated. Both the guest and host
kernel are 5.8 Linus' tree. (Note, if you fails to boot with
--overcommit cpu-pm=on, you can comments out commit e72436bc3a52, I
have a report here, https://lkml.org/lkml/2020/7/8/308)

IPI microbenchmark(https://lkml.org/lkml/2017/12/19/141, Destination
Shorthand is All excluding self)

avic0_xapic:   12313907508.50 ns
avic1_xapic:   19106424733.30 ns
avic0_x2apic: 13073988486.00 ns


ebizzy -M  (Destination Shorthand is All excluding self)

avic0_xapic
9416 records/s
real 10.00 s
user  4.80 s
sys  1693.25 s

avic1_xapic
18157 records/s
real 10.00 s
user 10.69 s
sys  1779.80 s

avic0_x2apic
74507 records/s
real 10.00 s
user 48.98 s
sys  1752.12 s


./hackbench -l 1000000  (Destination Shorthand is Destination)

avic0_xapic
Time: 121.339

avic1_xapic
Time: 117.840

avic0_x2apic
Time: 118.753

Any thoughts?

    Wanpeng

Re: IPI broadcast latency in the guest is worse when AVIC is enabled

[Paolo Bonzini]

On 11/08/20 04:04, Wanpeng Li wrote:
> We found that the IPI broadcast latency in the guest when AVIC=1,
> exposing xapic is worse than when AVIC=0, exposing xapic. The host is
> AMD ROME, 2 sockets, 96 cores, 192 threads, the VM is 180 vCPUs. The
> guest boots with kvm-hint-dedicated=on, --overcommit cpu-pm=on, -smp
> 180,sockets=2,cores=45,threads=2, l3-cache=on qemu command-line, the
> pCPU which vCPU is running on is isolated. Both the guest and host
> kernel are 5.8 Linus' tree. (Note, if you fails to boot with
> --overcommit cpu-pm=on, you can comments out commit e72436bc3a52, I
> have a report here, https://lkml.org/lkml/2020/7/8/308)
> 
> IPI microbenchmark(https://lkml.org/lkml/2017/12/19/141, Destination
> Shorthand is All excluding self)
> 
> avic0_xapic:   12313907508.50 ns
> avic1_xapic:   19106424733.30 ns
> avic0_x2apic: 13073988486.00 ns

I think it depends on the microarchitecture implementation of AVIC?

Paolo

Re: IPI broadcast latency in the guest is worse when AVIC is enabled

[Wanpeng Li]

Suravee, any inputs?
On Tue, 11 Aug 2020 at 16:29, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 11/08/20 04:04, Wanpeng Li wrote:
> > We found that the IPI broadcast latency in the guest when AVIC=1,
> > exposing xapic is worse than when AVIC=0, exposing xapic. The host is
> > AMD ROME, 2 sockets, 96 cores, 192 threads, the VM is 180 vCPUs. The
> > guest boots with kvm-hint-dedicated=on, --overcommit cpu-pm=on, -smp
> > 180,sockets=2,cores=45,threads=2, l3-cache=on qemu command-line, the
> > pCPU which vCPU is running on is isolated. Both the guest and host
> > kernel are 5.8 Linus' tree. (Note, if you fails to boot with
> > --overcommit cpu-pm=on, you can comments out commit e72436bc3a52, I
> > have a report here, https://lkml.org/lkml/2020/7/8/308)
> >
> > IPI microbenchmark(https://lkml.org/lkml/2017/12/19/141, Destination
> > Shorthand is All excluding self)
> >
> > avic0_xapic:   12313907508.50 ns
> > avic1_xapic:   19106424733.30 ns
> > avic0_x2apic: 13073988486.00 ns
>
> I think it depends on the microarchitecture implementation of AVIC?
>
> Paolo
>