GICv3 stride issue with many VCPUs

GICv3 stride issue with many VCPUs

[Christoffer Dall]

Hi,

I recently ran kvm-unit-tests on TX2, and I got an error from gicv3-ipi,
with this in the log:

  % cat logs/gicv3-ipi.log
  timeout -k 1s --foreground 90s ../src/qemu/build/aarch64-softmmu/qemu-system-aarch64 -nodefaults -machine virt,gic-version=host,accel=kvm -cpu host -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -device pci-testdev -display none -serial stdio -kernel arm/gic.flat -smp 224 -machine gic-version=3 -append ipi # -initrd /tmp/tmp.Xy3T5zjn9n
  lib/arm/gic-v3.c:25: assert failed: 0
          STACK:


Which is an assert in gicv3_set_redist_base().

I suspect this is related to the number of vcpus (112) and our GICv3 configuration.

Reporting it here in case anyone else is seeing this and wondering what's going on.


Thanks,

    Christoffer

Re: GICv3 stride issue with many VCPUs

[Auger Eric]

Hi Christoffer,

On 09/03/2018 01:11 PM, Christoffer Dall wrote:
> Hi,
> 
> I recently ran kvm-unit-tests on TX2, and I got an error from gicv3-ipi,
> with this in the log:
> 
>   % cat logs/gicv3-ipi.log
>   timeout -k 1s --foreground 90s ../src/qemu/build/aarch64-softmmu/qemu-system-aarch64 -nodefaults -machine virt,gic-version=host,accel=kvm -cpu host -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -device pci-testdev -display none -serial stdio -kernel arm/gic.flat -smp 224 -machine gic-version=3 -append ipi # -initrd /tmp/tmp.Xy3T5zjn9n
>   lib/arm/gic-v3.c:25: assert failed: 0
>           STACK:

Quickly looking at gicv3_set_redist_base(), I noted a single contiguous
redistributor region seems to be used: void *ptr =
gicv3_data.redist_base[0];

Supporting more than 123 vcpus requires to use the 2 separate
redistributor regions exposed by lastest QEMU mach-virt. I suspect the
kvm-unit-test is not using both redist regions yet.

Thanks

Eric


> 
> 
> Which is an assert in gicv3_set_redist_base().
> 
> I suspect this is related to the number of vcpus (112) and our GICv3 configuration.
> 
> Reporting it here in case anyone else is seeing this and wondering what's going on.
> 
> 
> Thanks,
> 
>     Christoffer
> 

Re: GICv3 stride issue with many VCPUs

[Andrew Jones]

On Mon, Sep 03, 2018 at 01:39:12PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 09/03/2018 01:11 PM, Christoffer Dall wrote:
> > Hi,
> > 
> > I recently ran kvm-unit-tests on TX2, and I got an error from gicv3-ipi,
> > with this in the log:
> > 
> >   % cat logs/gicv3-ipi.log
> >   timeout -k 1s --foreground 90s ../src/qemu/build/aarch64-softmmu/qemu-system-aarch64 -nodefaults -machine virt,gic-version=host,accel=kvm -cpu host -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -device pci-testdev -display none -serial stdio -kernel arm/gic.flat -smp 224 -machine gic-version=3 -append ipi # -initrd /tmp/tmp.Xy3T5zjn9n
> >   lib/arm/gic-v3.c:25: assert failed: 0
> >           STACK:
> 
> Quickly looking at gicv3_set_redist_base(), I noted a single contiguous
> redistributor region seems to be used: void *ptr =
> gicv3_data.redist_base[0];
> 
> Supporting more than 123 vcpus requires to use the 2 separate
> redistributor regions exposed by lastest QEMU mach-virt. I suspect the
> kvm-unit-test is not using both redist regions yet.

Nope. I didn't have any hardware to test that on yet :-) I can write a
patch, though, and send it to Christoffer to test. Or, if Christoffer
doesn't mind hacking kvm-unit-tests, I'd be happy to review his patch :-)

Thanks,
drew

> 
> Thanks
> 
> Eric
> 
> 
> > 
> > 
> > Which is an assert in gicv3_set_redist_base().
> > 
> > I suspect this is related to the number of vcpus (112) and our GICv3 configuration.
> > 
> > Reporting it here in case anyone else is seeing this and wondering what's going on.
> > 
> > 
> > Thanks,
> > 
> >     Christoffer
> > 

Re: GICv3 stride issue with many VCPUs

[Christoffer Dall]

On Mon, Sep 03, 2018 at 02:31:19PM +0200, Andrew Jones wrote:
> On Mon, Sep 03, 2018 at 01:39:12PM +0200, Auger Eric wrote:
> > Hi Christoffer,
> > 
> > On 09/03/2018 01:11 PM, Christoffer Dall wrote:
> > > Hi,
> > > 
> > > I recently ran kvm-unit-tests on TX2, and I got an error from gicv3-ipi,
> > > with this in the log:
> > > 
> > >   % cat logs/gicv3-ipi.log
> > >   timeout -k 1s --foreground 90s ../src/qemu/build/aarch64-softmmu/qemu-system-aarch64 -nodefaults -machine virt,gic-version=host,accel=kvm -cpu host -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -device pci-testdev -display none -serial stdio -kernel arm/gic.flat -smp 224 -machine gic-version=3 -append ipi # -initrd /tmp/tmp.Xy3T5zjn9n
> > >   lib/arm/gic-v3.c:25: assert failed: 0
> > >           STACK:
> > 
> > Quickly looking at gicv3_set_redist_base(), I noted a single contiguous
> > redistributor region seems to be used: void *ptr =
> > gicv3_data.redist_base[0];
> > 
> > Supporting more than 123 vcpus requires to use the 2 separate
> > redistributor regions exposed by lastest QEMU mach-virt. I suspect the
> > kvm-unit-test is not using both redist regions yet.
> 
> Nope. I didn't have any hardware to test that on yet :-) I can write a
> patch, though, and send it to Christoffer to test. Or, if Christoffer
> doesn't mind hacking kvm-unit-tests, I'd be happy to review his patch :-)
> 

I'm quite caugt up at the moment, so if you could write a patch, I'll be
happy to test.

Thanks,

    Christoffer

Re: GICv3 stride issue with many VCPUs

[Andrew Jones]

On Mon, Sep 03, 2018 at 03:15:07PM +0200, Christoffer Dall wrote:
> On Mon, Sep 03, 2018 at 02:31:19PM +0200, Andrew Jones wrote:
> > On Mon, Sep 03, 2018 at 01:39:12PM +0200, Auger Eric wrote:
> > > Hi Christoffer,
> > > 
> > > On 09/03/2018 01:11 PM, Christoffer Dall wrote:
> > > > Hi,
> > > > 
> > > > I recently ran kvm-unit-tests on TX2, and I got an error from gicv3-ipi,
> > > > with this in the log:
> > > > 
> > > >   % cat logs/gicv3-ipi.log
> > > >   timeout -k 1s --foreground 90s ../src/qemu/build/aarch64-softmmu/qemu-system-aarch64 -nodefaults -machine virt,gic-version=host,accel=kvm -cpu host -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -device pci-testdev -display none -serial stdio -kernel arm/gic.flat -smp 224 -machine gic-version=3 -append ipi # -initrd /tmp/tmp.Xy3T5zjn9n
> > > >   lib/arm/gic-v3.c:25: assert failed: 0
> > > >           STACK:
> > > 
> > > Quickly looking at gicv3_set_redist_base(), I noted a single contiguous
> > > redistributor region seems to be used: void *ptr =
> > > gicv3_data.redist_base[0];
> > > 
> > > Supporting more than 123 vcpus requires to use the 2 separate
> > > redistributor regions exposed by lastest QEMU mach-virt. I suspect the
> > > kvm-unit-test is not using both redist regions yet.
> > 
> > Nope. I didn't have any hardware to test that on yet :-) I can write a
> > patch, though, and send it to Christoffer to test. Or, if Christoffer
> > doesn't mind hacking kvm-unit-tests, I'd be happy to review his patch :-)
> > 
> 
> I'm quite caugt up at the moment, so if you could write a patch, I'll be
> happy to test.
>

No problem. I'll try to pull it together soon.

Thanks,
drew