* [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
@ 2013-10-04 8:31 Kashyap Chamarthy
2013-10-04 9:33 ` Kashyap Chamarthy
0 siblings, 1 reply; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-04 8:31 UTC (permalink / raw)
To: kvm; +Cc: Gleb Natapov
I have EPT enabled on L0, disabled on L1, and when attempted to boot
an L2 guest, it results in cursor just blinking on the serial console.
(All of them are Fedora 19 minimal guests.)
To be clear, by "disable" I mean, the below KVM parameter is N
$ cat /sys/module/kvm_intel/parameters/ept
N
I don't see any boot messages on serial console either. Running
$ virsh start nguest-01 --console
results in cursor just waiting there and hung.
I thought I'll see what's going on in L2 logs by mounting the disk
image read-only via guestfish
$ guestfish --ro -d nguest-01 -i
This is hung too.
NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
can see all the boot messages.
Any hints on what might be going on here?
Details about test environment below.
Setup info:
----------------
L0 (Intel Haswell):
- 4 pCPU 16G pMEM.
- Version (same for both L0 and L1):
$ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
3.12.0-0.rc3.git1.2.fc21.x86_64
qemu-kvm-1.4.2-7.fc19.x86_64
libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
libguestfs-1.22.5-1.fc19.x86_64
KVM Parameters in L0:
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
Y
$ cat /sys/module/kvm_intel/parameters/ept
Y
KVM Parameters in L1 (Note - EPT is disabled below):
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
N
$ cat /sys/module/kvm_intel/parameters/ept
N
L1 qemu-kvm command line:
----------------------------------------
$ ps -ef | grep qemu
qemu 4329 1 30 03:35 ? 00:13:47
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
-machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
4,sockets=4,cores=1,threads=1 -uuid
4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -boot c -usb -drive
file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
L2 qemu-kvm command line:
----------------------------------------
$ ps -ef | grep qemu
qemu 1174 1 99 04:13 ? 00:06:12
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
-machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
2,sockets=2,cores=1,threads=1 -uuid
b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
/kashyap
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-04 8:31 [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot Kashyap Chamarthy
@ 2013-10-04 9:33 ` Kashyap Chamarthy
2013-10-04 9:39 ` Gleb Natapov
0 siblings, 1 reply; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-04 9:33 UTC (permalink / raw)
To: kvm; +Cc: Gleb Natapov
Discussed with Gleb on IRC: I'll have to try with kvm.git and a simple
qemu-kvm command line.
Thanks, Gleb.
On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> I have EPT enabled on L0, disabled on L1, and when attempted to boot
> an L2 guest, it results in cursor just blinking on the serial console.
> (All of them are Fedora 19 minimal guests.)
>
> To be clear, by "disable" I mean, the below KVM parameter is N
>
> $ cat /sys/module/kvm_intel/parameters/ept
> N
>
> I don't see any boot messages on serial console either. Running
>
> $ virsh start nguest-01 --console
>
> results in cursor just waiting there and hung.
>
> I thought I'll see what's going on in L2 logs by mounting the disk
> image read-only via guestfish
>
> $ guestfish --ro -d nguest-01 -i
>
> This is hung too.
>
> NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
> can see all the boot messages.
>
> Any hints on what might be going on here?
>
> Details about test environment below.
>
> Setup info:
> ----------------
>
> L0 (Intel Haswell):
> - 4 pCPU 16G pMEM.
>
> - Version (same for both L0 and L1):
> $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
> 3.12.0-0.rc3.git1.2.fc21.x86_64
> qemu-kvm-1.4.2-7.fc19.x86_64
> libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
> libguestfs-1.22.5-1.fc19.x86_64
>
> KVM Parameters in L0:
>
> $ cat /sys/module/kvm_intel/parameters/nested
> Y
> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> Y
> $ cat /sys/module/kvm_intel/parameters/ept
> Y
>
>
> KVM Parameters in L1 (Note - EPT is disabled below):
>
> $ cat /sys/module/kvm_intel/parameters/nested
> Y
> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> N
> $ cat /sys/module/kvm_intel/parameters/ept
> N
>
>
> L1 qemu-kvm command line:
> ----------------------------------------
>
> $ ps -ef | grep qemu
> qemu 4329 1 30 03:35 ? 00:13:47
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
> -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
> 4,sockets=4,cores=1,threads=1 -uuid
> 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
> -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -boot c -usb -drive
> file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>
>
> L2 qemu-kvm command line:
> ----------------------------------------
> $ ps -ef | grep qemu
> qemu 1174 1 99 04:13 ? 00:06:12
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
> -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
> 2,sockets=2,cores=1,threads=1 -uuid
> b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
> -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>
>
> /kashyap
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-04 9:33 ` Kashyap Chamarthy
@ 2013-10-04 9:39 ` Gleb Natapov
2013-10-04 12:33 ` Kashyap Chamarthy
0 siblings, 1 reply; 15+ messages in thread
From: Gleb Natapov @ 2013-10-04 9:39 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm
On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
> Discussed with Gleb on IRC: I'll have to try with kvm.git and a simple
> qemu-kvm command line.
>
And try to disable shadow vmcs on the host just to rule it out.
> Thanks, Gleb.
>
> On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> > I have EPT enabled on L0, disabled on L1, and when attempted to boot
> > an L2 guest, it results in cursor just blinking on the serial console.
> > (All of them are Fedora 19 minimal guests.)
> >
> > To be clear, by "disable" I mean, the below KVM parameter is N
> >
> > $ cat /sys/module/kvm_intel/parameters/ept
> > N
> >
> > I don't see any boot messages on serial console either. Running
> >
> > $ virsh start nguest-01 --console
> >
> > results in cursor just waiting there and hung.
> >
> > I thought I'll see what's going on in L2 logs by mounting the disk
> > image read-only via guestfish
> >
> > $ guestfish --ro -d nguest-01 -i
> >
> > This is hung too.
> >
> > NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
> > can see all the boot messages.
> >
> > Any hints on what might be going on here?
> >
> > Details about test environment below.
> >
> > Setup info:
> > ----------------
> >
> > L0 (Intel Haswell):
> > - 4 pCPU 16G pMEM.
> >
> > - Version (same for both L0 and L1):
> > $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
> > 3.12.0-0.rc3.git1.2.fc21.x86_64
> > qemu-kvm-1.4.2-7.fc19.x86_64
> > libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
> > libguestfs-1.22.5-1.fc19.x86_64
> >
> > KVM Parameters in L0:
> >
> > $ cat /sys/module/kvm_intel/parameters/nested
> > Y
> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> > Y
> > $ cat /sys/module/kvm_intel/parameters/ept
> > Y
> >
> >
> > KVM Parameters in L1 (Note - EPT is disabled below):
> >
> > $ cat /sys/module/kvm_intel/parameters/nested
> > Y
> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> > N
> > $ cat /sys/module/kvm_intel/parameters/ept
> > N
> >
> >
> > L1 qemu-kvm command line:
> > ----------------------------------------
> >
> > $ ps -ef | grep qemu
> > qemu 4329 1 30 03:35 ? 00:13:47
> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
> > -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
> > 4,sockets=4,cores=1,threads=1 -uuid
> > 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
> > -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> > -no-shutdown -boot c -usb -drive
> > file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> > -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
> > -chardev pty,id=charserial0 -device
> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >
> >
> > L2 qemu-kvm command line:
> > ----------------------------------------
> > $ ps -ef | grep qemu
> > qemu 1174 1 99 04:13 ? 00:06:12
> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
> > -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
> > 2,sockets=2,cores=1,threads=1 -uuid
> > b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
> > -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
> > -chardev pty,id=charserial0 -device
> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >
> >
> > /kashyap
--
Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-04 9:39 ` Gleb Natapov
@ 2013-10-04 12:33 ` Kashyap Chamarthy
2013-10-04 13:05 ` Gleb Natapov
0 siblings, 1 reply; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-04 12:33 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
>> Discussed with Gleb on IRC: I'll have to try with kvm.git
I just compiled kernel from kvm.git queue on both L0 and L1.
For completeness' sake, this how I did:
$ git describe
for-linus-12017-g8a3c1a33
$ git log | head -1
commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
$ git branch -d test_ept1 origin/queue
$ make -j6 && make bzImage && make modules
$ sudo make modules_install && make install
(Rebooted both L0 and L1 into the new kernel)
> and a simple
>> qemu-kvm command line.
>>
> And try to disable shadow vmcs on the host just to rule it out.
$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
N
The above didn't have any effect.
No, booting L2 is still stuck.
Not sure what's broken (if at all) here. Gleb, you mentioned you were
able to test the MMU combination 'shadow on EPT' just fine with an F19
liveCD without libvirt in-place. If you have still have that test
environment, can you please post your qemu-kvm command line?
Thanks.
>
>> Thanks, Gleb.
>>
>> On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>> > I have EPT enabled on L0, disabled on L1, and when attempted to boot
>> > an L2 guest, it results in cursor just blinking on the serial console.
>> > (All of them are Fedora 19 minimal guests.)
>> >
>> > To be clear, by "disable" I mean, the below KVM parameter is N
>> >
>> > $ cat /sys/module/kvm_intel/parameters/ept
>> > N
>> >
>> > I don't see any boot messages on serial console either. Running
>> >
>> > $ virsh start nguest-01 --console
>> >
>> > results in cursor just waiting there and hung.
>> >
>> > I thought I'll see what's going on in L2 logs by mounting the disk
>> > image read-only via guestfish
>> >
>> > $ guestfish --ro -d nguest-01 -i
>> >
>> > This is hung too.
>> >
>> > NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
>> > can see all the boot messages.
>> >
>> > Any hints on what might be going on here?
>> >
>> > Details about test environment below.
>> >
>> > Setup info:
>> > ----------------
>> >
>> > L0 (Intel Haswell):
>> > - 4 pCPU 16G pMEM.
>> >
>> > - Version (same for both L0 and L1):
>> > $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
>> > 3.12.0-0.rc3.git1.2.fc21.x86_64
>> > qemu-kvm-1.4.2-7.fc19.x86_64
>> > libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
>> > libguestfs-1.22.5-1.fc19.x86_64
>> >
>> > KVM Parameters in L0:
>> >
>> > $ cat /sys/module/kvm_intel/parameters/nested
>> > Y
>> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> > Y
>> > $ cat /sys/module/kvm_intel/parameters/ept
>> > Y
>> >
>> >
>> > KVM Parameters in L1 (Note - EPT is disabled below):
>> >
>> > $ cat /sys/module/kvm_intel/parameters/nested
>> > Y
>> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> > N
>> > $ cat /sys/module/kvm_intel/parameters/ept
>> > N
>> >
>> >
>> > L1 qemu-kvm command line:
>> > ----------------------------------------
>> >
>> > $ ps -ef | grep qemu
>> > qemu 4329 1 30 03:35 ? 00:13:47
>> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
>> > -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
>> > 4,sockets=4,cores=1,threads=1 -uuid
>> > 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
>> > -nodefaults -chardev
>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>> > -no-shutdown -boot c -usb -drive
>> > file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
>> > -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
>> > -chardev pty,id=charserial0 -device
>> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
>> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> >
>> >
>> > L2 qemu-kvm command line:
>> > ----------------------------------------
>> > $ ps -ef | grep qemu
>> > qemu 1174 1 99 04:13 ? 00:06:12
>> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
>> > -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
>> > 2,sockets=2,cores=1,threads=1 -uuid
>> > b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
>> > -nodefaults -chardev
>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>> > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>> > -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
>> > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
>> > -chardev pty,id=charserial0 -device
>> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
>> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> >
>> >
>> > /kashyap
>
> --
> Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-04 12:33 ` Kashyap Chamarthy
@ 2013-10-04 13:05 ` Gleb Natapov
2013-10-04 13:08 ` Gleb Natapov
0 siblings, 1 reply; 15+ messages in thread
From: Gleb Natapov @ 2013-10-04 13:05 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm
On Fri, Oct 04, 2013 at 06:03:33PM +0530, Kashyap Chamarthy wrote:
> On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
> >> Discussed with Gleb on IRC: I'll have to try with kvm.git
>
> I just compiled kernel from kvm.git queue on both L0 and L1.
>
> For completeness' sake, this how I did:
>
> $ git describe
> for-linus-12017-g8a3c1a33
> $ git log | head -1
> commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
>
> $ git branch -d test_ept1 origin/queue
> $ make -j6 && make bzImage && make modules
> $ sudo make modules_install && make install
>
> (Rebooted both L0 and L1 into the new kernel)
>
> > and a simple
> >> qemu-kvm command line.
> >>
> > And try to disable shadow vmcs on the host just to rule it out.
>
> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> N
>
> The above didn't have any effect.
>
> No, booting L2 is still stuck.
>
> Not sure what's broken (if at all) here. Gleb, you mentioned you were
> able to test the MMU combination 'shadow on EPT' just fine with an F19
> liveCD without libvirt in-place. If you have still have that test
> environment, can you please post your qemu-kvm command line?
>
Something as basic as "qemu -m 2G -cdrom /path/to/image -cpu host
> Thanks.
>
> >
> >> Thanks, Gleb.
> >>
> >> On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> >> > I have EPT enabled on L0, disabled on L1, and when attempted to boot
> >> > an L2 guest, it results in cursor just blinking on the serial console.
> >> > (All of them are Fedora 19 minimal guests.)
> >> >
> >> > To be clear, by "disable" I mean, the below KVM parameter is N
> >> >
> >> > $ cat /sys/module/kvm_intel/parameters/ept
> >> > N
> >> >
> >> > I don't see any boot messages on serial console either. Running
> >> >
> >> > $ virsh start nguest-01 --console
> >> >
> >> > results in cursor just waiting there and hung.
> >> >
> >> > I thought I'll see what's going on in L2 logs by mounting the disk
> >> > image read-only via guestfish
> >> >
> >> > $ guestfish --ro -d nguest-01 -i
> >> >
> >> > This is hung too.
> >> >
> >> > NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
> >> > can see all the boot messages.
> >> >
> >> > Any hints on what might be going on here?
> >> >
> >> > Details about test environment below.
> >> >
> >> > Setup info:
> >> > ----------------
> >> >
> >> > L0 (Intel Haswell):
> >> > - 4 pCPU 16G pMEM.
> >> >
> >> > - Version (same for both L0 and L1):
> >> > $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
> >> > 3.12.0-0.rc3.git1.2.fc21.x86_64
> >> > qemu-kvm-1.4.2-7.fc19.x86_64
> >> > libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
> >> > libguestfs-1.22.5-1.fc19.x86_64
> >> >
> >> > KVM Parameters in L0:
> >> >
> >> > $ cat /sys/module/kvm_intel/parameters/nested
> >> > Y
> >> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >> > Y
> >> > $ cat /sys/module/kvm_intel/parameters/ept
> >> > Y
> >> >
> >> >
> >> > KVM Parameters in L1 (Note - EPT is disabled below):
> >> >
> >> > $ cat /sys/module/kvm_intel/parameters/nested
> >> > Y
> >> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >> > N
> >> > $ cat /sys/module/kvm_intel/parameters/ept
> >> > N
> >> >
> >> >
> >> > L1 qemu-kvm command line:
> >> > ----------------------------------------
> >> >
> >> > $ ps -ef | grep qemu
> >> > qemu 4329 1 30 03:35 ? 00:13:47
> >> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
> >> > -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
> >> > 4,sockets=4,cores=1,threads=1 -uuid
> >> > 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
> >> > -nodefaults -chardev
> >> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
> >> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> >> > -no-shutdown -boot c -usb -drive
> >> > file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> >> > -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> >> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> >> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
> >> > -chardev pty,id=charserial0 -device
> >> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> >> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >> >
> >> >
> >> > L2 qemu-kvm command line:
> >> > ----------------------------------------
> >> > $ ps -ef | grep qemu
> >> > qemu 1174 1 99 04:13 ? 00:06:12
> >> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
> >> > -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
> >> > 2,sockets=2,cores=1,threads=1 -uuid
> >> > b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
> >> > -nodefaults -chardev
> >> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
> >> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> >> > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> >> > -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> >> > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> >> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> >> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
> >> > -chardev pty,id=charserial0 -device
> >> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> >> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >> >
> >> >
> >> > /kashyap
> >
> > --
> > Gleb.
--
Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-04 13:05 ` Gleb Natapov
@ 2013-10-04 13:08 ` Gleb Natapov
2013-10-04 15:28 ` Kashyap Chamarthy
0 siblings, 1 reply; 15+ messages in thread
From: Gleb Natapov @ 2013-10-04 13:08 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm
On Fri, Oct 04, 2013 at 04:05:50PM +0300, Gleb Natapov wrote:
> On Fri, Oct 04, 2013 at 06:03:33PM +0530, Kashyap Chamarthy wrote:
> > On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > > On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
> > >> Discussed with Gleb on IRC: I'll have to try with kvm.git
> >
> > I just compiled kernel from kvm.git queue on both L0 and L1.
> >
> > For completeness' sake, this how I did:
> >
> > $ git describe
> > for-linus-12017-g8a3c1a33
> > $ git log | head -1
> > commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
> >
> > $ git branch -d test_ept1 origin/queue
> > $ make -j6 && make bzImage && make modules
> > $ sudo make modules_install && make install
> >
> > (Rebooted both L0 and L1 into the new kernel)
> >
> > > and a simple
> > >> qemu-kvm command line.
> > >>
> > > And try to disable shadow vmcs on the host just to rule it out.
> >
> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> > N
> >
> > The above didn't have any effect.
> >
> > No, booting L2 is still stuck.
> >
> > Not sure what's broken (if at all) here. Gleb, you mentioned you were
> > able to test the MMU combination 'shadow on EPT' just fine with an F19
> > liveCD without libvirt in-place. If you have still have that test
> > environment, can you please post your qemu-kvm command line?
> >
> Something as basic as "qemu -m 2G -cdrom /path/to/image -cpu host
>
And Live CD I use is XFCE based, not sure if it matters.
--
Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-04 13:08 ` Gleb Natapov
@ 2013-10-04 15:28 ` Kashyap Chamarthy
2013-10-07 12:59 ` Kashyap Chamarthy
0 siblings, 1 reply; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-04 15:28 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Fri, Oct 4, 2013 at 6:38 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Fri, Oct 04, 2013 at 04:05:50PM +0300, Gleb Natapov wrote:
>> On Fri, Oct 04, 2013 at 06:03:33PM +0530, Kashyap Chamarthy wrote:
>> > On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> > > On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
>> > >> Discussed with Gleb on IRC: I'll have to try with kvm.git
>> >
>> > I just compiled kernel from kvm.git queue on both L0 and L1.
>> >
>> > For completeness' sake, this how I did:
>> >
>> > $ git describe
>> > for-linus-12017-g8a3c1a33
>> > $ git log | head -1
>> > commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
>> >
>> > $ git branch -d test_ept1 origin/queue
>> > $ make -j6 && make bzImage && make modules
>> > $ sudo make modules_install && make install
>> >
>> > (Rebooted both L0 and L1 into the new kernel)
>> >
>> > > and a simple
>> > >> qemu-kvm command line.
>> > >>
>> > > And try to disable shadow vmcs on the host just to rule it out.
>> >
>> > $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> > N
>> >
>> > The above didn't have any effect.
>> >
>> > No, booting L2 is still stuck.
>> >
>> > Not sure what's broken (if at all) here. Gleb, you mentioned you were
>> > able to test the MMU combination 'shadow on EPT' just fine with an F19
>> > liveCD without libvirt in-place. If you have still have that test
>> > environment, can you please post your qemu-kvm command line?
>> >
>> Something as basic as "qemu -m 2G -cdrom /path/to/image -cpu host
Thanks. I'll see if I can figure further what's going on here,
otherwise I'll just write a bug for now with all the details, just so
it's tracked. If it turns out to be a false alarm, I can just close
it.
>>
> And Live CD I use is XFCE based, not sure if it matters.
I'm using no X. So, it shouldn't matter :-)
>
> --
> Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-04 15:28 ` Kashyap Chamarthy
@ 2013-10-07 12:59 ` Kashyap Chamarthy
2013-10-07 15:18 ` Gleb Natapov
2013-10-08 5:38 ` Kashyap Chamarthy
0 siblings, 2 replies; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-07 12:59 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
stuck with shadow on EPT
Ensure, EPT is enabled on L0 & disabled on L1
On L0:
---------
$ cat /sys/module/kvm_intel/parameters/ept
Y
On L1
---------
$ cat /sys/module/kvm_intel/parameters/ept
N
Build and install trace-cmd on L1:
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
trace-cmd
$ make && sudo make-install
Run trace command (on L1) to capture kvmmmu event:
$ trace-cmd record -b 2000 -e kvmmmu
/sys/kernel/debug/tracing/events/kvmmmu/filter
/sys/kernel/debug/tracing/events/*/kvmmmu/filter
Hit Ctrl^C to stop recording
Boot L2 guest:
$ virsh start nguest-01 --console
After a minute or two, ctl-C the trace-cmd, generate the report:
$ trace-cmd report
version = 6
trace-cmd: No such file or directory
function is_writable_pte not defined
CPU 1 is empty
CPU 2 is empty
CPU 3 is empty
cpus=4
qemu-system-x86-2554 [000] 90667.995345:
kvm_mmu_invalidate_zap_all_pages: kvm-mmu-valid-gen 0 used_pages 0
qemu-system-x86-2557 [000] 90668.078978: kvm_mmu_get_page: new
sp gfn 0 0/2 q0 direct --- !pge !nxe root 0 sync
qemu-system-x86-2557 [000] 90668.078980: kvm_mmu_get_page: new
sp gfn 40000 0/2 q0 direct --- !pge !nxe root 0 sync
qemu-system-x86-2557 [000] 90668.078980: kvm_mmu_get_page: new
sp gfn 80000 0/2 q0 direct --- !pge !nxe root 0 sync
qemu-system-x86-2557 [000] 90668.078981: kvm_mmu_get_page: new
sp gfn c0000 0/2 q0 direct --- !pge !nxe root 0 sync
qemu-system-x86-2557 [000] 90668.079025: kvm_mmu_get_page: new
sp gfn 0 0/1 q0 direct --- !pge !nxe root 0 sync
Side question: I wonder why would trace-cmd say "trace-cmd: No such
file or directory"?
/kashyap
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-07 12:59 ` Kashyap Chamarthy
@ 2013-10-07 15:18 ` Gleb Natapov
2013-10-08 5:38 ` Kashyap Chamarthy
1 sibling, 0 replies; 15+ messages in thread
From: Gleb Natapov @ 2013-10-07 15:18 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm
On Mon, Oct 07, 2013 at 06:29:30PM +0530, Kashyap Chamarthy wrote:
> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
> stuck with shadow on EPT
>
> Ensure, EPT is enabled on L0 & disabled on L1
>
> On L0:
> ---------
> $ cat /sys/module/kvm_intel/parameters/ept
> Y
>
> On L1
> ---------
> $ cat /sys/module/kvm_intel/parameters/ept
> N
>
> Build and install trace-cmd on L1:
>
> $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
> trace-cmd
> $ make && sudo make-install
>
> Run trace command (on L1) to capture kvmmmu event:
>
> $ trace-cmd record -b 2000 -e kvmmmu
You need to trace kvm events too and make buffer bigger
trace-cmd record -b 100000 -e kvmmmu -e kvm.
> /sys/kernel/debug/tracing/events/kvmmmu/filter
> /sys/kernel/debug/tracing/events/*/kvmmmu/filter
> Hit Ctrl^C to stop recording
>
> Boot L2 guest:
>
> $ virsh start nguest-01 --console
>
> After a minute or two, ctl-C the trace-cmd, generate the report:
>
> $ trace-cmd report
> version = 6
> trace-cmd: No such file or directory
> function is_writable_pte not defined
> CPU 1 is empty
> CPU 2 is empty
> CPU 3 is empty
> cpus=4
> qemu-system-x86-2554 [000] 90667.995345:
> kvm_mmu_invalidate_zap_all_pages: kvm-mmu-valid-gen 0 used_pages 0
> qemu-system-x86-2557 [000] 90668.078978: kvm_mmu_get_page: new
> sp gfn 0 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.078980: kvm_mmu_get_page: new
> sp gfn 40000 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.078980: kvm_mmu_get_page: new
> sp gfn 80000 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.078981: kvm_mmu_get_page: new
> sp gfn c0000 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.079025: kvm_mmu_get_page: new
> sp gfn 0 0/1 q0 direct --- !pge !nxe root 0 sync
Is this all?
>
> Side question: I wonder why would trace-cmd say "trace-cmd: No such
> file or directory"?
>
Ignore it.
--
Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-07 12:59 ` Kashyap Chamarthy
2013-10-07 15:18 ` Gleb Natapov
@ 2013-10-08 5:38 ` Kashyap Chamarthy
2013-10-08 13:03 ` Paolo Bonzini
1 sibling, 1 reply; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-08 5:38 UTC (permalink / raw)
To: Gleb Natapov, Paolo Bonzini; +Cc: kvm
On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
> stuck with shadow on EPT
Paolo, were you able to reproduce this again? Yesterday, on #qemu you
mentioned you'll test it again :-)
I was using kvm.git queue on both L0 and L1:
$ git describe ; git log | head -1
for-linus-11227-g8a3c1a33
commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
/kashyap
>
> Ensure, EPT is enabled on L0 & disabled on L1
>
> On L0:
> ---------
> $ cat /sys/module/kvm_intel/parameters/ept
> Y
>
> On L1
> ---------
> $ cat /sys/module/kvm_intel/parameters/ept
> N
>
> Build and install trace-cmd on L1:
>
> $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
> trace-cmd
> $ make && sudo make-install
>
> Run trace command (on L1) to capture kvmmmu event:
>
> $ trace-cmd record -b 2000 -e kvmmmu
> /sys/kernel/debug/tracing/events/kvmmmu/filter
> /sys/kernel/debug/tracing/events/*/kvmmmu/filter
> Hit Ctrl^C to stop recording
>
> Boot L2 guest:
>
> $ virsh start nguest-01 --console
>
> After a minute or two, ctl-C the trace-cmd, generate the report:
>
> $ trace-cmd report
> version = 6
> trace-cmd: No such file or directory
> function is_writable_pte not defined
> CPU 1 is empty
> CPU 2 is empty
> CPU 3 is empty
> cpus=4
> qemu-system-x86-2554 [000] 90667.995345:
> kvm_mmu_invalidate_zap_all_pages: kvm-mmu-valid-gen 0 used_pages 0
> qemu-system-x86-2557 [000] 90668.078978: kvm_mmu_get_page: new
> sp gfn 0 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.078980: kvm_mmu_get_page: new
> sp gfn 40000 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.078980: kvm_mmu_get_page: new
> sp gfn 80000 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.078981: kvm_mmu_get_page: new
> sp gfn c0000 0/2 q0 direct --- !pge !nxe root 0 sync
> qemu-system-x86-2557 [000] 90668.079025: kvm_mmu_get_page: new
> sp gfn 0 0/1 q0 direct --- !pge !nxe root 0 sync
>
> Side question: I wonder why would trace-cmd say "trace-cmd: No such
> file or directory"?
>
> /kashyap
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-08 5:38 ` Kashyap Chamarthy
@ 2013-10-08 13:03 ` Paolo Bonzini
2013-10-08 14:48 ` Paolo Bonzini
0 siblings, 1 reply; 15+ messages in thread
From: Paolo Bonzini @ 2013-10-08 13:03 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: Gleb Natapov, kvm
Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>> stuck with shadow on EPT
>
> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
> mentioned you'll test it again :-)
Yes, I could reproduce it too.
>> Boot L2 guest:
Here L2 doesn't go past the second instruction. It gets a page fault
even though the spte is present, and KVM then loops on a page fault
for 0xfe05b.
Here is an annotated function_graph trace of L1.
It's possible that L0 is injecting the same fault repeatedly, i.e.
they are not different faults from the processor. I'll get an L0
trace next.
---- KVM executes at 0xfffffff0 via emulation
kvm_cpu_has_pending_timer [kvm]() {
apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]();
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]() {
}
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
vmx_interrupt_allowed [kvm_intel]();
x86_emulate_instruction [kvm]() {
init_emulate_ctxt [kvm]() {
vmx_get_cs_db_l_bits [kvm_intel]() {
vmx_read_guest_seg_ar [kvm_intel]();
}
vmx_get_rflags [kvm_intel]();
}
x86_decode_insn [kvm]() {
do_insn_fetch [kvm]() {
...
kvm_fetch_guest_virt [kvm]() {
vmx_get_cpl [kvm_intel]();
kvm_read_guest_virt_helper [kvm]() {
nonpaging_gva_to_gpa [kvm]();
kvm_read_guest [kvm]() {
kvm_read_guest_page [kvm]() {
gfn_to_hva_prot [kvm]() {
__gfn_to_hva_many [kvm]();
}
}
}
} /* kvm_read_guest_virt_helper [kvm] */
}
}
...
}
x86_emulate_insn [kvm]() {
/* kvm_emulate_insn: ffff0000:fff0:ea 5b e0 00 f0 (real) */
em_jmp_far [kvm]() {
load_segment_descriptor [kvm]() {
emulator_get_segment [kvm]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
}
emulator_set_segment [kvm]() {
vmx_set_segment [kvm_intel]() {
fix_rmode_seg [kvm_intel]() {
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
vmcs_writel [kvm_intel]();
}
emulation_required [kvm_intel]() {
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
} /* rmode_segment_valid [kvm_intel] */
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
}
}
}
}
}
writeback [kvm]();
writeback_registers [kvm]();
}
vmx_get_interrupt_shadow [kvm_intel]();
vmx_set_interrupt_shadow [kvm_intel]();
vmx_get_rflags [kvm_intel]();
kvm_set_rflags [kvm]() {
vmx_set_rflags [kvm_intel]() {
vmcs_writel [kvm_intel]();
}
}
}
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
emulation_required [kvm_intel]() {
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
}
}
---- KVM executes at f000:e05b and gets a page fault
kvm_cpu_has_pending_timer [kvm]() {
apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
kvm_apic_accept_events [kvm]();
kvm_cpu_has_injectable_intr [kvm]() {
kvm_apic_accept_pic_intr [kvm]();
}
vmx_interrupt_allowed [kvm_intel]();
kvm_cpu_has_injectable_intr [kvm]() {
kvm_apic_accept_pic_intr [kvm]();
}
enable_irq_window [kvm_intel]() {
vmcs_writel [kvm_intel]();
}
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]() {
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
perf_guest_get_msrs();
/* kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e */
}
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]();
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
handle_exception [kvm_intel]() {
/* kvm_page_fault: address fe05b error_code 14 */
kvm_mmu_page_fault [kvm]() {
nonpaging_page_fault [kvm]() {
mmu_topup_memory_caches [kvm]();
gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
gfn_to_memslot [kvm]();
}
mapping_level.isra.86 [kvm]() {
kvm_host_page_size [kvm]() {
gfn_to_hva [kvm]() {
__gfn_to_hva_many [kvm]();
}
down_read() {
_cond_resched();
}
find_vma();
vma_kernel_pagesize();
up_read();
}
}
try_async_pf [kvm]() {
gfn_to_pfn_async [kvm]() {
__gfn_to_pfn [kvm]() {
__gfn_to_pfn_memslot [kvm]() {
__gfn_to_hva_many [kvm]();
__get_user_pages_fast() {
gup_pud_range() {
gup_pte_range();
}
}
}
}
}
}
handle_abnormal_pfn [kvm]();
_raw_spin_lock();
make_mmu_pages_available.isra.80 [kvm]();
transparent_hugepage_adjust.isra.91 [kvm]() {
kvm_is_mmio_pfn [kvm]();
}
__direct_map.isra.104 [kvm]() {
shadow_walk_init [kvm]();
kvm_mmu_get_page [kvm]() {
pte_list_add [kvm]();
/* kvm_mmu_get_page: sp gen 2 gfn 0 1 q0 direct wux !nxe root 0 sync new */
}
link_shadow_page.isra.63 [kvm]() {
mmu_spte_set [kvm]();
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
__direct_pte_prefetch [kvm]() {
kvm_mmu_page_get_gfn [kvm]();
gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
gfn_to_memslot [kvm]();
}
gfn_to_page_many_atomic [kvm]() {
__gfn_to_hva_many [kvm]();
__get_user_pages_fast() {
gup_pud_range() {
gup_pte_range();
}
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
}
}
}
}
}
}
---- up to this point the trace is identical for shadow-on-shadow (working)
---- and shadow-on-EPT (not working)
----
---- for shadow-on-EPT L1 gets another page fault even though the spte is
---- present. The error code is identical.
kvm_cpu_has_pending_timer [kvm]() {
apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]() {
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
perf_guest_get_msrs();
/* kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e */
}
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]();
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
handle_exception [kvm_intel]() {
/* kvm_page_fault: address fe05b error_code 14 */
kvm_mmu_page_fault [kvm]() {
nonpaging_page_fault [kvm]() {
mmu_topup_memory_caches [kvm]();
gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
gfn_to_memslot [kvm]();
}
mapping_level.isra.86 [kvm]() {
kvm_host_page_size [kvm]() {
gfn_to_hva [kvm]() {
__gfn_to_hva_many [kvm]();
}
down_read() {
_cond_resched();
}
find_vma();
vma_kernel_pagesize();
up_read();
}
}
try_async_pf [kvm]() {
gfn_to_pfn_async [kvm]() {
__gfn_to_pfn [kvm]() {
__gfn_to_pfn_memslot [kvm]() {
__gfn_to_hva_many [kvm]();
__get_user_pages_fast() {
gup_pud_range() {
gup_pte_range();
}
}
}
}
}
}
handle_abnormal_pfn [kvm]();
_raw_spin_lock();
make_mmu_pages_available.isra.80 [kvm]();
transparent_hugepage_adjust.isra.91 [kvm]() {
kvm_is_mmio_pfn [kvm]();
}
__direct_map.isra.104 [kvm]() {
shadow_walk_init [kvm]();
>>>>>> here starts the difference with the second part,
>>>>>> this shows that the spte is present
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
spte_has_volatile_bits [kvm]();
}
}
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
__direct_pte_prefetch [kvm]();
}
}
}
}
}
---- this part of the trace then loops endlessly
Paolo
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-08 13:03 ` Paolo Bonzini
@ 2013-10-08 14:48 ` Paolo Bonzini
2013-10-09 6:22 ` Kashyap Chamarthy
0 siblings, 1 reply; 15+ messages in thread
From: Paolo Bonzini @ 2013-10-08 14:48 UTC (permalink / raw)
Cc: Kashyap Chamarthy, Gleb Natapov, kvm
Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>>> stuck with shadow on EPT
>>
>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
>> mentioned you'll test it again :-)
>
> Yes, I could reproduce it too.
>
>>> Boot L2 guest:
>
> Here L2 doesn't go past the second instruction. It gets a page fault
> even though the spte is present, and KVM then loops on a page fault
> for 0xfe05b.
>
> Here is an annotated function_graph trace of L1.
>
> It's possible that L0 is injecting the same fault repeatedly, i.e.
> they are not different faults from the processor. I'll get an L0
> trace next.
>
The L0 trace is not particularly helpful (and probably would not be
particularly helpful even if there were a specific tracepoint for
VMREAD):
287.534156: kvm_exit: reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
287.534160: kvm_mmu_get_page: sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
287.534161: kvm_entry: vcpu 0
287.534162: kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e
287.534170: kvm_mmu_get_page: sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
287.534171: kvm_entry: vcpu 0
287.534172: kvm_exit: reason VMREAD rip 0xffffffffa021f97d info 0 0
287.534173: kvm_entry: vcpu 0
287.534174: kvm_exit: reason VMREAD rip 0xffffffffa021f996 info 0 0
287.534174: kvm_entry: vcpu 0
287.534175: kvm_exit: reason VMREAD rip 0xffffffffa021f9b5 info 0 0
287.534175: kvm_entry: vcpu 0
287.534177: kvm_exit: reason VMREAD rip 0xffffffffa021b377 info 0 0
287.534177: kvm_entry: vcpu 0
287.534178: kvm_exit: reason VMREAD rip 0xffffffffa021b5ce info 0 0
287.534179: kvm_entry: vcpu 0
287.534180: kvm_exit: reason VMREAD rip 0xffffffffa0222c95 info 0 0
287.534180: kvm_entry: vcpu 0
287.534181: kvm_exit: reason VMREAD rip 0xffffffffa0222e1c info 0 0
287.534182: kvm_entry: vcpu 0
287.534185: kvm_exit: reason MSR_READ rip 0xffffffff8104c2b6 info 0 0
287.534185: kvm_msr: msr_read 1d9 = 0x0
287.534185: kvm_entry: vcpu 0
And then it repeats:
287.534186: kvm_exit: reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
287.534191: kvm_mmu_get_page: sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
287.534192: kvm_entry: vcpu 0
Trying to add function_graph loses a lot of events.
Paolo
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-08 14:48 ` Paolo Bonzini
@ 2013-10-09 6:22 ` Kashyap Chamarthy
2013-10-09 8:16 ` Gleb Natapov
0 siblings, 1 reply; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-09 6:22 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Gleb Natapov, kvm
On Tue, Oct 8, 2013 at 8:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
>> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
>>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>>>> stuck with shadow on EPT
>>>
>>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
>>> mentioned you'll test it again :-)
>>
>> Yes, I could reproduce it too.
>>
>>>> Boot L2 guest:
>>
>> Here L2 doesn't go past the second instruction. It gets a page fault
>> even though the spte is present, and KVM then loops on a page fault
>> for 0xfe05b.
>>
>> Here is an annotated function_graph trace of L1.
>>
>> It's possible that L0 is injecting the same fault repeatedly, i.e.
>> they are not different faults from the processor. I'll get an L0
>> trace next.
>>
>
> The L0 trace is not particularly helpful (and probably would not be
> particularly helpful even if there were a specific tracepoint for
> VMREAD):
>
> 287.534156: kvm_exit: reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
> 287.534160: kvm_mmu_get_page: sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
> 287.534161: kvm_entry: vcpu 0
> 287.534162: kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e
> 287.534170: kvm_mmu_get_page: sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
> 287.534171: kvm_entry: vcpu 0
> 287.534172: kvm_exit: reason VMREAD rip 0xffffffffa021f97d info 0 0
> 287.534173: kvm_entry: vcpu 0
> 287.534174: kvm_exit: reason VMREAD rip 0xffffffffa021f996 info 0 0
> 287.534174: kvm_entry: vcpu 0
> 287.534175: kvm_exit: reason VMREAD rip 0xffffffffa021f9b5 info 0 0
> 287.534175: kvm_entry: vcpu 0
> 287.534177: kvm_exit: reason VMREAD rip 0xffffffffa021b377 info 0 0
> 287.534177: kvm_entry: vcpu 0
> 287.534178: kvm_exit: reason VMREAD rip 0xffffffffa021b5ce info 0 0
> 287.534179: kvm_entry: vcpu 0
> 287.534180: kvm_exit: reason VMREAD rip 0xffffffffa0222c95 info 0 0
> 287.534180: kvm_entry: vcpu 0
> 287.534181: kvm_exit: reason VMREAD rip 0xffffffffa0222e1c info 0 0
> 287.534182: kvm_entry: vcpu 0
> 287.534185: kvm_exit: reason MSR_READ rip 0xffffffff8104c2b6 info 0 0
> 287.534185: kvm_msr: msr_read 1d9 = 0x0
> 287.534185: kvm_entry: vcpu 0
>
> And then it repeats:
>
> 287.534186: kvm_exit: reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
> 287.534191: kvm_mmu_get_page: sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
> 287.534192: kvm_entry: vcpu 0
>
> Trying to add function_graph loses a lot of events.
>
> Paolo
Paolo, here's L0's trace with shadow-on-EPT.
I invoked with the below command line on L0
$ /usr/bin/trace-cmd record -b 10000 -e kvmmmu
then, started the L2 guest.
Report:
$ trace-cmd report 2>&1 | less
The trace data file was large (150MB), just pasting the repeating message here:
---------------------
.
.
.
qemu-system-x86-2006 [000] 57931.272466: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
qemu-system-x86-2006 [000] 57931.272468: kvm_mmu_paging_element: pte
1c0f067 level 4
qemu-system-x86-2006 [000] 57931.272469: kvm_mmu_paging_element: pte
1c10063 level 3
qemu-system-x86-2006 [000] 57931.272469: kvm_mmu_paging_element: pte
12001e1 level 2
qemu-system-x86-2006 [000] 57931.272497: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
qemu-system-x86-2006 [000] 57931.272497: kvm_mmu_paging_element: pte
1c0f067 level 4
qemu-system-x86-2006 [000] 57931.272497: kvm_mmu_paging_element: pte
1c10063 level 3
qemu-system-x86-2006 [000] 57931.272497: kvm_mmu_paging_element: pte
12001e1 level 2
qemu-system-x86-2006 [000] 57931.272516: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
qemu-system-x86-2006 [000] 57931.272516: kvm_mmu_paging_element: pte
1c0f067 level 4
qemu-system-x86-2006 [000] 57931.272516: kvm_mmu_paging_element: pte
1c10063 level 3
qemu-system-x86-2006 [000] 57931.272517: kvm_mmu_paging_element: pte
12001e1 level 2
qemu-system-x86-2006 [000] 57931.272525: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
qemu-system-x86-2006 [000] 57931.272525: kvm_mmu_paging_element: pte
1c0f067 level 4
qemu-system-x86-2006 [000] 57931.272525: kvm_mmu_paging_element: pte
1c10063 level 3
qemu-system-x86-2006 [000] 57931.272525: kvm_mmu_paging_element: pte
12001e1 level 2
qemu-system-x86-2006 [000] 57931.272540: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
qemu-system-x86-2006 [000] 57931.272540: kvm_mmu_paging_element: pte
1c0f067 level 4
qemu-system-x86-2006 [000] 57931.272540: kvm_mmu_paging_element: pte
1c10063 level 3
qemu-system-x86-2006 [000] 57931.272541: kvm_mmu_paging_element: pte
12001e1 level 2
qemu-system-x86-2006 [000] 57931.272580: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
qemu-system-x86-2006 [000] 57931.272581: kvm_mmu_paging_element: pte
1c0f067 level 4
qemu-system-x86-2006 [000] 57931.272581: kvm_mmu_paging_element: pte
1c10063 level 3
qemu-system-x86-2006 [000] 57931.272581: kvm_mmu_paging_element: pte
12001e1 level 2
qemu-system-x86-2006 [000] 57931.272593: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
qemu-system-x86-2006 [000] 57931.272594: kvm_mmu_paging_element: pte
1c0f067 level 4
qemu-system-x86-2006 [000] 57931.272594: kvm_mmu_paging_element: pte
1c10063 level 3
qemu-system-x86-2006 [000] 57931.272594: kvm_mmu_paging_element: pte
12001e1 level 2
.
.
.
.
qemu-system-x86-2007 [000] 57967.047692: kvm_mmu_get_page: sp gfn
0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047698: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047701: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047706: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047710: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047724: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047727: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047732: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047736: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
qemu-system-x86-2007 [000] 57967.047741: kvm_mmu_get_page: sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3s
---------------------
/kashyap
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-09 6:22 ` Kashyap Chamarthy
@ 2013-10-09 8:16 ` Gleb Natapov
2013-10-09 17:46 ` Kashyap Chamarthy
0 siblings, 1 reply; 15+ messages in thread
From: Gleb Natapov @ 2013-10-09 8:16 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: Paolo Bonzini, kvm
On Wed, Oct 09, 2013 at 11:52:29AM +0530, Kashyap Chamarthy wrote:
> On Tue, Oct 8, 2013 at 8:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
> >> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
> >>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> >>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
> >>>> stuck with shadow on EPT
> >>>
> >>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
> >>> mentioned you'll test it again :-)
> >>
> >> Yes, I could reproduce it too.
72f857950f6f19cba42a9ded078bbc99f10aa667 causes it and I ran kernel
without this commit. Will look into it.
--
Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
2013-10-09 8:16 ` Gleb Natapov
@ 2013-10-09 17:46 ` Kashyap Chamarthy
0 siblings, 0 replies; 15+ messages in thread
From: Kashyap Chamarthy @ 2013-10-09 17:46 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Paolo Bonzini, kvm
On Wed, Oct 9, 2013 at 1:46 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Wed, Oct 09, 2013 at 11:52:29AM +0530, Kashyap Chamarthy wrote:
>> On Tue, Oct 8, 2013 at 8:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> > Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
>> >> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
>> >>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>> >>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>> >>>> stuck with shadow on EPT
>> >>>
>> >>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
>> >>> mentioned you'll test it again :-)
>> >>
>> >> Yes, I could reproduce it too.
>
> 72f857950f6f19cba42a9ded078bbc99f10aa667 causes it and I ran kernel
> without this commit. Will look into it.
Thank you, this fixed it. I tested by applying your patch from the
other email.[1]
I'm now able to boot L2 guest with shadow-on-EPT with the above fix.
Thank you.
/kashyap
[1] http://www.mail-archive.com/kvm@vger.kernel.org/msg96484.html
>
> --
> Gleb.
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2013-10-09 17:46 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-10-04 8:31 [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot Kashyap Chamarthy
2013-10-04 9:33 ` Kashyap Chamarthy
2013-10-04 9:39 ` Gleb Natapov
2013-10-04 12:33 ` Kashyap Chamarthy
2013-10-04 13:05 ` Gleb Natapov
2013-10-04 13:08 ` Gleb Natapov
2013-10-04 15:28 ` Kashyap Chamarthy
2013-10-07 12:59 ` Kashyap Chamarthy
2013-10-07 15:18 ` Gleb Natapov
2013-10-08 5:38 ` Kashyap Chamarthy
2013-10-08 13:03 ` Paolo Bonzini
2013-10-08 14:48 ` Paolo Bonzini
2013-10-09 6:22 ` Kashyap Chamarthy
2013-10-09 8:16 ` Gleb Natapov
2013-10-09 17:46 ` Kashyap Chamarthy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.