Re: R/W HG memory mappings with kvm?

[Stephen Donnelly <sfdonnelly@gmail.com>]

On Wed, Jul 29, 2009 at 11:06 AM, Stephen Donnelly<sfdonnelly@gmail.com> wrote:
> On Tue, Jul 28, 2009 at 8:54 PM, Avi Kivity<avi@redhat.com> wrote:
>> On 07/28/2009 12:32 AM, Stephen Donnelly wrote:

>> You need a variant of qemu_ram_alloc() that accepts an fd and offset and
>> mmaps that.

I had a go at this, creating qemu_ram_mmap() using qemu_ram_alloc() as
a template, but I'm still seeing the same BUG.

>> A less intrusive, but uglier, alternative is to call
>> qemu_ram_alloc() and them mmap(MAP_FIXED) on top of that.
>
> I did try this, but ended up with a BUG on the host in
> /var/lib/dkms/kvm/84/build/x86/kvm_main.c:1266 gfn_to_pfn() on the
> line "BUG_ON(!kvm_is_mmio_pfn(pfn));" when the guest accesses the bar.

It looks to me from the call trace like the guest is writing to the
memory, gfn_to_pfn() from mmu_guess_page_from_pte_write() gets
confused because of the mapping.

Inside gfn_to_pfn:

addr = gfn_to_hva(kvm, gfn); correctly returns the host virtual
address of the external memory mapping.

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT,
presumably because (vma->vm_flags & (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't
understand what it is doing here: pfn = ((addr - vma->vm_start) >>
PAGE_SHIFT) + vma->vm_pgoff;

In my case addr == vma->vm_start, and vma->vm_pgoff == 0, so pfn ==0.
BUG_ON(!kvm_is_mmio_pfn(pfn)) then triggers.

Instrumenting inside gfn_to_pfn I see:
gfn_to_pfn: gfn f2010 gpte f2010000 hva 7f3eac2b0000 pfn 0 npages -14
gfn_to_pfn: vma ffff88022142af18 start 7f3eac2b0000 pgoff 0

Any suggestions what should be happening here?

[ 1826.807846] ------------[ cut here ]------------
[ 1826.807907] kernel BUG at
/build/buildd/linux-2.6.28/arch/x86/kvm/../../../virt/kvm/kvm_main.c:1001!
[ 1826.807985] invalid opcode: 0000 [#1] SMP
[ 1826.808102] last sysfs file: /sys/module/nf_nat/initstate
[ 1826.808159] Dumping ftrace buffer:
[ 1826.808213]    (ftrace buffer empty)
[ 1826.808266] CPU 3
[ 1826.808347] Modules linked in: tun softcard_driver(P)
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ip
v4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables
x_tables kvm_intel kvm input_polldev video output
 bridge stp lp parport iTCO_wdt iTCO_vendor_support psmouse pcspkr
serio_raw joydev i5000_edac edac_core shpchp e1000 us
bhid usb_storage e1000e floppy raid10 raid456 async_xor async_memcpy
async_tx xor raid1 raid0 multipath linear fbcon til
eblit font bitblit softcursor
[ 1826.810269] Pid: 9353, comm: qemu-system-x86 Tainted: P
2.6.28-13-server #45-Ubuntu
[ 1826.810344] RIP: 0010:[<ffffffffa01da853>]  [<ffffffffa01da853>]
gfn_to_pfn+0x153/0x160 [kvm]
[ 1826.810463] RSP: 0018:ffff88022d857958  EFLAGS: 00010246
[ 1826.810518] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88022d4d32a0
[ 1826.810577] RDX: 0000000000000000 RSI: 0000000000000282 RDI: 0000000000000000
[ 1826.810636] RBP: ffff88022d857978 R08: 0000000000000001 R09: ffff88022d857958
[ 1826.810694] R10: 0000000000000003 R11: 0000000000000001 R12: 00000000000f2010
[ 1826.810753] R13: ffff880212cb0000 R14: ffff880212cb0000 R15: ffff880212cb0000
[ 1826.810812] FS:  00007f5253bfd950(0000) GS:ffff88022f1fa380(0000)
knlGS:0000000000000000
[ 1826.810887] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 1826.810943] CR2: 00000000b7eb2044 CR3: 0000000212cac000 CR4: 00000000000026a0
[ 1826.811002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1826.811061] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1826.811120] Process qemu-system-x86 (pid: 9353, threadinfo
ffff88022d856000, task ffff88022e0cd980)
[ 1826.811196] Stack:
[ 1826.811246]  ffff88022d857968 0000000000000004 0000000000000004
0000000000000000
[ 1826.811401]  ffff88022d8579b8 ffffffffa01e7ccf ffff88022d8579b8
00000000f2010073
[ 1826.811634]  0000000000000004 ffff880212cb15b0 000000001f402b00
ffff880212cb0000
[ 1826.811913] Call Trace:
[ 1826.811964]  [<ffffffffa01e7ccf>]
mmu_guess_page_from_pte_write+0xaf/0x190 [kvm]
[ 1826.812076]  [<ffffffffa01e820f>] kvm_mmu_pte_write+0x3f/0x4f0 [kvm]
[ 1826.812172]  [<ffffffffa01da9f1>] ? mark_page_dirty+0x11/0x90 [kvm]
[ 1826.812268]  [<ffffffffa01dabe8>] ? kvm_write_guest+0x48/0x90 [kvm]
[ 1826.812364]  [<ffffffffa01de427>] emulator_write_phys+0x47/0x70 [kvm]
[ 1826.812460]  [<ffffffffa01e0e26>]
emulator_write_emulated_onepage+0x66/0x120 [kvm]
[ 1826.812571]  [<ffffffffa01e0f50>] emulator_write_emulated+0x70/0x90 [kvm]
[ 1826.812668]  [<ffffffffa01eb36f>] x86_emulate_insn+0x4ef/0x32e0 [kvm]
[ 1826.812764]  [<ffffffffa01e950e>] ? do_insn_fetch+0x8e/0x100 [kvm]
[ 1826.812860]  [<ffffffffa01e9454>] ? seg_override_base+0x24/0x50 [kvm]
[ 1826.812955]  [<ffffffffa01eacb0>] ? x86_decode_insn+0x7a0/0x970 [kvm]
[ 1826.813051]  [<ffffffffa01e221f>] emulate_instruction+0x15f/0x2f0 [kvm]
[ 1826.813148]  [<ffffffffa01e7bd5>] kvm_mmu_page_fault+0x65/0xb0 [kvm]
[ 1826.813243]  [<ffffffffa020ac5f>] handle_exception+0x2ef/0x360 [kvm_intel]
[ 1826.813338]  [<ffffffffa01eb0a3>] ? x86_emulate_insn+0x223/0x32e0 [kvm]
[ 1826.813434]  [<ffffffffa0209c25>] kvm_handle_exit+0xb5/0x1d0 [kvm_intel]
[ 1826.813526]  [<ffffffff80699643>] ? __down_read+0xc3/0xce
[ 1826.813618]  [<ffffffffa01dd958>] vcpu_enter_guest+0x1f8/0x400 [kvm]
[ 1826.813714]  [<ffffffffa01dfc29>] __vcpu_run+0x69/0x2d0 [kvm]
[ 1826.813751]  [<ffffffffa01e38ea>] kvm_arch_vcpu_ioctl_run+0x8a/0x1f0 [kvm]
[ 1826.813751]  [<ffffffffa01d8582>] kvm_vcpu_ioctl+0x2e2/0x5a0 [kvm]
[ 1826.813751]  [<ffffffff802f6091>] vfs_ioctl+0x31/0xa0
[ 1826.813751]  [<ffffffff802f6445>] do_vfs_ioctl+0x75/0x230
[ 1826.813751]  [<ffffffff802e8216>] ? generic_file_llseek+0x56/0x70
[ 1826.813751]  [<ffffffff802f6699>] sys_ioctl+0x99/0xa0
[ 1826.813751]  [<ffffffff802e70d2>] ? sys_lseek+0x52/0x90
[ 1826.813751]  [<ffffffff8021253a>] system_call_fastpath+0x16/0x1b
[ 1826.813751] Code: 00 00 65 48 8b 04 25 00 00 00 00 48 8b b8 38 02
00 00 48 83 c7 60 e8 dd 23 09 e0 48 89 df e8 45 fe ff ff 85 c0 0f 85
08 ff ff ff <0f> 0b eb fe 66 0f 1f 84 00 00 00 00 00 55 65 8b 14 25 24
00 00
[ 1826.813751] RIP  [<ffffffffa01da853>] gfn_to_pfn+0x153/0x160 [kvm]
[ 1826.813751]  RSP <ffff88022d857958>
[ 1826.816899] ---[ end trace 2437a1197b66fb45 ]---

Stephen.