IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

Hello,
NetBSD works as dom0 up to Xen 4.11. I'm trying to get it working
on 4.13.0. I added the support for gntdev operations,  but I'm stuck with
privcmd IOCTL_PRIVCMD_MMAPBATCH. It seems to work fine for PV and PVH domUs,
but with HVM domUs, MMU_NORMAL_PT_UPDATE returns -22 (EINVAL) and
qemu-dm dumps core (as expected; the page is not mapped).
Of course this works fine in 4.11

In the Xen kernel, I tracked it down to arch/x86/mm.c near line 2229,
in mod_l1_entry():
        /* Translate foreign guest address. */
        if ( cmd != MMU_PT_UPDATE_NO_TRANSLATE &&
             paging_mode_translate(pg_dom) )
        {
            p2m_type_t p2mt;
            p2m_query_t q = l1e_get_flags(nl1e) & _PAGE_RW ?
                            P2M_ALLOC | P2M_UNSHARE : P2M_ALLOC;

            page = get_page_from_gfn(pg_dom, l1e_get_pfn(nl1e), &p2mt, q);

            if ( p2m_is_paged(p2mt) )
            {
                if ( page )
                    put_page(page);
                p2m_mem_paging_populate(pg_dom, l1e_get_pfn(nl1e));
                return -ENOENT;
            }

            if ( p2mt == p2m_ram_paging_in && !page )
                return -ENOENT;

            /* Did our attempt to unshare fail? */
            if ( (q & P2M_UNSHARE) && p2m_is_shared(p2mt) )
            {
                /* We could not have obtained a page ref. */
                ASSERT(!page);
                /* And mem_sharing_notify has already been called. */
                return -ENOMEM;
            }

            if ( !page ) {
                gdprintk(XENLOG_WARNING, "translate but no page\n");
                return -EINVAL;
            }                        
            nl1e = l1e_from_page(page, l1e_get_flags(nl1e));
        }

the gdprintk() I added in the ( !page) case fires, so this is the
cause of the EINVAL.
Is it expected for a HVM domU ? If so, how should the dom0 code be
changed to get it working ? I failed to see where our code is different
from linux ...

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Andrew Cooper]

On 15/05/2020 21:29, Manuel Bouyer wrote:
> Hello,
> NetBSD works as dom0 up to Xen 4.11. I'm trying to get it working
> on 4.13.0. I added the support for gntdev operations,  but I'm stuck with
> privcmd IOCTL_PRIVCMD_MMAPBATCH. It seems to work fine for PV and PVH domUs,
> but with HVM domUs, MMU_NORMAL_PT_UPDATE returns -22 (EINVAL) and
> qemu-dm dumps core (as expected; the page is not mapped).
> Of course this works fine in 4.11
>
> In the Xen kernel, I tracked it down to arch/x86/mm.c near line 2229,
> in mod_l1_entry():
>         /* Translate foreign guest address. */
>         if ( cmd != MMU_PT_UPDATE_NO_TRANSLATE &&
>              paging_mode_translate(pg_dom) )
>         {
>             p2m_type_t p2mt;
>             p2m_query_t q = l1e_get_flags(nl1e) & _PAGE_RW ?
>                             P2M_ALLOC | P2M_UNSHARE : P2M_ALLOC;
>
>             page = get_page_from_gfn(pg_dom, l1e_get_pfn(nl1e), &p2mt, q);
>
>             if ( p2m_is_paged(p2mt) )
>             {
>                 if ( page )
>                     put_page(page);
>                 p2m_mem_paging_populate(pg_dom, l1e_get_pfn(nl1e));
>                 return -ENOENT;
>             }
>
>             if ( p2mt == p2m_ram_paging_in && !page )
>                 return -ENOENT;
>
>             /* Did our attempt to unshare fail? */
>             if ( (q & P2M_UNSHARE) && p2m_is_shared(p2mt) )
>             {
>                 /* We could not have obtained a page ref. */
>                 ASSERT(!page);
>                 /* And mem_sharing_notify has already been called. */
>                 return -ENOMEM;
>             }
>
>             if ( !page ) {
>                 gdprintk(XENLOG_WARNING, "translate but no page\n");
>                 return -EINVAL;
>             }                        
>             nl1e = l1e_from_page(page, l1e_get_flags(nl1e));
>         }
>
> the gdprintk() I added in the ( !page) case fires, so this is the
> cause of the EINVAL.
> Is it expected for a HVM domU ? If so, how should the dom0 code be
> changed to get it working ? I failed to see where our code is different
> from linux ...

What is qemu doing at the time?  Is it by any chance trying to map the
IOREQ server frame?

~Andrew

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

On Fri, May 15, 2020 at 10:00:07PM +0100, Andrew Cooper wrote:
> What is qemu doing at the time?  Is it by any chance trying to map the
> IOREQ server frame?

Here's what gdb says about it:
Core was generated by `qemu-dm'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000046997d in cpu_x86_init (
    cpu_model=cpu_model@entry=0x4d622d "qemu32")
    at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/i386-dm/helper2.c:156
156                 rc = xenevtchn_bind_interdomain(
--Type <RET> for more, q to quit, c to continue without paging--
[Current thread is 1 (process 1480)]
(gdb) where
#0  0x000000000046997d in cpu_x86_init (
    cpu_model=cpu_model@entry=0x4d622d "qemu32")
    at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/i386-dm/helper2.c:156
#1  0x000000000043628d in pc_init1 (ram_size=<optimized out>, 
    vga_ram_size=4194304, boot_device=0x7f7fff460397 "cda", pci_enabled=1, 
    cpu_model=0x4d622d "qemu32", initrd_filename=<optimized out>, 
    kernel_cmdline=<optimized out>, kernel_filename=<optimized out>)
    at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/hw/pc.c:829
#2  0x00000000004636e7 in xen_init_fv (ram_size=0, vga_ram_size=4194304, 
    boot_device=0x7f7fff460397 "cda", kernel_filename=0x0, 
    kernel_cmdline=0x4abff6 "", initrd_filename=0x0, cpu_model=0x0, 
    direct_pci=0x0)
    at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/hw/xen_machine_fv.c:405
#3  0x00000000004a975b in main (argc=23, argv=0x7f7fff45fc78, 
    envp=<optimized out>)
    at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/vl.c:6014

Does it help ?

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Andrew Cooper]

On 15/05/2020 22:06, Manuel Bouyer wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
>
> On Fri, May 15, 2020 at 10:00:07PM +0100, Andrew Cooper wrote:
>> What is qemu doing at the time?  Is it by any chance trying to map the
>> IOREQ server frame?
> Here's what gdb says about it:
> Core was generated by `qemu-dm'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x000000000046997d in cpu_x86_init (
>     cpu_model=cpu_model@entry=0x4d622d "qemu32")
>     at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/i386-dm/helper2.c:156
> 156                 rc = xenevtchn_bind_interdomain(
> --Type <RET> for more, q to quit, c to continue without paging--
> [Current thread is 1 (process 1480)]
> (gdb) where
> #0  0x000000000046997d in cpu_x86_init (
>     cpu_model=cpu_model@entry=0x4d622d "qemu32")
>     at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/i386-dm/helper2.c:156
> #1  0x000000000043628d in pc_init1 (ram_size=<optimized out>, 
>     vga_ram_size=4194304, boot_device=0x7f7fff460397 "cda", pci_enabled=1, 
>     cpu_model=0x4d622d "qemu32", initrd_filename=<optimized out>, 
>     kernel_cmdline=<optimized out>, kernel_filename=<optimized out>)
>     at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/hw/pc.c:829
> #2  0x00000000004636e7 in xen_init_fv (ram_size=0, vga_ram_size=4194304, 
>     boot_device=0x7f7fff460397 "cda", kernel_filename=0x0, 
>     kernel_cmdline=0x4abff6 "", initrd_filename=0x0, cpu_model=0x0, 
>     direct_pci=0x0)
>     at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/hw/xen_machine_fv.c:405
> #3  0x00000000004a975b in main (argc=23, argv=0x7f7fff45fc78, 
>     envp=<optimized out>)
>     at /home/bouyer/pkgbuild/current/sysutils/xentools413/work/xen-4.13.0/tools/qemu-xen-traditional/vl.c:6014
>
> Does it help ?

Yes and no.  This is collateral damage of earlier bug.

What failed was xen_init_fv()'s

    shared_page = xc_map_foreign_range(xc_handle, domid, XC_PAGE_SIZE,
                                       PROT_READ|PROT_WRITE, ioreq_pfn);
    if (shared_page == NULL) {
        fprintf(logfile, "map shared IO page returned error %d\n", errno);
        exit(-1);
    }

because we've ended up with a non-NULL pointer with no mapping behind
it, hence the SIGSEGV for the first time we try to use the pointer.

Whatever logic is behind xc_map_foreign_range() should have returned
NULL or a real mapping.

ioreq_pfn ought to be something just below the 4G boundary, and the
toolstack ought to put memory there in the first place.  Can you
identify what value ioreq_pfn has, and whether it matches up with the
magic gfn range as reported by `xl create -vvv` for the guest?

~Andrew

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

On Fri, May 15, 2020 at 10:38:13PM +0100, Andrew Cooper wrote:
> > [...]
> > Does it help ?
> 
> Yes and no.  This is collateral damage of earlier bug.
> 
> What failed was xen_init_fv()'s
> 
>     shared_page = xc_map_foreign_range(xc_handle, domid, XC_PAGE_SIZE,
>                                        PROT_READ|PROT_WRITE, ioreq_pfn);
>     if (shared_page == NULL) {
>         fprintf(logfile, "map shared IO page returned error %d\n", errno);
>         exit(-1);
>     }
> 
> because we've ended up with a non-NULL pointer with no mapping behind
> it, hence the SIGSEGV for the first time we try to use the pointer.
> 
> Whatever logic is behind xc_map_foreign_range() should have returned
> NULL or a real mapping.

What's strange is that the mapping is validated, by mapping it in
the dom0 kernel space. But when we try to remap in in the process's
space, it fails.

> 
> ioreq_pfn ought to be something just below the 4G boundary, and the
> toolstack ought to put memory there in the first place.  Can you
> identify what value ioreq_pfn has,

You mean, something like:
(gdb) print/x ioreq_pfn
$2 = 0xfeff0

> and whether it matches up with the
> magic gfn range as reported by `xl create -vvv` for the guest?

I guess you mean
xl -vvv create
the output is attached

The kernel says it tries to map 0xfeff0000 to virtual address 0x79656f951000.


-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

[-- Attachment #2: typescript --]
[-- Type: text/plain, Size: 13735 bytes --]

Script started on Fri May 15 23:47:25 2020
# xl -vvv create nb1=-\b \b\b \b-hbvm\b \b\b \b\b \bvm
Parsing config from nb1-hvm
libxl: debug: libxl_create.c:1819:do_domain_create: Domain 0:ao 0x75de79e9b000: create: how=0x0 callback=0x0 poller=0x75de79ec60a0
libxl: detail: libxl_create.c:584:libxl__domain_make: passthrough: disabled
libxl: debug: libxl_device.c:380:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=unknown
libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=hda, using backend phy
libxl: debug: libxl_create.c:1148:initiate_domain_create: Domain 5:running bootloader
libxl: debug: libxl_bootloader.c:328:libxl__bootloader_run: Domain 5:not a PV/PVH domain, skipping bootloader
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79e64cd0: deregister unregistered
domainbuilder: detail: xc_dom_allocate: cmdline="", features=""
domainbuilder: detail: xc_dom_kernel_file: filename="/usr/pkg/libexec/xen/boot/hvmloader"
domainbuilder: detail: xc_dom_malloc_filemap    : 337 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.13, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ... 
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ... 
domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ... 
domainbuilder: detail: loader probe OK
xc: detail: ELF: phdr: paddr=0x100000 memsz=0x5c844
xc: detail: ELF: memory: 0x100000 -> 0x15c844
domainbuilder: detail: xc_dom_mem_init: mem 1020 MB, pages 0x3fc00 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x3fc00 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: range: start=0x0 end=0x3fc00000
domainbuilder: detail: xc_dom_malloc            : 2040 kB
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000001fd
xc: detail:   1GB PAGES: 0x0000000000000000
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x5d at 0x75de79b31000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x100000 -> 0x15d000  (pfn 0x100 + 0x5d pages)
xc: detail: ELF: phdr 0 at 0x75de79ad4000 -> 0x75de79b26ca0
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x15d+0x1 at 0x75de79e1d000
domainbuilder: detail: xc_dom_alloc_segment:   HVM start info : 0x15d000 -> 0x15e000  (pfn 0x15d + 0x1 pages)
domainbuilder: detail: alloc_pgtables_hvm: doing nothing
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x15e000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 2045 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 337 kB
domainbuilder: detail:       domU mmap          : 376 kB
domainbuilder: detail: vcpu_hvm: called
domainbuilder: detail: compat_gnttab_hvm_seed: d5: pfn=0xff000
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[0] -> d0 0xfefff
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[1] -> d0 0xfeffc
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:380:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=phy
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x75de79bfd4d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/0: register slotnum=3
libxl: debug: libxl_create.c:1856:do_domain_create: Domain 0:ao 0x75de79e9b000: inprogress: poller=0x75de79ec60a0, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x75de79bfd4d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/0: event epath=/local/domain/0/backend/vbd/5/768/state
libxl: debug: libxl_event.c:881:devstate_callback: backend /local/domain/0/backend/vbd/5/768/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x75de79bfd4d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/0: event epath=/local/domain/0/backend/vbd/5/768/state
libxl: debug: libxl_event.c:877:devstate_callback: backend /local/domain/0/backend/vbd/5/768/state wanted state 2 ok
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x75de79bfd4d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/0: deregister slotnum=3
libxl: debug: libxl_device.c:1090:device_backend_callback: Domain 5:calling device_backend_cleanup
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79bfd4d0: deregister unregistered
libxl: debug: libxl_device.c:1191:device_hotplug: Domain 5:calling hotplug script: /usr/pkg/etc/xen/scripts/block /local/domain/0/backend/vbd/5/768
libxl: debug: libxl_device.c:1192:device_hotplug: Domain 5:extra args:
libxl: debug: libxl_device.c:1198:device_hotplug: Domain 5:     2
libxl: debug: libxl_device.c:1200:device_hotplug: Domain 5:env:
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /usr/pkg/etc/xen/scripts/block /local/domain/0/backend/vbd/5/768 
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79bfd5d0: deregister unregistered
libxl: debug: libxl_netbsd.c:74:libxl__get_hotplug_script_info: Domain 5:num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1176:device_hotplug: Domain 5:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79bfd5d0: deregister unregistered
libxl: debug: libxl_dm.c:2626:libxl__spawn_local_dm: Domain 5:Spawning device-model /usr/pkg/libexec/xen/bin/qemu-dm with arguments:
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  /usr/pkg/libexec/xen/bin/qemu-dm
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -d
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  5
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -domain-name
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  nb1
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -vnc
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  132.227.103.47:1
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -videoram
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  4
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -boot
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  cda
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -usb
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -usbdevice
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  tablet
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -acpi
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -vcpu_avail
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  0x01
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -net
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  nic,vlan=0,macaddr=00:16:3e:00:10:54,model=e1000
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -net
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  tap,vlan=0,ifname=xvif5i0-emu,bridge=bridge0,script=/usr/pkg/etc/xen/scripts/qemu-ifup,downscript=/usr/pkg/etc/xen/scripts/qemu-ifup
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  -M
libxl: debug: libxl_dm.c:2628:libxl__spawn_local_dm: Domain 5:  xenfv
libxl: debug: libxl_dm.c:2630:libxl__spawn_local_dm: Domain 5:Spawning device-model /usr/pkg/libexec/xen/bin/qemu-dm with additional environment:
libxl: debug: libxl_dm.c:2632:libxl__spawn_local_dm: Domain 5:  XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x75de79e64fc8 wpath=/local/domain/0/device-model/5/state token=3/1: register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x75de79e64fc8 wpath=/local/domain/0/device-model/5/state token=3/1: event epath=/local/domain/0/device-model/5/state
libxl: debug: libxl_exec.c:407:spawn_watch_event: domain 5 device model: spawn watch p=(null)
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x75de79e64fc8 wpath=/local/domain/0/device-model/5/state token=3/1: deregister slotnum=3
libxl: error: libxl_dm.c:2783:device_model_spawn_outcome: Domain 5:domain 5 device model: spawn failed (rc=-3)
libxl: error: libxl_dm.c:2999:device_model_postconfig_done: Domain 5:Post DM startup configs failed, rc=-3
libxl: debug: libxl_qmp.c:1896:libxl__ev_qmp_dispose: Domain 0: ev 0x75de79e64fe0
libxl: error: libxl_create.c:1676:domcreate_devmodel_started: Domain 5:device model did not start: -3
libxl: debug: libxl_dm.c:3237:libxl__destroy_device_model: Domain 5:Didn't find dm UID; destroying by pid
libxl: error: libxl_dm.c:3103:kill_device_model: Device Model already exited
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x75de79bfe0d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/2: register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x75de79bfe0d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/2: event epath=/local/domain/0/backend/vbd/5/768/state
libxl: debug: libxl_event.c:881:devstate_callback: backend /local/domain/0/backend/vbd/5/768/state wanted state 6 still waiting state 5
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x75de79bfe0d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/2: event epath=/local/domain/0/backend/vbd/5/768/state
libxl: debug: libxl_event.c:877:devstate_callback: backend /local/domain/0/backend/vbd/5/768/state wanted state 6 ok
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x75de79bfe0d0 wpath=/local/domain/0/backend/vbd/5/768/state token=3/2: deregister slotnum=3
libxl: debug: libxl_device.c:1090:device_backend_callback: Domain 5:calling device_backend_cleanup
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79bfe0d0: deregister unregistered
libxl: debug: libxl_device.c:1191:device_hotplug: Domain 5:calling hotplug script: /usr/pkg/etc/xen/scripts/block /local/domain/0/backend/vbd/5/768
libxl: debug: libxl_device.c:1192:device_hotplug: Domain 5:extra args:
libxl: debug: libxl_device.c:1198:device_hotplug: Domain 5:     6
libxl: debug: libxl_device.c:1200:device_hotplug: Domain 5:env:
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /usr/pkg/etc/xen/scripts/block /local/domain/0/backend/vbd/5/768 
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79bfe1d0: deregister unregistered
libxl: debug: libxl_netbsd.c:74:libxl__get_hotplug_script_info: Domain 5:num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1176:device_hotplug: Domain 5:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79bfe1d0: deregister unregistered
libxl: debug: libxl_device.c:1176:device_hotplug: Domain 5:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x75de79bfe5d0: deregister unregistered
libxl: debug: libxl_domain.c:1355:devices_destroy_cb: Domain 5:Forked pid 2095 for destroy of domain
libxl: debug: libxl_event.c:1893:libxl__ao_complete: ao 0x75de79e9b000: complete, rc=-3
libxl: debug: libxl_event.c:1862:libxl__ao__destroy: ao 0x75de79e9b000: destroy
libxl: debug: libxl_domain.c:1040:libxl_domain_destroy: Domain 5:ao 0x75de79e9b000: create: how=0x0 callback=0x0 poller=0x75de79ec60a0
libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 5:Non-existant domain
libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 5:Unable to destroy guest
libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 5:Destruction of domain failed
libxl: debug: libxl_event.c:1893:libxl__ao_complete: ao 0x75de79e9b000: complete, rc=-21
libxl: debug: libxl_domain.c:1049:libxl_domain_destroy: Domain 5:ao 0x75de79e9b000: inprogress: poller=0x75de79ec60a0, flags=ic
libxl: debug: libxl_event.c:1862:libxl__ao__destroy: ao 0x75de79e9b000: destroy
xencall:buffer: debug: total allocations:400 total releases:400
xencall:buffer: debug: current allocations:0 maximum allocations:3
xencall:buffer: debug: cache current size:3
xencall:buffer: debug: cache hits:383 misses:3 toobig:14
xencall:buffer: debug: total allocations:0 total releases:0
xencall:buffer: debug: current allocations:0 maximum allocations:0
xencall:buffer: debug: cache current size:0
xencall:buffer: debug: cache hits:0 misses:0 toobig:0
# 
Script done on Fri May 15 23:47:46 2020

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Andrew Cooper]

On 15/05/2020 22:53, Manuel Bouyer wrote:
> On Fri, May 15, 2020 at 10:38:13PM +0100, Andrew Cooper wrote:
>>> [...]
>>> Does it help ?
>> Yes and no.  This is collateral damage of earlier bug.
>>
>> What failed was xen_init_fv()'s
>>
>>     shared_page = xc_map_foreign_range(xc_handle, domid, XC_PAGE_SIZE,
>>                                        PROT_READ|PROT_WRITE, ioreq_pfn);
>>     if (shared_page == NULL) {
>>         fprintf(logfile, "map shared IO page returned error %d\n", errno);
>>         exit(-1);
>>     }
>>
>> because we've ended up with a non-NULL pointer with no mapping behind
>> it, hence the SIGSEGV for the first time we try to use the pointer.
>>
>> Whatever logic is behind xc_map_foreign_range() should have returned
>> NULL or a real mapping.
> What's strange is that the mapping is validated, by mapping it in
> the dom0 kernel space. But when we try to remap in in the process's
> space, it fails.

Hmm - this sounds like a kernel bug I'm afraid.

>> ioreq_pfn ought to be something just below the 4G boundary, and the
>> toolstack ought to put memory there in the first place.  Can you
>> identify what value ioreq_pfn has,
> You mean, something like:
> (gdb) print/x ioreq_pfn
> $2 = 0xfeff0
>
>> and whether it matches up with the
>> magic gfn range as reported by `xl create -vvv` for the guest?
> I guess you mean
> xl -vvv create
> the output is attached
>
> The kernel says it tries to map 0xfeff0000 to virtual address 0x79656f951000.

The value looks right, and the logs look normal.

~Andrew

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

On Sat, May 16, 2020 at 05:18:45PM +0100, Andrew Cooper wrote:
> On 15/05/2020 22:53, Manuel Bouyer wrote:
> > On Fri, May 15, 2020 at 10:38:13PM +0100, Andrew Cooper wrote:
> >>> [...]
> >>> Does it help ?
> >> Yes and no.  This is collateral damage of earlier bug.
> >>
> >> What failed was xen_init_fv()'s
> >>
> >>     shared_page = xc_map_foreign_range(xc_handle, domid, XC_PAGE_SIZE,
> >>                                        PROT_READ|PROT_WRITE, ioreq_pfn);
> >>     if (shared_page == NULL) {
> >>         fprintf(logfile, "map shared IO page returned error %d\n", errno);
> >>         exit(-1);
> >>     }
> >>
> >> because we've ended up with a non-NULL pointer with no mapping behind
> >> it, hence the SIGSEGV for the first time we try to use the pointer.
> >>
> >> Whatever logic is behind xc_map_foreign_range() should have returned
> >> NULL or a real mapping.
> > What's strange is that the mapping is validated, by mapping it in
> > the dom0 kernel space. But when we try to remap in in the process's
> > space, it fails.
> 
> Hmm - this sounds like a kernel bug I'm afraid.

no I don't think it is. It works with Xen 4.11, and it works with 4.13 for
PV/PVH domUs. Maybe there's a missing flag in the PTE of some sort that is
mandatory for 4.13 but not 4.11 for userland PTE but it doens't look obvious.

The difference could be that the kernel page tables are active when
mapping the foreing page in the dom0's kernel space, but the
user process page table it not (obviously, as we're in the kernel
when doing the mapping).

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

I've been looking a bit deeper in the Xen kernel.
The mapping is failed in ./arch/x86/mm/p2m.c:p2m_get_page_from_gfn(),
        /* Error path: not a suitable GFN at all */
	if ( !p2m_is_ram(*t) && !p2m_is_paging(*t) && !p2m_is_pod(*t) ) {
	    gdprintk(XENLOG_ERR, "p2m_get_page_from_gfn2: %d is_ram %ld is_paging %ld is_pod %ld\n", *t, p2m_is_ram(*t), p2m_is_paging(*t), p2m_is_pod(*t) );
	    return NULL;
	}

*t is 4, which translates to p2m_mmio_dm

it looks like p2m_get_page_from_gfn() is not ready to handle this case
for dom0.

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

On Sun, May 17, 2020 at 07:32:59PM +0200, Manuel Bouyer wrote:
> I've been looking a bit deeper in the Xen kernel.
> The mapping is failed in ./arch/x86/mm/p2m.c:p2m_get_page_from_gfn(),
>         /* Error path: not a suitable GFN at all */
> 	if ( !p2m_is_ram(*t) && !p2m_is_paging(*t) && !p2m_is_pod(*t) ) {
> 	    gdprintk(XENLOG_ERR, "p2m_get_page_from_gfn2: %d is_ram %ld is_paging %ld is_pod %ld\n", *t, p2m_is_ram(*t), p2m_is_paging(*t), p2m_is_pod(*t) );
> 	    return NULL;
> 	}
> 
> *t is 4, which translates to p2m_mmio_dm
> 
> it looks like p2m_get_page_from_gfn() is not ready to handle this case
> for dom0.

And so it looks like I need to implement osdep_xenforeignmemory_map_resource()
for NetBSD

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

RE: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Paul Durrant]

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Manuel Bouyer
> Sent: 17 May 2020 18:56
> To: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: xen-devel@lists.xenproject.org
> Subject: Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0
> 
> On Sun, May 17, 2020 at 07:32:59PM +0200, Manuel Bouyer wrote:
> > I've been looking a bit deeper in the Xen kernel.
> > The mapping is failed in ./arch/x86/mm/p2m.c:p2m_get_page_from_gfn(),
> >         /* Error path: not a suitable GFN at all */
> > 	if ( !p2m_is_ram(*t) && !p2m_is_paging(*t) && !p2m_is_pod(*t) ) {
> > 	    gdprintk(XENLOG_ERR, "p2m_get_page_from_gfn2: %d is_ram %ld is_paging %ld is_pod %ld\n", *t,
> p2m_is_ram(*t), p2m_is_paging(*t), p2m_is_pod(*t) );
> > 	    return NULL;
> > 	}
> >
> > *t is 4, which translates to p2m_mmio_dm
> >
> > it looks like p2m_get_page_from_gfn() is not ready to handle this case
> > for dom0.
> 
> And so it looks like I need to implement osdep_xenforeignmemory_map_resource()
> for NetBSD
> 

It would be a good idea but you shouldn't have to. Also, qemu-trad won't use it even if it is there.

  Paul

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

On Mon, May 18, 2020 at 08:36:24AM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Manuel Bouyer
> > Sent: 17 May 2020 18:56
> > To: Andrew Cooper <andrew.cooper3@citrix.com>
> > Cc: xen-devel@lists.xenproject.org
> > Subject: Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0
> > 
> > On Sun, May 17, 2020 at 07:32:59PM +0200, Manuel Bouyer wrote:
> > > I've been looking a bit deeper in the Xen kernel.
> > > The mapping is failed in ./arch/x86/mm/p2m.c:p2m_get_page_from_gfn(),
> > >         /* Error path: not a suitable GFN at all */
> > > 	if ( !p2m_is_ram(*t) && !p2m_is_paging(*t) && !p2m_is_pod(*t) ) {
> > > 	    gdprintk(XENLOG_ERR, "p2m_get_page_from_gfn2: %d is_ram %ld is_paging %ld is_pod %ld\n", *t,
> > p2m_is_ram(*t), p2m_is_paging(*t), p2m_is_pod(*t) );
> > > 	    return NULL;
> > > 	}
> > >
> > > *t is 4, which translates to p2m_mmio_dm
> > >
> > > it looks like p2m_get_page_from_gfn() is not ready to handle this case
> > > for dom0.
> > 
> > And so it looks like I need to implement osdep_xenforeignmemory_map_resource()
> > for NetBSD
> > 
> 
> It would be a good idea but you shouldn't have to.

This is how I read the code too, it should fallback to mmapbatch.
But mmapbatch doesn't work in all cases on 4.13.0

> Also, qemu-trad won't use it even if it is there.

Looks like it still use mmapbatch for some mappings, indeeed.
But now it goes a bit further, but still end up failing to map the guest's
memory.

Also, with some fix I got qemu-xen working building on NetBSD, it starts
but fails to load the BIOS rom. Once again p2m_get_page_from_gfn() fails
to find a page, and thinks the type is p2m_mmio_dm.

From what I found it seems that all unallocated memory is tagged p2m_mmio_dm,
is it right ? That would point to some issue allocating RAM for
the domU in qemu, I would need to find where this happens in qemu.

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Jan Beulich]

On 18.05.2020 19:31, Manuel Bouyer wrote:
> From what I found it seems that all unallocated memory is tagged p2m_mmio_dm,
> is it right ?

Yes. For many years there has been a plan to better separate this from
p2m_invalid ...

Jan

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

On Tue, May 19, 2020 at 09:34:30AM +0200, Jan Beulich wrote:
> On 18.05.2020 19:31, Manuel Bouyer wrote:
> > From what I found it seems that all unallocated memory is tagged p2m_mmio_dm,
> > is it right ?
> 
> Yes. For many years there has been a plan to better separate this from
> p2m_invalid ...

thanks.

So for some reason, MMU_NORMAL_PT_UPDATE thinks that the memory is not
allocated for this domain. This is true for both the ioreq page, and
when trying to load the BIOS rom in the guest memory.
I traced the hypercall in the tools and the memory is allocated with
XENMEM_populate_physmap (and the gfn returned by XENMEM_populate_physmap
and passed to MMU_NORMAL_PT_UPDATE do match).

Still looking ...

Note that I'm using the 4.13.0 release sources, not the top of branch.
Is it something that could have been fixed after the release ?

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Jan Beulich]

On 19.05.2020 10:46, Manuel Bouyer wrote:
> Note that I'm using the 4.13.0 release sources, not the top of branch.
> Is it something that could have been fixed after the release ?

I don't recall anything, but switching to 4.13.1 would still seem like
a helpful thing for you to do.

Jan

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Roger Pau Monné]

On Sun, May 17, 2020 at 07:56:07PM +0200, Manuel Bouyer wrote:
> On Sun, May 17, 2020 at 07:32:59PM +0200, Manuel Bouyer wrote:
> > I've been looking a bit deeper in the Xen kernel.
> > The mapping is failed in ./arch/x86/mm/p2m.c:p2m_get_page_from_gfn(),
> >         /* Error path: not a suitable GFN at all */
> > 	if ( !p2m_is_ram(*t) && !p2m_is_paging(*t) && !p2m_is_pod(*t) ) {
> > 	    gdprintk(XENLOG_ERR, "p2m_get_page_from_gfn2: %d is_ram %ld is_paging %ld is_pod %ld\n", *t, p2m_is_ram(*t), p2m_is_paging(*t), p2m_is_pod(*t) );
> > 	    return NULL;
> > 	}
> > 
> > *t is 4, which translates to p2m_mmio_dm
> > 
> > it looks like p2m_get_page_from_gfn() is not ready to handle this case
> > for dom0.
> 
> And so it looks like I need to implement osdep_xenforeignmemory_map_resource()
> for NetBSD

FWIW, FreeBSD doesn't have osdep_xenforeignmemory_map_resource
implemented and still works fine with 4.13.0 (is able to create HVM
guests), but that's a PVH dom0, not a PV one.

Regards, Roger.

Re: IOCTL_PRIVCMD_MMAPBATCH on Xen 4.13.0

[Manuel Bouyer]

On Tue, May 19, 2020 at 11:54:07AM +0200, Roger Pau Monné wrote:
> FWIW, FreeBSD doesn't have osdep_xenforeignmemory_map_resource
> implemented and still works fine with 4.13.0 (is able to create HVM
> guests), but that's a PVH dom0, not a PV one.

Yes, FreeBSD is PVH-nnly. This implies different code paths (the dom0
kernel has to map the foreing pages in its physical space, which PV
doesn't have to do (and can't do))

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--