Stefan Hajnoczi writes: > On Thu, Jul 18, 2019 at 05:21:46PM +0200, Sergio Lopez wrote: >> >> Stefan Hajnoczi writes: >> >> > On Tue, Jul 02, 2019 at 02:11:02PM +0200, Sergio Lopez wrote: >> >> Microvm is a machine type inspired by both NEMU and Firecracker, and >> >> constructed after the machine model implemented by the latter. >> >> >> >> It's main purpose is providing users a KVM-only machine type with fast >> >> boot times, minimal attack surface (measured as the number of IO ports >> >> and MMIO regions exposed to the Guest) and small footprint (specially >> >> when combined with the ongoing QEMU modularization effort). >> >> >> >> Normally, other than the device support provided by KVM itself, >> >> microvm only supports virtio-mmio devices. Microvm also includes a >> >> legacy mode, which adds an ISA bus with a 16550A serial port, useful >> >> for being able to see the early boot kernel messages. >> >> >> >> Microvm only supports booting PVH-enabled Linux ELF images. Booting >> >> other PVH-enabled kernels may be possible, but due to the lack of ACPI >> >> and firmware, we're relying on the command line for specifying the >> >> location of the virtio-mmio transports. If there's an interest on >> >> using this machine type with other kernels, we'll try to find some >> >> kind of middle ground solution. >> >> >> >> This is the list of the exposed IO ports and MMIO regions when running >> >> in non-legacy mode: >> >> >> >> address-space: memory >> >> 00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio >> >> 00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio >> >> 00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio >> >> 00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio >> >> 00000000d0000800-00000000d00009ff (prio 0, i/o): virtio-mmio >> >> 00000000d0000a00-00000000d0000bff (prio 0, i/o): virtio-mmio >> >> 00000000d0000c00-00000000d0000dff (prio 0, i/o): virtio-mmio >> >> 00000000d0000e00-00000000d0000fff (prio 0, i/o): virtio-mmio >> >> 00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi >> >> >> >> address-space: I/O >> >> 0000000000000000-000000000000ffff (prio 0, i/o): io >> >> 0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic >> >> 0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit >> >> 000000000000007e-000000000000007f (prio 0, i/o): kvmvapic >> >> 00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic >> >> 00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr >> >> 00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr >> >> >> >> A QEMU instance with the microvm machine type can be invoked this way: >> >> >> >> - Normal mode: >> >> >> >> qemu-system-x86_64 -M microvm -m 512m -smp 2 \ >> >> -kernel vmlinux -append "console=hvc0 root=/dev/vda" \ >> >> -nodefaults -no-user-config \ >> >> -chardev pty,id=virtiocon0,server \ >> >> -device virtio-serial-device \ >> >> -device virtconsole,chardev=virtiocon0 \ >> >> -drive id=test,file=test.img,format=raw,if=none \ >> >> -device virtio-blk-device,drive=test \ >> >> -netdev tap,id=tap0,script=no,downscript=no \ >> >> -device virtio-net-device,netdev=tap0 >> >> >> >> - Legacy mode: >> >> >> >> qemu-system-x86_64 -M microvm,legacy -m 512m -smp 2 \ >> >> -kernel vmlinux -append "console=ttyS0 root=/dev/vda" \ >> >> -nodefaults -no-user-config \ >> >> -drive id=test,file=test.img,format=raw,if=none \ >> >> -device virtio-blk-device,drive=test \ >> >> -netdev tap,id=tap0,script=no,downscript=no \ >> >> -device virtio-net-device,netdev=tap0 \ >> >> -serial stdio >> > >> > Please post metrics that compare this against a minimal Q35. >> > >> > With qboot it was later found that SeaBIOS can achieve comparable boot >> > times, so it wasn't worth maintaining qboot. >> > >> > Data is needed to show that microvm is really a significant improvement >> > over a minimal Q35. >> >> I've just ran some numbers using Stefano Garzarella's qemu-boot-time >> scripts [1] on a server with 2xIntel Xeon Silver 4114 2.20GHz, using the >> upstream QEMU (474f3938d79ab36b9231c9ad3b5a9314c2aeacde) built with >> minimal features [2]. The VM boots a minimal kernel [3] without initrd, >> using a kata container image as root via virtio-blk (though this isn't >> really relevant, as we're just taking measurements until the kernel is >> about to exec init). >> >> To try to make the comparison as fair as possible, I've used a minimal >> q35 machine with as few devices as possible. Disabling HPET and PIT at >> the same time caused the kernel to get stuck on boot, so I ran two >> iterations, one without HPET and the other without PIT: >> >> >> ----------------- >> | Q35 with HPET | >> ----------------- >> >> Command line: >> >> ./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M q35,smbus=off,nvdimm=off,pit=off,vmport=off,sata=off,usb=off,graphics=off -kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev pty,id=virtiocon0,server -device virtio-serial -device virtconsole,chardev=virtiocon0 -drive id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device virtio-blk,drive=test >> >> Average boot times after 10 consecutive runs: >> >> qemu_init_end: 77.637936 >> linux_start_kernel: 117.082526 (+39.44459) >> linux_start_user: 364.629972 (+247.547446) >> >> Memory tree: >> >> address-space: memory >> 0000000000000000-ffffffffffffffff (prio 0, i/o): system >> 0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g @pc.ram 0000000000000000-000000001fffffff >> 0000000000000000-ffffffffffffffff (prio -1, i/o): pci >> 00000000000c0000-00000000000dffff (prio 1, rom): pc.rom >> 00000000000e0000-00000000000fffff (prio 1, i/o): alias isa-bios @pc.bios 0000000000020000-000000000003ffff >> 00000000febf4000-00000000febf7fff (prio 1, i/o): virtio-pci >> 00000000febf4000-00000000febf4fff (prio 0, i/o): virtio-pci-common >> 00000000febf5000-00000000febf5fff (prio 0, i/o): virtio-pci-isr >> 00000000febf6000-00000000febf6fff (prio 0, i/o): virtio-pci-device >> 00000000febf7000-00000000febf7fff (prio 0, i/o): virtio-pci-notify >> 00000000febf8000-00000000febfbfff (prio 1, i/o): virtio-pci >> 00000000febf8000-00000000febf8fff (prio 0, i/o): virtio-pci-common >> 00000000febf9000-00000000febf9fff (prio 0, i/o): virtio-pci-isr >> 00000000febfa000-00000000febfafff (prio 0, i/o): virtio-pci-device >> 00000000febfb000-00000000febfbfff (prio 0, i/o): virtio-pci-notify >> 00000000febfe000-00000000febfefff (prio 1, i/o): virtio-serial-pci-msix >> 00000000febfe000-00000000febfe01f (prio 0, i/o): msix-table >> 00000000febfe800-00000000febfe807 (prio 0, i/o): msix-pba >> 00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix >> 00000000febff000-00000000febff01f (prio 0, i/o): msix-table >> 00000000febff800-00000000febff807 (prio 0, i/o): msix-pba >> 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios >> 00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff >> 00000000000c0000-00000000000c2fff (prio 1000, i/o): alias kvmvapic-rom @pc.ram 00000000000c0000-00000000000c2fff >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000c0000-00000000000c3fff [disabled] >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000c0000-00000000000c3fff [disabled] >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pci 00000000000c0000-00000000000c3fff [disabled] >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000c4000-00000000000c7fff [disabled] >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000c4000-00000000000c7fff [disabled] >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pci 00000000000c4000-00000000000c7fff [disabled] >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-ram @pc.ram 00000000000c8000-00000000000cbfff [disabled] >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pc.ram 00000000000c8000-00000000000cbfff [disabled] >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pci 00000000000c8000-00000000000cbfff [disabled] >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-ram @pc.ram 00000000000cc000-00000000000cffff [disabled] >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pc.ram 00000000000cc000-00000000000cffff [disabled] >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pci 00000000000cc000-00000000000cffff [disabled] >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000d0000-00000000000d3fff [disabled] >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000d0000-00000000000d3fff [disabled] >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pci 00000000000d0000-00000000000d3fff [disabled] >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000d4000-00000000000d7fff [disabled] >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000d4000-00000000000d7fff [disabled] >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pci 00000000000d4000-00000000000d7fff [disabled] >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-ram @pc.ram 00000000000d8000-00000000000dbfff [disabled] >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pc.ram 00000000000d8000-00000000000dbfff [disabled] >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pci 00000000000d8000-00000000000dbfff [disabled] >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-ram @pc.ram 00000000000dc000-00000000000dffff [disabled] >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pc.ram 00000000000dc000-00000000000dffff [disabled] >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pci 00000000000dc000-00000000000dffff [disabled] >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000e0000-00000000000e3fff [disabled] >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000e0000-00000000000e3fff [disabled] >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pci 00000000000e0000-00000000000e3fff [disabled] >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000e4000-00000000000e7fff [disabled] >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000e4000-00000000000e7fff [disabled] >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pci 00000000000e4000-00000000000e7fff [disabled] >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pc.ram 00000000000e8000-00000000000ebfff [disabled] >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-rom @pc.ram 00000000000e8000-00000000000ebfff [disabled] >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pci 00000000000e8000-00000000000ebfff [disabled] >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-ram @pc.ram 00000000000ec000-00000000000effff >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pc.ram 00000000000ec000-00000000000effff [disabled] >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-rom @pc.ram 00000000000ec000-00000000000effff [disabled] >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pci 00000000000ec000-00000000000effff [disabled] >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-ram @pc.ram 00000000000f0000-00000000000fffff [disabled] >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pc.ram 00000000000f0000-00000000000fffff [disabled] >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pci 00000000000f0000-00000000000fffff [disabled] >> 0000000020000000-0000000020000000 (prio 1, i/o): tseg-blackhole [disabled] >> 00000000b0000000-00000000bfffffff (prio 0, i/o): pcie-mmcfg-mmio >> 00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic >> 00000000fed00000-00000000fed003ff (prio 0, i/o): hpet >> 00000000fed1c000-00000000fed1ffff (prio 1, i/o): lpc-rcrb-mmio >> 00000000feda0000-00000000fedbffff (prio 1, i/o): alias smram-open-high @pc.ram 00000000000a0000-00000000000bffff [disabled] >> 00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi >> >> address-space: I/O >> 0000000000000000-000000000000ffff (prio 0, i/o): io >> 0000000000000000-0000000000000007 (prio 0, i/o): dma-chan >> 0000000000000008-000000000000000f (prio 0, i/o): dma-cont >> 0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic >> 0000000000000060-0000000000000060 (prio 0, i/o): i8042-data >> 0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd >> 0000000000000070-0000000000000071 (prio 0, i/o): rtc >> 0000000000000070-0000000000000070 (prio 0, i/o): rtc-index >> 000000000000007e-000000000000007f (prio 0, i/o): kvmvapic >> 0000000000000080-0000000000000080 (prio 0, i/o): ioport80 >> 0000000000000081-0000000000000083 (prio 0, i/o): dma-page >> 0000000000000087-0000000000000087 (prio 0, i/o): dma-page >> 0000000000000089-000000000000008b (prio 0, i/o): dma-page >> 000000000000008f-000000000000008f (prio 0, i/o): dma-page >> 0000000000000092-0000000000000092 (prio 0, i/o): port92 >> 00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic >> 00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io >> 00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan >> 00000000000000d0-00000000000000df (prio 0, i/o): dma-cont >> 00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0 >> 00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr >> 00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr >> 0000000000000510-0000000000000511 (prio 0, i/o): fwcfg >> 0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma >> 0000000000000600-000000000000067f (prio 0, i/o): ich9-pm >> 0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt >> 0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt >> 0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr >> 0000000000000620-000000000000062f (prio 0, i/o): acpi-gpe0 >> 0000000000000630-0000000000000637 (prio 0, i/o): acpi-smi >> 0000000000000660-000000000000067f (prio 0, i/o): sm-tco >> 0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug >> 0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx >> 0000000000000cf9-0000000000000cf9 (prio 1, i/o): lpc-reset-control >> 0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data >> 000000000000c000-000000000000c07f (prio 1, i/o): virtio-pci >> 000000000000c080-000000000000c0bf (prio 1, i/o): virtio-pci >> >> >> ---------------- >> | Q35 with PIT | >> ---------------- >> >> Command line: >> >> ./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M q35,smbus=off,nvdimm=off,pit=on,vmport=off,sata=off,usb=off,graphics=off -no-hpet -kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev pty,id=virtiocon0,server -device virtio-serial -device virtconsole,chardev=virtiocon0 -drive id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device virtio-blk,drive=test >> >> Average boot times after 10 consecutive runs: >> >> qemu_init_end: 77.467852 >> linux_start_kernel: 116.688472 (+39.22062) >> linux_start_user: 363.033365 (+246.344893) >> >> Memory tree: >> >> address-space: memory >> 0000000000000000-ffffffffffffffff (prio 0, i/o): system >> 0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g @pc.ram 0000000000000000-000000001fffffff >> 0000000000000000-ffffffffffffffff (prio -1, i/o): pci >> 00000000000c0000-00000000000dffff (prio 1, rom): pc.rom >> 00000000000e0000-00000000000fffff (prio 1, i/o): alias isa-bios @pc.bios 0000000000020000-000000000003ffff >> 00000000febf4000-00000000febf7fff (prio 1, i/o): virtio-pci >> 00000000febf4000-00000000febf4fff (prio 0, i/o): virtio-pci-common >> 00000000febf5000-00000000febf5fff (prio 0, i/o): virtio-pci-isr >> 00000000febf6000-00000000febf6fff (prio 0, i/o): virtio-pci-device >> 00000000febf7000-00000000febf7fff (prio 0, i/o): virtio-pci-notify >> 00000000febf8000-00000000febfbfff (prio 1, i/o): virtio-pci >> 00000000febf8000-00000000febf8fff (prio 0, i/o): virtio-pci-common >> 00000000febf9000-00000000febf9fff (prio 0, i/o): virtio-pci-isr >> 00000000febfa000-00000000febfafff (prio 0, i/o): virtio-pci-device >> 00000000febfb000-00000000febfbfff (prio 0, i/o): virtio-pci-notify >> 00000000febfe000-00000000febfefff (prio 1, i/o): virtio-serial-pci-msix >> 00000000febfe000-00000000febfe01f (prio 0, i/o): msix-table >> 00000000febfe800-00000000febfe807 (prio 0, i/o): msix-pba >> 00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix >> 00000000febff000-00000000febff01f (prio 0, i/o): msix-table >> 00000000febff800-00000000febff807 (prio 0, i/o): msix-pba >> 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios >> 00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff >> 00000000000c0000-00000000000c2fff (prio 1000, i/o): alias kvmvapic-rom @pc.ram 00000000000c0000-00000000000c2fff >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000c0000-00000000000c3fff [disabled] >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000c0000-00000000000c3fff [disabled] >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff >> 00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pci 00000000000c0000-00000000000c3fff [disabled] >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000c4000-00000000000c7fff [disabled] >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000c4000-00000000000c7fff [disabled] >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff >> 00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pci 00000000000c4000-00000000000c7fff [disabled] >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-ram @pc.ram 00000000000c8000-00000000000cbfff [disabled] >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pc.ram 00000000000c8000-00000000000cbfff [disabled] >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff >> 00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pci 00000000000c8000-00000000000cbfff [disabled] >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-ram @pc.ram 00000000000cc000-00000000000cffff [disabled] >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pc.ram 00000000000cc000-00000000000cffff [disabled] >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff >> 00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pci 00000000000cc000-00000000000cffff [disabled] >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000d0000-00000000000d3fff [disabled] >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000d0000-00000000000d3fff [disabled] >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff >> 00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pci 00000000000d0000-00000000000d3fff [disabled] >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000d4000-00000000000d7fff [disabled] >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000d4000-00000000000d7fff [disabled] >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff >> 00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pci 00000000000d4000-00000000000d7fff [disabled] >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-ram @pc.ram 00000000000d8000-00000000000dbfff [disabled] >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pc.ram 00000000000d8000-00000000000dbfff [disabled] >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff >> 00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pci 00000000000d8000-00000000000dbfff [disabled] >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-ram @pc.ram 00000000000dc000-00000000000dffff [disabled] >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pc.ram 00000000000dc000-00000000000dffff [disabled] >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff >> 00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pci 00000000000dc000-00000000000dffff [disabled] >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000e0000-00000000000e3fff [disabled] >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000e0000-00000000000e3fff [disabled] >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff >> 00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pci 00000000000e0000-00000000000e3fff [disabled] >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-ram @pc.ram 00000000000e4000-00000000000e7fff [disabled] >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pc.ram 00000000000e4000-00000000000e7fff [disabled] >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff >> 00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pci 00000000000e4000-00000000000e7fff [disabled] >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pc.ram 00000000000e8000-00000000000ebfff [disabled] >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-rom @pc.ram 00000000000e8000-00000000000ebfff [disabled] >> 00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pci 00000000000e8000-00000000000ebfff [disabled] >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-ram @pc.ram 00000000000ec000-00000000000effff >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pc.ram 00000000000ec000-00000000000effff [disabled] >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-rom @pc.ram 00000000000ec000-00000000000effff [disabled] >> 00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pci 00000000000ec000-00000000000effff [disabled] >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-ram @pc.ram 00000000000f0000-00000000000fffff [disabled] >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pc.ram 00000000000f0000-00000000000fffff [disabled] >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff >> 00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pci 00000000000f0000-00000000000fffff [disabled] >> 0000000020000000-0000000020000000 (prio 1, i/o): tseg-blackhole [disabled] >> 00000000b0000000-00000000bfffffff (prio 0, i/o): pcie-mmcfg-mmio >> 00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic >> 00000000fed1c000-00000000fed1ffff (prio 1, i/o): lpc-rcrb-mmio >> 00000000feda0000-00000000fedbffff (prio 1, i/o): alias smram-open-high @pc.ram 00000000000a0000-00000000000bffff [disabled] >> 00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi >> >> address-space: I/O >> 0000000000000000-000000000000ffff (prio 0, i/o): io >> 0000000000000000-0000000000000007 (prio 0, i/o): dma-chan >> 0000000000000008-000000000000000f (prio 0, i/o): dma-cont >> 0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic >> 0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit >> 0000000000000060-0000000000000060 (prio 0, i/o): i8042-data >> 0000000000000061-0000000000000061 (prio 0, i/o): pcspk >> 0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd >> 0000000000000070-0000000000000071 (prio 0, i/o): rtc >> 0000000000000070-0000000000000070 (prio 0, i/o): rtc-index >> 000000000000007e-000000000000007f (prio 0, i/o): kvmvapic >> 0000000000000080-0000000000000080 (prio 0, i/o): ioport80 >> 0000000000000081-0000000000000083 (prio 0, i/o): dma-page >> 0000000000000087-0000000000000087 (prio 0, i/o): dma-page >> 0000000000000089-000000000000008b (prio 0, i/o): dma-page >> 000000000000008f-000000000000008f (prio 0, i/o): dma-page >> 0000000000000092-0000000000000092 (prio 0, i/o): port92 >> 00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic >> 00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io >> 00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan >> 00000000000000d0-00000000000000df (prio 0, i/o): dma-cont >> 00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0 >> 00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr >> 00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr >> 0000000000000510-0000000000000511 (prio 0, i/o): fwcfg >> 0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma >> 0000000000000600-000000000000067f (prio 0, i/o): ich9-pm >> 0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt >> 0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt >> 0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr >> 0000000000000620-000000000000062f (prio 0, i/o): acpi-gpe0 >> 0000000000000630-0000000000000637 (prio 0, i/o): acpi-smi >> 0000000000000660-000000000000067f (prio 0, i/o): sm-tco >> 0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug >> 0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx >> 0000000000000cf9-0000000000000cf9 (prio 1, i/o): lpc-reset-control >> 0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data >> 000000000000c000-000000000000c07f (prio 1, i/o): virtio-pci >> 000000000000c080-000000000000c0bf (prio 1, i/o): virtio-pci >> >> >> ----------- >> | microvm | >> ----------- >> >> Command line: >> >> ./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M microvm -kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev pty,id=virtiocon0,server -device virtio-serial-device -device virtconsole,chardev=virtiocon0 -drive id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device virtio-blk-device,drive=test >> >> Average boot times after 10 consecutive runs: >> >> qemu_init_end: 64.043264 >> linux_start_kernel: 65.481782 (+1.438518) >> linux_start_user: 114.938353 (+49.456571) >> >> Memory tree: >> >> address-space: memory >> 0000000000000000-ffffffffffffffff (prio 0, i/o): system >> 0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g @microvm.ram 0000000000000000-000000001fffffff >> 00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio >> 00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio >> 00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio >> 00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio >> 00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic >> 00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi >> >> address-space: I/O >> 0000000000000000-000000000000ffff (prio 0, i/o): io >> 000000000000007e-000000000000007f (prio 0, i/o): kvmvapic >> >> >> -------------- >> | Conclusion | >> -------------- >> >> The average boot time of microvm is a third of Q35's (115ms vs. 363ms), >> and is smaller on all sections (QEMU initialization, firmware overhead >> and kernel start-to-user). >> >> Microvm's memory tree is also visibly simpler, significantly reducing >> the exposed surface to the guest. >> >> While we can certainly work on making Q35 smaller, I definitely think >> it's better (and way safer!) having a specialized machine type for a >> specific use case, than a minimal Q35 whose behavior significantly >> diverges from a conventional Q35. > > Interesting, so not a 10x difference! This might be amenable to > optimization. > > My concern with microvm is that it's so limited that few users will be > able to benefit from the reduced attack surface and faster startup time. > I think it's worth investigating slimming down Q35 further first. > > In terms of startup time the first step would be profiling Q35 kernel > startup to find out what's taking so long (firmware initialization, PCI > probing, etc)? Some findings: 1. Exposing the TSC_DEADLINE CPU flag (i.e. using "-cpu host") saves a whooping 120ms by avoiding the APIC timer calibration at arch/x86/kernel/apic/apic.c:calibrate_APIC_clock Average boot time with "-cpu host" qemu_init_end: 76.408950 linux_start_kernel: 116.166142 (+39.757192) linux_start_user: 242.954347 (+126.788205) Average boot time with default "cpu" qemu_init_end: 77.467852 linux_start_kernel: 116.688472 (+39.22062) linux_start_user: 363.033365 (+246.344893) 2. The other 130ms are a direct result of PCI and ACPI presence (tested with a kernel without support for those elements). I'll publish some detailed numbers next week. Sergio.