PVH dom0 creation fails - the system freezes

PVH dom0 creation fails - the system freezes

[bercarug]

[-- Attachment #1.1: Type: text/plain, Size: 3813 bytes --]

Hello,

For the last few days, I have been trying to get a PVH dom0 running,
however I encountered the following problem: the system seems to
freeze after the hypervisor boots, the screen goes black. I have tried to
debug it via a serial console (using Minicom) and managed to get some
more Xen output, after the screen turns black.

I mention that I have tried to boot the PVH dom0 using different kernel
images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).

Below I attached my system / hypervisor configuration, as well as the
output captured through the serial console, corresponding to the latest
versions for Xen and the Linux Kernel (Xen staging and Kernel from the
xen/tip tree).


OS + Distro: Linux / Debian 9 Stretch
Kernel Version: 4.17-rc5, tagged with for-linus-4.18-rc5-tag from the
xen/tip tree.
Xen Version: 4.12, commit id e3f667bc5f51d0aa44357a64ca134cd952679c81
of the Xen tree.
Host system: attached cpuinfo.log
Serial console output: attached boot.log
My grub configuration file, containing the Xen command line arguments: 
attached
grub.log

I can provide additional info as requested.
Any ideas why this happens? Do you have any recommendations for additional
debugging?

Here are the last few lines of the boot log. The last (separated) ones 
were only
visible though the serial console, since at that point the screen was 
completely
black.

(XEN) *** Building a PVH Dom0 ***
(XEN) [VT-D]d0:Hostbridge: skip 0000:00:00.0 map
(XEN) [VT-D]d0:PCI: map 0000:00:14.0
(XEN) [VT-D]d0:PCI: map 0000:00:14.2
(XEN) [VT-D]d0:PCI: map 0000:00:16.0
(XEN) [VT-D]d0:PCI: map 0000:00:16.1
(XEN) [VT-D]d0:PCI: map 0000:00:17.0
(XEN) [VT-D]d0:PCI: map 0000:00:1f.0
(XEN) [VT-D]d0:PCI: map 0000:00:1f.2
(XEN) [VT-D]d0:PCI: map 0000:00:1f.4
(XEN) [VT-D]d0:PCIe: map 0000:01:00.0
(XEN) [VT-D]d0:PCIe: map 0000:02:00.0
(XEN) [VT-D]d0:PCIe: map 0000:03:00.0
(XEN) [VT-D]d0:PCIe: map 0000:04:00.0
(XEN) [VT-D]iommu_enable_translation: iommu->reg = ffff82c00021b000
(XEN) WARNING: PVH is an experimental mode with limited functionality
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs
(XEN) 
...................................................................................................................................done.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) This option is intended to aid debugging of Xen by ensuring
(XEN) that all output is synchronously delivered on the serial line.
(XEN) However it can introduce SIGNIFICANT latencies and affect
(XEN) timekeeping. It is NOT recommended for production use!
(XEN) ***************************************************
(XEN) 3... 2... 1...

(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch 
input to Xen)
(XEN) Freed 468kB init memory
(XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 
8deb3000, iommu reg = ffff82c00021b000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
(XEN) root_entry[00] = 1021c60001
(XEN) context[a0] = 2_1021d6d001
(XEN) l4[000] = 9c00001021d6c107
(XEN) l3[002] = 9c00001021d3e107
(XEN) l2[06f] = 9c000010218c0107
(XEN) l1[0b3] = 8000000000000000
(XEN) l1[0b3] not present
(XEN) Dom0 callback via changed to Direct Vector 0xf3

Thanks,
Gabriel




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #1.2: Type: text/html, Size: 5455 bytes --]

[-- Attachment #2: boot.log --]
[-- Type: text/x-log, Size: 12019 bytes --]

(XEN) Xen version 4.12-unstable (root@) (gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516) debug=y  Mon Jul 16 09:20:26 EDT 2018
(XEN) Latest ChangeSet: Thu Jul 12 18:48:06 2018 +0200 git:e3f667bc5f
(XEN) Console output is synchronous.
(XEN) Bootloader: GRUB 2.02~beta3-5
(XEN) Command line: placeholder dom0=pvh dom0_mem=4096M loglvl=all sync_console console_to_ring=true console=com1,vga com1=115200,8n1 iommu=debug,verbose,workaround_bios_bug iommu_inclusive_mapping=true
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 0000000000098c00 (usable)
(XEN)  0000000000098c00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000008c1c4000 (usable)
(XEN)  000000008c1c4000 - 000000008c1c5000 (ACPI NVS)
(XEN)  000000008c1c5000 - 000000008c20f000 (reserved)
(XEN)  000000008c20f000 - 000000008c281000 (usable)
(XEN)  000000008c281000 - 000000008dec1000 (reserved)
(XEN)  000000008dec1000 - 000000008df9a000 (ACPI NVS)
(XEN)  000000008df9a000 - 000000008dfff000 (ACPI data)
(XEN)  000000008dfff000 - 000000008e000000 (usable)
(XEN)  000000008e000000 - 0000000090000000 (reserved)
(XEN)  0000000094000000 - 000000009a000000 (reserved)
(XEN)  000000009df00000 - 00000000a0000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fd000000 - 00000000fe800000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed01000 (reserved)
(XEN)  00000000fed10000 - 00000000fed1a000 (reserved)
(XEN)  00000000fed84000 - 00000000fed85000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff400000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001060000000 (usable)
(XEN) New Xen image base address: 0x8ba00000
(XEN) ACPI: RSDP 000F0510, 0024 (r2 INTEL )
(XEN) ACPI: XSDT 8DFB7188, 00EC (r1 INTEL  S1200SPO        0 INTL  1000013)
(XEN) ACPI: FACP 8DFF3000, 00F4 (r5 INTEL  S1200SPO        0 INTL 20091013)
(XEN) ACPI: DSDT 8DFC3000, 29241 (r2 INTEL  S1200SPO        0 INTL 20091013)
(XEN) ACPI: FACS 8DF6D000, 0040
(XEN) ACPI: HPET 8DFF2000, 0038 (r1 INTEL  S1200SPO        1 INTL 20091013)
(XEN) ACPI: APIC 8DFF1000, 00BC (r3 INTEL  S1200SPO        1 INTL 20091013)
(XEN) ACPI: MCFG 8DFF0000, 003C (r1 INTEL  S1200SPO        1 INTL 20091013)
(XEN) ACPI: SPMI 8DFEE000, 0042 (r5 INTEL  S1200SPO        0 INTL 20091013)
(XEN) ACPI: WDDT 8DFED000, 0040 (r1 INTEL  S1200SPO        0 INTL 20091013)
(XEN) ACPI: SSDT 8DFC0000, 2BAE (r2 INTEL  S1200SPO     1000 INTL 20091013)
(XEN) ACPI: SSDT 8DFBF000, 0BE3 (r2 INTEL  S1200SPO     1000 INTL 20091013)
(XEN) ACPI: SSDT 8DFBE000, 019A (r2 INTEL  S1200SPO     1000 INTL 20091013)
(XEN) ACPI: SSDT 8DFBD000, 04A3 (r2 INTEL  S1200SPO     1000 INTL 20091013)
(XEN) ACPI: TCPA 8DFFC000, 0064 (r2 INTEL  S1200SPO        2 INTL  1000013)
(XEN) ACPI: TPM2 8DFFA000, 0034 (r3 INTEL  S1200SPO        2 INTL  1000013)
(XEN) ACPI: SSDT 8DFF4000, 5328 (r2 INTEL  S1200SPO     3000 INTL 20141107)
(XEN) ACPI: SSDT 8DFBC000, 0E73 (r2 INTEL  S1200SPO     3000 INTL 20141107)
(XEN) ACPI: SSDT 8DFBA000, 0064 (r2 INTEL  S1200SPO        2 INTL 20141107)
(XEN) ACPI: DMAR 8DFB8000, 0070 (r1 INTEL  S1200SPO        1 INTL        1)
(XEN) ACPI: HEST 8DFFD000, 00A8 (r1 INTEL  S1200SPO        1 INTL        1)
(XEN) ACPI: ERST 8DFB5000, 0230 (r1 INTEL  S1200SPO        1 INTL        1)
(XEN) ACPI: SSDT 8DFFB000, 03A7 (r2 INTEL  S1200SPO     1000 INTL 20141107)
(XEN) ACPI: SSDT 8DFBB000, 0B79 (r2 INTEL  S1200SPO        2 INTL 20141107)
(XEN) ACPI: BERT 8DFB6000, 0030 (r1 INTEL  S1200SPO        1 INTL        1)
(XEN) ACPI: UEFI 8DF82000, 0042 (r1 INTEL  S1200SPO        2 INTL  1000013)
(XEN) ACPI: PRAD 8DFB9000, 0102 (r2 INTEL  S1200SPO        2 INTL 20141107)
(XEN) ACPI: EINJ 8DFB4000, 0130 (r1 INTEL  S1200SPO        1 INTL        1)
(XEN) ACPI: SPCR 8DFEF000, 0050 (r1 INTEL  S1200SPO        0 INTL 20091013)
(XEN) System RAM: 65217MB (66783036kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000001060000000
(XEN) Domain heap initialised
(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 158 (0x9e), Stepping 9 (raw 000906e9)
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x1808 (32 bits)
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1804,1:0], pm1x_evt[1:1800,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 8df6d000/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[8df6d00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [VT-D]Host address width 39
(XEN) [VT-D]found ACPI_DMAR_DRHD:
(XEN) [VT-D]  dmaru->address = fed90000
(XEN) [VT-D]drhd->address = fed90000 iommu->reg = ffff82c00021b000
(XEN) [VT-D]cap = d2008c40660462 ecap = f050da
(XEN) [VT-D] IOAPIC: 0000:f0:1f.0
(XEN) [VT-D] MSI HPET: 0000:00:1f.0
(XEN) [VT-D]  flags: INCLUDE_ALL
(XEN) [VT-D]found ACPI_DMAR_RMRR:
(XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved memory; need "iommu_inclusive_mapping=1"?
(XEN) [VT-D] endpoint: 0000:00:14.0
(XEN) [VT-D]dmar.c:638:   RMRR region: base_addr 3e2e0000 end_addr 3e2fffff
(XEN) Xen ERST support is initialized.
(XEN) HEST: Table parsing has been initialized
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 120 GSI, 1432 MSI/MSI-X
(XEN) Not enabling x2APIC (upon firmware request)
(XEN) xstate: size: 0x440 and states: 0x1f
(XEN) mce_intel.c:782: MCA Capability: firstbank 0, extended MCE MSR 0, BCAST, CMCI
(XEN) CPU0: Intel machine check reporting enabled
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features:
(XEN)   Compiled-in support: INDIRECT_THUNK
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: No, Other:
(XEN)   Support for VMs: PV: RSB EAGER_FPU, HVM: RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Platform timer is 23.999MHz HPET
(XEN) Detected 3792.189 MHz processor.
(XEN) Initing memory sharing.
(XEN) alt table ffff82d080465858 -> ffff82d08046759a
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) Allocated console ring of 64 KiB.
(XEN) mwait-idle: MWAIT substates: 0x142120
(XEN) mwait-idle: v0.4.1 model 0x9e
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN)  - VMCS shadowing
(XEN)  - VM Functions
(XEN)  - Virtualisation Exceptions
(XEN)  - Page Modification Logging
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 8 CPUs
(XEN) build-id: 217e72f0df3db05d72d6fa8b460b643bad6c230e
(XEN) Running stub recovery selftests...
(XEN) traps.c:1570: GPF (0000): ffff82d0bffff041 [ffff82d0bffff041] -> ffff82d08037b412
(XEN) traps.c:755: Trap 12: ffff82d0bffff040 [ffff82d0bffff040] -> ffff82d08037b412
(XEN) traps.c:1097: Trap 3: ffff82d0bffff041 [ffff82d0bffff041] -> ffff82d08037b412
(XEN) ACPI sleep modes: S3
(XEN) VPMU: disabled
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) Dom0 has maximum 888 PIRQs
(XEN) NX (Execute Disable) protection active
(XEN) *** Building a PVH Dom0 ***
(XEN) [VT-D]d0:Hostbridge: skip 0000:00:00.0 map
(XEN) [VT-D]d0:PCI: map 0000:00:14.0
(XEN) [VT-D]d0:PCI: map 0000:00:14.2
(XEN) [VT-D]d0:PCI: map 0000:00:16.0
(XEN) [VT-D]d0:PCI: map 0000:00:16.1
(XEN) [VT-D]d0:PCI: map 0000:00:17.0
(XEN) [VT-D]d0:PCI: map 0000:00:1f.0
(XEN) [VT-D]d0:PCI: map 0000:00:1f.2
(XEN) [VT-D]d0:PCI: map 0000:00:1f.4
(XEN) [VT-D]d0:PCIe: map 0000:01:00.0
(XEN) [VT-D]d0:PCIe: map 0000:02:00.0
(XEN) [VT-D]d0:PCIe: map 0000:03:00.0
(XEN) [VT-D]d0:PCIe: map 0000:04:00.0
(XEN) [VT-D]iommu_enable_translation: iommu->reg = ffff82c00021b000
(XEN) WARNING: PVH is an experimental mode with limited functionality
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs
(XEN) ...................................................................................................................................done.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) This option is intended to aid debugging of Xen by ensuring
(XEN) that all output is synchronously delivered on the serial line.
(XEN) However it can introduce SIGNIFICANT latencies and affect
(XEN) timekeeping. It is NOT recommended for production use!
(XEN) ***************************************************
(XEN) 3... 2... 1... 
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 468kB init memory
(XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
(XEN)     root_entry[00] = 1021c60001
(XEN)     context[a0] = 2_1021d6d001
(XEN)     l4[000] = 9c00001021d6c107
(XEN)     l3[002] = 9c00001021d3e107
(XEN)     l2[06f] = 9c000010218c0107
(XEN)     l1[0b3] = 8000000000000000
(XEN)     l1[0b3] not present
(XEN) Dom0 callback via changed to Direct Vector 0xf3

[-- Attachment #3: cpuinfo.log --]
[-- Type: text/x-log, Size: 10009 bytes --]

cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 800.031
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 799.716
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 800.285
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 800.068
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 801.970
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 800.277
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 800.104
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
stepping	: 9
microcode	: 0x5e
cpu MHz		: 800.171
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 7584.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

[-- Attachment #4: grub.log --]
[-- Type: text/x-log, Size: 1473 bytes --]

cat /etc/default/grub 
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="console=tty0,115200n8 console=tty1,115200n8 console=ttyS0,115200n8 console=ttyS1,115200n8"
GRUB_CMDLINE_XEN="dom0=pvh dom0_mem=4096M loglvl=all sync_console console_to_ring=true console=com1,vga com1=115200,8n1 iommu=debug,verbose,workaround_bios_bug iommu_inclusive_mapping=true"

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

[-- Attachment #5: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Jan Beulich]

>>> On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
> For the last few days, I have been trying to get a PVH dom0 running,
> however I encountered the following problem: the system seems to
> freeze after the hypervisor boots, the screen goes black. I have tried to
> debug it via a serial console (using Minicom) and managed to get some
> more Xen output, after the screen turns black.
> 
> I mention that I have tried to boot the PVH dom0 using different kernel
> images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).
> 
> Below I attached my system / hypervisor configuration, as well as the
> output captured through the serial console, corresponding to the latest
> versions for Xen and the Linux Kernel (Xen staging and Kernel from the
> xen/tip tree).
> [...]
> (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
> (XEN) root_entry[00] = 1021c60001
> (XEN) context[a0] = 2_1021d6d001
> (XEN) l4[000] = 9c00001021d6c107
> (XEN) l3[002] = 9c00001021d3e107
> (XEN) l2[06f] = 9c000010218c0107
> (XEN) l1[0b3] = 8000000000000000
> (XEN) l1[0b3] not present
> (XEN) Dom0 callback via changed to Direct Vector 0xf3

This might be a hint at a missing RMRR entry in the ACPI tables, as
we've seen to be the case for a number of systems (I dare to guess
that 0000:00:14.0 is a USB controller, perhaps one with a keyboard
and/or mouse connected). You may want to play with the respective
command line option ("rmrr="). Note that "iommu_inclusive_mapping"
as you're using it does not have any meaning for PVH (see
intel_iommu_hwdom_init()).

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

[-- Attachment #1: Type: text/plain, Size: 3510 bytes --]

On 07/24/2018 12:54 PM, Jan Beulich wrote:
>>>> On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
>> For the last few days, I have been trying to get a PVH dom0 running,
>> however I encountered the following problem: the system seems to
>> freeze after the hypervisor boots, the screen goes black. I have tried to
>> debug it via a serial console (using Minicom) and managed to get some
>> more Xen output, after the screen turns black.
>>
>> I mention that I have tried to boot the PVH dom0 using different kernel
>> images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).
>>
>> Below I attached my system / hypervisor configuration, as well as the
>> output captured through the serial console, corresponding to the latest
>> versions for Xen and the Linux Kernel (Xen staging and Kernel from the
>> xen/tip tree).
>> [...]
>> (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
>> (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000
>> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>> (XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
>> (XEN) root_entry[00] = 1021c60001
>> (XEN) context[a0] = 2_1021d6d001
>> (XEN) l4[000] = 9c00001021d6c107
>> (XEN) l3[002] = 9c00001021d3e107
>> (XEN) l2[06f] = 9c000010218c0107
>> (XEN) l1[0b3] = 8000000000000000
>> (XEN) l1[0b3] not present
>> (XEN) Dom0 callback via changed to Direct Vector 0xf3
> This might be a hint at a missing RMRR entry in the ACPI tables, as
> we've seen to be the case for a number of systems (I dare to guess
> that 0000:00:14.0 is a USB controller, perhaps one with a keyboard
> and/or mouse connected). You may want to play with the respective
> command line option ("rmrr="). Note that "iommu_inclusive_mapping"
> as you're using it does not have any meaning for PVH (see
> intel_iommu_hwdom_init()).
>
> Jan
>
>
>
Hello,

Following Roger's advice, I rebuilt Xen (4.12) using the staging branch 
and I managed to get a PVH dom0 starting. However, some other problems 
appeared:

1) The USB devices are not usable anymore (keyboard and mouse), so the 
system is only accessible through the serial port.

2) I can run any usual command in dom0, but the ones involving xl 
(except for xl info) will make the system run out of memory very fast. 
Eventually, when there is no more free memory available, the OOM killer 
begins removing processes until the system auto reboots.

I attached a file containing the output of a lsusb, as well as the 
output of xl info and xl list -l.
After xl list -l, the “free -m” commands show the available memory 
decreasing.
Each command has a timestamp appended, so it can be seen how fast the 
available memory decreases.

I removed much of the process killing logs and kept the last one, since 
they were following the same pattern.

Dom0 still appears to be of type PV (output of xl list -l), however 
during boot, the following messages were displayed: “Building a PVH 
Dom0” and “Booting paravirtualized kernel on Xen PVH”.

I mention that I had to add “workaround_bios_bug” in GRUB_CMDLINE_XEN 
for iommu to get dom0 running.

What could be causing the available memory loss problem?

Thank you,
Gabriel




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #2: lsusb.txt --]
[-- Type: text/plain, Size: 17884 bytes --]

lsusb && tim\b\b\bdate +%c
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Wed 25 Jul 2018 04:59:25 AM EDT
root@debian:/home/test# lsusb && date +%c\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\a\a\a\a\axl \binfo
host                   : debian
release                : 4.17.0-rc5
version                : #4 SMP Tue Jul 24 06:12:21 EDT 2018
machine                : x86_64
nr_cpus                : 8
max_cpu_id             : 7
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 3792.227
hw_caps                : bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 65217
free_memory            : 56242
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 12
xen_extra              : -unstable
xen_version            : 4.12-unstable
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Thu Jun 28 10:54:01 2018 +0300 git:61bdddb821
xen_commandline        : placeholder dom0=pvh dom0_mem=8192M loglvl=all sync_console console_to_ring=true console=com1,vga com1=115200,8n1 iommu=debug,verbose,workaround_bios_bug
cc_compiler            : gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
cc_compile_by          : root
cc_compile_domain      : 
cc_compile_date        : Tue Jul 24 04:03:02 EDT 2018
build_id               : e6d3e802a6420aae9e2e25dd5941c5d24adad026
xend_config_format     : 4
Wed 25 Jul 2018 04:59:38 AM EDT
root@debian:/home/test# xl info && date +%c\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\a\a\a\a\afree \b-m
              total        used        free      shared  buff/cache   available
Mem:           7977         179        7549           8         248        7560
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:44 AM EDT
root@debian:/home/test# free -m && date +%c\a\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\a\a\a\axl \blist \b-l
[
    {
        "domid": 0,
        "config": {
            "c_info": {
                "type": "pv",
                "name": "Domain-0"
            },
            "b_info": {
                "max_memkb": 17179869180,
                "target_memkb": 8387743,
                "sched_params": {
                    "sched": "credit",
                    "weight": 256,
                    "cap": 0
                },
                "type.pv": {

                },
                "arch_arm": {

                }
            }
        }
    }
]

Wed 25 Jul 2018 04:59:52 AM EDT
root@debian:/home/test# xl list -l && date +%c\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bfree -m
              total        used        free      shared  buff/cache   available
Mem:           7129         180        6701           8         248        6711
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:53 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           6441         180        6012           8         248        6023
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:54 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           5789         180        5360           8         248        5371
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:55 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           5007         181        4578           8         248        4589
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:56 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           4317         180        3888           8         248        3899
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:57 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           3603         181        3174           8         248        3184
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:57 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           2863         181        2434           8         248        2444
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:58 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           2169         181        1739           8         248        1750
Swap:         65120           0       65120
Wed 25 Jul 2018 04:59:59 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:           1495         182        1064           8         248        1075
Swap:         65120           0       65120
Wed 25 Jul 2018 05:00:00 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:            775         182         343           8         248         354
Swap:         65120           0       65120
Wed 25 Jul 2018 05:00:00 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:            293         151         117           1          24          46
Swap:         65120          57       65063
Wed 25 Jul 2018 05:00:01 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:            293         139         129           1          24           4
Swap:         65120          55       65065
Wed 25 Jul 2018 05:00:02 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:            293         139         126           1          26           2
Swap:         65120          55       65065
Wed 25 Jul 2018 05:00:03 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:            246         110         116           0          18          43
Swap:         65120          89       65031
Wed 25 Jul 2018 05:00:04 AM EDT
root@debian:/home/test# free -m && date +%c
              total        used        free      shared  buff/cache   available
Mem:            246         111         116           0          18          42
Swap:         65120          90       65030
Wed 25 Jul 2018 05:00:05 AM EDT
root@debian:/home/test# free -m && date +%c

[...]

[  255.133877] Out of memory: Kill process 971 (systemd-cgroups) score 0 or sacrifice child
[  255.142990] Killed process 971 (systemd-cgroups) total-vm:4804kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[  255.184192] systemd invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[  255.196535] systemd cpuset=/ mems_allowed=0
[  255.201282] CPU: 7 PID: 1 Comm: systemd Not tainted 4.17.0-rc5 #4
[  255.208161] Hardware name:  , BIOS  
[  255.212232] Call Trace:
[  255.215048]  dump_stack+0x5c/0x7b
[  255.218829]  dump_header+0x6b/0x28c
[  255.222801]  ? find_lock_task_mm+0x52/0x80
[  255.227457]  ? oom_unkillable_task+0x9b/0xc0
[  255.232304]  out_of_memory+0x328/0x480
[  255.236569]  __alloc_pages_slowpath+0xd25/0xe00
[  255.241707]  ? __do_page_cache_readahead+0x129/0x2e0
[  255.247329]  __alloc_pages_nodemask+0x212/0x250
[  255.252469]  filemap_fault+0x3a0/0x650
[  255.256737]  ? alloc_set_pte+0x39c/0x520
[  255.261195]  ? filemap_map_pages+0x182/0x330
[  255.266049]  ext4_filemap_fault+0x2c/0x40 [ext4]
[  255.271279]  __do_fault+0x1f/0xb3
[  255.275059]  __handle_mm_fault+0xbdf/0x1110
[  255.279809]  handle_mm_fault+0xfc/0x1f0
[  255.284171]  __do_page_fault+0x255/0x4f0
[  255.288632]  ? exit_to_usermode_loop+0xa3/0xc0
[  255.293672]  ? page_fault+0x8/0x30
[  255.297551]  page_fault+0x1e/0x30
[  255.301332] RIP: 0033:0x7f537ebdad50
[  255.305408] RSP: 002b:00007ffec5e9bdb8 EFLAGS: 00010202
[  255.311318] RAX: 0000000000000000 RBX: 00005601f8de19e0 RCX: 00007f537d85db00
[  255.319365] RDX: 00005601f8de19e0 RSI: 00007f537ecdd76c RDI: 00005601f8de19e0
[  255.327412] RBP: 00007f537ecdd76c R08: 00007f537d85dbb8 R09: 0000000000000060
[  255.335460] R10: 00007f537efb0940 R11: 0000000000000206 R12: 00007ffec5e9bde0
[  255.343508] R13: 00007ffec5e9bef0 R14: 0000000000000000 R15: 0000000000000009
[  255.351563] Mem-Info:
[  255.354178] active_anon:10 inactive_anon:4 isolated_anon:0
[  255.354178]  active_file:129 inactive_file:6 isolated_file:0
[  255.354178]  unevictable:0 dirty:0 writeback:0 unstable:0
[  255.354178]  slab_reclaimable:2925 slab_unreclaimable:4868
[  255.354178]  mapped:0 shmem:0 pagetables:65 bounce:0
[  255.354178]  free:26541 free_pcp:0 free_cma:0
[  255.389666] Node 0 active_anon:40kB inactive_anon:16kB active_file:516kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[  255.418748] Node 0 DMA free:15880kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15964kB managed:15880kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  255.447834] lowmem_reserve[]: 0 1889 7707 7707 7707
[  255.453359] Node 0 DMA32 free:39428kB min:16532kB low:20664kB high:24796kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2279640kB managed:44552kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  255.483415] lowmem_reserve[]: 0 0 5818 5818 5818
[  255.488651] Node 0 Normal free:50856kB min:50912kB low:63640kB high:76368kB active_anon:40kB inactive_anon:16kB active_file:708kB inactive_file:104kB unevictable:0kB writepending:0kB present:6092996kB managed:139648kB mlocked:0kB kernel_stack:2240kB pagetables:260kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  255.519966] lowmem_reserve[]: 0 0 0 0 0
[  255.524326] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15880kB
[  255.538966] Node 0 DMA32: 9*4kB (UM) 6*8kB (UM) 3*16kB (UM) 4*32kB (M) 6*64kB (M) 7*128kB (UM) 4*256kB (M) 6*512kB (M) 3*1024kB (M) 1*2048kB (U) 7*4096kB (UM) = 39428kB
[  255.555839] Node 0 Normal: 551*4kB (UME) 356*8kB (UME) 185*16kB (UME) 90*32kB (ME) 50*64kB (UME) 36*128kB (UME) 12*256kB (UM) 6*512kB (M) 8*1024kB (UM) 9*2048kB (M) 0*4096kB = 51468kB
[  255.574160] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[  255.583952] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  255.593453] 206 total pagecache pages
[  255.597628] 6 pages in swap cache
[  255.601404] Swap cache stats: add 127798, delete 127855, find 93647/162354
[  255.609164] Free swap  = 66670076kB
[  255.613138] Total swap = 66683900kB
[  255.617115] 2097150 pages RAM
[  255.620500] 0 pages HighMem/MovableOnly
[  255.624867] 2047130 pages reserved
[  255.628746] 0 pages hwpoisoned
[  255.632230] Unreclaimable slab info:
[  255.636308] Name                      Used          Total
[  255.642418] scsi_sense_cache          41KB         56KB
[  255.648329] ip6_dst_cache              3KB         15KB
[  255.654245] RAWv6                     27KB         31KB
[  255.660161] sgpool-128                 8KB          8KB
[  255.666072] cfq_io_cq                  7KB         19KB
[  255.671988] cfq_queue                 11KB         23KB
[  255.677901] mqueue_inode_cache          0KB          3KB
[  255.683910] dnotify_struct             0KB          3KB
[  255.689828] secpath_cache              0KB          8KB
[  255.695738] RAW                       30KB         30KB
[  255.701654] hugetlbfs_inode_cache          1KB          7KB
[  255.707955] eventpoll_pwq              4KB         23KB
[  255.713872] eventpoll_epi              7KB         32KB
[  255.719785] request_queue              4KB         12KB
[  255.725698] blkdev_ioc                 6KB         15KB
[  255.731613] biovec-max                96KB         96KB
[  255.737526] biovec-128                 4KB          4KB
[  255.743442] biovec-64                293KB        328KB
[  255.749357] dmaengine-unmap-256          2KB          6KB
[  255.755466] dmaengine-unmap-128          3KB         22KB
[  255.761570] dmaengine-unmap-16          6KB          7KB
[  255.767581] dmaengine-unmap-2          0KB          3KB
[  255.773497] skbuff_fclone_cache         51KB         84KB
[  255.779604] skbuff_head_cache         74KB        148KB
[  255.785518] net_namespace              6KB          6KB
[  255.791431] shmem_inode_cache        609KB        665KB
[  255.797348] taskstats                  3KB          3KB
[  255.803259] proc_dir_entry           176KB        192KB
[  255.809176] pde_opener                 0KB          3KB
[  255.815088] seq_file                   2KB          8KB
[  255.821002] sigqueue                   7KB         11KB
[  255.826915] kernfs_node_cache       2792KB       2812KB
[  255.832832] mnt_cache                 29KB         48KB
[  255.838745] filp                      83KB        352KB
[  255.844661] names_cache               56KB         56KB
[  255.850572] vm_area_struct           108KB        566KB
[  255.856486] mm_struct                 76KB         96KB
[  255.862404] files_cache               29KB         45KB
[  255.868314] signal_cache             203KB        232KB
[  255.874227] sighand_cache            406KB        420KB
[  255.880146] task_struct              655KB        655KB
[  255.886057] cred_jar                  57KB        165KB
[  255.891972] anon_vma                  17KB        105KB
[  255.897885] pid                       49KB        276KB
[  255.903798] Acpi-Operand             590KB        606KB
[  255.909715] Acpi-Parse                 4KB         15KB
[  255.915630] Acpi-State                 5KB         19KB
[  255.921545] Acpi-Namespace           221KB        228KB
[  255.927455] numa_policy                0KB          3KB
[  255.933370] trace_event_file         114KB        126KB
[  255.939284] ftrace_event_field        148KB        159KB
[  255.945297] pool_workqueue           115KB        328KB
[  255.951211] task_group                12KB         27KB
[  255.957124] kmalloc-2097152         2048KB       2048KB
[  255.963038] kmalloc-262144           768KB        768KB
[  255.968953] kmalloc-131072           128KB        128KB
[  255.974864] kmalloc-32768            288KB        288KB
[  255.980778] kmalloc-16384            384KB        384KB
[  255.986697] kmalloc-8192             712KB        712KB
[  255.992607] kmalloc-4096             596KB        600KB
[  255.998521] kmalloc-2048            1546KB       1592KB
[  256.004436] kmalloc-1024            1158KB       1216KB
[  256.010352] kmalloc-512              522KB        604KB
[  256.016264] kmalloc-256              125KB        132KB
[  256.022179] kmalloc-192              236KB        267KB
[  256.028092] kmalloc-96               178KB        252KB
[  256.034009] kmalloc-64               298KB        360KB
[  256.039924] kmalloc-32               411KB        430KB
[  256.045839] kmalloc-128              104KB        132KB
[  256.051749] kmem_cache                33KB         40KB
[  256.057665] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[  256.067170] [  273]     0   273    11680        1   122880      363         -1000 systemd-udevd
[  256.076962] Kernel panic - not syncing: Out of memory and no killable processes...
[  256.076962] 
[  256.087232] CPU: 7 PID: 1 Comm: systemd Not tainted 4.17.0-rc5 #4
[  256.094113] Hardware name:  , BIOS  
[  256.098185] Call Trace:
[  256.100999]  dump_stack+0x5c/0x7b
[  256.104780]  panic+0xe4/0x252
[  256.108173]  ? dump_header+0x189/0x28c
[  256.112437]  out_of_memory+0x334/0x480
[  256.116705]  __alloc_pages_slowpath+0xd25/0xe00
[  256.121843]  ? __do_page_cache_readahead+0x129/0x2e0
[  256.127465]  __alloc_pages_nodemask+0x212/0x250
[  256.132603]  filemap_fault+0x3a0/0x650
[  256.136871]  ? alloc_set_pte+0x39c/0x520
[  256.141330]  ? filemap_map_pages+0x182/0x330
[  256.146184]  ext4_filemap_fault+0x2c/0x40 [ext4]
[  256.151412]  __do_fault+0x1f/0xb3
[  256.155197]  __handle_mm_fault+0xbdf/0x1110
[  256.159948]  handle_mm_fault+0xfc/0x1f0
[  256.164306]  __do_page_fault+0x255/0x4f0
[  256.168769]  ? exit_to_usermode_loop+0xa3/0xc0
[  256.173807]  ? page_fault+0x8/0x30
[  256.177687]  page_fault+0x1e/0x30
[  256.181467] RIP: 0033:0x7f537ebdad50
[  256.185538] RSP: 002b:00007ffec5e9bdb8 EFLAGS: 00010202
[  256.191455] RAX: 0000000000000000 RBX: 00005601f8de19e0 RCX: 00007f537d85db00
[  256.199500] RDX: 00005601f8de19e0 RSI: 00007f537ecdd76c RDI: 00005601f8de19e0
[  256.207546] RBP: 00007f537ecdd76c R08: 00007f537d85dbb8 R09: 0000000000000060
[  256.215593] R10: 00007f537efb0940 R11: 0000000000000206 R12: 00007ffec5e9bde0
[  256.223642] R13: 00007ffec5e9bef0 R14: 0000000000000000 R15: 0000000000000009
[  256.231752] Kernel Offset: disabled
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Wei Liu]

On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com wrote:
> On 07/24/2018 12:54 PM, Jan Beulich wrote:
> > > > > On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
> > > For the last few days, I have been trying to get a PVH dom0 running,
> > > however I encountered the following problem: the system seems to
> > > freeze after the hypervisor boots, the screen goes black. I have tried to
> > > debug it via a serial console (using Minicom) and managed to get some
> > > more Xen output, after the screen turns black.
> > > 
> > > I mention that I have tried to boot the PVH dom0 using different kernel
> > > images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).
> > > 
> > > Below I attached my system / hypervisor configuration, as well as the
> > > output captured through the serial console, corresponding to the latest
> > > versions for Xen and the Linux Kernel (Xen staging and Kernel from the
> > > xen/tip tree).
> > > [...]
> > > (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
> > > (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
> > > (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000
> > > (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> > > (XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
> > > (XEN) root_entry[00] = 1021c60001
> > > (XEN) context[a0] = 2_1021d6d001
> > > (XEN) l4[000] = 9c00001021d6c107
> > > (XEN) l3[002] = 9c00001021d3e107
> > > (XEN) l2[06f] = 9c000010218c0107
> > > (XEN) l1[0b3] = 8000000000000000
> > > (XEN) l1[0b3] not present
> > > (XEN) Dom0 callback via changed to Direct Vector 0xf3
> > This might be a hint at a missing RMRR entry in the ACPI tables, as
> > we've seen to be the case for a number of systems (I dare to guess
> > that 0000:00:14.0 is a USB controller, perhaps one with a keyboard
> > and/or mouse connected). You may want to play with the respective
> > command line option ("rmrr="). Note that "iommu_inclusive_mapping"
> > as you're using it does not have any meaning for PVH (see
> > intel_iommu_hwdom_init()).
> > 
> > Jan
> > 
> > 
> > 
> Hello,
> 
> Following Roger's advice, I rebuilt Xen (4.12) using the staging branch and
> I managed to get a PVH dom0 starting. However, some other problems appeared:
> 
> 1) The USB devices are not usable anymore (keyboard and mouse), so the
> system is only accessible through the serial port.
> 
> 2) I can run any usual command in dom0, but the ones involving xl (except
> for xl info) will make the system run out of memory very fast. Eventually,
> when there is no more free memory available, the OOM killer begins removing
> processes until the system auto reboots.

Any chance you can run valgrind with xl?

I'm intrigued. by this PVH-only memory leak.  :-)

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 25/07/18 12:06, bercarug@amazon.com wrote:
> On 07/24/2018 12:54 PM, Jan Beulich wrote:
>>>>> On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
>>> For the last few days, I have been trying to get a PVH dom0 running,
>>> however I encountered the following problem: the system seems to
>>> freeze after the hypervisor boots, the screen goes black. I have
>>> tried to
>>> debug it via a serial console (using Minicom) and managed to get some
>>> more Xen output, after the screen turns black.
>>>
>>> I mention that I have tried to boot the PVH dom0 using different kernel
>>> images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11,
>>> 4.12).
>>>
>>> Below I attached my system / hypervisor configuration, as well as the
>>> output captured through the serial console, corresponding to the latest
>>> versions for Xen and the Linux Kernel (Xen staging and Kernel from the
>>> xen/tip tree).
>>> [...]
>>> (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
>>> (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr
>>> 8deb3000, iommu reg = ffff82c00021b000
>>> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>>> (XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
>>> (XEN) root_entry[00] = 1021c60001
>>> (XEN) context[a0] = 2_1021d6d001
>>> (XEN) l4[000] = 9c00001021d6c107
>>> (XEN) l3[002] = 9c00001021d3e107
>>> (XEN) l2[06f] = 9c000010218c0107
>>> (XEN) l1[0b3] = 8000000000000000
>>> (XEN) l1[0b3] not present
>>> (XEN) Dom0 callback via changed to Direct Vector 0xf3
>> This might be a hint at a missing RMRR entry in the ACPI tables, as
>> we've seen to be the case for a number of systems (I dare to guess
>> that 0000:00:14.0 is a USB controller, perhaps one with a keyboard
>> and/or mouse connected). You may want to play with the respective
>> command line option ("rmrr="). Note that "iommu_inclusive_mapping"
>> as you're using it does not have any meaning for PVH (see
>> intel_iommu_hwdom_init()).
>>
>> Jan
>>
>>
>>
> Hello,
> 
> Following Roger's advice, I rebuilt Xen (4.12) using the staging branch
> and I managed to get a PVH dom0 starting. However, some other problems
> appeared:
> 
> 1) The USB devices are not usable anymore (keyboard and mouse), so the
> system is only accessible through the serial port.
> 
> 2) I can run any usual command in dom0, but the ones involving xl
> (except for xl info) will make the system run out of memory very fast.
> Eventually, when there is no more free memory available, the OOM killer
> begins removing processes until the system auto reboots.
> 
> I attached a file containing the output of a lsusb, as well as the
> output of xl info and xl list -l.
> After xl list -l, the “free -m” commands show the available memory
> decreasing.
> Each command has a timestamp appended, so it can be seen how fast the
> available memory decreases.
> 
> I removed much of the process killing logs and kept the last one, since
> they were following the same pattern.
> 
> Dom0 still appears to be of type PV (output of xl list -l), however
> during boot, the following messages were displayed: “Building a PVH
> Dom0” and “Booting paravirtualized kernel on Xen PVH”.

This is a problem in xen-init-dom0, which will set "PV" for dom0
unconditionally.

> I mention that I had to add “workaround_bios_bug” in GRUB_CMDLINE_XEN
> for iommu to get dom0 running.
> 
> What could be causing the available memory loss problem?

Hmm, good question.

xl list is using domctl hypercalls, while xl info is using sysctl ones.
Can you test whether xl cpupool-list is causing memory loss, too? That's
another sysctl based command. As is xl dmesg.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com wrote:
> On 07/24/2018 12:54 PM, Jan Beulich wrote:
> > > > > On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
> > > For the last few days, I have been trying to get a PVH dom0 running,
> > > however I encountered the following problem: the system seems to
> > > freeze after the hypervisor boots, the screen goes black. I have tried to
> > > debug it via a serial console (using Minicom) and managed to get some
> > > more Xen output, after the screen turns black.
> > > 
> > > I mention that I have tried to boot the PVH dom0 using different kernel
> > > images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).
> > > 
> > > Below I attached my system / hypervisor configuration, as well as the
> > > output captured through the serial console, corresponding to the latest
> > > versions for Xen and the Linux Kernel (Xen staging and Kernel from the
> > > xen/tip tree).
> > > [...]
> > > (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
> > > (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
> > > (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000

Can you figure out which PCI device is 00:14.0?

> > > (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> > > (XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
> > > (XEN) root_entry[00] = 1021c60001
> > > (XEN) context[a0] = 2_1021d6d001
> > > (XEN) l4[000] = 9c00001021d6c107
> > > (XEN) l3[002] = 9c00001021d3e107
> > > (XEN) l2[06f] = 9c000010218c0107
> > > (XEN) l1[0b3] = 8000000000000000
> > > (XEN) l1[0b3] not present
> > > (XEN) Dom0 callback via changed to Direct Vector 0xf3
> > This might be a hint at a missing RMRR entry in the ACPI tables, as
> > we've seen to be the case for a number of systems (I dare to guess
> > that 0000:00:14.0 is a USB controller, perhaps one with a keyboard
> > and/or mouse connected). You may want to play with the respective
> > command line option ("rmrr="). Note that "iommu_inclusive_mapping"
> > as you're using it does not have any meaning for PVH (see
> > intel_iommu_hwdom_init()).
> > 
> > Jan
> > 
> > 
> > 
> Hello,
> 
> Following Roger's advice, I rebuilt Xen (4.12) using the staging branch and
> I managed to get a PVH dom0 starting. However, some other problems appeared:
> 
> 1) The USB devices are not usable anymore (keyboard and mouse), so the
> system is only accessible through the serial port.

Can you boot with iommu=debug and see if you get any extra IOMMU
information on the serial console?

> 2) I can run any usual command in dom0, but the ones involving xl (except
> for xl info) will make the system run out of memory very fast. Eventually,
> when there is no more free memory available, the OOM killer begins removing
> processes until the system auto reboots.
> 
> I attached a file containing the output of a lsusb, as well as the output of
> xl info and xl list -l.
> After xl list -l, the “free -m” commands show the available memory
> decreasing.
> Each command has a timestamp appended, so it can be seen how fast the
> available memory decreases.
> 
> I removed much of the process killing logs and kept the last one, since they
> were following the same pattern.
> 
> Dom0 still appears to be of type PV (output of xl list -l), however during
> boot, the following messages were displayed: “Building a PVH Dom0” and
> “Booting paravirtualized kernel on Xen PVH”.
> 
> I mention that I had to add “workaround_bios_bug” in GRUB_CMDLINE_XEN for
> iommu to get dom0 running.

It seems to me like your ACPI DMAR table contains errors, and I
wouldn't be surprised if those also cause the USB devices to
malfunction.

> 
> What could be causing the available memory loss problem?

That seems to be Linux aggressively ballooning out memory, you go from
7129M total memory to 246M. Are you creating a lot of domains?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 25/07/18 15:35, Roger Pau Monné wrote:
> On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com wrote:
>> On 07/24/2018 12:54 PM, Jan Beulich wrote:
>>>>>> On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
>>>> For the last few days, I have been trying to get a PVH dom0 running,
>>>> however I encountered the following problem: the system seems to
>>>> freeze after the hypervisor boots, the screen goes black. I have tried to
>>>> debug it via a serial console (using Minicom) and managed to get some
>>>> more Xen output, after the screen turns black.
>>>>
>>>> I mention that I have tried to boot the PVH dom0 using different kernel
>>>> images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).
>>>>
>>>> Below I attached my system / hypervisor configuration, as well as the
>>>> output captured through the serial console, corresponding to the latest
>>>> versions for Xen and the Linux Kernel (Xen staging and Kernel from the
>>>> xen/tip tree).
>>>> [...]
>>>> (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
>>>> (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
>>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000
> 
> Can you figure out which PCI device is 00:14.0?
> 
>>>> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>>>> (XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
>>>> (XEN) root_entry[00] = 1021c60001
>>>> (XEN) context[a0] = 2_1021d6d001
>>>> (XEN) l4[000] = 9c00001021d6c107
>>>> (XEN) l3[002] = 9c00001021d3e107
>>>> (XEN) l2[06f] = 9c000010218c0107
>>>> (XEN) l1[0b3] = 8000000000000000
>>>> (XEN) l1[0b3] not present
>>>> (XEN) Dom0 callback via changed to Direct Vector 0xf3
>>> This might be a hint at a missing RMRR entry in the ACPI tables, as
>>> we've seen to be the case for a number of systems (I dare to guess
>>> that 0000:00:14.0 is a USB controller, perhaps one with a keyboard
>>> and/or mouse connected). You may want to play with the respective
>>> command line option ("rmrr="). Note that "iommu_inclusive_mapping"
>>> as you're using it does not have any meaning for PVH (see
>>> intel_iommu_hwdom_init()).
>>>
>>> Jan
>>>
>>>
>>>
>> Hello,
>>
>> Following Roger's advice, I rebuilt Xen (4.12) using the staging branch and
>> I managed to get a PVH dom0 starting. However, some other problems appeared:
>>
>> 1) The USB devices are not usable anymore (keyboard and mouse), so the
>> system is only accessible through the serial port.
> 
> Can you boot with iommu=debug and see if you get any extra IOMMU
> information on the serial console?
> 
>> 2) I can run any usual command in dom0, but the ones involving xl (except
>> for xl info) will make the system run out of memory very fast. Eventually,
>> when there is no more free memory available, the OOM killer begins removing
>> processes until the system auto reboots.
>>
>> I attached a file containing the output of a lsusb, as well as the output of
>> xl info and xl list -l.
>> After xl list -l, the “free -m” commands show the available memory
>> decreasing.
>> Each command has a timestamp appended, so it can be seen how fast the
>> available memory decreases.
>>
>> I removed much of the process killing logs and kept the last one, since they
>> were following the same pattern.
>>
>> Dom0 still appears to be of type PV (output of xl list -l), however during
>> boot, the following messages were displayed: “Building a PVH Dom0” and
>> “Booting paravirtualized kernel on Xen PVH”.
>>
>> I mention that I had to add “workaround_bios_bug” in GRUB_CMDLINE_XEN for
>> iommu to get dom0 running.
> 
> It seems to me like your ACPI DMAR table contains errors, and I
> wouldn't be surprised if those also cause the USB devices to
> malfunction.
> 
>>
>> What could be causing the available memory loss problem?
> 
> That seems to be Linux aggressively ballooning out memory, you go from
> 7129M total memory to 246M. Are you creating a lot of domains?

This might be related to the tools thinking dom0 is a PV domain.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Wei Liu]

On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> On 25/07/18 15:35, Roger Pau Monné wrote:
> > 
> >>
> >> What could be causing the available memory loss problem?
> > 
> > That seems to be Linux aggressively ballooning out memory, you go from
> > 7129M total memory to 246M. Are you creating a lot of domains?
> 
> This might be related to the tools thinking dom0 is a PV domain.

Good point.

In that case, xenstore-ls -fp would also be useful. The output should
show the balloon target for Dom0.

You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
if it makes any difference.

Wei.

> 
> 
> Juergen
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

On 07/25/2018 05:02 PM, Wei Liu wrote:
> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>> What could be causing the available memory loss problem?
>>> That seems to be Linux aggressively ballooning out memory, you go from
>>> 7129M total memory to 246M. Are you creating a lot of domains?
>> This might be related to the tools thinking dom0 is a PV domain.
> Good point.
>
> In that case, xenstore-ls -fp would also be useful. The output should
> show the balloon target for Dom0.
>
> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
> if it makes any difference.
>
> Wei.
Also tried setting autoballooning off, but it had no effect.

Gabriel
>
>>
>> Juergen
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xenproject.org
>> https://lists.xenproject.org/mailman/listinfo/xen-devel





Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Wei Liu]

On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
> On 07/25/2018 05:02 PM, Wei Liu wrote:
> > On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> > > On 25/07/18 15:35, Roger Pau Monné wrote:
> > > > > What could be causing the available memory loss problem?
> > > > That seems to be Linux aggressively ballooning out memory, you go from
> > > > 7129M total memory to 246M. Are you creating a lot of domains?
> > > This might be related to the tools thinking dom0 is a PV domain.
> > Good point.
> > 
> > In that case, xenstore-ls -fp would also be useful. The output should
> > show the balloon target for Dom0.
> > 
> > You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
> > if it makes any difference.
> > 
> > Wei.
> Also tried setting autoballooning off, but it had no effect.

What does xenstore-ls -fp says? Is the target for Dom0 something
sensible? If yes, then the leak is not related to ballooning.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
> On 07/25/2018 05:02 PM, Wei Liu wrote:
> > On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> > > On 25/07/18 15:35, Roger Pau Monné wrote:
> > > > > What could be causing the available memory loss problem?
> > > > That seems to be Linux aggressively ballooning out memory, you go from
> > > > 7129M total memory to 246M. Are you creating a lot of domains?
> > > This might be related to the tools thinking dom0 is a PV domain.
> > Good point.
> > 
> > In that case, xenstore-ls -fp would also be useful. The output should
> > show the balloon target for Dom0.
> > 
> > You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
> > if it makes any difference.
> > 
> > Wei.
> Also tried setting autoballooning off, but it had no effect.

This is a Linux/libxl issue that I'm not sure what's the best way to
solve. Linux has the following 'workaround' in the balloon driver:

err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
		   &static_max);
if (err != 1)
	static_max = new_target;
else
	static_max >>= PAGE_SHIFT - 10;
target_diff = xen_pv_domain() ? 0
		: static_max - balloon_stats.target_pages;

I suppose this is used to cope with the memory reporting mismatch
usually seen on HVM guests. This however interacts quite badly on a
PVH Dom0 that has for example:

/local/domain/0/memory/target = "8391840"   (n0)
/local/domain/0/memory/static-max = "17179869180"   (n0)

One way to solve this is to set target and static-max to the same
value initially, so that target_diff on Linux is 0. Another option
would be to force target_diff = 0 for Dom0.

I'm attaching a patch for libxl that should solve this, could you
please give it a try and report back?

I'm still unsure however about the best way to fix this, need to think
about it.

Roger.
---8<---
diff --git a/tools/libxl/libxl_mem.c b/tools/libxl/libxl_mem.c
index e551e09fed..2c984993d8 100644
--- a/tools/libxl/libxl_mem.c
+++ b/tools/libxl/libxl_mem.c
@@ -151,7 +151,9 @@ retry_transaction:
         *target_memkb = info.current_memkb;
     }
     if (staticmax == NULL) {
-        libxl__xs_printf(gc, t, max_path, "%"PRIu64, info.max_memkb);
+        libxl__xs_printf(gc, t, max_path, "%"PRIu64,
+                         libxl__domain_type(gc, 0) == LIBXL_DOMAIN_TYPE_PV ?
+                         info.max_memkb : info.current_memkb);
         *max_memkb = info.max_memkb;
     }
 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 25/07/18 18:12, Roger Pau Monné wrote:
> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>> What could be causing the available memory loss problem?
>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>> This might be related to the tools thinking dom0 is a PV domain.
>>> Good point.
>>>
>>> In that case, xenstore-ls -fp would also be useful. The output should
>>> show the balloon target for Dom0.
>>>
>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>> if it makes any difference.
>>>
>>> Wei.
>> Also tried setting autoballooning off, but it had no effect.
> 
> This is a Linux/libxl issue that I'm not sure what's the best way to
> solve. Linux has the following 'workaround' in the balloon driver:
> 
> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
> 		   &static_max);
> if (err != 1)
> 	static_max = new_target;
> else
> 	static_max >>= PAGE_SHIFT - 10;
> target_diff = xen_pv_domain() ? 0
> 		: static_max - balloon_stats.target_pages;

Hmm, shouldn't PVH behave the same way as PV here? I don't think
there is memory missing for PVH, opposed to HVM's firmware memory.

Adding Boris for a second opinion.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Andrew Cooper]

On 25/07/18 17:29, Juergen Gross wrote:
> On 25/07/18 18:12, Roger Pau Monné wrote:
>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>> What could be causing the available memory loss problem?
>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>> Good point.
>>>>
>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>> show the balloon target for Dom0.
>>>>
>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>>> if it makes any difference.
>>>>
>>>> Wei.
>>> Also tried setting autoballooning off, but it had no effect.
>> This is a Linux/libxl issue that I'm not sure what's the best way to
>> solve. Linux has the following 'workaround' in the balloon driver:
>>
>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>> 		   &static_max);
>> if (err != 1)
>> 	static_max = new_target;
>> else
>> 	static_max >>= PAGE_SHIFT - 10;
>> target_diff = xen_pv_domain() ? 0
>> 		: static_max - balloon_stats.target_pages;
> Hmm, shouldn't PVH behave the same way as PV here? I don't think
> there is memory missing for PVH, opposed to HVM's firmware memory.
>
> Adding Boris for a second opinion.

/sigh

<rant>

Ballooning, and guest memory accounting is a known, growing, clustermess
of swamps.  The ballooning protocol itself is sufficiently broken as to
be useless outside of contrived scenarios, owing to the lack of any
ability to nack the request and the guest not knowing or being able to
work out how much RAM it actually has.

The Xen/toolstack/qemu-{trad,upstream}/hvmloader guessathon contributes
to lots of corner cases where things explode spectacularly on migration,
such as having more than 4 network cards, or having vram != 64M, or
generally anything involving PCI Passthrough.

Can we take this hint that maybe its time to try fixing the problem
properly rather than applying even more duct tape?  I'd like to remind
people that there is a design which has been discussed at various
conferences in the past, not not overly objected to.

</rant>

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Boris Ostrovsky]

On 07/25/2018 02:56 PM, Andrew Cooper wrote:
> On 25/07/18 17:29, Juergen Gross wrote:
>> On 25/07/18 18:12, Roger Pau Monné wrote:
>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>>> What could be causing the available memory loss problem?
>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>>> Good point.
>>>>>
>>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>>> show the balloon target for Dom0.
>>>>>
>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>>>> if it makes any difference.
>>>>>
>>>>> Wei.
>>>> Also tried setting autoballooning off, but it had no effect.
>>> This is a Linux/libxl issue that I'm not sure what's the best way to
>>> solve. Linux has the following 'workaround' in the balloon driver:
>>>
>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>>> 		   &static_max);
>>> if (err != 1)
>>> 	static_max = new_target;
>>> else
>>> 	static_max >>= PAGE_SHIFT - 10;
>>> target_diff = xen_pv_domain() ? 0
>>> 		: static_max - balloon_stats.target_pages;
>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
>> there is memory missing for PVH, opposed to HVM's firmware memory.
>>
>> Adding Boris for a second opinion.

(Notwithstanding Andrews' rant below ;-))

I am trying to remember --- what memory were we trying not to online for
HVM here?


-boris


> /sigh
>
> <rant>
>
> Ballooning, and guest memory accounting is a known, growing, clustermess
> of swamps.  The ballooning protocol itself is sufficiently broken as to
> be useless outside of contrived scenarios, owing to the lack of any
> ability to nack the request and the guest not knowing or being able to
> work out how much RAM it actually has.
>
> The Xen/toolstack/qemu-{trad,upstream}/hvmloader guessathon contributes
> to lots of corner cases where things explode spectacularly on migration,
> such as having more than 4 network cards, or having vram != 64M, or
> generally anything involving PCI Passthrough.
>
> Can we take this hint that maybe its time to try fixing the problem
> properly rather than applying even more duct tape?  I'd like to remind
> people that there is a design which has been discussed at various
> conferences in the past, not not overly objected to.
>
> </rant>
>
> ~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 26/07/18 01:07, Boris Ostrovsky wrote:
> On 07/25/2018 02:56 PM, Andrew Cooper wrote:
>> On 25/07/18 17:29, Juergen Gross wrote:
>>> On 25/07/18 18:12, Roger Pau Monné wrote:
>>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>>>> What could be causing the available memory loss problem?
>>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>>>> Good point.
>>>>>>
>>>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>>>> show the balloon target for Dom0.
>>>>>>
>>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>>>>> if it makes any difference.
>>>>>>
>>>>>> Wei.
>>>>> Also tried setting autoballooning off, but it had no effect.
>>>> This is a Linux/libxl issue that I'm not sure what's the best way to
>>>> solve. Linux has the following 'workaround' in the balloon driver:
>>>>
>>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>>>> 		   &static_max);
>>>> if (err != 1)
>>>> 	static_max = new_target;
>>>> else
>>>> 	static_max >>= PAGE_SHIFT - 10;
>>>> target_diff = xen_pv_domain() ? 0
>>>> 		: static_max - balloon_stats.target_pages;
>>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
>>> there is memory missing for PVH, opposed to HVM's firmware memory.
>>>
>>> Adding Boris for a second opinion.
> 
> (Notwithstanding Andrews' rant below ;-))
> 
> I am trying to remember --- what memory were we trying not to online for
> HVM here?

The problem was to not online memory automatically which was added later
via hotplug (especially NVDIMMs). This happened as the reported memory
sizes via E820 map and hypervisor differed due to HVM's firmware memory.
So the initial target size was larger than the initial memory size
reported via E820 map. This led to automatic ballooning up when a NVDIMM
was added to the HVM guest.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[George Dunlap]

On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
<boris.ostrovsky@oracle.com> wrote:
> On 07/25/2018 02:56 PM, Andrew Cooper wrote:
>> On 25/07/18 17:29, Juergen Gross wrote:
>>> On 25/07/18 18:12, Roger Pau Monné wrote:
>>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>>>> What could be causing the available memory loss problem?
>>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>>>> Good point.
>>>>>>
>>>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>>>> show the balloon target for Dom0.
>>>>>>
>>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>>>>> if it makes any difference.
>>>>>>
>>>>>> Wei.
>>>>> Also tried setting autoballooning off, but it had no effect.
>>>> This is a Linux/libxl issue that I'm not sure what's the best way to
>>>> solve. Linux has the following 'workaround' in the balloon driver:
>>>>
>>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>>>>                &static_max);
>>>> if (err != 1)
>>>>     static_max = new_target;
>>>> else
>>>>     static_max >>= PAGE_SHIFT - 10;
>>>> target_diff = xen_pv_domain() ? 0
>>>>             : static_max - balloon_stats.target_pages;
>>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
>>> there is memory missing for PVH, opposed to HVM's firmware memory.
>>>
>>> Adding Boris for a second opinion.
>
> (Notwithstanding Andrews' rant below ;-))
>
> I am trying to remember --- what memory were we trying not to online for
> HVM here?

My general memory of the situation is this:

* Balloon drivers are told to reach a "target" value for max_pages.
* max_pages includes all memory assigned to the guest, including video
ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
devices, and so on.
* Unfortunately, the balloon driver doesn't know what their max_pages
value is and can't read it.
* So what the balloon drivers do at the moment (as I understand it) is
look at the memory *reported as RAM*, and do a calculation:
  visible_ram - target_max_pages = pages_in_balloon

You can probably see why this won't work -- the result is that the
guest balloons down to (target_max_pages + non_ram_pages).  This is
kind of messy for normal guests, but when you have a
populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
in the guest.  The hypervisor then spends a huge amount of work
swapping the PoD pages around under the guest's feet, until it can't
find any more zeroed guest pages to use, and it crashes the guest.

The kludge we have right now is to make up a number for HVM guests
which is slightly larger than non_ram_pages, and tell the guest to aim
for *that* instead.

I think what we need is for the *toolstack* to calculate the size of
the balloon rather than the guest, and tell the balloon driver how big
to make its balloon, rather than the balloon driver trying to figure
that out on its own.

We also need to get a handle on making the allocation and tracking of
all the random "non-RAM" pages allocated to a guest; but that's a
slightly different region of the swamp.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Thu, Jul 26, 2018 at 10:45:08AM +0100, George Dunlap wrote:
> On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
> <boris.ostrovsky@oracle.com> wrote:
> > On 07/25/2018 02:56 PM, Andrew Cooper wrote:
> >> On 25/07/18 17:29, Juergen Gross wrote:
> >>> On 25/07/18 18:12, Roger Pau Monné wrote:
> >>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
> >>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
> >>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> >>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
> >>>>>>>>> What could be causing the available memory loss problem?
> >>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
> >>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
> >>>>>>> This might be related to the tools thinking dom0 is a PV domain.
> >>>>>> Good point.
> >>>>>>
> >>>>>> In that case, xenstore-ls -fp would also be useful. The output should
> >>>>>> show the balloon target for Dom0.
> >>>>>>
> >>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
> >>>>>> if it makes any difference.
> >>>>>>
> >>>>>> Wei.
> >>>>> Also tried setting autoballooning off, but it had no effect.
> >>>> This is a Linux/libxl issue that I'm not sure what's the best way to
> >>>> solve. Linux has the following 'workaround' in the balloon driver:
> >>>>
> >>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
> >>>>                &static_max);
> >>>> if (err != 1)
> >>>>     static_max = new_target;
> >>>> else
> >>>>     static_max >>= PAGE_SHIFT - 10;
> >>>> target_diff = xen_pv_domain() ? 0
> >>>>             : static_max - balloon_stats.target_pages;
> >>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
> >>> there is memory missing for PVH, opposed to HVM's firmware memory.
> >>>
> >>> Adding Boris for a second opinion.
> >
> > (Notwithstanding Andrews' rant below ;-))
> >
> > I am trying to remember --- what memory were we trying not to online for
> > HVM here?
> 
> My general memory of the situation is this:
> 
> * Balloon drivers are told to reach a "target" value for max_pages.
> * max_pages includes all memory assigned to the guest, including video
> ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
> devices, and so on.
> * Unfortunately, the balloon driver doesn't know what their max_pages
> value is and can't read it.
> * So what the balloon drivers do at the moment (as I understand it) is
> look at the memory *reported as RAM*, and do a calculation:
>   visible_ram - target_max_pages = pages_in_balloon
> 
> You can probably see why this won't work -- the result is that the
> guest balloons down to (target_max_pages + non_ram_pages).  This is
> kind of messy for normal guests, but when you have a
> populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
> in the guest.  The hypervisor then spends a huge amount of work
> swapping the PoD pages around under the guest's feet, until it can't
> find any more zeroed guest pages to use, and it crashes the guest.
> 
> The kludge we have right now is to make up a number for HVM guests
> which is slightly larger than non_ram_pages, and tell the guest to aim
> for *that* instead.
> 
> I think what we need is for the *toolstack* to calculate the size of
> the balloon rather than the guest, and tell the balloon driver how big
> to make its balloon, rather than the balloon driver trying to figure
> that out on its own.

Maybe the best option would be for the toolstack to fetch the e820
memory map and set the target based on the size of the RAM regions in
there for PVH Dom0? That would certainly match the expectations of the
guest.

Note that for DomUs if hvmloader (or any other component) inside of
the guest changes the memory map it would also have to adjust the
value in the xenstore 'target' node.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 26/07/18 13:11, Roger Pau Monné wrote:
> On Thu, Jul 26, 2018 at 10:45:08AM +0100, George Dunlap wrote:
>> On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
>> <boris.ostrovsky@oracle.com> wrote:
>>> On 07/25/2018 02:56 PM, Andrew Cooper wrote:
>>>> On 25/07/18 17:29, Juergen Gross wrote:
>>>>> On 25/07/18 18:12, Roger Pau Monné wrote:
>>>>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>>>>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>>>>>> What could be causing the available memory loss problem?
>>>>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>>>>>> Good point.
>>>>>>>>
>>>>>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>>>>>> show the balloon target for Dom0.
>>>>>>>>
>>>>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>>>>>>> if it makes any difference.
>>>>>>>>
>>>>>>>> Wei.
>>>>>>> Also tried setting autoballooning off, but it had no effect.
>>>>>> This is a Linux/libxl issue that I'm not sure what's the best way to
>>>>>> solve. Linux has the following 'workaround' in the balloon driver:
>>>>>>
>>>>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>>>>>>                &static_max);
>>>>>> if (err != 1)
>>>>>>     static_max = new_target;
>>>>>> else
>>>>>>     static_max >>= PAGE_SHIFT - 10;
>>>>>> target_diff = xen_pv_domain() ? 0
>>>>>>             : static_max - balloon_stats.target_pages;
>>>>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
>>>>> there is memory missing for PVH, opposed to HVM's firmware memory.
>>>>>
>>>>> Adding Boris for a second opinion.
>>>
>>> (Notwithstanding Andrews' rant below ;-))
>>>
>>> I am trying to remember --- what memory were we trying not to online for
>>> HVM here?
>>
>> My general memory of the situation is this:
>>
>> * Balloon drivers are told to reach a "target" value for max_pages.
>> * max_pages includes all memory assigned to the guest, including video
>> ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
>> devices, and so on.
>> * Unfortunately, the balloon driver doesn't know what their max_pages
>> value is and can't read it.
>> * So what the balloon drivers do at the moment (as I understand it) is
>> look at the memory *reported as RAM*, and do a calculation:
>>   visible_ram - target_max_pages = pages_in_balloon
>>
>> You can probably see why this won't work -- the result is that the
>> guest balloons down to (target_max_pages + non_ram_pages).  This is
>> kind of messy for normal guests, but when you have a
>> populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
>> in the guest.  The hypervisor then spends a huge amount of work
>> swapping the PoD pages around under the guest's feet, until it can't
>> find any more zeroed guest pages to use, and it crashes the guest.
>>
>> The kludge we have right now is to make up a number for HVM guests
>> which is slightly larger than non_ram_pages, and tell the guest to aim
>> for *that* instead.
>>
>> I think what we need is for the *toolstack* to calculate the size of
>> the balloon rather than the guest, and tell the balloon driver how big
>> to make its balloon, rather than the balloon driver trying to figure
>> that out on its own.
> 
> Maybe the best option would be for the toolstack to fetch the e820
> memory map and set the target based on the size of the RAM regions in
> there for PVH Dom0? That would certainly match the expectations of the
> guest.
> 
> Note that for DomUs if hvmloader (or any other component) inside of
> the guest changes the memory map it would also have to adjust the
> value in the xenstore 'target' node.

How would it do that later when the guest is already running?

I believe the right way would be to design a proper ballooning interface
suitable for all kinds of guests from scratch. This should include how
to deal with hotplug of memory or booting with mem < mem_max. Whether
PoD should be included should be discussed, too.

After defining that interface we can look after a proper way to select
the correct interface (old or new) in the gust and how to communicate
that selection to the host.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[George Dunlap]

On Thu, Jul 26, 2018 at 12:22 PM, Juergen Gross <jgross@suse.com> wrote:
> I believe the right way would be to design a proper ballooning interface
> suitable for all kinds of guests from scratch. This should include how
> to deal with hotplug of memory or booting with mem < mem_max. Whether
> PoD should be included should be discussed, too.

Juergen, what do you think of this interface:  The toolstack looks at
all the available information about the guest and determines what size
the guest's balloon needs to be.  It then writes
memory/target_balloon_size.  The guest balloon driver simply attempts
to allocate / free memory to make the balloon that size.

We'll have to leave memory/target handling as is until all old
versions of the balloon driver disappear.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 26/07/18 13:27, George Dunlap wrote:
> On Thu, Jul 26, 2018 at 12:22 PM, Juergen Gross <jgross@suse.com> wrote:
>> I believe the right way would be to design a proper ballooning interface
>> suitable for all kinds of guests from scratch. This should include how
>> to deal with hotplug of memory or booting with mem < mem_max. Whether
>> PoD should be included should be discussed, too.
> 
> Juergen, what do you think of this interface:  The toolstack looks at
> all the available information about the guest and determines what size
> the guest's balloon needs to be.  It then writes
> memory/target_balloon_size.  The guest balloon driver simply attempts
> to allocate / free memory to make the balloon that size.

This should be per numa node. So memory/node<n>/target-balloon-size
(I don't like the underscores in the node name).

When adding node specific memory hotplug this scheme could be used
for that purpose, too.

So any memory hotplugged will immediately be used by the guest.

That's just stating a fact, no criticism.

> We'll have to leave memory/target handling as is until all old
> versions of the balloon driver disappear.

Sure.

I'd like the guest to write xenstore node
control/feature-per-node-memory to indicate it will use the new
node(s) instead of the memory/target node.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[George Dunlap]

On Thu, Jul 26, 2018 at 1:19 PM, Juergen Gross <jgross@suse.com> wrote:
> On 26/07/18 13:27, George Dunlap wrote:
>> On Thu, Jul 26, 2018 at 12:22 PM, Juergen Gross <jgross@suse.com> wrote:
>>> I believe the right way would be to design a proper ballooning interface
>>> suitable for all kinds of guests from scratch. This should include how
>>> to deal with hotplug of memory or booting with mem < mem_max. Whether
>>> PoD should be included should be discussed, too.
>>
>> Juergen, what do you think of this interface:  The toolstack looks at
>> all the available information about the guest and determines what size
>> the guest's balloon needs to be.  It then writes
>> memory/target_balloon_size.  The guest balloon driver simply attempts
>> to allocate / free memory to make the balloon that size.
>
> This should be per numa node. So memory/node<n>/target-balloon-size
> (I don't like the underscores in the node name).

This should be vnode then, to emphasize that it's a node from the
*guest's* perspective, not the host's perspective.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Thu, Jul 26, 2018 at 01:22:33PM +0200, Juergen Gross wrote:
> On 26/07/18 13:11, Roger Pau Monné wrote:
> > On Thu, Jul 26, 2018 at 10:45:08AM +0100, George Dunlap wrote:
> >> On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
> >> <boris.ostrovsky@oracle.com> wrote:
> >>> On 07/25/2018 02:56 PM, Andrew Cooper wrote:
> >>>> On 25/07/18 17:29, Juergen Gross wrote:
> >>>>> On 25/07/18 18:12, Roger Pau Monné wrote:
> >>>>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
> >>>>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
> >>>>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> >>>>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
> >>>>>>>>>>> What could be causing the available memory loss problem?
> >>>>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
> >>>>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
> >>>>>>>>> This might be related to the tools thinking dom0 is a PV domain.
> >>>>>>>> Good point.
> >>>>>>>>
> >>>>>>>> In that case, xenstore-ls -fp would also be useful. The output should
> >>>>>>>> show the balloon target for Dom0.
> >>>>>>>>
> >>>>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
> >>>>>>>> if it makes any difference.
> >>>>>>>>
> >>>>>>>> Wei.
> >>>>>>> Also tried setting autoballooning off, but it had no effect.
> >>>>>> This is a Linux/libxl issue that I'm not sure what's the best way to
> >>>>>> solve. Linux has the following 'workaround' in the balloon driver:
> >>>>>>
> >>>>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
> >>>>>>                &static_max);
> >>>>>> if (err != 1)
> >>>>>>     static_max = new_target;
> >>>>>> else
> >>>>>>     static_max >>= PAGE_SHIFT - 10;
> >>>>>> target_diff = xen_pv_domain() ? 0
> >>>>>>             : static_max - balloon_stats.target_pages;
> >>>>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
> >>>>> there is memory missing for PVH, opposed to HVM's firmware memory.
> >>>>>
> >>>>> Adding Boris for a second opinion.
> >>>
> >>> (Notwithstanding Andrews' rant below ;-))
> >>>
> >>> I am trying to remember --- what memory were we trying not to online for
> >>> HVM here?
> >>
> >> My general memory of the situation is this:
> >>
> >> * Balloon drivers are told to reach a "target" value for max_pages.
> >> * max_pages includes all memory assigned to the guest, including video
> >> ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
> >> devices, and so on.
> >> * Unfortunately, the balloon driver doesn't know what their max_pages
> >> value is and can't read it.
> >> * So what the balloon drivers do at the moment (as I understand it) is
> >> look at the memory *reported as RAM*, and do a calculation:
> >>   visible_ram - target_max_pages = pages_in_balloon
> >>
> >> You can probably see why this won't work -- the result is that the
> >> guest balloons down to (target_max_pages + non_ram_pages).  This is
> >> kind of messy for normal guests, but when you have a
> >> populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
> >> in the guest.  The hypervisor then spends a huge amount of work
> >> swapping the PoD pages around under the guest's feet, until it can't
> >> find any more zeroed guest pages to use, and it crashes the guest.
> >>
> >> The kludge we have right now is to make up a number for HVM guests
> >> which is slightly larger than non_ram_pages, and tell the guest to aim
> >> for *that* instead.
> >>
> >> I think what we need is for the *toolstack* to calculate the size of
> >> the balloon rather than the guest, and tell the balloon driver how big
> >> to make its balloon, rather than the balloon driver trying to figure
> >> that out on its own.
> > 
> > Maybe the best option would be for the toolstack to fetch the e820
> > memory map and set the target based on the size of the RAM regions in
> > there for PVH Dom0? That would certainly match the expectations of the
> > guest.
> > 
> > Note that for DomUs if hvmloader (or any other component) inside of
> > the guest changes the memory map it would also have to adjust the
> > value in the xenstore 'target' node.
> 
> How would it do that later when the guest is already running?

hvmloader should modify the 'target' xenstore node if it changes the
memory map.

So the value provided by the toolstack would match the amount of RAM
in the memory map up to the point where the guest is started, from
there on anything inside the guest changing the memory map should also
change the xenstore value.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 26/07/18 15:50, Roger Pau Monné wrote:
> On Thu, Jul 26, 2018 at 01:22:33PM +0200, Juergen Gross wrote:
>> On 26/07/18 13:11, Roger Pau Monné wrote:
>>> On Thu, Jul 26, 2018 at 10:45:08AM +0100, George Dunlap wrote:
>>>> On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
>>>> <boris.ostrovsky@oracle.com> wrote:
>>>>> On 07/25/2018 02:56 PM, Andrew Cooper wrote:
>>>>>> On 25/07/18 17:29, Juergen Gross wrote:
>>>>>>> On 25/07/18 18:12, Roger Pau Monné wrote:
>>>>>>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>>>>>>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>>>>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>>>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>>>>>>>> What could be causing the available memory loss problem?
>>>>>>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>>>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>>>>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>>>>>>>> Good point.
>>>>>>>>>>
>>>>>>>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>>>>>>>> show the balloon target for Dom0.
>>>>>>>>>>
>>>>>>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>>>>>>>>> if it makes any difference.
>>>>>>>>>>
>>>>>>>>>> Wei.
>>>>>>>>> Also tried setting autoballooning off, but it had no effect.
>>>>>>>> This is a Linux/libxl issue that I'm not sure what's the best way to
>>>>>>>> solve. Linux has the following 'workaround' in the balloon driver:
>>>>>>>>
>>>>>>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>>>>>>>>                &static_max);
>>>>>>>> if (err != 1)
>>>>>>>>     static_max = new_target;
>>>>>>>> else
>>>>>>>>     static_max >>= PAGE_SHIFT - 10;
>>>>>>>> target_diff = xen_pv_domain() ? 0
>>>>>>>>             : static_max - balloon_stats.target_pages;
>>>>>>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
>>>>>>> there is memory missing for PVH, opposed to HVM's firmware memory.
>>>>>>>
>>>>>>> Adding Boris for a second opinion.
>>>>>
>>>>> (Notwithstanding Andrews' rant below ;-))
>>>>>
>>>>> I am trying to remember --- what memory were we trying not to online for
>>>>> HVM here?
>>>>
>>>> My general memory of the situation is this:
>>>>
>>>> * Balloon drivers are told to reach a "target" value for max_pages.
>>>> * max_pages includes all memory assigned to the guest, including video
>>>> ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
>>>> devices, and so on.
>>>> * Unfortunately, the balloon driver doesn't know what their max_pages
>>>> value is and can't read it.
>>>> * So what the balloon drivers do at the moment (as I understand it) is
>>>> look at the memory *reported as RAM*, and do a calculation:
>>>>   visible_ram - target_max_pages = pages_in_balloon
>>>>
>>>> You can probably see why this won't work -- the result is that the
>>>> guest balloons down to (target_max_pages + non_ram_pages).  This is
>>>> kind of messy for normal guests, but when you have a
>>>> populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
>>>> in the guest.  The hypervisor then spends a huge amount of work
>>>> swapping the PoD pages around under the guest's feet, until it can't
>>>> find any more zeroed guest pages to use, and it crashes the guest.
>>>>
>>>> The kludge we have right now is to make up a number for HVM guests
>>>> which is slightly larger than non_ram_pages, and tell the guest to aim
>>>> for *that* instead.
>>>>
>>>> I think what we need is for the *toolstack* to calculate the size of
>>>> the balloon rather than the guest, and tell the balloon driver how big
>>>> to make its balloon, rather than the balloon driver trying to figure
>>>> that out on its own.
>>>
>>> Maybe the best option would be for the toolstack to fetch the e820
>>> memory map and set the target based on the size of the RAM regions in
>>> there for PVH Dom0? That would certainly match the expectations of the
>>> guest.
>>>
>>> Note that for DomUs if hvmloader (or any other component) inside of
>>> the guest changes the memory map it would also have to adjust the
>>> value in the xenstore 'target' node.
>>
>> How would it do that later when the guest is already running?
> 
> hvmloader should modify the 'target' xenstore node if it changes the
> memory map.
> 
> So the value provided by the toolstack would match the amount of RAM
> in the memory map up to the point where the guest is started, from
> there on anything inside the guest changing the memory map should also
> change the xenstore value.

So what should libxl write into target when the user specifies a new
value via "xl mem-set" then? It doesn't know whether the guest is still
trying to reach the old target, so it can't trust the current memory
allocated and target value in Xenstore to match.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Thu, Jul 26, 2018 at 03:58:43PM +0200, Juergen Gross wrote:
> On 26/07/18 15:50, Roger Pau Monné wrote:
> > On Thu, Jul 26, 2018 at 01:22:33PM +0200, Juergen Gross wrote:
> >> On 26/07/18 13:11, Roger Pau Monné wrote:
> >>> On Thu, Jul 26, 2018 at 10:45:08AM +0100, George Dunlap wrote:
> >>>> On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
> >>>> <boris.ostrovsky@oracle.com> wrote:
> >>>>> On 07/25/2018 02:56 PM, Andrew Cooper wrote:
> >>>>>> On 25/07/18 17:29, Juergen Gross wrote:
> >>>>>>> On 25/07/18 18:12, Roger Pau Monné wrote:
> >>>>>>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
> >>>>>>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
> >>>>>>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> >>>>>>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
> >>>>>>>>>>>>> What could be causing the available memory loss problem?
> >>>>>>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
> >>>>>>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
> >>>>>>>>>>> This might be related to the tools thinking dom0 is a PV domain.
> >>>>>>>>>> Good point.
> >>>>>>>>>>
> >>>>>>>>>> In that case, xenstore-ls -fp would also be useful. The output should
> >>>>>>>>>> show the balloon target for Dom0.
> >>>>>>>>>>
> >>>>>>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
> >>>>>>>>>> if it makes any difference.
> >>>>>>>>>>
> >>>>>>>>>> Wei.
> >>>>>>>>> Also tried setting autoballooning off, but it had no effect.
> >>>>>>>> This is a Linux/libxl issue that I'm not sure what's the best way to
> >>>>>>>> solve. Linux has the following 'workaround' in the balloon driver:
> >>>>>>>>
> >>>>>>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
> >>>>>>>>                &static_max);
> >>>>>>>> if (err != 1)
> >>>>>>>>     static_max = new_target;
> >>>>>>>> else
> >>>>>>>>     static_max >>= PAGE_SHIFT - 10;
> >>>>>>>> target_diff = xen_pv_domain() ? 0
> >>>>>>>>             : static_max - balloon_stats.target_pages;
> >>>>>>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
> >>>>>>> there is memory missing for PVH, opposed to HVM's firmware memory.
> >>>>>>>
> >>>>>>> Adding Boris for a second opinion.
> >>>>>
> >>>>> (Notwithstanding Andrews' rant below ;-))
> >>>>>
> >>>>> I am trying to remember --- what memory were we trying not to online for
> >>>>> HVM here?
> >>>>
> >>>> My general memory of the situation is this:
> >>>>
> >>>> * Balloon drivers are told to reach a "target" value for max_pages.
> >>>> * max_pages includes all memory assigned to the guest, including video
> >>>> ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
> >>>> devices, and so on.
> >>>> * Unfortunately, the balloon driver doesn't know what their max_pages
> >>>> value is and can't read it.
> >>>> * So what the balloon drivers do at the moment (as I understand it) is
> >>>> look at the memory *reported as RAM*, and do a calculation:
> >>>>   visible_ram - target_max_pages = pages_in_balloon
> >>>>
> >>>> You can probably see why this won't work -- the result is that the
> >>>> guest balloons down to (target_max_pages + non_ram_pages).  This is
> >>>> kind of messy for normal guests, but when you have a
> >>>> populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
> >>>> in the guest.  The hypervisor then spends a huge amount of work
> >>>> swapping the PoD pages around under the guest's feet, until it can't
> >>>> find any more zeroed guest pages to use, and it crashes the guest.
> >>>>
> >>>> The kludge we have right now is to make up a number for HVM guests
> >>>> which is slightly larger than non_ram_pages, and tell the guest to aim
> >>>> for *that* instead.
> >>>>
> >>>> I think what we need is for the *toolstack* to calculate the size of
> >>>> the balloon rather than the guest, and tell the balloon driver how big
> >>>> to make its balloon, rather than the balloon driver trying to figure
> >>>> that out on its own.
> >>>
> >>> Maybe the best option would be for the toolstack to fetch the e820
> >>> memory map and set the target based on the size of the RAM regions in
> >>> there for PVH Dom0? That would certainly match the expectations of the
> >>> guest.
> >>>
> >>> Note that for DomUs if hvmloader (or any other component) inside of
> >>> the guest changes the memory map it would also have to adjust the
> >>> value in the xenstore 'target' node.
> >>
> >> How would it do that later when the guest is already running?
> > 
> > hvmloader should modify the 'target' xenstore node if it changes the
> > memory map.
> > 
> > So the value provided by the toolstack would match the amount of RAM
> > in the memory map up to the point where the guest is started, from
> > there on anything inside the guest changing the memory map should also
> > change the xenstore value.
> 
> So what should libxl write into target when the user specifies a new
> value via "xl mem-set" then?

I thought the problem was at guest boot, where the OS sees a target
value different than the amount of RAM in the memory map, but I see
that the same problem would happen when doing a mem-set because the
target set in xenstore would match d->tot_pages, but the guest won't
be able to reach it due to memory being used by firmware. Sorry for
the noise.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

[George Dunlap]

On Thu, Jul 26, 2018 at 12:11 PM, Roger Pau Monné <roger.pau@citrix.com> wrote:
> On Thu, Jul 26, 2018 at 10:45:08AM +0100, George Dunlap wrote:
>> On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
>> <boris.ostrovsky@oracle.com> wrote:
>> > On 07/25/2018 02:56 PM, Andrew Cooper wrote:
>> >> On 25/07/18 17:29, Juergen Gross wrote:
>> >>> On 25/07/18 18:12, Roger Pau Monné wrote:
>> >>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>> >>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>> >>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>> >>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>> >>>>>>>>> What could be causing the available memory loss problem?
>> >>>>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>> >>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>> >>>>>>> This might be related to the tools thinking dom0 is a PV domain.
>> >>>>>> Good point.
>> >>>>>>
>> >>>>>> In that case, xenstore-ls -fp would also be useful. The output should
>> >>>>>> show the balloon target for Dom0.
>> >>>>>>
>> >>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>> >>>>>> if it makes any difference.
>> >>>>>>
>> >>>>>> Wei.
>> >>>>> Also tried setting autoballooning off, but it had no effect.
>> >>>> This is a Linux/libxl issue that I'm not sure what's the best way to
>> >>>> solve. Linux has the following 'workaround' in the balloon driver:
>> >>>>
>> >>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>> >>>>                &static_max);
>> >>>> if (err != 1)
>> >>>>     static_max = new_target;
>> >>>> else
>> >>>>     static_max >>= PAGE_SHIFT - 10;
>> >>>> target_diff = xen_pv_domain() ? 0
>> >>>>             : static_max - balloon_stats.target_pages;
>> >>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
>> >>> there is memory missing for PVH, opposed to HVM's firmware memory.
>> >>>
>> >>> Adding Boris for a second opinion.
>> >
>> > (Notwithstanding Andrews' rant below ;-))
>> >
>> > I am trying to remember --- what memory were we trying not to online for
>> > HVM here?
>>
>> My general memory of the situation is this:
>>
>> * Balloon drivers are told to reach a "target" value for max_pages.
>> * max_pages includes all memory assigned to the guest, including video
>> ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
>> devices, and so on.
>> * Unfortunately, the balloon driver doesn't know what their max_pages
>> value is and can't read it.
>> * So what the balloon drivers do at the moment (as I understand it) is
>> look at the memory *reported as RAM*, and do a calculation:
>>   visible_ram - target_max_pages = pages_in_balloon
>>
>> You can probably see why this won't work -- the result is that the
>> guest balloons down to (target_max_pages + non_ram_pages).  This is
>> kind of messy for normal guests, but when you have a
>> populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
>> in the guest.  The hypervisor then spends a huge amount of work
>> swapping the PoD pages around under the guest's feet, until it can't
>> find any more zeroed guest pages to use, and it crashes the guest.
>>
>> The kludge we have right now is to make up a number for HVM guests
>> which is slightly larger than non_ram_pages, and tell the guest to aim
>> for *that* instead.
>>
>> I think what we need is for the *toolstack* to calculate the size of
>> the balloon rather than the guest, and tell the balloon driver how big
>> to make its balloon, rather than the balloon driver trying to figure
>> that out on its own.
>
> Maybe the best option would be for the toolstack to fetch the e820
> memory map and set the target based on the size of the RAM regions in
> there for PVH Dom0? That would certainly match the expectations of the
> guest.

Right; so:
* Expecting the guest do to calculate its own balloon size was always
an architectural mistake.
* What we're tripping over now is that the hack we used to paper over
the architectural mistake for HVM doesn't apply for PVH.

We can either:
1. Extend the hack to paper it over for PVH as well
2. Fix it properly by addressing the underlying architectural mistake.

Given how long it's been that nobody's had bandwidth to do #2, I think
at the moment I think we don't really have any option except to do #1.
But we shouldn't forget that we need to do #2 at some point.

> Note that for DomUs if hvmloader (or any other component) inside of
> the guest changes the memory map it would also have to adjust the
> value in the xenstore 'target' node.

Yes, this sort of thing is part of the reason the swamp hasn't been
drained yet, just fences put up in a few places to keep the alligators
in.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Jul 25, 2018 at 06:29:25PM +0200, Juergen Gross wrote:
> On 25/07/18 18:12, Roger Pau Monné wrote:
> > On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
> >> On 07/25/2018 05:02 PM, Wei Liu wrote:
> >>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> >>>> On 25/07/18 15:35, Roger Pau Monné wrote:
> >>>>>> What could be causing the available memory loss problem?
> >>>>> That seems to be Linux aggressively ballooning out memory, you go from
> >>>>> 7129M total memory to 246M. Are you creating a lot of domains?
> >>>> This might be related to the tools thinking dom0 is a PV domain.
> >>> Good point.
> >>>
> >>> In that case, xenstore-ls -fp would also be useful. The output should
> >>> show the balloon target for Dom0.
> >>>
> >>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
> >>> if it makes any difference.
> >>>
> >>> Wei.
> >> Also tried setting autoballooning off, but it had no effect.
> > 
> > This is a Linux/libxl issue that I'm not sure what's the best way to
> > solve. Linux has the following 'workaround' in the balloon driver:
> > 
> > err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
> > 		   &static_max);
> > if (err != 1)
> > 	static_max = new_target;
> > else
> > 	static_max >>= PAGE_SHIFT - 10;
> > target_diff = xen_pv_domain() ? 0
> > 		: static_max - balloon_stats.target_pages;
> 
> Hmm, shouldn't PVH behave the same way as PV here? I don't think
> there is memory missing for PVH, opposed to HVM's firmware memory.

There's memory missing for PVH, eg: the console page, the TSS for real
mode (if required), the identity page tables for running in
protected mode (if required) and the memory used to store the ACPI
tables.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

On 07/25/2018 07:12 PM, Roger Pau Monné wrote:
> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>> What could be causing the available memory loss problem?
>>>>> That seems to be Linux aggressively ballooning out memory, you go from
>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>> This might be related to the tools thinking dom0 is a PV domain.
>>> Good point.
>>>
>>> In that case, xenstore-ls -fp would also be useful. The output should
>>> show the balloon target for Dom0.
>>>
>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to see
>>> if it makes any difference.
>>>
>>> Wei.
>> Also tried setting autoballooning off, but it had no effect.
> This is a Linux/libxl issue that I'm not sure what's the best way to
> solve. Linux has the following 'workaround' in the balloon driver:
>
> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
> 		   &static_max);
> if (err != 1)
> 	static_max = new_target;
> else
> 	static_max >>= PAGE_SHIFT - 10;
> target_diff = xen_pv_domain() ? 0
> 		: static_max - balloon_stats.target_pages;
>
> I suppose this is used to cope with the memory reporting mismatch
> usually seen on HVM guests. This however interacts quite badly on a
> PVH Dom0 that has for example:
>
> /local/domain/0/memory/target = "8391840"   (n0)
> /local/domain/0/memory/static-max = "17179869180"   (n0)
>
> One way to solve this is to set target and static-max to the same
> value initially, so that target_diff on Linux is 0. Another option
> would be to force target_diff = 0 for Dom0.
>
> I'm attaching a patch for libxl that should solve this, could you
> please give it a try and report back?
>
> I'm still unsure however about the best way to fix this, need to think
> about it.
>
> Roger.
> ---8<---
> diff --git a/tools/libxl/libxl_mem.c b/tools/libxl/libxl_mem.c
> index e551e09fed..2c984993d8 100644
> --- a/tools/libxl/libxl_mem.c
> +++ b/tools/libxl/libxl_mem.c
> @@ -151,7 +151,9 @@ retry_transaction:
>           *target_memkb = info.current_memkb;
>       }
>       if (staticmax == NULL) {
> -        libxl__xs_printf(gc, t, max_path, "%"PRIu64, info.max_memkb);
> +        libxl__xs_printf(gc, t, max_path, "%"PRIu64,
> +                         libxl__domain_type(gc, 0) == LIBXL_DOMAIN_TYPE_PV ?
> +                         info.max_memkb : info.current_memkb);
>           *max_memkb = info.max_memkb;
>       }
>   
>
>
I have tried Roger's patch and it fixed the memory decrease problem. "xl 
list -l"

no longer causes any issue.

The output of "xenstore-ls -fp" shows that both target and static-max 
are now

set to the same value.


Gabriel




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Juergen Gross]

On 26/07/18 10:15, bercarug@amazon.com wrote:
> On 07/25/2018 07:12 PM, Roger Pau Monné wrote:
>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>> What could be causing the available memory loss problem?
>>>>>> That seems to be Linux aggressively ballooning out memory, you go
>>>>>> from
>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>> Good point.
>>>>
>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>> show the balloon target for Dom0.
>>>>
>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to
>>>> see
>>>> if it makes any difference.
>>>>
>>>> Wei.
>>> Also tried setting autoballooning off, but it had no effect.
>> This is a Linux/libxl issue that I'm not sure what's the best way to
>> solve. Linux has the following 'workaround' in the balloon driver:
>>
>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>>            &static_max);
>> if (err != 1)
>>     static_max = new_target;
>> else
>>     static_max >>= PAGE_SHIFT - 10;
>> target_diff = xen_pv_domain() ? 0
>>         : static_max - balloon_stats.target_pages;
>>
>> I suppose this is used to cope with the memory reporting mismatch
>> usually seen on HVM guests. This however interacts quite badly on a
>> PVH Dom0 that has for example:
>>
>> /local/domain/0/memory/target = "8391840"   (n0)
>> /local/domain/0/memory/static-max = "17179869180"   (n0)
>>
>> One way to solve this is to set target and static-max to the same
>> value initially, so that target_diff on Linux is 0. Another option
>> would be to force target_diff = 0 for Dom0.
>>
>> I'm attaching a patch for libxl that should solve this, could you
>> please give it a try and report back?
>>
>> I'm still unsure however about the best way to fix this, need to think
>> about it.
>>
>> Roger.
>> ---8<---
>> diff --git a/tools/libxl/libxl_mem.c b/tools/libxl/libxl_mem.c
>> index e551e09fed..2c984993d8 100644
>> --- a/tools/libxl/libxl_mem.c
>> +++ b/tools/libxl/libxl_mem.c
>> @@ -151,7 +151,9 @@ retry_transaction:
>>           *target_memkb = info.current_memkb;
>>       }
>>       if (staticmax == NULL) {
>> -        libxl__xs_printf(gc, t, max_path, "%"PRIu64, info.max_memkb);
>> +        libxl__xs_printf(gc, t, max_path, "%"PRIu64,
>> +                         libxl__domain_type(gc, 0) ==
>> LIBXL_DOMAIN_TYPE_PV ?
>> +                         info.max_memkb : info.current_memkb);
>>           *max_memkb = info.max_memkb;
>>       }
>>  
>>
> I have tried Roger's patch and it fixed the memory decrease problem. "xl
> list -l"
> 
> no longer causes any issue.
> 
> The output of "xenstore-ls -fp" shows that both target and static-max
> are now
> 
> set to the same value.

Right.

Meaning that it will be impossible to add memory to PVH dom0 e.g. after
memory hotplug.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Thu, Jul 26, 2018 at 10:31:21AM +0200, Juergen Gross wrote:
> On 26/07/18 10:15, bercarug@amazon.com wrote:
> > On 07/25/2018 07:12 PM, Roger Pau Monné wrote:
> >> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@amazon.com wrote:
> >>> On 07/25/2018 05:02 PM, Wei Liu wrote:
> >>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
> >>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
> >>>>>>> What could be causing the available memory loss problem?
> >>>>>> That seems to be Linux aggressively ballooning out memory, you go
> >>>>>> from
> >>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
> >>>>> This might be related to the tools thinking dom0 is a PV domain.
> >>>> Good point.
> >>>>
> >>>> In that case, xenstore-ls -fp would also be useful. The output should
> >>>> show the balloon target for Dom0.
> >>>>
> >>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to
> >>>> see
> >>>> if it makes any difference.
> >>>>
> >>>> Wei.
> >>> Also tried setting autoballooning off, but it had no effect.
> >> This is a Linux/libxl issue that I'm not sure what's the best way to
> >> solve. Linux has the following 'workaround' in the balloon driver:
> >>
> >> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
> >>            &static_max);
> >> if (err != 1)
> >>     static_max = new_target;
> >> else
> >>     static_max >>= PAGE_SHIFT - 10;
> >> target_diff = xen_pv_domain() ? 0
> >>         : static_max - balloon_stats.target_pages;
> >>
> >> I suppose this is used to cope with the memory reporting mismatch
> >> usually seen on HVM guests. This however interacts quite badly on a
> >> PVH Dom0 that has for example:
> >>
> >> /local/domain/0/memory/target = "8391840"   (n0)
> >> /local/domain/0/memory/static-max = "17179869180"   (n0)
> >>
> >> One way to solve this is to set target and static-max to the same
> >> value initially, so that target_diff on Linux is 0. Another option
> >> would be to force target_diff = 0 for Dom0.
> >>
> >> I'm attaching a patch for libxl that should solve this, could you
> >> please give it a try and report back?
> >>
> >> I'm still unsure however about the best way to fix this, need to think
> >> about it.
> >>
> >> Roger.
> >> ---8<---
> >> diff --git a/tools/libxl/libxl_mem.c b/tools/libxl/libxl_mem.c
> >> index e551e09fed..2c984993d8 100644
> >> --- a/tools/libxl/libxl_mem.c
> >> +++ b/tools/libxl/libxl_mem.c
> >> @@ -151,7 +151,9 @@ retry_transaction:
> >>           *target_memkb = info.current_memkb;
> >>       }
> >>       if (staticmax == NULL) {
> >> -        libxl__xs_printf(gc, t, max_path, "%"PRIu64, info.max_memkb);
> >> +        libxl__xs_printf(gc, t, max_path, "%"PRIu64,
> >> +                         libxl__domain_type(gc, 0) ==
> >> LIBXL_DOMAIN_TYPE_PV ?
> >> +                         info.max_memkb : info.current_memkb);
> >>           *max_memkb = info.max_memkb;
> >>       }
> >>  
> >>
> > I have tried Roger's patch and it fixed the memory decrease problem. "xl
> > list -l"
> > 
> > no longer causes any issue.
> > 
> > The output of "xenstore-ls -fp" shows that both target and static-max
> > are now
> > 
> > set to the same value.
> 
> Right.
> 
> Meaning that it will be impossible to add memory to PVH dom0 e.g. after
> memory hotplug.

Likely. HVM guests ATM can only boot ballooned down (when target !=
max) by using PoD IIRC.

Right now if the user doesn't specify a 'max' value in the command
line for a PV(H) Dom0 it's set to LONG_MAX.

Maybe a better option would be to set max == current if no max is
specified on the command line?

This however doesn't fully solve the problem, since setting target !=
static-max for a Linux PVH guest will cause the balloon driver to go
nuts.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

On 07/25/2018 04:35 PM, Roger Pau Monné wrote:
> On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com wrote:
>> On 07/24/2018 12:54 PM, Jan Beulich wrote:
>>>>>> On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
>>>> For the last few days, I have been trying to get a PVH dom0 running,
>>>> however I encountered the following problem: the system seems to
>>>> freeze after the hypervisor boots, the screen goes black. I have tried to
>>>> debug it via a serial console (using Minicom) and managed to get some
>>>> more Xen output, after the screen turns black.
>>>>
>>>> I mention that I have tried to boot the PVH dom0 using different kernel
>>>> images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).
>>>>
>>>> Below I attached my system / hypervisor configuration, as well as the
>>>> output captured through the serial console, corresponding to the latest
>>>> versions for Xen and the Linux Kernel (Xen staging and Kernel from the
>>>> xen/tip tree).
>>>> [...]
>>>> (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
>>>> (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
>>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000
> Can you figure out which PCI device is 00:14.0?
This is the output of lspci -vvv for device 00:14.0:

00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI 
Controller (rev 31) (prog-if 30 [XHCI])
         Subsystem: Intel Corporation Sunrise Point-H USB 3.0 xHCI 
Controller
         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium 
 >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
         Latency: 0
         Interrupt: pin A routed to IRQ 178
         Region 0: Memory at a2e00000 (64-bit, non-prefetchable) [size=64K]
         Capabilities: [70] Power Management version 2
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA 
PME(D0-,D1-,D2-,D3hot+,D3cold+)
                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
         Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
                 Address: 00000000fee0e000  Data: 4021
         Kernel driver in use: xhci_hcd
         Kernel modules: xhci_pci
>>>> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>>>> (XEN) print_vtd_entries: iommu #0 dev 0000:00:14.0 gmfn 8deb3
>>>> (XEN) root_entry[00] = 1021c60001
>>>> (XEN) context[a0] = 2_1021d6d001
>>>> (XEN) l4[000] = 9c00001021d6c107
>>>> (XEN) l3[002] = 9c00001021d3e107
>>>> (XEN) l2[06f] = 9c000010218c0107
>>>> (XEN) l1[0b3] = 8000000000000000
>>>> (XEN) l1[0b3] not present
>>>> (XEN) Dom0 callback via changed to Direct Vector 0xf3
>>> This might be a hint at a missing RMRR entry in the ACPI tables, as
>>> we've seen to be the case for a number of systems (I dare to guess
>>> that 0000:00:14.0 is a USB controller, perhaps one with a keyboard
>>> and/or mouse connected). You may want to play with the respective
>>> command line option ("rmrr="). Note that "iommu_inclusive_mapping"
>>> as you're using it does not have any meaning for PVH (see
>>> intel_iommu_hwdom_init()).
>>>
>>> Jan
>>>
>>>
>>>
>> Hello,
>>
>> Following Roger's advice, I rebuilt Xen (4.12) using the staging branch and
>> I managed to get a PVH dom0 starting. However, some other problems appeared:
>>
>> 1) The USB devices are not usable anymore (keyboard and mouse), so the
>> system is only accessible through the serial port.
> Can you boot with iommu=debug and see if you get any extra IOMMU
> information on the serial console?
The debug flag was already set, so the log I attached on the first
message already contains the IOMMU info.
In Xen's command line I used iommu=debug,verbose,workaround_bios_bug.
>
>> 2) I can run any usual command in dom0, but the ones involving xl (except
>> for xl info) will make the system run out of memory very fast. Eventually,
>> when there is no more free memory available, the OOM killer begins removing
>> processes until the system auto reboots.
>>
>> I attached a file containing the output of a lsusb, as well as the output of
>> xl info and xl list -l.
>> After xl list -l, the “free -m” commands show the available memory
>> decreasing.
>> Each command has a timestamp appended, so it can be seen how fast the
>> available memory decreases.
>>
>> I removed much of the process killing logs and kept the last one, since they
>> were following the same pattern.
>>
>> Dom0 still appears to be of type PV (output of xl list -l), however during
>> boot, the following messages were displayed: “Building a PVH Dom0” and
>> “Booting paravirtualized kernel on Xen PVH”.
>>
>> I mention that I had to add “workaround_bios_bug” in GRUB_CMDLINE_XEN for
>> iommu to get dom0 running.
> It seems to me like your ACPI DMAR table contains errors, and I
> wouldn't be surprised if those also cause the USB devices to
> malfunction.
>
>> What could be causing the available memory loss problem?
> That seems to be Linux aggressively ballooning out memory, you go from
> 7129M total memory to 246M. Are you creating a lot of domains?
>
> Roger.
>
I did not create any guest before issuing "xl list -l". However, creating

a PVH domU will work - "xl create <cfg_file>" does not produce this 
behavior.


Gabriel




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Jul 25, 2018 at 04:57:23PM +0300, bercarug@amazon.com wrote:
> On 07/25/2018 04:35 PM, Roger Pau Monné wrote:
> > On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com wrote:
> > > On 07/24/2018 12:54 PM, Jan Beulich wrote:
> > > > > > > On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
> > > > > For the last few days, I have been trying to get a PVH dom0 running,
> > > > > however I encountered the following problem: the system seems to
> > > > > freeze after the hypervisor boots, the screen goes black. I have tried to
> > > > > debug it via a serial console (using Minicom) and managed to get some
> > > > > more Xen output, after the screen turns black.
> > > > > 
> > > > > I mention that I have tried to boot the PVH dom0 using different kernel
> > > > > images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11, 4.12).
> > > > > 
> > > > > Below I attached my system / hypervisor configuration, as well as the
> > > > > output captured through the serial console, corresponding to the latest
> > > > > versions for Xen and the Linux Kernel (Xen staging and Kernel from the
> > > > > xen/tip tree).
> > > > > [...]
> > > > > (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
> > > > > (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending Fault
> > > > > (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault addr 8deb3000, iommu reg = ffff82c00021b000
> > Can you figure out which PCI device is 00:14.0?
> This is the output of lspci -vvv for device 00:14.0:
> 
> 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI
> Controller (rev 31) (prog-if 30 [XHCI])
>         Subsystem: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 178
>         Region 0: Memory at a2e00000 (64-bit, non-prefetchable) [size=64K]
>         Capabilities: [70] Power Management version 2
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
>                 Address: 00000000fee0e000  Data: 4021
>         Kernel driver in use: xhci_hcd
>         Kernel modules: xhci_pci

I'm afraid your USB controller is missing RMRR entries in the DMAR
ACPI tables, thus causing the IOMMU faults and not working properly.

You could try to manually add some extra rmrr regions by appending:

rmrr=0x8deb3=0:0:14.0

To the Xen command line, and keep adding any address that pops up in
the iommu faults. This is of course quite cumbersome, but there's no
way to get the required memory addresses if the data in RMRR is
wrong/incomplete.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Paul Durrant]

> -----Original Message-----
> From: Xen-devel [mailto:xen-devel-bounces@lists.xenproject.org] On Behalf
> Of Roger Pau Monné
> Sent: 25 July 2018 15:12
> To: bercarug@amazon.com
> Cc: xen-devel <xen-devel@lists.xenproject.org>; David Woodhouse
> <dwmw2@infradead.org>; Jan Beulich <JBeulich@suse.com>;
> abelgun@amazon.com
> Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
> 
> On Wed, Jul 25, 2018 at 04:57:23PM +0300, bercarug@amazon.com wrote:
> > On 07/25/2018 04:35 PM, Roger Pau Monné wrote:
> > > On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com
> wrote:
> > > > On 07/24/2018 12:54 PM, Jan Beulich wrote:
> > > > > > > > On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
> > > > > > For the last few days, I have been trying to get a PVH dom0 running,
> > > > > > however I encountered the following problem: the system seems
> to
> > > > > > freeze after the hypervisor boots, the screen goes black. I have
> tried to
> > > > > > debug it via a serial console (using Minicom) and managed to get
> some
> > > > > > more Xen output, after the screen turns black.
> > > > > >
> > > > > > I mention that I have tried to boot the PVH dom0 using different
> kernel
> > > > > > images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11,
> 4.12).
> > > > > >
> > > > > > Below I attached my system / hypervisor configuration, as well as
> the
> > > > > > output captured through the serial console, corresponding to the
> latest
> > > > > > versions for Xen and the Linux Kernel (Xen staging and Kernel from
> the
> > > > > > xen/tip tree).
> > > > > > [...]
> > > > > > (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
> > > > > > (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending
> Fault
> > > > > > (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault
> addr 8deb3000, iommu reg = ffff82c00021b000
> > > Can you figure out which PCI device is 00:14.0?
> > This is the output of lspci -vvv for device 00:14.0:
> >
> > 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI
> > Controller (rev 31) (prog-if 30 [XHCI])
> >         Subsystem: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller
> >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr+
> > Stepping- SERR+ FastB2B- DisINTx+
> >         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> > <TAbort- <MAbort+ >SERR- <PERR- INTx-
> >         Latency: 0
> >         Interrupt: pin A routed to IRQ 178
> >         Region 0: Memory at a2e00000 (64-bit, non-prefetchable) [size=64K]
> >         Capabilities: [70] Power Management version 2
> >                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> > PME(D0-,D1-,D2-,D3hot+,D3cold+)
> >                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> >         Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> >                 Address: 00000000fee0e000  Data: 4021
> >         Kernel driver in use: xhci_hcd
> >         Kernel modules: xhci_pci
> 
> I'm afraid your USB controller is missing RMRR entries in the DMAR
> ACPI tables, thus causing the IOMMU faults and not working properly.
> 
> You could try to manually add some extra rmrr regions by appending:
> 
> rmrr=0x8deb3=0:0:14.0
> 
> To the Xen command line, and keep adding any address that pops up in
> the iommu faults. This is of course quite cumbersome, but there's no
> way to get the required memory addresses if the data in RMRR is
> wrong/incomplete.
> 

You could just add all E820 reserved regions in there. That will almost certainly cover it.

  Paul

> Roger.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Jul 25, 2018 at 05:19:03PM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Xen-devel [mailto:xen-devel-bounces@lists.xenproject.org] On Behalf
> > Of Roger Pau Monné
> > Sent: 25 July 2018 15:12
> > To: bercarug@amazon.com
> > Cc: xen-devel <xen-devel@lists.xenproject.org>; David Woodhouse
> > <dwmw2@infradead.org>; Jan Beulich <JBeulich@suse.com>;
> > abelgun@amazon.com
> > Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
> > 
> > On Wed, Jul 25, 2018 at 04:57:23PM +0300, bercarug@amazon.com wrote:
> > > On 07/25/2018 04:35 PM, Roger Pau Monné wrote:
> > > > On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com
> > wrote:
> > > > > On 07/24/2018 12:54 PM, Jan Beulich wrote:
> > > > > > > > > On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
> > > > > > > For the last few days, I have been trying to get a PVH dom0 running,
> > > > > > > however I encountered the following problem: the system seems
> > to
> > > > > > > freeze after the hypervisor boots, the screen goes black. I have
> > tried to
> > > > > > > debug it via a serial console (using Minicom) and managed to get
> > some
> > > > > > > more Xen output, after the screen turns black.
> > > > > > >
> > > > > > > I mention that I have tried to boot the PVH dom0 using different
> > kernel
> > > > > > > images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11,
> > 4.12).
> > > > > > >
> > > > > > > Below I attached my system / hypervisor configuration, as well as
> > the
> > > > > > > output captured through the serial console, corresponding to the
> > latest
> > > > > > > versions for Xen and the Linux Kernel (Xen staging and Kernel from
> > the
> > > > > > > xen/tip tree).
> > > > > > > [...]
> > > > > > > (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
> > > > > > > (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending
> > Fault
> > > > > > > (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault
> > addr 8deb3000, iommu reg = ffff82c00021b000
> > > > Can you figure out which PCI device is 00:14.0?
> > > This is the output of lspci -vvv for device 00:14.0:
> > >
> > > 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI
> > > Controller (rev 31) (prog-if 30 [XHCI])
> > >         Subsystem: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller
> > >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > ParErr+
> > > Stepping- SERR+ FastB2B- DisINTx+
> > >         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> > > <TAbort- <MAbort+ >SERR- <PERR- INTx-
> > >         Latency: 0
> > >         Interrupt: pin A routed to IRQ 178
> > >         Region 0: Memory at a2e00000 (64-bit, non-prefetchable) [size=64K]
> > >         Capabilities: [70] Power Management version 2
> > >                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> > > PME(D0-,D1-,D2-,D3hot+,D3cold+)
> > >                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> > >         Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> > >                 Address: 00000000fee0e000  Data: 4021
> > >         Kernel driver in use: xhci_hcd
> > >         Kernel modules: xhci_pci
> > 
> > I'm afraid your USB controller is missing RMRR entries in the DMAR
> > ACPI tables, thus causing the IOMMU faults and not working properly.
> > 
> > You could try to manually add some extra rmrr regions by appending:
> > 
> > rmrr=0x8deb3=0:0:14.0
> > 
> > To the Xen command line, and keep adding any address that pops up in
> > the iommu faults. This is of course quite cumbersome, but there's no
> > way to get the required memory addresses if the data in RMRR is
> > wrong/incomplete.
> > 
> 
> You could just add all E820 reserved regions in there. That will almost certainly cover it.

I have a prototype patch for this that attempts to identity map all
reserved regions below 4GB to the p2m. It's still a WIP, but if you
could give it a try that would help me figure out whether this fixes
your issues and is indeed something that would be good to have.

I don't really like the patch as-is because it doesn't check whether
the reserved regions added to the p2m overlap with the LAPIC page or
the PCIe MCFG regions for example, I will continue to work on a safer
version.

If you can give this a shot, please remove any rmrr options from the
command line and use iommu=debug in order to catch any issues.

Thanks, Roger.
---8<---
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 2c44fabf99..76a1fd6681 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -21,6 +21,8 @@
 #include <xen/keyhandler.h>
 #include <xsm/xsm.h>
 
+#include <asm/setup.h>
+
 static int parse_iommu_param(const char *s);
 static void iommu_dump_p2m_table(unsigned char key);
 
@@ -47,6 +49,8 @@ integer_param("iommu_dev_iotlb_timeout", iommu_dev_iotlb_timeout);
  *   no-igfx                    Disable VT-d for IGD devices (insecure)
  *   no-amd-iommu-perdev-intremap Don't use per-device interrupt remapping
  *                              tables (insecure)
+ *   inclusive                  Include any memory ranges below 4GB not used
+ *                              by Xen or unusable to the iommu page tables.
  */
 custom_param("iommu", parse_iommu_param);
 bool_t __initdata iommu_enable = 1;
@@ -60,6 +64,7 @@ bool_t __read_mostly iommu_passthrough;
 bool_t __read_mostly iommu_snoop = 1;
 bool_t __read_mostly iommu_qinval = 1;
 bool_t __read_mostly iommu_intremap = 1;
+bool __read_mostly iommu_inclusive = true;
 
 /*
  * In the current implementation of VT-d posted interrupts, in some extreme
@@ -126,6 +131,8 @@ static int __init parse_iommu_param(const char *s)
             iommu_dom0_strict = val;
         else if ( !strncmp(s, "sharept", ss - s) )
             iommu_hap_pt_share = val;
+        else if ( !strncmp(s, "inclusive", ss - s) )
+            iommu_inclusive = val;
         else
             rc = -EINVAL;
 
@@ -165,6 +172,85 @@ static void __hwdom_init check_hwdom_reqs(struct domain *d)
     iommu_dom0_strict = 1;
 }
 
+static void __hwdom_init setup_inclusive_mappings(struct domain *d)
+{
+    unsigned long i, j, tmp, top, max_pfn;
+
+    BUG_ON(!is_hardware_domain(d));
+
+    max_pfn = (GB(4) >> PAGE_SHIFT) - 1;
+    top = max(max_pdx, pfn_to_pdx(max_pfn) + 1);
+
+    for ( i = 0; i < top; i++ )
+    {
+        unsigned long pfn = pdx_to_pfn(i);
+        bool map;
+        int rc = 0;
+
+        /*
+         * Set up 1:1 mapping for dom0. Default to include only
+         * conventional RAM areas and let RMRRs include needed reserved
+         * regions. When set, the inclusive mapping additionally maps in
+         * every pfn up to 4GB except those that fall in unusable ranges.
+         */
+        if ( pfn > max_pfn && !mfn_valid(_mfn(pfn)) )
+            continue;
+
+        if ( is_pv_domain(d) && iommu_inclusive && pfn <= max_pfn )
+            map = !page_is_ram_type(pfn, RAM_TYPE_UNUSABLE);
+        else if ( is_hvm_domain(d) && iommu_inclusive )
+            map = page_is_ram_type(pfn, RAM_TYPE_RESERVED);
+        else
+            map = page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL);
+
+        if ( !map )
+            continue;
+
+        /* Exclude Xen bits */
+        if ( xen_in_range(pfn) )
+            continue;
+
+        /*
+         * If dom0-strict mode is enabled or guest type is HVM/PVH then exclude
+         * conventional RAM and let the common code map dom0's pages.
+         */
+        if ( (iommu_dom0_strict || is_hvm_domain(d)) &&
+             page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL) )
+            continue;
+
+        /* For HVM avoid memory below 1MB because that's already mapped. */
+        if ( is_hvm_domain(d) && pfn < PFN_DOWN(MB(1)) )
+            continue;
+
+        tmp = 1 << (PAGE_SHIFT - PAGE_SHIFT_4K);
+        for ( j = 0; j < tmp; j++ )
+        {
+            int ret;
+
+            if ( iommu_use_hap_pt(d) )
+            {
+                ASSERT(is_hvm_domain(d));
+                ret = set_identity_p2m_entry(d, pfn * tmp + j, p2m_access_rw,
+                                             0);
+            }
+            else
+                ret = iommu_map_page(d, pfn * tmp + j, pfn * tmp + j,
+                                     IOMMUF_readable|IOMMUF_writable);
+
+            if ( !rc )
+               rc = ret;
+        }
+
+        if ( rc )
+            printk(XENLOG_WARNING " d%d: IOMMU mapping failed: %d\n",
+                   d->domain_id, rc);
+
+        if (!(i & (0xfffff >> (PAGE_SHIFT - PAGE_SHIFT_4K))))
+            process_pending_softirqs();
+    }
+
+}
+
 void __hwdom_init iommu_hwdom_init(struct domain *d)
 {
     const struct domain_iommu *hd = dom_iommu(d);
@@ -207,7 +293,10 @@ void __hwdom_init iommu_hwdom_init(struct domain *d)
                    d->domain_id, rc);
     }
 
-    return hd->platform_ops->hwdom_init(d);
+    hd->platform_ops->hwdom_init(d);
+
+    if ( !iommu_passthrough )
+        setup_inclusive_mappings(d);
 }
 
 void iommu_teardown(struct domain *d)
diff --git a/xen/drivers/passthrough/vtd/extern.h b/xen/drivers/passthrough/vtd/extern.h
index fb7edfaef9..91cadc602e 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -99,6 +99,4 @@ void pci_vtd_quirk(const struct pci_dev *);
 bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);
 
-void vtd_set_hwdom_mapping(struct domain *d);
-
 #endif // _VTD_EXTERN_H_
diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c
index 1710256823..569ec4aec2 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1304,12 +1304,6 @@ static void __hwdom_init intel_iommu_hwdom_init(struct domain *d)
 {
     struct acpi_drhd_unit *drhd;
 
-    if ( !iommu_passthrough && is_pv_domain(d) )
-    {
-        /* Set up 1:1 page table for hardware domain. */
-        vtd_set_hwdom_mapping(d);
-    }
-
     setup_hwdom_pci_devices(d, setup_hwdom_device);
     setup_hwdom_rmrr(d);
 
diff --git a/xen/drivers/passthrough/vtd/x86/vtd.c b/xen/drivers/passthrough/vtd/x86/vtd.c
index cc2bfea162..9971915349 100644
--- a/xen/drivers/passthrough/vtd/x86/vtd.c
+++ b/xen/drivers/passthrough/vtd/x86/vtd.c
@@ -32,11 +32,9 @@
 #include "../extern.h"
 
 /*
- * iommu_inclusive_mapping: when set, all memory below 4GB is included in dom0
- * 1:1 iommu mappings except xen and unusable regions.
+ * iommu_inclusive_mapping: superseded by iommu=inclusive.
  */
-static bool_t __hwdom_initdata iommu_inclusive_mapping = 1;
-boolean_param("iommu_inclusive_mapping", iommu_inclusive_mapping);
+boolean_param("iommu_inclusive_mapping", iommu_inclusive);
 
 void *map_vtd_domain_page(u64 maddr)
 {
@@ -107,67 +105,3 @@ void hvm_dpci_isairq_eoi(struct domain *d, unsigned int isairq)
     }
     spin_unlock(&d->event_lock);
 }
-
-void __hwdom_init vtd_set_hwdom_mapping(struct domain *d)
-{
-    unsigned long i, j, tmp, top, max_pfn;
-
-    BUG_ON(!is_hardware_domain(d));
-
-    max_pfn = (GB(4) >> PAGE_SHIFT) - 1;
-    top = max(max_pdx, pfn_to_pdx(max_pfn) + 1);
-
-    for ( i = 0; i < top; i++ )
-    {
-        unsigned long pfn = pdx_to_pfn(i);
-        bool map;
-        int rc = 0;
-
-        /*
-         * Set up 1:1 mapping for dom0. Default to include only
-         * conventional RAM areas and let RMRRs include needed reserved
-         * regions. When set, the inclusive mapping additionally maps in
-         * every pfn up to 4GB except those that fall in unusable ranges.
-         */
-        if ( pfn > max_pfn && !mfn_valid(_mfn(pfn)) )
-            continue;
-
-        if ( iommu_inclusive_mapping && pfn <= max_pfn )
-            map = !page_is_ram_type(pfn, RAM_TYPE_UNUSABLE);
-        else
-            map = page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL);
-
-        if ( !map )
-            continue;
-
-        /* Exclude Xen bits */
-        if ( xen_in_range(pfn) )
-            continue;
-
-        /*
-         * If dom0-strict mode is enabled then exclude conventional RAM
-         * and let the common code map dom0's pages.
-         */
-        if ( iommu_dom0_strict &&
-             page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL) )
-            continue;
-
-        tmp = 1 << (PAGE_SHIFT - PAGE_SHIFT_4K);
-        for ( j = 0; j < tmp; j++ )
-        {
-            int ret = iommu_map_page(d, pfn * tmp + j, pfn * tmp + j,
-                                     IOMMUF_readable|IOMMUF_writable);
-
-            if ( !rc )
-               rc = ret;
-        }
-
-        if ( rc )
-            printk(XENLOG_WARNING VTDPREFIX " d%d: IOMMU mapping failed: %d\n",
-                   d->domain_id, rc);
-
-        if (!(i & (0xfffff >> (PAGE_SHIFT - PAGE_SHIFT_4K))))
-            process_pending_softirqs();
-    }
-}
-
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 6b42e3b876..15d6584837 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -35,6 +35,7 @@ extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
 extern bool_t iommu_hap_pt_share;
 extern bool_t iommu_debug;
 extern bool_t amd_iommu_perdev_intremap;
+extern bool iommu_inclusive;
 
 extern unsigned int iommu_dev_iotlb_timeout;
 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Bercaru, Gabriel]

[-- Attachment #1: Type: text/plain, Size: 14539 bytes --]

I tried the patch and it fixes the unusable USB devices problem.
However, I captured the boot messages and the "IOMMU mapping failed" printk
seems to have been executed on each iteration of the loop.

I attached a small section of the boot log. As I said, the warning log was displayed
many more times, I removed a lot of them to keep the attached file short.

Gabriel
________________________________________
From: Roger Pau Monné <roger.pau@citrix.com>
Sent: Thursday, July 26, 2018 7:46 PM
To: Paul Durrant
Cc: Bercaru, Gabriel; xen-devel; David Woodhouse; Jan Beulich; Belgun, Adrian
Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes

On Wed, Jul 25, 2018 at 05:19:03PM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Xen-devel [mailto:xen-devel-bounces@lists.xenproject.org] On Behalf
> > Of Roger Pau Monné
> > Sent: 25 July 2018 15:12
> > To: bercarug@amazon.com
> > Cc: xen-devel <xen-devel@lists.xenproject.org>; David Woodhouse
> > <dwmw2@infradead.org>; Jan Beulich <JBeulich@suse.com>;
> > abelgun@amazon.com
> > Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
> >
> > On Wed, Jul 25, 2018 at 04:57:23PM +0300, bercarug@amazon.com wrote:
> > > On 07/25/2018 04:35 PM, Roger Pau Monné wrote:
> > > > On Wed, Jul 25, 2018 at 01:06:43PM +0300, bercarug@amazon.com
> > wrote:
> > > > > On 07/24/2018 12:54 PM, Jan Beulich wrote:
> > > > > > > > > On 23.07.18 at 13:50, <bercarug@amazon.com> wrote:
> > > > > > > For the last few days, I have been trying to get a PVH dom0 running,
> > > > > > > however I encountered the following problem: the system seems
> > to
> > > > > > > freeze after the hypervisor boots, the screen goes black. I have
> > tried to
> > > > > > > debug it via a serial console (using Minicom) and managed to get
> > some
> > > > > > > more Xen output, after the screen turns black.
> > > > > > >
> > > > > > > I mention that I have tried to boot the PVH dom0 using different
> > kernel
> > > > > > > images (from 4.9.0 to 4.18-rc3), different Xen  versions (4.10, 4.11,
> > 4.12).
> > > > > > >
> > > > > > > Below I attached my system / hypervisor configuration, as well as
> > the
> > > > > > > output captured through the serial console, corresponding to the
> > latest
> > > > > > > versions for Xen and the Linux Kernel (Xen staging and Kernel from
> > the
> > > > > > > xen/tip tree).
> > > > > > > [...]
> > > > > > > (XEN) [VT-D]iommu.c:919: iommu_fault_status: Fault Overflow
> > > > > > > (XEN) [VT-D]iommu.c:921: iommu_fault_status: Primary Pending
> > Fault
> > > > > > > (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:14.0] fault
> > addr 8deb3000, iommu reg = ffff82c00021b000
> > > > Can you figure out which PCI device is 00:14.0?
> > > This is the output of lspci -vvv for device 00:14.0:
> > >
> > > 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI
> > > Controller (rev 31) (prog-if 30 [XHCI])
> > >         Subsystem: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller
> > >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > ParErr+
> > > Stepping- SERR+ FastB2B- DisINTx+
> > >         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> > > <TAbort- <MAbort+ >SERR- <PERR- INTx-
> > >         Latency: 0
> > >         Interrupt: pin A routed to IRQ 178
> > >         Region 0: Memory at a2e00000 (64-bit, non-prefetchable) [size=64K]
> > >         Capabilities: [70] Power Management version 2
> > >                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> > > PME(D0-,D1-,D2-,D3hot+,D3cold+)
> > >                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> > >         Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> > >                 Address: 00000000fee0e000  Data: 4021
> > >         Kernel driver in use: xhci_hcd
> > >         Kernel modules: xhci_pci
> >
> > I'm afraid your USB controller is missing RMRR entries in the DMAR
> > ACPI tables, thus causing the IOMMU faults and not working properly.
> >
> > You could try to manually add some extra rmrr regions by appending:
> >
> > rmrr=0x8deb3=0:0:14.0
> >
> > To the Xen command line, and keep adding any address that pops up in
> > the iommu faults. This is of course quite cumbersome, but there's no
> > way to get the required memory addresses if the data in RMRR is
> > wrong/incomplete.
> >
>
> You could just add all E820 reserved regions in there. That will almost certainly cover it.

I have a prototype patch for this that attempts to identity map all
reserved regions below 4GB to the p2m. It's still a WIP, but if you
could give it a try that would help me figure out whether this fixes
your issues and is indeed something that would be good to have.

I don't really like the patch as-is because it doesn't check whether
the reserved regions added to the p2m overlap with the LAPIC page or
the PCIe MCFG regions for example, I will continue to work on a safer
version.

If you can give this a shot, please remove any rmrr options from the
command line and use iommu=debug in order to catch any issues.

Thanks, Roger.
---8<---
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 2c44fabf99..76a1fd6681 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -21,6 +21,8 @@
 #include <xen/keyhandler.h>
 #include <xsm/xsm.h>

+#include <asm/setup.h>
+
 static int parse_iommu_param(const char *s);
 static void iommu_dump_p2m_table(unsigned char key);

@@ -47,6 +49,8 @@ integer_param("iommu_dev_iotlb_timeout", iommu_dev_iotlb_timeout);
  *   no-igfx                    Disable VT-d for IGD devices (insecure)
  *   no-amd-iommu-perdev-intremap Don't use per-device interrupt remapping
  *                              tables (insecure)
+ *   inclusive                  Include any memory ranges below 4GB not used
+ *                              by Xen or unusable to the iommu page tables.
  */
 custom_param("iommu", parse_iommu_param);
 bool_t __initdata iommu_enable = 1;
@@ -60,6 +64,7 @@ bool_t __read_mostly iommu_passthrough;
 bool_t __read_mostly iommu_snoop = 1;
 bool_t __read_mostly iommu_qinval = 1;
 bool_t __read_mostly iommu_intremap = 1;
+bool __read_mostly iommu_inclusive = true;

 /*
  * In the current implementation of VT-d posted interrupts, in some extreme
@@ -126,6 +131,8 @@ static int __init parse_iommu_param(const char *s)
             iommu_dom0_strict = val;
         else if ( !strncmp(s, "sharept", ss - s) )
             iommu_hap_pt_share = val;
+        else if ( !strncmp(s, "inclusive", ss - s) )
+            iommu_inclusive = val;
         else
             rc = -EINVAL;

@@ -165,6 +172,85 @@ static void __hwdom_init check_hwdom_reqs(struct domain *d)
     iommu_dom0_strict = 1;
 }

+static void __hwdom_init setup_inclusive_mappings(struct domain *d)
+{
+    unsigned long i, j, tmp, top, max_pfn;
+
+    BUG_ON(!is_hardware_domain(d));
+
+    max_pfn = (GB(4) >> PAGE_SHIFT) - 1;
+    top = max(max_pdx, pfn_to_pdx(max_pfn) + 1);
+
+    for ( i = 0; i < top; i++ )
+    {
+        unsigned long pfn = pdx_to_pfn(i);
+        bool map;
+        int rc = 0;
+
+        /*
+         * Set up 1:1 mapping for dom0. Default to include only
+         * conventional RAM areas and let RMRRs include needed reserved
+         * regions. When set, the inclusive mapping additionally maps in
+         * every pfn up to 4GB except those that fall in unusable ranges.
+         */
+        if ( pfn > max_pfn && !mfn_valid(_mfn(pfn)) )
+            continue;
+
+        if ( is_pv_domain(d) && iommu_inclusive && pfn <= max_pfn )
+            map = !page_is_ram_type(pfn, RAM_TYPE_UNUSABLE);
+        else if ( is_hvm_domain(d) && iommu_inclusive )
+            map = page_is_ram_type(pfn, RAM_TYPE_RESERVED);
+        else
+            map = page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL);
+
+        if ( !map )
+            continue;
+
+        /* Exclude Xen bits */
+        if ( xen_in_range(pfn) )
+            continue;
+
+        /*
+         * If dom0-strict mode is enabled or guest type is HVM/PVH then exclude
+         * conventional RAM and let the common code map dom0's pages.
+         */
+        if ( (iommu_dom0_strict || is_hvm_domain(d)) &&
+             page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL) )
+            continue;
+
+        /* For HVM avoid memory below 1MB because that's already mapped. */
+        if ( is_hvm_domain(d) && pfn < PFN_DOWN(MB(1)) )
+            continue;
+
+        tmp = 1 << (PAGE_SHIFT - PAGE_SHIFT_4K);
+        for ( j = 0; j < tmp; j++ )
+        {
+            int ret;
+
+            if ( iommu_use_hap_pt(d) )
+            {
+                ASSERT(is_hvm_domain(d));
+                ret = set_identity_p2m_entry(d, pfn * tmp + j, p2m_access_rw,
+                                             0);
+            }
+            else
+                ret = iommu_map_page(d, pfn * tmp + j, pfn * tmp + j,
+                                     IOMMUF_readable|IOMMUF_writable);
+
+            if ( !rc )
+               rc = ret;
+        }
+
+        if ( rc )
+            printk(XENLOG_WARNING " d%d: IOMMU mapping failed: %d\n",
+                   d->domain_id, rc);
+
+        if (!(i & (0xfffff >> (PAGE_SHIFT - PAGE_SHIFT_4K))))
+            process_pending_softirqs();
+    }
+
+}
+
 void __hwdom_init iommu_hwdom_init(struct domain *d)
 {
     const struct domain_iommu *hd = dom_iommu(d);
@@ -207,7 +293,10 @@ void __hwdom_init iommu_hwdom_init(struct domain *d)
                    d->domain_id, rc);
     }

-    return hd->platform_ops->hwdom_init(d);
+    hd->platform_ops->hwdom_init(d);
+
+    if ( !iommu_passthrough )
+        setup_inclusive_mappings(d);
 }

 void iommu_teardown(struct domain *d)
diff --git a/xen/drivers/passthrough/vtd/extern.h b/xen/drivers/passthrough/vtd/extern.h
index fb7edfaef9..91cadc602e 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -99,6 +99,4 @@ void pci_vtd_quirk(const struct pci_dev *);
 bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);

-void vtd_set_hwdom_mapping(struct domain *d);
-
 #endif // _VTD_EXTERN_H_
diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c
index 1710256823..569ec4aec2 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1304,12 +1304,6 @@ static void __hwdom_init intel_iommu_hwdom_init(struct domain *d)
 {
     struct acpi_drhd_unit *drhd;

-    if ( !iommu_passthrough && is_pv_domain(d) )
-    {
-        /* Set up 1:1 page table for hardware domain. */
-        vtd_set_hwdom_mapping(d);
-    }
-
     setup_hwdom_pci_devices(d, setup_hwdom_device);
     setup_hwdom_rmrr(d);

diff --git a/xen/drivers/passthrough/vtd/x86/vtd.c b/xen/drivers/passthrough/vtd/x86/vtd.c
index cc2bfea162..9971915349 100644
--- a/xen/drivers/passthrough/vtd/x86/vtd.c
+++ b/xen/drivers/passthrough/vtd/x86/vtd.c
@@ -32,11 +32,9 @@
 #include "../extern.h"

 /*
- * iommu_inclusive_mapping: when set, all memory below 4GB is included in dom0
- * 1:1 iommu mappings except xen and unusable regions.
+ * iommu_inclusive_mapping: superseded by iommu=inclusive.
  */
-static bool_t __hwdom_initdata iommu_inclusive_mapping = 1;
-boolean_param("iommu_inclusive_mapping", iommu_inclusive_mapping);
+boolean_param("iommu_inclusive_mapping", iommu_inclusive);

 void *map_vtd_domain_page(u64 maddr)
 {
@@ -107,67 +105,3 @@ void hvm_dpci_isairq_eoi(struct domain *d, unsigned int isairq)
     }
     spin_unlock(&d->event_lock);
 }
-
-void __hwdom_init vtd_set_hwdom_mapping(struct domain *d)
-{
-    unsigned long i, j, tmp, top, max_pfn;
-
-    BUG_ON(!is_hardware_domain(d));
-
-    max_pfn = (GB(4) >> PAGE_SHIFT) - 1;
-    top = max(max_pdx, pfn_to_pdx(max_pfn) + 1);
-
-    for ( i = 0; i < top; i++ )
-    {
-        unsigned long pfn = pdx_to_pfn(i);
-        bool map;
-        int rc = 0;
-
-        /*
-         * Set up 1:1 mapping for dom0. Default to include only
-         * conventional RAM areas and let RMRRs include needed reserved
-         * regions. When set, the inclusive mapping additionally maps in
-         * every pfn up to 4GB except those that fall in unusable ranges.
-         */
-        if ( pfn > max_pfn && !mfn_valid(_mfn(pfn)) )
-            continue;
-
-        if ( iommu_inclusive_mapping && pfn <= max_pfn )
-            map = !page_is_ram_type(pfn, RAM_TYPE_UNUSABLE);
-        else
-            map = page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL);
-
-        if ( !map )
-            continue;
-
-        /* Exclude Xen bits */
-        if ( xen_in_range(pfn) )
-            continue;
-
-        /*
-         * If dom0-strict mode is enabled then exclude conventional RAM
-         * and let the common code map dom0's pages.
-         */
-        if ( iommu_dom0_strict &&
-             page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL) )
-            continue;
-
-        tmp = 1 << (PAGE_SHIFT - PAGE_SHIFT_4K);
-        for ( j = 0; j < tmp; j++ )
-        {
-            int ret = iommu_map_page(d, pfn * tmp + j, pfn * tmp + j,
-                                     IOMMUF_readable|IOMMUF_writable);
-
-            if ( !rc )
-               rc = ret;
-        }
-
-        if ( rc )
-            printk(XENLOG_WARNING VTDPREFIX " d%d: IOMMU mapping failed: %d\n",
-                   d->domain_id, rc);
-
-        if (!(i & (0xfffff >> (PAGE_SHIFT - PAGE_SHIFT_4K))))
-            process_pending_softirqs();
-    }
-}
-
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 6b42e3b876..15d6584837 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -35,6 +35,7 @@ extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
 extern bool_t iommu_hap_pt_share;
 extern bool_t iommu_debug;
 extern bool_t amd_iommu_perdev_intremap;
+extern bool iommu_inclusive;

 extern unsigned int iommu_dev_iotlb_timeout;






Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #2: iommu.txt --]
[-- Type: text/plain, Size: 1732 bytes --]

(XEN) *** Building a PVH Dom0 ***
(XEN) [VT-D]d0:Hostbridge: skip 0000:00:00.0 map
(XEN) [VT-D]d0:PCI: map 0000:00:14.0
(XEN) [VT-D]d0:PCI: map 0000:00:14.2
(XEN) [VT-D]d0:PCI: map 0000:00:16.0
(XEN) [VT-D]d0:PCI: map 0000:00:16.1
(XEN) [VT-D]d0:PCI: map 0000:00:17.0
(XEN) [VT-D]d0:PCI: map 0000:00:1f.0
(XEN) [VT-D]d0:PCI: map 0000:00:1f.2
(XEN) [VT-D]d0:PCI: map 0000:00:1f.4
(XEN) [VT-D]d0:PCIe: map 0000:01:00.0
(XEN) [VT-D]d0:PCIe: map 0000:02:00.0
(XEN) [VT-D]d0:PCIe: map 0000:03:00.0
(XEN) [VT-D]d0:PCIe: map 0000:04:00.0
(XEN) [VT-D]iommu_enable_translation: iommu->reg = ffff82c00021b000
(XEN) hap.c:289: d0 failed to allocate from HAP pool
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN) Cannot setup identity map d0:fee00, gfn already mapped to 1021c63.
(XEN)  d0: IOMMU mapping failed: -16
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN)  d0: IOMMU mapping failed: -2
(XEN) WARNING: PVH is an experimental mode with limited functionality

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Fri, Jul 27, 2018 at 08:48:32AM +0000, Bercaru, Gabriel wrote:
> I tried the patch and it fixes the unusable USB devices problem.
> However, I captured the boot messages and the "IOMMU mapping failed" printk
> seems to have been executed on each iteration of the loop.
> 
> I attached a small section of the boot log. As I said, the warning log was displayed
> many more times, I removed a lot of them to keep the attached file short.

The patch is still a WIP, but it's good to know it solves your USB
issues :).

I think you are likely missing patch:

http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=173c7803592065d27bf2e60d50e08e197a0efa83

Can you try to update to latest staging or apply the patch and try
again?

I think if you have this patch applied the number of errors reported
by the IOMMU initialization should go down.

Thank, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Bercaru, Gabriel]

[-- Attachment #1: Type: text/plain, Size: 4568 bytes --]

I applied the match mentioned, but the system fails to boot. Instead, it
drops to a BusyBox shell. It seems to be a file system issue.

Following is a sequence of the boot log regarding the file system issue.

Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Begin: Waiting for suspend/resume device ... [    4.892146] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x6d5282a38d0, max_idle_ns: 881591194934 ns
[    4.903311] clocksource: Switched to clocksource tsc
Begin: Running /scripts/local-block ... done.
...
Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for suspend/resume device
done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root file system device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  UUID=93bcf82b-b9b9-4362-936a-d0c100a920da does not exist.  Dropping to a shell!


BusyBox v1.22.1 (Debian 1:1.22.0-19+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) ls /dev
char                snapshot            tty29               tty53
console             stderr              tty3                tty54
core                stdin               tty30               tty55
cpu_dma_latency     stdout              tty31               tty56
fd                  tty                 tty32               tty57
full                tty0                tty33               tty58
hpet                tty1                tty34               tty59
hvc0                tty10               tty35               tty6
hvc1                tty11               tty36               tty60
hvc2                tty12               tty37               tty61
hvc3                tty13               tty38               tty62
hvc4                tty14               tty39               tty63
hvc5                tty15               tty4                tty7
hvc6                tty16               tty40               tty8
hvc7                tty17               tty41               tty9
input               tty18               tty42               ttyS0
kmsg                tty19               tty43               ttyS1
mem                 tty2                tty44               ttyS2
memory_bandwidth    tty20               tty45               ttyS3
network_latency     tty21               tty46               urandom
network_throughput  tty22               tty47               vcs
null                tty23               tty48               vcs1
port                tty24               tty49               vcsa
psaux               tty25               tty5                vcsa1
ptmx                tty26               tty50               vga_arbiter
pts                 tty27               tty51               xen
random              tty28               tty52               zero
(initramfs)


I attached the full boot log.

Gabriel

________________________________________
From: Roger Pau Monné <roger.pau@citrix.com>
Sent: Friday, July 27, 2018 12:11 PM
To: Bercaru, Gabriel
Cc: Paul Durrant; xen-devel; David Woodhouse; Jan Beulich; Belgun, Adrian
Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes

On Fri, Jul 27, 2018 at 08:48:32AM +0000, Bercaru, Gabriel wrote:
> I tried the patch and it fixes the unusable USB devices problem.
> However, I captured the boot messages and the "IOMMU mapping failed" printk
> seems to have been executed on each iteration of the loop.
>
> I attached a small section of the boot log. As I said, the warning log was displayed
> many more times, I removed a lot of them to keep the attached file short.

The patch is still a WIP, but it's good to know it solves your USB
issues :).

I think you are likely missing patch:

http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=173c7803592065d27bf2e60d50e08e197a0efa83

Can you try to update to latest staging or apply the patch and try
again?

I think if you have this patch applied the number of errors reported
by the IOMMU initialization should go down.

Thank, Roger.



Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #2: fserror.cap --]
[-- Type: application/vnd.tcpdump.pcap, Size: 49404 bytes --]

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

Please try to avoid top posting.

On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
> I applied the match mentioned, but the system fails to boot. Instead, it
> drops to a BusyBox shell. It seems to be a file system issue.

So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
causes a regression for you?

As I understand it, before applying 173c780359206 you where capable of
booting the PVH Dom0, albeit with non-working USB?

And after applying 173c780359206 you are no longer able to boot?

> Following is a sequence of the boot log regarding the file system issue.

At least part of the issue seems to be that the IO-APIC information
provided to Dom0 is wrong (from the attached log):

[    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-0
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
[    0.000000] Failed to find ioapic for gsi : 2
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
[    0.000000] Failed to find ioapic for gsi : 9
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
[    0.000000] ERROR: Unable to locate IOAPIC for GSI 15

Can you try to boot with just the staging branch (current commit is
008a8fb249b9d433) and see how that goes?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

[-- Attachment #1: Type: text/plain, Size: 3650 bytes --]

On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
> Please try to avoid top posting.
>
> On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
>> I applied the match mentioned, but the system fails to boot. Instead, it
>> drops to a BusyBox shell. It seems to be a file system issue.
> So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
> causes a regression for you?
>
> As I understand it, before applying 173c780359206 you where capable of
> booting the PVH Dom0, albeit with non-working USB?
>
> And after applying 173c780359206 you are no longer able to boot?
Right, after applying 173c780359206 the system fails to boot (it drops 
to the BusyBox shell).
>> Following is a sequence of the boot log regarding the file system issue.
> At least part of the issue seems to be that the IO-APIC information
> provided to Dom0 is wrong (from the attached log):
>
> [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-0
> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> [    0.000000] Failed to find ioapic for gsi : 2
> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> [    0.000000] Failed to find ioapic for gsi : 9
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
>
> Can you try to boot with just the staging branch (current commit is
> 008a8fb249b9d433) and see how that goes?
>
> Thanks, Roger.
>
I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and 
the system boots,

however the USB problem persists. I was able to log in using the serial 
port and after executing

"xl list -l" the memory decrease problem appeared.


I attached the boot log. Following are some lines extracted from the 
log, regarding the USB

devices problem:

[    5.864084] usb 1-1: device descriptor read/64, error -71

[    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
[    7.571347] usb 1-1: Device not responding to setup address.

[    8.008031] usb 1-1: device not accepting address 4, error -71

[    8.609623] usb 1-1: device not accepting address 5, error -71


At the beginning of the log, there is a message regarding 
iommu_inclusive_mapping:


(XEN) [VT-D]found ACPI_DMAR_RMRR:
(XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved 
memory; need "iommu_inclusive_mapping=1"?
(XEN) [VT-D] endpoint: 0000:00:14.0


I mention that I tried to boot again using this command line option, but 
this message and the

USB messages persist.


Gabriel




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #2: staging.cap --]
[-- Type: application/vnd.tcpdump.pcap, Size: 68757 bytes --]

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Aug 08, 2018 at 10:46:40AM +0300, bercarug@amazon.com wrote:
> On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
> > Please try to avoid top posting.
> > 
> > On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
> > > I applied the match mentioned, but the system fails to boot. Instead, it
> > > drops to a BusyBox shell. It seems to be a file system issue.
> > So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
> > causes a regression for you?
> > 
> > As I understand it, before applying 173c780359206 you where capable of
> > booting the PVH Dom0, albeit with non-working USB?
> > 
> > And after applying 173c780359206 you are no longer able to boot?
> Right, after applying 173c780359206 the system fails to boot (it drops to
> the BusyBox shell).
> > > Following is a sequence of the boot log regarding the file system issue.
> > At least part of the issue seems to be that the IO-APIC information
> > provided to Dom0 is wrong (from the attached log):
> > 
> > [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-0
> > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > [    0.000000] Failed to find ioapic for gsi : 2
> > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > [    0.000000] Failed to find ioapic for gsi : 9
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
> > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
> > 
> > Can you try to boot with just the staging branch (current commit is
> > 008a8fb249b9d433) and see how that goes?
> > 
> > Thanks, Roger.
> > 
> I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and the
> system boots,

OK, so your issues where not caused by 173c780359206 then?

008a8fb249b9d433 already contains 173c780359206 because it was
committed earlier. In any case it's good to know you are able to boot
(albeit with issues) using the current staging branch.

> however the USB problem persists. I was able to log in using the serial port
> and after executing

Yes, the fixes for this issue have not been committed yet, see:

https://lists.xenproject.org/archives/html/xen-devel/2018-08/msg00528.html

If you want you can give this branch a try, it should hopefully solve
your USB issues.

> "xl list -l" the memory decrease problem appeared.

Yup, I will look into this now in order to find some kind of
workaround.

> I attached the boot log. Following are some lines extracted from the log,
> regarding the USB
> 
> devices problem:
> 
> [    5.864084] usb 1-1: device descriptor read/64, error -71
> 
> [    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
> [    7.571347] usb 1-1: Device not responding to setup address.
> 
> [    8.008031] usb 1-1: device not accepting address 4, error -71
> 
> [    8.609623] usb 1-1: device not accepting address 5, error -71
> 
> 
> At the beginning of the log, there is a message regarding
> iommu_inclusive_mapping:
> 
> 
> (XEN) [VT-D]found ACPI_DMAR_RMRR:
> (XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved memory;
> need "iommu_inclusive_mapping=1"?
> (XEN) [VT-D] endpoint: 0000:00:14.0
> 
> 
> I mention that I tried to boot again using this command line option, but
> this message and the
> 
> USB messages persist.

Yes, iommu_inclusive_mapping doesn't work for PVH, that's what my
patch series is trying to address. The error is caused by
missing/wrong RMRR regions in the ACPI tables.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

On 08/08/2018 11:08 AM, Roger Pau Monné wrote:
> On Wed, Aug 08, 2018 at 10:46:40AM +0300, bercarug@amazon.com wrote:
>> On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
>>> Please try to avoid top posting.
>>>
>>> On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
>>>> I applied the match mentioned, but the system fails to boot. Instead, it
>>>> drops to a BusyBox shell. It seems to be a file system issue.
>>> So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
>>> causes a regression for you?
>>>
>>> As I understand it, before applying 173c780359206 you where capable of
>>> booting the PVH Dom0, albeit with non-working USB?
>>>
>>> And after applying 173c780359206 you are no longer able to boot?
>> Right, after applying 173c780359206 the system fails to boot (it drops to
>> the BusyBox shell).
>>>> Following is a sequence of the boot log regarding the file system issue.
>>> At least part of the issue seems to be that the IO-APIC information
>>> provided to Dom0 is wrong (from the attached log):
>>>
>>> [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-0
>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
>>> [    0.000000] Failed to find ioapic for gsi : 2
>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
>>> [    0.000000] Failed to find ioapic for gsi : 9
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
>>>
>>> Can you try to boot with just the staging branch (current commit is
>>> 008a8fb249b9d433) and see how that goes?
>>>
>>> Thanks, Roger.
>>>
>> I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and the
>> system boots,
> OK, so your issues where not caused by 173c780359206 then?
>
> 008a8fb249b9d433 already contains 173c780359206 because it was
> committed earlier. In any case it's good to know you are able to boot
> (albeit with issues) using the current staging branch.
>
>> however the USB problem persists. I was able to log in using the serial port
>> and after executing
> Yes, the fixes for this issue have not been committed yet, see:
>
> https://lists.xenproject.org/archives/html/xen-devel/2018-08/msg00528.html
>
> If you want you can give this branch a try, it should hopefully solve
> your USB issues.
>
>> "xl list -l" the memory decrease problem appeared.
> Yup, I will look into this now in order to find some kind of
> workaround.
>
>> I attached the boot log. Following are some lines extracted from the log,
>> regarding the USB
>>
>> devices problem:
>>
>> [    5.864084] usb 1-1: device descriptor read/64, error -71
>>
>> [    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
>> [    7.571347] usb 1-1: Device not responding to setup address.
>>
>> [    8.008031] usb 1-1: device not accepting address 4, error -71
>>
>> [    8.609623] usb 1-1: device not accepting address 5, error -71
>>
>>
>> At the beginning of the log, there is a message regarding
>> iommu_inclusive_mapping:
>>
>>
>> (XEN) [VT-D]found ACPI_DMAR_RMRR:
>> (XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved memory;
>> need "iommu_inclusive_mapping=1"?
>> (XEN) [VT-D] endpoint: 0000:00:14.0
>>
>>
>> I mention that I tried to boot again using this command line option, but
>> this message and the
>>
>> USB messages persist.
> Yes, iommu_inclusive_mapping doesn't work for PVH, that's what my
> patch series is trying to address. The error is caused by
> missing/wrong RMRR regions in the ACPI tables.
>
> Thanks, Roger.

I tried compiling from the branch mentioned. I changed the command line by

adding "dom0-iommu=reserved", but I get the same error messages about USB

during boot.


Gabriel




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Paul Durrant]

> -----Original Message-----
> From: Roger Pau Monne
> Sent: 08 August 2018 09:08
> To: bercarug@amazon.com
> Cc: Paul Durrant <Paul.Durrant@citrix.com>; xen-devel <xen-
> devel@lists.xenproject.org>; David Woodhouse <dwmw2@infradead.org>;
> Jan Beulich <JBeulich@suse.com>; Belgun, Adrian <abelgun@amazon.com>
> Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
> 
> On Wed, Aug 08, 2018 at 10:46:40AM +0300, bercarug@amazon.com wrote:
> > On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
> > > Please try to avoid top posting.
> > >
> > > On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
> > > > I applied the match mentioned, but the system fails to boot. Instead, it
> > > > drops to a BusyBox shell. It seems to be a file system issue.
> > > So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
> > > causes a regression for you?
> > >
> > > As I understand it, before applying 173c780359206 you where capable of
> > > booting the PVH Dom0, albeit with non-working USB?
> > >
> > > And after applying 173c780359206 you are no longer able to boot?
> > Right, after applying 173c780359206 the system fails to boot (it drops to
> > the BusyBox shell).
> > > > Following is a sequence of the boot log regarding the file system issue.
> > > At least part of the issue seems to be that the IO-APIC information
> > > provided to Dom0 is wrong (from the attached log):
> > >
> > > [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-
> 0
> > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > > [    0.000000] Failed to find ioapic for gsi : 2
> > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > > [    0.000000] Failed to find ioapic for gsi : 9
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
> > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
> > >
> > > Can you try to boot with just the staging branch (current commit is
> > > 008a8fb249b9d433) and see how that goes?
> > >
> > > Thanks, Roger.
> > >
> > I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and
> the
> > system boots,
> 
> OK, so your issues where not caused by 173c780359206 then?
> 
> 008a8fb249b9d433 already contains 173c780359206 because it was
> committed earlier. In any case it's good to know you are able to boot
> (albeit with issues) using the current staging branch.
> 
> > however the USB problem persists. I was able to log in using the serial port
> > and after executing
> 
> Yes, the fixes for this issue have not been committed yet, see:
> 
> https://lists.xenproject.org/archives/html/xen-devel/2018-
> 08/msg00528.html
> 
> If you want you can give this branch a try, it should hopefully solve
> your USB issues.
> 
> > "xl list -l" the memory decrease problem appeared.
> 
> Yup, I will look into this now in order to find some kind of
> workaround.
> 
> > I attached the boot log. Following are some lines extracted from the log,
> > regarding the USB
> >
> > devices problem:
> >
> > [    5.864084] usb 1-1: device descriptor read/64, error -71
> >
> > [    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
> > [    7.571347] usb 1-1: Device not responding to setup address.
> >
> > [    8.008031] usb 1-1: device not accepting address 4, error -71
> >
> > [    8.609623] usb 1-1: device not accepting address 5, error -71
> >
> >
> > At the beginning of the log, there is a message regarding
> > iommu_inclusive_mapping:
> >
> >
> > (XEN) [VT-D]found ACPI_DMAR_RMRR:
> > (XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved
> memory;
> > need "iommu_inclusive_mapping=1"?
> > (XEN) [VT-D] endpoint: 0000:00:14.0
> >
> >
> > I mention that I tried to boot again using this command line option, but
> > this message and the
> >
> > USB messages persist.
> 
> Yes, iommu_inclusive_mapping doesn't work for PVH, that's what my
> patch series is trying to address. The error is caused by
> missing/wrong RMRR regions in the ACPI tables.
> 

Looks like this warning is suggesting that there is an RMRR that falls outside of an E820 reserved region. For PV I guess that 'inclusive' will fix this, but for PVH 'reserved' isn't going to fix it. I hope that the range at least falls in a hole, so maybe we also need a dom0_iommu=holes option too? Ugly, but maybe necessary.

  Paul

> Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Aug 08, 2018 at 09:43:39AM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Roger Pau Monne
> > Sent: 08 August 2018 09:08
> > To: bercarug@amazon.com
> > Cc: Paul Durrant <Paul.Durrant@citrix.com>; xen-devel <xen-
> > devel@lists.xenproject.org>; David Woodhouse <dwmw2@infradead.org>;
> > Jan Beulich <JBeulich@suse.com>; Belgun, Adrian <abelgun@amazon.com>
> > Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
> > 
> > On Wed, Aug 08, 2018 at 10:46:40AM +0300, bercarug@amazon.com wrote:
> > > On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
> > > > Please try to avoid top posting.
> > > >
> > > > On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
> > > > > I applied the match mentioned, but the system fails to boot. Instead, it
> > > > > drops to a BusyBox shell. It seems to be a file system issue.
> > > > So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
> > > > causes a regression for you?
> > > >
> > > > As I understand it, before applying 173c780359206 you where capable of
> > > > booting the PVH Dom0, albeit with non-working USB?
> > > >
> > > > And after applying 173c780359206 you are no longer able to boot?
> > > Right, after applying 173c780359206 the system fails to boot (it drops to
> > > the BusyBox shell).
> > > > > Following is a sequence of the boot log regarding the file system issue.
> > > > At least part of the issue seems to be that the IO-APIC information
> > > > provided to Dom0 is wrong (from the attached log):
> > > >
> > > > [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-
> > 0
> > > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > > > [    0.000000] Failed to find ioapic for gsi : 2
> > > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > > > [    0.000000] Failed to find ioapic for gsi : 9
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
> > > >
> > > > Can you try to boot with just the staging branch (current commit is
> > > > 008a8fb249b9d433) and see how that goes?
> > > >
> > > > Thanks, Roger.
> > > >
> > > I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and
> > the
> > > system boots,
> > 
> > OK, so your issues where not caused by 173c780359206 then?
> > 
> > 008a8fb249b9d433 already contains 173c780359206 because it was
> > committed earlier. In any case it's good to know you are able to boot
> > (albeit with issues) using the current staging branch.
> > 
> > > however the USB problem persists. I was able to log in using the serial port
> > > and after executing
> > 
> > Yes, the fixes for this issue have not been committed yet, see:
> > 
> > https://lists.xenproject.org/archives/html/xen-devel/2018-
> > 08/msg00528.html
> > 
> > If you want you can give this branch a try, it should hopefully solve
> > your USB issues.
> > 
> > > "xl list -l" the memory decrease problem appeared.
> > 
> > Yup, I will look into this now in order to find some kind of
> > workaround.
> > 
> > > I attached the boot log. Following are some lines extracted from the log,
> > > regarding the USB
> > >
> > > devices problem:
> > >
> > > [    5.864084] usb 1-1: device descriptor read/64, error -71
> > >
> > > [    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
> > > [    7.571347] usb 1-1: Device not responding to setup address.
> > >
> > > [    8.008031] usb 1-1: device not accepting address 4, error -71
> > >
> > > [    8.609623] usb 1-1: device not accepting address 5, error -71
> > >
> > >
> > > At the beginning of the log, there is a message regarding
> > > iommu_inclusive_mapping:
> > >
> > >
> > > (XEN) [VT-D]found ACPI_DMAR_RMRR:
> > > (XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved
> > memory;
> > > need "iommu_inclusive_mapping=1"?
> > > (XEN) [VT-D] endpoint: 0000:00:14.0
> > >
> > >
> > > I mention that I tried to boot again using this command line option, but
> > > this message and the
> > >
> > > USB messages persist.

Does USB work despite of the warning message?

> > Yes, iommu_inclusive_mapping doesn't work for PVH, that's what my
> > patch series is trying to address. The error is caused by
> > missing/wrong RMRR regions in the ACPI tables.
> > 
> 
> Looks like this warning is suggesting that there is an RMRR that falls outside of an E820 reserved region. For PV I guess that 'inclusive' will fix this, but for PVH 'reserved' isn't going to fix it. I hope that the range at least falls in a hole, so maybe we also need a dom0_iommu=holes option too? Ugly, but maybe necessary.

I wanted to avoid adding such option because I think it's going to
interact quite badly with BAR mappings.

Maybe it would make sense to add RMRR regions as long as they don't
reside in a RAM region and just print the warning message?

As a side note, it would be good that you report this to the vendor,
the firmware they are providing is broken and should be fixed.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

On 08/08/2018 11:51 AM, Roger Pau Monné wrote:
> On Wed, Aug 08, 2018 at 09:43:39AM +0100, Paul Durrant wrote:
>>> -----Original Message-----
>>> From: Roger Pau Monne
>>> Sent: 08 August 2018 09:08
>>> To: bercarug@amazon.com
>>> Cc: Paul Durrant <Paul.Durrant@citrix.com>; xen-devel <xen-
>>> devel@lists.xenproject.org>; David Woodhouse <dwmw2@infradead.org>;
>>> Jan Beulich <JBeulich@suse.com>; Belgun, Adrian <abelgun@amazon.com>
>>> Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
>>>
>>> On Wed, Aug 08, 2018 at 10:46:40AM +0300, bercarug@amazon.com wrote:
>>>> On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
>>>>> Please try to avoid top posting.
>>>>>
>>>>> On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
>>>>>> I applied the match mentioned, but the system fails to boot. Instead, it
>>>>>> drops to a BusyBox shell. It seems to be a file system issue.
>>>>> So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
>>>>> causes a regression for you?
>>>>>
>>>>> As I understand it, before applying 173c780359206 you where capable of
>>>>> booting the PVH Dom0, albeit with non-working USB?
>>>>>
>>>>> And after applying 173c780359206 you are no longer able to boot?
>>>> Right, after applying 173c780359206 the system fails to boot (it drops to
>>>> the BusyBox shell).
>>>>>> Following is a sequence of the boot log regarding the file system issue.
>>>>> At least part of the issue seems to be that the IO-APIC information
>>>>> provided to Dom0 is wrong (from the attached log):
>>>>>
>>>>> [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-
>>> 0
>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
>>>>> [    0.000000] Failed to find ioapic for gsi : 2
>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
>>>>> [    0.000000] Failed to find ioapic for gsi : 9
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
>>>>> [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
>>>>>
>>>>> Can you try to boot with just the staging branch (current commit is
>>>>> 008a8fb249b9d433) and see how that goes?
>>>>>
>>>>> Thanks, Roger.
>>>>>
>>>> I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and
>>> the
>>>> system boots,
>>> OK, so your issues where not caused by 173c780359206 then?
>>>
>>> 008a8fb249b9d433 already contains 173c780359206 because it was
>>> committed earlier. In any case it's good to know you are able to boot
>>> (albeit with issues) using the current staging branch.
>>>
>>>> however the USB problem persists. I was able to log in using the serial port
>>>> and after executing
>>> Yes, the fixes for this issue have not been committed yet, see:
>>>
>>> https://lists.xenproject.org/archives/html/xen-devel/2018-
>>> 08/msg00528.html
>>>
>>> If you want you can give this branch a try, it should hopefully solve
>>> your USB issues.
>>>
>>>> "xl list -l" the memory decrease problem appeared.
>>> Yup, I will look into this now in order to find some kind of
>>> workaround.
>>>
>>>> I attached the boot log. Following are some lines extracted from the log,
>>>> regarding the USB
>>>>
>>>> devices problem:
>>>>
>>>> [    5.864084] usb 1-1: device descriptor read/64, error -71
>>>>
>>>> [    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
>>>> [    7.571347] usb 1-1: Device not responding to setup address.
>>>>
>>>> [    8.008031] usb 1-1: device not accepting address 4, error -71
>>>>
>>>> [    8.609623] usb 1-1: device not accepting address 5, error -71
>>>>
>>>>
>>>> At the beginning of the log, there is a message regarding
>>>> iommu_inclusive_mapping:
>>>>
>>>>
>>>> (XEN) [VT-D]found ACPI_DMAR_RMRR:
>>>> (XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved
>>> memory;
>>>> need "iommu_inclusive_mapping=1"?
>>>> (XEN) [VT-D] endpoint: 0000:00:14.0
>>>>
>>>>
>>>> I mention that I tried to boot again using this command line option, but
>>>> this message and the
>>>>
>>>> USB messages persist.
> Does USB work despite of the warning message?
No, it does not.
>
>>> Yes, iommu_inclusive_mapping doesn't work for PVH, that's what my
>>> patch series is trying to address. The error is caused by
>>> missing/wrong RMRR regions in the ACPI tables.
>>>
>> Looks like this warning is suggesting that there is an RMRR that falls outside of an E820 reserved region. For PV I guess that 'inclusive' will fix this, but for PVH 'reserved' isn't going to fix it. I hope that the range at least falls in a hole, so maybe we also need a dom0_iommu=holes option too? Ugly, but maybe necessary.
> I wanted to avoid adding such option because I think it's going to
> interact quite badly with BAR mappings.
>
> Maybe it would make sense to add RMRR regions as long as they don't
> reside in a RAM region and just print the warning message?
>
> As a side note, it would be good that you report this to the vendor,
> the firmware they are providing is broken and should be fixed.
>
> Thanks, Roger.





Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Aug 08, 2018 at 11:54:40AM +0300, bercarug@amazon.com wrote:
> On 08/08/2018 11:51 AM, Roger Pau Monné wrote:
> > On Wed, Aug 08, 2018 at 09:43:39AM +0100, Paul Durrant wrote:
> > > > -----Original Message-----
> > > > From: Roger Pau Monne
> > > > Sent: 08 August 2018 09:08
> > > > To: bercarug@amazon.com
> > > > Cc: Paul Durrant <Paul.Durrant@citrix.com>; xen-devel <xen-
> > > > devel@lists.xenproject.org>; David Woodhouse <dwmw2@infradead.org>;
> > > > Jan Beulich <JBeulich@suse.com>; Belgun, Adrian <abelgun@amazon.com>
> > > > Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
> > > > 
> > > > On Wed, Aug 08, 2018 at 10:46:40AM +0300, bercarug@amazon.com wrote:
> > > > > On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
> > > > > > Please try to avoid top posting.
> > > > > > 
> > > > > > On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
> > > > > > > I applied the match mentioned, but the system fails to boot. Instead, it
> > > > > > > drops to a BusyBox shell. It seems to be a file system issue.
> > > > > > So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
> > > > > > causes a regression for you?
> > > > > > 
> > > > > > As I understand it, before applying 173c780359206 you where capable of
> > > > > > booting the PVH Dom0, albeit with non-working USB?
> > > > > > 
> > > > > > And after applying 173c780359206 you are no longer able to boot?
> > > > > Right, after applying 173c780359206 the system fails to boot (it drops to
> > > > > the BusyBox shell).
> > > > > > > Following is a sequence of the boot log regarding the file system issue.
> > > > > > At least part of the issue seems to be that the IO-APIC information
> > > > > > provided to Dom0 is wrong (from the attached log):
> > > > > > 
> > > > > > [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 0-
> > > > 0
> > > > > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > > > > > [    0.000000] Failed to find ioapic for gsi : 2
> > > > > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > > > > > [    0.000000] Failed to find ioapic for gsi : 9
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
> > > > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
> > > > > > 
> > > > > > Can you try to boot with just the staging branch (current commit is
> > > > > > 008a8fb249b9d433) and see how that goes?
> > > > > > 
> > > > > > Thanks, Roger.
> > > > > > 
> > > > > I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and
> > > > the
> > > > > system boots,
> > > > OK, so your issues where not caused by 173c780359206 then?
> > > > 
> > > > 008a8fb249b9d433 already contains 173c780359206 because it was
> > > > committed earlier. In any case it's good to know you are able to boot
> > > > (albeit with issues) using the current staging branch.
> > > > 
> > > > > however the USB problem persists. I was able to log in using the serial port
> > > > > and after executing
> > > > Yes, the fixes for this issue have not been committed yet, see:
> > > > 
> > > > https://lists.xenproject.org/archives/html/xen-devel/2018-
> > > > 08/msg00528.html
> > > > 
> > > > If you want you can give this branch a try, it should hopefully solve
> > > > your USB issues.
> > > > 
> > > > > "xl list -l" the memory decrease problem appeared.
> > > > Yup, I will look into this now in order to find some kind of
> > > > workaround.
> > > > 
> > > > > I attached the boot log. Following are some lines extracted from the log,
> > > > > regarding the USB
> > > > > 
> > > > > devices problem:
> > > > > 
> > > > > [    5.864084] usb 1-1: device descriptor read/64, error -71
> > > > > 
> > > > > [    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
> > > > > [    7.571347] usb 1-1: Device not responding to setup address.
> > > > > 
> > > > > [    8.008031] usb 1-1: device not accepting address 4, error -71
> > > > > 
> > > > > [    8.609623] usb 1-1: device not accepting address 5, error -71
> > > > > 
> > > > > 
> > > > > At the beginning of the log, there is a message regarding
> > > > > iommu_inclusive_mapping:
> > > > > 
> > > > > 
> > > > > (XEN) [VT-D]found ACPI_DMAR_RMRR:
> > > > > (XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved
> > > > memory;
> > > > > need "iommu_inclusive_mapping=1"?
> > > > > (XEN) [VT-D] endpoint: 0000:00:14.0
> > > > > 
> > > > > 
> > > > > I mention that I tried to boot again using this command line option, but
> > > > > this message and the
> > > > > 
> > > > > USB messages persist.
> > Does USB work despite of the warning message?
> No, it does not.

I just realized that I've dropped a chunk from my series while
rebasing, could you please try again with the following diff applied
on top of my series?

diff --git a/xen/drivers/passthrough/x86/iommu.c b/xen/drivers/passthrough/x86/iommu.c
index 6aec43ed1a..6271d8b671 100644
--- a/xen/drivers/passthrough/x86/iommu.c
+++ b/xen/drivers/passthrough/x86/iommu.c
@@ -209,7 +209,13 @@ void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
         if ( !hwdom_iommu_map(d, pfn, max_pfn) )
             continue;
 
-        rc = iommu_map_page(d, pfn, pfn, IOMMUF_readable|IOMMUF_writable);
+        if ( iommu_use_hap_pt(d) )
+        {
+            ASSERT(is_hvm_domain(d));
+            rc = set_identity_p2m_entry(d, pfn, p2m_access_rw, 0);
+        }
+        else
+            rc = iommu_map_page(d, pfn, pfn, IOMMUF_readable|IOMMUF_writable);
         if ( rc )
             printk(XENLOG_WARNING " d%d: IOMMU mapping failed: %d\n",
                    d->domain_id, rc);

> > > > Yes, iommu_inclusive_mapping doesn't work for PVH, that's what my
> > > > patch series is trying to address. The error is caused by
> > > > missing/wrong RMRR regions in the ACPI tables.
> > > > 
> > > Looks like this warning is suggesting that there is an RMRR that falls outside of an E820 reserved region. For PV I guess that 'inclusive' will fix this, but for PVH 'reserved' isn't going to fix it. I hope that the range at least falls in a hole, so maybe we also need a dom0_iommu=holes option too? Ugly, but maybe necessary.
> > I wanted to avoid adding such option because I think it's going to
> > interact quite badly with BAR mappings.
> > 
> > Maybe it would make sense to add RMRR regions as long as they don't
> > reside in a RAM region and just print the warning message?

I see that the RMRR region is mapped regardless of the error message,
so it means that the firmware hos both wrong RMRR regions and missing
entries...

I can try to make something like 'inclusive' work for PVH if the
current series doesn't solve your issues.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Roger Pau Monné]

On Wed, Aug 08, 2018 at 11:44:28AM +0200, Roger Pau Monné wrote:
> I just realized that I've dropped a chunk from my series while
> rebasing, could you please try again with the following diff applied
> on top of my series?
> 
> diff --git a/xen/drivers/passthrough/x86/iommu.c b/xen/drivers/passthrough/x86/iommu.c
> index 6aec43ed1a..6271d8b671 100644
> --- a/xen/drivers/passthrough/x86/iommu.c
> +++ b/xen/drivers/passthrough/x86/iommu.c
> @@ -209,7 +209,13 @@ void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
>          if ( !hwdom_iommu_map(d, pfn, max_pfn) )
>              continue;
>  
> -        rc = iommu_map_page(d, pfn, pfn, IOMMUF_readable|IOMMUF_writable);
> +        if ( iommu_use_hap_pt(d) )
> +        {
> +            ASSERT(is_hvm_domain(d));
> +            rc = set_identity_p2m_entry(d, pfn, p2m_access_rw, 0);
> +        }
> +        else
> +            rc = iommu_map_page(d, pfn, pfn, IOMMUF_readable|IOMMUF_writable);
>          if ( rc )
>              printk(XENLOG_WARNING " d%d: IOMMU mapping failed: %d\n",
>                     d->domain_id, rc);

I've pushed a new version that has this chunk, so it might be easier
for you to just fetch and test:

git://xenbits.xen.org/people/royger/xen.git iommu_inclusive_v4

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[bercarug]

On 08/08/2018 01:11 PM, Roger Pau Monné wrote:
> On Wed, Aug 08, 2018 at 11:44:28AM +0200, Roger Pau Monné wrote:
>> I just realized that I've dropped a chunk from my series while
>> rebasing, could you please try again with the following diff applied
>> on top of my series?
>>
>> diff --git a/xen/drivers/passthrough/x86/iommu.c b/xen/drivers/passthrough/x86/iommu.c
>> index 6aec43ed1a..6271d8b671 100644
>> --- a/xen/drivers/passthrough/x86/iommu.c
>> +++ b/xen/drivers/passthrough/x86/iommu.c
>> @@ -209,7 +209,13 @@ void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
>>           if ( !hwdom_iommu_map(d, pfn, max_pfn) )
>>               continue;
>>   
>> -        rc = iommu_map_page(d, pfn, pfn, IOMMUF_readable|IOMMUF_writable);
>> +        if ( iommu_use_hap_pt(d) )
>> +        {
>> +            ASSERT(is_hvm_domain(d));
>> +            rc = set_identity_p2m_entry(d, pfn, p2m_access_rw, 0);
>> +        }
>> +        else
>> +            rc = iommu_map_page(d, pfn, pfn, IOMMUF_readable|IOMMUF_writable);
>>           if ( rc )
>>               printk(XENLOG_WARNING " d%d: IOMMU mapping failed: %d\n",
>>                      d->domain_id, rc);
> I've pushed a new version that has this chunk, so it might be easier
> for you to just fetch and test:
>
> git://xenbits.xen.org/people/royger/xen.git iommu_inclusive_v4
>
> Thanks, Roger.
>
I already recompiled Xen using this patch, and the USB devices are 
functional again.


Thanks,

Gabriel




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PVH dom0 creation fails - the system freezes

[Jan Beulich]

>>> On 08.08.18 at 10:51, <roger.pau@citrix.com> wrote:
> On Wed, Aug 08, 2018 at 09:43:39AM +0100, Paul Durrant wrote:
>> > -----Original Message-----
>> > From: Roger Pau Monne
>> > Sent: 08 August 2018 09:08
>> > To: bercarug@amazon.com 
>> > Cc: Paul Durrant <Paul.Durrant@citrix.com>; xen-devel <xen-
>> > devel@lists.xenproject.org>; David Woodhouse <dwmw2@infradead.org>;
>> > Jan Beulich <JBeulich@suse.com>; Belgun, Adrian <abelgun@amazon.com>
>> > Subject: Re: [Xen-devel] PVH dom0 creation fails - the system freezes
>> > 
>> > On Wed, Aug 08, 2018 at 10:46:40AM +0300, bercarug@amazon.com wrote:
>> > > On 08/02/2018 04:55 PM, Roger Pau Monné wrote:
>> > > > Please try to avoid top posting.
>> > > >
>> > > > On Thu, Aug 02, 2018 at 11:36:26AM +0000, Bercaru, Gabriel wrote:
>> > > > > I applied the match mentioned, but the system fails to boot. Instead, it
>> > > > > drops to a BusyBox shell. It seems to be a file system issue.
>> > > > So you have applied 173c7803592065d27bf2e60d50e08e197a0efa83 and it
>> > > > causes a regression for you?
>> > > >
>> > > > As I understand it, before applying 173c780359206 you where capable of
>> > > > booting the PVH Dom0, albeit with non-working USB?
>> > > >
>> > > > And after applying 173c780359206 you are no longer able to boot?
>> > > Right, after applying 173c780359206 the system fails to boot (it drops to
>> > > the BusyBox shell).
>> > > > > Following is a sequence of the boot log regarding the file system issue.
>> > > > At least part of the issue seems to be that the IO-APIC information
>> > > > provided to Dom0 is wrong (from the attached log):
>> > > >
>> > > > [    0.000000] IOAPIC[0]: apic_id 2, version 152, address 0xfec00000, GSI 
> 0-
>> > 0
>> > > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
>> > > > [    0.000000] Failed to find ioapic for gsi : 2
>> > > > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high 
> level)
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
>> > > > [    0.000000] Failed to find ioapic for gsi : 9
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 1
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 2
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 3
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 4
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 5
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 6
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 7
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 8
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 9
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 10
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 11
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 12
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 13
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 14
>> > > > [    0.000000] ERROR: Unable to locate IOAPIC for GSI 15
>> > > >
>> > > > Can you try to boot with just the staging branch (current commit is
>> > > > 008a8fb249b9d433) and see how that goes?
>> > > >
>> > > > Thanks, Roger.
>> > > >
>> > > I recompiled Xen using the staging branch, commit 008a8fb249b9d433 and
>> > the
>> > > system boots,
>> > 
>> > OK, so your issues where not caused by 173c780359206 then?
>> > 
>> > 008a8fb249b9d433 already contains 173c780359206 because it was
>> > committed earlier. In any case it's good to know you are able to boot
>> > (albeit with issues) using the current staging branch.
>> > 
>> > > however the USB problem persists. I was able to log in using the serial 
> port
>> > > and after executing
>> > 
>> > Yes, the fixes for this issue have not been committed yet, see:
>> > 
>> > https://lists.xenproject.org/archives/html/xen-devel/2018- 
>> > 08/msg00528.html
>> > 
>> > If you want you can give this branch a try, it should hopefully solve
>> > your USB issues.
>> > 
>> > > "xl list -l" the memory decrease problem appeared.
>> > 
>> > Yup, I will look into this now in order to find some kind of
>> > workaround.
>> > 
>> > > I attached the boot log. Following are some lines extracted from the log,
>> > > regarding the USB
>> > >
>> > > devices problem:
>> > >
>> > > [    5.864084] usb 1-1: device descriptor read/64, error -71
>> > >
>> > > [    7.564025] usb 1-1: new full-speed USB device number 4 using xhci_hcd
>> > > [    7.571347] usb 1-1: Device not responding to setup address.
>> > >
>> > > [    8.008031] usb 1-1: device not accepting address 4, error -71
>> > >
>> > > [    8.609623] usb 1-1: device not accepting address 5, error -71
>> > >
>> > >
>> > > At the beginning of the log, there is a message regarding
>> > > iommu_inclusive_mapping:
>> > >
>> > >
>> > > (XEN) [VT-D]found ACPI_DMAR_RMRR:
>> > > (XEN) [VT-D]  RMRR address range 3e2e0000..3e2fffff not in reserved
>> > memory;
>> > > need "iommu_inclusive_mapping=1"?
>> > > (XEN) [VT-D] endpoint: 0000:00:14.0
>> > >
>> > >
>> > > I mention that I tried to boot again using this command line option, but
>> > > this message and the
>> > >
>> > > USB messages persist.
> 
> Does USB work despite of the warning message?
> 
>> > Yes, iommu_inclusive_mapping doesn't work for PVH, that's what my
>> > patch series is trying to address. The error is caused by
>> > missing/wrong RMRR regions in the ACPI tables.
>> > 
>> 
>> Looks like this warning is suggesting that there is an RMRR that falls 
> outside of an E820 reserved region. For PV I guess that 'inclusive' will fix 
> this, but for PVH 'reserved' isn't going to fix it. I hope that the range at 
> least falls in a hole, so maybe we also need a dom0_iommu=holes option too? 
> Ugly, but maybe necessary.
> 
> I wanted to avoid adding such option because I think it's going to
> interact quite badly with BAR mappings.
> 
> Maybe it would make sense to add RMRR regions as long as they don't
> reside in a RAM region and just print the warning message?

But the BAR problem would then still exist, unless we hand Dom0 a
fixed up E820 table.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel