Xen 4.6 --with-ovmf, on UEFI Dom0 host up/running, but logging repeated "d1v0 Over-allocation for domain" ?

Xen 4.6 --with-ovmf, on UEFI Dom0 host up/running, but logging repeated "d1v0 Over-allocation for domain" ?

[PGNet Dev]

On UEFI Dom0 with

	xl info | egrep "release|version"
		release                : 4.5.0-2.gb2c9ae5-default
		version                : #1 SMP PREEMPT Wed Mar 16 17:30:21 UTC 2016 
(b2c9ae5)
		xen_version            : 4.620160318T070646

with Xen built with ovmf enabled, and guest booting UEFI,

my Host & Guests are, atm, up & running.

At Dom0 host, I'm seeing these constantly repeating (every 30-32 secs) 
messages

	...
	(XEN) [2016-03-18 14:54:35] d1v0 Over-allocation for domain 1: 524545 > 
524544
	...

I've only found one mention of this exact error, in an OVMF-related post,

	[Xen-devel] Regression in OVMF + RMRR series
	http://lists.xen.org/archives/html/xen-devel/2015-07/msg04901.html

but it was not the primary focus of that issue.

I understand EFI/OVMF dev is in heavy flux atm ...

Is this currently known/expected?  Or, more info needed?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.6 --with-ovmf, on UEFI Dom0 host up/running, but logging repeated "d1v0 Over-allocation for domain" ?

[Wei Liu]

On Fri, Mar 18, 2016 at 08:21:40AM -0700, PGNet Dev wrote:
> On UEFI Dom0 with
> 
> 	xl info | egrep "release|version"
> 		release                : 4.5.0-2.gb2c9ae5-default
> 		version                : #1 SMP PREEMPT Wed Mar 16 17:30:21 UTC 2016
> (b2c9ae5)
> 		xen_version            : 4.620160318T070646
> 
> with Xen built with ovmf enabled, and guest booting UEFI,
> 
> my Host & Guests are, atm, up & running.
> 
> At Dom0 host, I'm seeing these constantly repeating (every 30-32 secs)
> messages
> 
> 	...
> 	(XEN) [2016-03-18 14:54:35] d1v0 Over-allocation for domain 1: 524545 >
> 524544
> 	...
> 
> I've only found one mention of this exact error, in an OVMF-related post,
> 
> 	[Xen-devel] Regression in OVMF + RMRR series
> 	http://lists.xen.org/archives/html/xen-devel/2015-07/msg04901.html
> 
> but it was not the primary focus of that issue.
> 
> I understand EFI/OVMF dev is in heavy flux atm ...
> 
> Is this currently known/expected?  Or, more info needed?
> 

The log message isn't necessarily indication for a bug.

OVMF is huge, so it pushes up total memory consumption of the guest.  It
is normal that you see that message when using OVMF during boot time.

What is unclear to me is why your guest keeps trying to ask for more
pages after it boots up.  Is your guest trying to balloon in pages? Do
you see the same message repeated when using seabios?


Wei.

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.6 --with-ovmf, on UEFI Dom0 host up/running, but logging repeated "d1v0 Over-allocation for domain" ?

[PGNet Dev]

On 03/18/2016 08:48 AM, Wei Liu wrote:> The log message isn't 
necessarily indication for a bug.
>
> OVMF is huge, so it pushes up total memory consumption of the guest.  It
> is normal that you see that message when using OVMF during boot time.
>
> What is unclear to me is why your guest keeps trying to ask for more
> pages after it boots up.  Is your guest trying to balloon in pages?

Re: ballooning. I don't _think_ so, but not sure how to verify that.

Currently, my DomO's cmd line is

	xen_commandline        : dom0_mem=4096M,max:4096M dom0_max_vcpus=1 
dom0_vcpus_pin=true cpuidle=1 cpufreq=xen clocksource=hpet iommu=verbose 
sched=credit vga=gfx-1920x1080x16 com1=115200,8n1,pci console=com1,vga 
console_timestamps sync_console console_to_ring conring_size=64 
earlyprintk=vga,keep log_buf_len=16M sched_debug apic_verbosity=verbose 
debug loglvl=debug guest_loglvl=debug

Yes, I'm sure it's subject to improvement.

On the guest,

	Kernel command line: BOOT_IMAGE=/vmlinuz root=LABEL=ROOT 
resume=LABEL=SWAP kbdtype=us headless text quiet nofb apparmor=0 edd=off 
splash=silent noshell showopts net.ifnames=0 console=ttyS0,19200n8 
systemd.log_level=info systemd.log_target=kmsg

> Do you see the same message repeated when using seabios?

Haven't yet checked.

Atm, I've only UEFI guests, setup.  To test, I'll just have to spin up a 
non-UEFI guest.  A bit later ...

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.6 --with-ovmf, on UEFI Dom0 host up/running, but logging repeated "d1v0 Over-allocation for domain" ?

[Wei Liu]

On Fri, Mar 18, 2016 at 08:55:16AM -0700, PGNet Dev wrote:
> On 03/18/2016 08:48 AM, Wei Liu wrote:> The log message isn't necessarily
> indication for a bug.
> >
> >OVMF is huge, so it pushes up total memory consumption of the guest.  It
> >is normal that you see that message when using OVMF during boot time.
> >
> >What is unclear to me is why your guest keeps trying to ask for more
> >pages after it boots up.  Is your guest trying to balloon in pages?
> 
> Re: ballooning. I don't _think_ so, but not sure how to verify that.
> 

In Dom0:

  xenstore-ls -f /local/domain/$guest_domid

and paste it here.

> Currently, my DomO's cmd line is
> 
> 	xen_commandline        : dom0_mem=4096M,max:4096M dom0_max_vcpus=1
> dom0_vcpus_pin=true cpuidle=1 cpufreq=xen clocksource=hpet iommu=verbose
> sched=credit vga=gfx-1920x1080x16 com1=115200,8n1,pci console=com1,vga
> console_timestamps sync_console console_to_ring conring_size=64
> earlyprintk=vga,keep log_buf_len=16M sched_debug apic_verbosity=verbose
> debug loglvl=debug guest_loglvl=debug
> 
> Yes, I'm sure it's subject to improvement.
> 
> On the guest,
> 
> 	Kernel command line: BOOT_IMAGE=/vmlinuz root=LABEL=ROOT resume=LABEL=SWAP
> kbdtype=us headless text quiet nofb apparmor=0 edd=off splash=silent noshell
> showopts net.ifnames=0 console=ttyS0,19200n8 systemd.log_level=info
> systemd.log_target=kmsg
> 

First observation is that the log message starts with d1v0, which means
domain 1 vcpu 0, so Dom0 is irrelevant in that case.

The guest kernel command line looks normal.

You can also verify if your guest is trying to balloon in memory by
looking at sysfs knobs.

According to Linux kernel documentation, the relevant knobs live under

/sys/devices/system/xen_memory0/*

> >Do you see the same message repeated when using seabios?
> 
> Haven't yet checked.
> 
> Atm, I've only UEFI guests, setup.  To test, I'll just have to spin up a
> non-UEFI guest.  A bit later ...

Yes, this would be helpful for identifying the issue.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.6 --with-ovmf, on UEFI Dom0 host up/running, but logging repeated "d1v0 Over-allocation for domain" ?

[PGNet Dev]

> In Dom0:
>
>   xenstore-ls -f /local/domain/$guest_domid
>
> and paste it here.

xl list
Name                                        ID   Mem VCPUs      State 
Time(s)
	Domain-0                                     0  4096     1     r----- 
   111.3
	test-template                                1  2049     1     -b---- 
   106.3
xenstore-ls -f /local/domain/1
	/local/domain/1/vm = "/vm/7088e288-a0ba-4848-80e6-19a05f1772a2"
	/local/domain/1/name = "test-template"
	/local/domain/1/cpu = ""
	/local/domain/1/cpu/0 = ""
	/local/domain/1/cpu/0/availability = "online"
	/local/domain/1/memory = ""
	/local/domain/1/memory/static-max = "2097152"
	/local/domain/1/memory/target = "2080768"
	/local/domain/1/memory/videoram = "16384"
	/local/domain/1/device = ""
	/local/domain/1/device/suspend = ""
	/local/domain/1/device/suspend/event-channel = ""
	/local/domain/1/device/vbd = ""
	/local/domain/1/device/vbd/51712 = ""
	/local/domain/1/device/vbd/51712/backend = 
"/local/domain/0/backend/vbd/1/51712"
	/local/domain/1/device/vbd/51712/backend-id = "0"
	/local/domain/1/device/vbd/51712/state = "4"
	/local/domain/1/device/vbd/51712/virtual-device = "51712"
	/local/domain/1/device/vbd/51712/device-type = "disk"
	/local/domain/1/device/vbd/51712/ring-ref = "8"
	/local/domain/1/device/vbd/51712/event-channel = "15"
	/local/domain/1/device/vbd/51712/protocol = "x86_64-abi"
	/local/domain/1/device/vbd/51712/feature-persistent = "1"
	/local/domain/1/device/vbd/51776 = ""
	/local/domain/1/device/vbd/51776/backend = 
"/local/domain/0/backend/vbd/1/51776"
	/local/domain/1/device/vbd/51776/backend-id = "0"
	/local/domain/1/device/vbd/51776/state = "4"
	/local/domain/1/device/vbd/51776/virtual-device = "51776"
	/local/domain/1/device/vbd/51776/device-type = "disk"
	/local/domain/1/device/vbd/51776/ring-ref = "9"
	/local/domain/1/device/vbd/51776/event-channel = "16"
	/local/domain/1/device/vbd/51776/protocol = "x86_64-abi"
	/local/domain/1/device/vbd/51776/feature-persistent = "1"
	/local/domain/1/device/vbd/51792 = ""
	/local/domain/1/device/vbd/51792/backend = 
"/local/domain/0/backend/vbd/1/51792"
	/local/domain/1/device/vbd/51792/backend-id = "0"
	/local/domain/1/device/vbd/51792/state = "4"
	/local/domain/1/device/vbd/51792/virtual-device = "51792"
	/local/domain/1/device/vbd/51792/device-type = "disk"
	/local/domain/1/device/vbd/51792/ring-ref = "10"
	/local/domain/1/device/vbd/51792/event-channel = "17"
	/local/domain/1/device/vbd/51792/protocol = "x86_64-abi"
	/local/domain/1/device/vbd/51792/feature-persistent = "1"
	/local/domain/1/device/vbd/51808 = ""
	/local/domain/1/device/vbd/51808/backend = 
"/local/domain/0/backend/vbd/1/51808"
	/local/domain/1/device/vbd/51808/backend-id = "0"
	/local/domain/1/device/vbd/51808/state = "4"
	/local/domain/1/device/vbd/51808/virtual-device = "51808"
	/local/domain/1/device/vbd/51808/device-type = "disk"
	/local/domain/1/device/vbd/51808/ring-ref = "11"
	/local/domain/1/device/vbd/51808/event-channel = "18"
	/local/domain/1/device/vbd/51808/protocol = "x86_64-abi"
	/local/domain/1/device/vbd/51808/feature-persistent = "1"
	/local/domain/1/device/vkbd = ""
	/local/domain/1/device/vkbd/0 = ""
	/local/domain/1/device/vkbd/0/backend = "/local/domain/0/backend/vkbd/1/0"
	/local/domain/1/device/vkbd/0/backend-id = "0"
	/local/domain/1/device/vkbd/0/state = "4"
	/local/domain/1/device/vkbd/0/request-abs-pointer = "1"
	/local/domain/1/device/vkbd/0/page-ref = "253746"
	/local/domain/1/device/vkbd/0/page-gref = "1249"
	/local/domain/1/device/vkbd/0/event-channel = "22"
	/local/domain/1/device/vif = ""
	/local/domain/1/device/vif/0 = ""
	/local/domain/1/device/vif/0/backend = "/local/domain/0/backend/vif/1/0"
	/local/domain/1/device/vif/0/backend-id = "0"
	/local/domain/1/device/vif/0/state = "4"
	/local/domain/1/device/vif/0/handle = "0"
	/local/domain/1/device/vif/0/mac = "00:16:3e:10:00:01"
	/local/domain/1/device/vif/0/multi-queue-num-queues = "1"
	/local/domain/1/device/vif/0/tx-ring-ref = "768"
	/local/domain/1/device/vif/0/rx-ring-ref = "769"
	/local/domain/1/device/vif/0/event-channel-tx = "20"
	/local/domain/1/device/vif/0/event-channel-rx = "21"
	/local/domain/1/device/vif/0/request-rx-copy = "1"
	/local/domain/1/device/vif/0/feature-rx-notify = "1"
	/local/domain/1/device/vif/0/feature-sg = "1"
	/local/domain/1/device/vif/0/feature-gso-tcpv4 = "1"
	/local/domain/1/device/vif/0/feature-gso-tcpv6 = "1"
	/local/domain/1/device/vif/0/feature-ipv6-csum-offload = "1"
	/local/domain/1/control = ""
	/local/domain/1/control/shutdown = ""
	/local/domain/1/control/platform-feature-multiprocessor-suspend = "1"
	/local/domain/1/control/platform-feature-xs_reset_watches = "1"
	/local/domain/1/hvmloader = ""
	/local/domain/1/hvmloader/bios = "ovmf"
	/local/domain/1/hvmloader/allow-memory-relocate = "0"
	/local/domain/1/data = ""
	/local/domain/1/domid = "1"
	/local/domain/1/store = ""
	/local/domain/1/store/port = "1"
	/local/domain/1/store/ring-ref = "1044476"
	/local/domain/1/platform = ""
	/local/domain/1/platform/acpi = "1"
	/local/domain/1/platform/acpi_s3 = "1"
	/local/domain/1/platform/acpi_s4 = "1"
	/local/domain/1/console = ""
	/local/domain/1/console/backend = "/local/domain/0/backend/console/1/0"
	/local/domain/1/console/backend-id = "0"
	/local/domain/1/console/limit = "1048576"
	/local/domain/1/console/type = "xenconsoled"
	/local/domain/1/console/output = "pty"
	/local/domain/1/console/tty = "/dev/pts/0"
	/local/domain/1/console/port = "2"
	/local/domain/1/console/ring-ref = "1044479"
	/local/domain/1/console/vnc-listen = "0.0.0.0"
	/local/domain/1/console/vnc-port = "5901"
	/local/domain/1/image = ""
	/local/domain/1/image/device-model-pid = "3405"
	/local/domain/1/serial = ""
	/local/domain/1/serial/0 = ""
	/local/domain/1/serial/0/tty = "/dev/pts/1"

> First observation is that the log message starts with d1v0, which means
> domain 1 vcpu 0, so Dom0 is irrelevant in that case.

ok

> The guest kernel command line looks normal.
>
> You can also verify if your guest is trying to balloon in memory by
> looking at sysfs knobs.
>
> According to Linux kernel documentation, the relevant knobs live under
>
> /sys/devices/system/xen_memory0/*

Here, anyway, at Guest

	/sys/devices/system/xen_memory/xen_memory0/*

	tree /sys/devices/system/xen_memory/xen_memory0
		/sys/devices/system/xen_memory/xen_memory0
		├── info
		│   ├── current_kb
		│   ├── high_kb
		│   └── low_kb
		├── max_retry_count
		├── max_schedule_delay
		├── power
		│   ├── async
		│   ├── autosuspend_delay_ms
		│   ├── control
		│   ├── runtime_active_kids
		│   ├── runtime_active_time
		│   ├── runtime_enabled
		│   ├── runtime_status
		│   ├── runtime_suspended_time
		│   └── runtime_usage
		├── retry_count
		├── schedule_delay
		├── subsystem -> ../../../../bus/xen_memory
		├── target
		├── target_kb
		└── uevent

IIUC (?), it's the *_kb that are relevant

	find . | grep _kb
		./info/current_kb
		./info/low_kb
		./info/high_kb
		./target_kb

	cat `find . | grep _kb`
		2079272
		131064
		0
		2080768

Comparing to that at the Dom0

	cat `find . | grep _kb`
		4194304
		121064
		0
		4194304

I note only that at the Guest, (target_kb) > (current_kb).

Fwiw, in my Guest config

	...
	bios   = 'ovmf'
	maxmem = 2048
	memory = 2048
	hap    = 1
	shadow_memory = 16
	...


>>> Do you see the same message repeated when using seabios?
>>
>> Haven't yet checked.
>>
>> Atm, I've only UEFI guests, setup.  To test, I'll just have to spin up a
>> non-UEFI guest.  A bit later ...
>
> Yes, this would be helpful for identifying the issue.

I'll post back when I have an answer.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel