Re: Xen 4.7 crash

From: Julien Grall <julien.grall@arm.com>
To: Aaron Cornelius <Aaron.Cornelius@dornerworks.com>,
	Xen-devel <xen-devel@lists.xenproject.org>
Subject: Re: Xen 4.7 crash
Date: Wed, 1 Jun 2016 23:35:04 +0100	[thread overview]
Message-ID: <c238f2ed-a399-6dbd-0ac8-b210aa144234@arm.com> (raw)
In-Reply-To: <2D1D4CC433A3D1448C10AC5545A3471DF1378C0B@Quimby.dw.local>

Hello Aaron,

On 01/06/2016 20:54, Aaron Cornelius wrote:
> I am doing some work with Xen 4.7 on the cubietruck (ARM32).  I've noticed some strange behavior after I create/destroy enough domains and put together a script to do the add/remove for me.  For this particular test I am creating a small mini-os (Mirage) domain with 32MB of RAM, deleting it, creating the new one, and so on.
>
> After running this for a while, I get the following error (with version 8478c9409a2c6726208e8dbc9f3e455b76725a33):
>
> (d846) Virtual -> physical offset = 3fc00000
> (d846) Checking DTB at 023ff000...
> (d846) [32;1mMirageOS booting...[0m
> (d846) Initialising console ... done.
> (d846) gnttab_stubs.c: initialised mini-os gntmap
> (d846) allocate_ondemand(1, 1) returning 2300000
> (d846) allocate_ondemand(1, 1) returning 2301000
> (XEN) grant_table.c:3288:d0v1 Grant release (0) ref:(9) flags:(2) dom:(0)
> (XEN) grant_table.c:3288:d0v1 Grant release (1) ref:(11) flags:(2) dom:(0)
> (XEN) p2m.c: dom1101: VMID pool exhausted
> (XEN) CPU0: Unexpected Trap: Data Abort
> (XEN) ----[ Xen-4.7.0-rc  arm32  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) PC:     0021fdd4 free_domheap_pages+0x1c/0x324
> (XEN) CPSR:   6001011a MODE:Hypervisor
> (XEN)      R0: 00000000 R1: 00000001 R2: 00000003 R3: 00304320
> (XEN)      R4: 41c57000 R5: 41c57188 R6: 00200200 R7: 00100100
> (XEN)      R8: 41c57180 R9: 43fdfe60 R10:00000000 R11:43fdfd5c R12:00000000
> (XEN) HYP: SP: 43fdfd2c LR: 0025b0cc
> (XEN)
> (XEN)   VTCR_EL2: 80003558
> (XEN)  VTTBR_EL2: 00010000bfb0e000
> (XEN)
> (XEN)  SCTLR_EL2: 30cd187f
> (XEN)    HCR_EL2: 000000000038663f
> (XEN)  TTBR0_EL2: 00000000bfafc000
> (XEN)
> (XEN)    ESR_EL2: 94000006
> (XEN)  HPFAR_EL2: 000000000001c810
> (XEN)      HDFAR: 00000014
> (XEN)      HIFAR: 84e37182
> (XEN)
> (XEN) Xen stack trace from sp=43fdfd2c:
> (XEN)    002cf1b7 43fdfd64 41c57000 00000100 41c57000 41c57188 00200200 00100100
> (XEN)    41c57180 43fdfe60 00000000 43fdfd7c 0025b0cc 41c57000 fffffff0 43fdfe60
> (XEN)    0000001f 0000044d 43fdfe60 43fdfd8c 0024f668 41c57000 fffffff0 43fdfda4
> (XEN)    0024f8f0 41c57000 00000000 00000000 0000001f 43fdfddc 0020854c 43fdfddc
> (XEN)    00000000 cccccccd 00304600 002822bc 00000000 b6f20004 0000044d 00304600
> (XEN)    00304320 d767a000 00000000 43fdfeec 00206d6c 43fdfe6c 00218f8c 00000000
> (XEN)    00000007 43fdfe30 43fdfe34 00000000 43fdfe20 00000002 43fdfe48 43fdfe78
> (XEN)    00000000 00000000 00000000 00007622 00002b0e 40023000 00000000 43fdfec8
> (XEN)    00000002 43fdfebc 00218f8c 00000001 0000000b 0000ffff b6eba880 0000000b
> (XEN)    5abab87d f34aab2c 6adc50b8 e1713cd0 00000000 00000000 00000000 00000000
> (XEN)    b6eba8d8 00000000 50043f00 b6eb5038 b6effba8 0000003e 00000000 000c3034
> (XEN)    000b9cb8 000bda30 000bda30 00000000 b6eba56c 0000003e b6effba8 b6effdb0
> (XEN)    be9558d4 000000d0 be9558d4 00000071 b6effba8 b6effd6c b6ed6fb4 4a000ea1
> (XEN)    c01077f8 43fdff58 002067b8 00305000 be9557c8 d767a000 00000000 43fdff54
> (XEN)    00260130 00000000 43fdfefc 43fdff1c 200f019a 400238f4 00000004 00000004
> (XEN)    002c9f00 00000000 00304600 c094c240 00000000 00305000 be9557a0 d767a000
> (XEN)    00000000 43fdff44 00000000 c094c240 00000000 00305000 be9557c8 d767a000
> (XEN)    00000000 43fdff58 00263b10 b6f20004 00000000 00000000 00000000 00000000
> (XEN)    c094c240 00000000 00305000 be9557c8 d767a000 00000000 00000001 00000024
> (XEN)    ffffffff b691ab34 c01077f8 60010013 00000000 be9557c4 c0a38600 c010c400
> (XEN) Xen call trace:
> (XEN)    [<0021fdd4>] free_domheap_pages+0x1c/0x324 (PC)
> (XEN)    [<0025b0cc>] p2m_teardown+0xa0/0x108 (LR)
> (XEN)    [<0025b0cc>] p2m_teardown+0xa0/0x108
> (XEN)    [<0024f668>] arch_domain_destroy+0x20/0x50
> (XEN)    [<0024f8f0>] arch_domain_create+0x258/0x284
> (XEN)    [<0020854c>] domain_create+0x2dc/0x510
> (XEN)    [<00206d6c>] do_domctl+0x5b4/0x1928
> (XEN)    [<00260130>] do_trap_hypervisor+0x1170/0x15b0
> (XEN)    [<00263b10>] entry.o#return_from_trap+0/0x4
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) CPU0: Unexpected Trap: Data Abort
> (XEN)
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
>
> I'm not 100% sure, from the "VMID pool exhausted" message it would appear that the p2m_init() function failed to allocate a VM ID, which caused domain creation to fail, and the NULL pointer dereference when trying to clean up the not-fully-created domain.
>
> However, since I only have 1 domain active at a time, I'm not sure why I should run out of VM IDs.

arch_domain_destroy (and p2m_teardown) is only called when all the 
reference on the given domain are released.

It may take a while to release all the resources. So if you launch the 
domain as the same time as you destroy the previous guest. You will have 
more than 1 domain active.

Can you detail how you create/destroy guest?

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel