Re: Xen 4.7 crash

From: Julien Grall <julien.grall@arm.com>
To: Aaron Cornelius <aaron.cornelius@dornerworks.com>,
	Xen-devel <xen-devel@lists.xenproject.org>,
	Jan Beulich <jbeulich@suse.com>
Subject: Re: Xen 4.7 crash
Date: Thu, 2 Jun 2016 10:07:34 +0100	[thread overview]
Message-ID: <21ecc73a-80fd-29df-87fd-b99ef570823e@arm.com> (raw)
In-Reply-To: <da971a8b-c7f6-9e0c-dbcb-48c0f34d534f@dornerworks.com>

Hello Aaron,

On 02/06/2016 02:32, Aaron Cornelius wrote:
> This is with a custom application, we use the libxl APIs to interact
> with Xen.  Domains are created using the libxl_domain_create_new()
> function, and domains are destroyed using the libxl_domain_destroy()
> function.
>
> The test in this case creates a domain, waits a minute, then
> deletes/creates the next domain, waits a minute, and so on.  So I
> wouldn't be surprised to see the VMID occasionally indicate there are 2
> active domains since there could be one being created and one being
> destroyed in a very short time.  However, I wouldn't expect to ever have
> 256 domains.

Your log has:

(XEN) grant_table.c:3288:d0v1 Grant release (0) ref:(9) flags:(2) dom:(0)
(XEN) grant_table.c:3288:d0v1 Grant release (1) ref:(11) flags:(2) dom:(0)

Which suggest that some grants are still mapped in DOM0.

>
> The CubieTruck only has 2GB of RAM, I allocate 512MB for dom0 which
> means that only 48 of the the Mirage domains (with 32MB of RAM) would
> work at the same time anyway.  Which doesn't account for the various
> inter-domain resources or the RAM used by Xen itself.

All the pages who belongs to the domain could have been freed except the 
one referenced by DOM0. So the footprint of this domain will be limited 
at the time.

I would recommend you to check how many domain are running at this time 
and if DOM0 effectively released all the resources.

> If the p2m_teardown() function checked for NULL it would prevent the
> crash, but I suspect Xen would be just as broken since all of my
> resources have leaked away.  More broken in fact, since if the board
> reboots at least the applications will restart and domains can be
> recreated.
>
> It certainly appears that some resources are leaking when domains are
> deleted (possibly only on the ARM or ARM32 platforms).  We will try to
> add some debug prints and see if we can discover exactly what is going on.

The leakage could also happen from DOM0. FWIW, I have been able to cycle 
2000 guests over the night on an ARM platforms.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel