Re: Xen 4.7 crash

[Aaron Cornelius <aaron.cornelius@dornerworks.com>]

On 6/2/2016 5:07 AM, Julien Grall wrote:
> Hello Aaron,
>
> On 02/06/2016 02:32, Aaron Cornelius wrote:
>> This is with a custom application, we use the libxl APIs to interact
>> with Xen.  Domains are created using the libxl_domain_create_new()
>> function, and domains are destroyed using the libxl_domain_destroy()
>> function.
>>
>> The test in this case creates a domain, waits a minute, then
>> deletes/creates the next domain, waits a minute, and so on.  So I
>> wouldn't be surprised to see the VMID occasionally indicate there are 2
>> active domains since there could be one being created and one being
>> destroyed in a very short time.  However, I wouldn't expect to ever have
>> 256 domains.
>
> Your log has:
>
> (XEN) grant_table.c:3288:d0v1 Grant release (0) ref:(9) flags:(2) dom:(0)
> (XEN) grant_table.c:3288:d0v1 Grant release (1) ref:(11) flags:(2) dom:(0)
>
> Which suggest that some grants are still mapped in DOM0.
>
>>
>> The CubieTruck only has 2GB of RAM, I allocate 512MB for dom0 which
>> means that only 48 of the the Mirage domains (with 32MB of RAM) would
>> work at the same time anyway.  Which doesn't account for the various
>> inter-domain resources or the RAM used by Xen itself.
>
> All the pages who belongs to the domain could have been freed except the
> one referenced by DOM0. So the footprint of this domain will be limited
> at the time.
>
> I would recommend you to check how many domain are running at this time
> and if DOM0 effectively released all the resources.
>
>> If the p2m_teardown() function checked for NULL it would prevent the
>> crash, but I suspect Xen would be just as broken since all of my
>> resources have leaked away.  More broken in fact, since if the board
>> reboots at least the applications will restart and domains can be
>> recreated.
>>
>> It certainly appears that some resources are leaking when domains are
>> deleted (possibly only on the ARM or ARM32 platforms).  We will try to
>> add some debug prints and see if we can discover exactly what is going on.
>
> The leakage could also happen from DOM0. FWIW, I have been able to cycle
> 2000 guests over the night on an ARM platforms.
>

We've done some more testing regarding this issue.  And further testing 
shows that it doesn't matter if we delete the vchans before the domains 
are deleted.  Those appear to be cleaned up correctly when the domain is 
destroyed.

What does stop this issue from happening (using the same version of Xen 
that the issue was detected on) is removing any non-standard xenstore 
references before deleting the domain.  In this case our application 
allocates permissions for created domains to non-standard xenstore 
paths.  Making sure to remove those domain permissions before deleting 
the domain prevents this issue from happening.

It does not appear to matter if we delete the standard domain xenstore 
path (/local/domain/<id>) since libxl handles removing this path when 
the domain is destroyed.

Based on this I would guess that the xenstore is hanging onto the VMID.

- Aaron Cornelius

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel