Re: Xen 4.7 crash

[Aaron Cornelius <aaron.cornelius@dornerworks.com>]

On 6/14/2016 9:15 AM, Wei Liu wrote:
> On Tue, Jun 14, 2016 at 09:11:47AM -0400, Aaron Cornelius wrote:
>> On 6/9/2016 7:14 AM, Ian Jackson wrote:
>>> Aaron Cornelius writes ("Re: [Xen-devel] Xen 4.7 crash"):
>>>> I am not that familiar with the xenstored code, but as far as I can tell
>>>> the grant mapping will be held by the xenstore until the xs_release()
>>>> function is called (which is not called by libxl, and I do not
>>>> explicitly call it in my software, although I might now just to be
>>>> safe), or until the last reference to a domain is released and the
>>>> registered destructor (destroy_domain), set by talloc_set_destructor(),
>>>> is called.
>>>
>>> I'm not sure I follow.  Or maybe I disagree.  ISTM that:
>>>
>>> The grant mapping is released by destroy_domain, which is called via
>>> the talloc destructor as a result of talloc_free(domain->conn) in
>>> domain_cleanup.  I don't see other references to domain->conn.
>>>
>>> domain_cleanup calls talloc_free on domain->conn when it sees the
>>> domain marked as dying in domain_cleanup.
>>>
>>> So I still think that your acl reference ought not to keep the grant
>>> mapping alive.
>>
>> It took a while to complete the testing, but we've finished trying to
>> reproduce the error using oxenstored instead of the C xenstored.  When the
>> condition occurs that caused the error with the C xenstored (on
>> 4.7.0-rc4/8478c9409a2c6726208e8dbc9f3e455b76725a33), oxenstored does not
>> cause the crash.
>>
>> So for whatever reason, it would appear that the C xenstored does keep the
>> grant allocations open, but oxenstored does not.
>>
>
> Can you provide some easy to follow steps to reproduce this issue?
>
> AFAICT your environment is very specialised, but we should be able to
> trigger the issue with plan xenstore-* utilities?

I am not sure if the plain xenstore-* utilities will work, but here are 
the steps to follow:

1. Create a non-standard xenstore path: /tool/test
2. Create a domU (mini-os/mirage/something small)
3. Add the new domU to the /tool/test permissions list (I'm not 100% 
sure how to do this with the xenstore-* utilities)
    a. call xs_get_permissions()
    b. realloc() the permissions block to add the new domain
    c. call xs_set_permissions()
4. Delete the domU from step 2
5. Repeat steps 2-4

Eventually the xs_set_permissions() function will return an E2BIG error 
because the list of domains has grown too large.  Sometime after that is 
when the crash occurs with the C xenstored and the 4.7.0-rc4 version of 
Xen.  It usually takes around 1200 or so iterations for the crash to occur.

- Aaron Cornelius

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel