From: Wei Liu <wei.liu2@citrix.com>
To: Aaron Cornelius <aaron.cornelius@dornerworks.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>,
Wei Liu <wei.liu2@citrix.com>,
Ian Jackson <Ian.Jackson@eu.citrix.com>,
Julien Grall <julien.grall@arm.com>,
Jan Beulich <jbeulich@suse.com>,
Xen-devel <xen-devel@lists.xenproject.org>
Subject: Re: Xen 4.7 crash
Date: Tue, 14 Jun 2016 14:47:24 +0100 [thread overview]
Message-ID: <20160614134724.GO28116@citrix.com> (raw)
In-Reply-To: <8403bbee-01f1-e11c-0549-afdf11a64c3a@dornerworks.com>
On Tue, Jun 14, 2016 at 09:38:22AM -0400, Aaron Cornelius wrote:
> On 6/14/2016 9:26 AM, Aaron Cornelius wrote:
> >On 6/14/2016 9:15 AM, Wei Liu wrote:
> >>On Tue, Jun 14, 2016 at 09:11:47AM -0400, Aaron Cornelius wrote:
> >>>On 6/9/2016 7:14 AM, Ian Jackson wrote:
> >>>>Aaron Cornelius writes ("Re: [Xen-devel] Xen 4.7 crash"):
> >>>>>I am not that familiar with the xenstored code, but as far as I can tell
> >>>>>the grant mapping will be held by the xenstore until the xs_release()
> >>>>>function is called (which is not called by libxl, and I do not
> >>>>>explicitly call it in my software, although I might now just to be
> >>>>>safe), or until the last reference to a domain is released and the
> >>>>>registered destructor (destroy_domain), set by talloc_set_destructor(),
> >>>>>is called.
> >>>>
> >>>>I'm not sure I follow. Or maybe I disagree. ISTM that:
> >>>>
> >>>>The grant mapping is released by destroy_domain, which is called via
> >>>>the talloc destructor as a result of talloc_free(domain->conn) in
> >>>>domain_cleanup. I don't see other references to domain->conn.
> >>>>
> >>>>domain_cleanup calls talloc_free on domain->conn when it sees the
> >>>>domain marked as dying in domain_cleanup.
> >>>>
> >>>>So I still think that your acl reference ought not to keep the grant
> >>>>mapping alive.
> >>>
> >>>It took a while to complete the testing, but we've finished trying to
> >>>reproduce the error using oxenstored instead of the C xenstored. When the
> >>>condition occurs that caused the error with the C xenstored (on
> >>>4.7.0-rc4/8478c9409a2c6726208e8dbc9f3e455b76725a33), oxenstored does not
> >>>cause the crash.
> >>>
> >>>So for whatever reason, it would appear that the C xenstored does keep the
> >>>grant allocations open, but oxenstored does not.
> >>>
> >>
> >>Can you provide some easy to follow steps to reproduce this issue?
> >>
> >>AFAICT your environment is very specialised, but we should be able to
> >>trigger the issue with plan xenstore-* utilities?
> >
> >I am not sure if the plain xenstore-* utilities will work, but here are
> >the steps to follow:
> >
> >1. Create a non-standard xenstore path: /tool/test
> >2. Create a domU (mini-os/mirage/something small)
> >3. Add the new domU to the /tool/test permissions list (I'm not 100%
> >sure how to do this with the xenstore-* utilities)
> > a. call xs_get_permissions()
> > b. realloc() the permissions block to add the new domain
> > c. call xs_set_permissions()
> >4. Delete the domU from step 2
> >5. Repeat steps 2-4
> >
> >Eventually the xs_set_permissions() function will return an E2BIG error
> >because the list of domains has grown too large. Sometime after that is
> >when the crash occurs with the C xenstored and the 4.7.0-rc4 version of
> >Xen. It usually takes around 1200 or so iterations for the crash to occur.
>
> After writing up those steps I suddenly realized that I think I have a bug
> in my test that might have been causing the bug in the first place. Once I
> get errors returned from xs_set_permissions() I was not properly cleaning up
> the created domains. So I think this was just a simple case of VMID
> exhaustion by creating more than 255 domUs at the same time.
>
> In which case this is completely unrelated to xenstore holding on to grant
> allocations, and the C xenstore most likely behaves correctly.
>
OK, so I will treat this issue as resolved for now. Let us know if you
discover something new.
Wei.
> - Aaron Cornelius
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
prev parent reply other threads:[~2016-06-14 13:47 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-01 19:54 Xen 4.7 crash Aaron Cornelius
2016-06-01 20:00 ` Andrew Cooper
2016-06-01 20:45 ` Aaron Cornelius
2016-06-01 21:24 ` Andrew Cooper
2016-06-01 22:18 ` Julien Grall
2016-06-01 22:26 ` Andrew Cooper
2016-06-01 21:35 ` Andrew Cooper
2016-06-01 22:24 ` Julien Grall
2016-06-01 22:31 ` Andrew Cooper
2016-06-02 8:47 ` Jan Beulich
2016-06-02 8:53 ` Andrew Cooper
2016-06-02 9:07 ` Jan Beulich
2016-06-01 22:35 ` Julien Grall
2016-06-02 1:32 ` Aaron Cornelius
2016-06-02 8:49 ` Jan Beulich
2016-06-02 9:07 ` Julien Grall
2016-06-06 13:58 ` Aaron Cornelius
2016-06-06 14:05 ` Julien Grall
2016-06-06 14:19 ` Wei Liu
2016-06-06 15:02 ` Aaron Cornelius
2016-06-07 9:53 ` Ian Jackson
2016-06-07 13:40 ` Aaron Cornelius
2016-06-07 15:13 ` Aaron Cornelius
2016-06-09 11:14 ` Ian Jackson
2016-06-14 13:11 ` Aaron Cornelius
2016-06-14 13:15 ` Wei Liu
2016-06-14 13:26 ` Aaron Cornelius
2016-06-14 13:38 ` Aaron Cornelius
2016-06-14 13:47 ` Wei Liu [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160614134724.GO28116@citrix.com \
--to=wei.liu2@citrix.com \
--cc=Ian.Jackson@eu.citrix.com \
--cc=aaron.cornelius@dornerworks.com \
--cc=jbeulich@suse.com \
--cc=julien.grall@arm.com \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).