Re: Xen crashing when killing a domain with no VCPUs allocated

From: Ian Campbell <Ian.Campbell@citrix.com>
To: Julien Grall <julien.grall@linaro.org>
Cc: jgross@suse.com,
	Stefano Stabellini <stefano.stabellini@eu.citrix.com>,
	Dario Faggioli <dario.faggioli@citrix.com>,
	Tim Deegan <tim@xen.org>,
	george.dunlap@citrix.com, xen-devel <xen-devel@lists.xen.org>
Subject: Re: Xen crashing when killing a domain with no VCPUs allocated
Date: Fri, 18 Jul 2014 17:39:20 +0100	[thread overview]
Message-ID: <1405701560.14973.1.camel@kazak.uk.xensource.com> (raw)
In-Reply-To: <53C920DD.6060300@linaro.org>

On Fri, 2014-07-18 at 14:27 +0100, Julien Grall wrote:
> Hi all,
> 
> I've been played with the function alloc_vcpu on ARM. And I hit one case
> where this function can failed.
> 
> During domain creation, the toolstack will call DOMCTL_max_vcpus which may
> fail, for instance because alloc_vcpu didn't succeed. In this case, the
> toolstack will call DOMCTL_domaindestroy. And I got the below stack trace.
> 
> It can be reproduced on Xen 4.5 (and I also suspect Xen 4.4) by returning
> in an error in vcpu_initialize.
> 
> I'm not sure how to correctly fix it.

I think a simple check at the head of the function would be ok.

Alternatively perhaps in sched_mode_domain, which could either detect
this or could detect a domain in pool0 being moved to pool0 and short
circuit.

[...]
> (XEN)    [<00226870>] sched_move_domain+0x3cc/0x42c
> (XEN)    [<0020925c>] domain_kill+0xc8/0x178

This call path surprised me but it is from:

commit bac6334b51d9bcfe57ecf4a4cb5288348fcf044a
Author: Juergen Gross <juergen.gross@ts.fujitsu.com>
Date:   Tue May 20 15:55:42 2014 +0200

    move domain to cpupool0 before destroying it

    Currently when a domain is destroyed it is removed from the domain_list
    before all of it's resources, including the cpupool membership, are freed.
    This can lead to a situation where the domain is still member of a cpupool
    without for_each_domain_in_cpupool() (or even for_each_domain()) being
    able to find it any more. This in turn can result in rejection of removing
    the last cpu from a cpupool, because there seems to be still a domain in
    the cpupool, even if it can't be found by scanning through all domains.

    This situation can be avoided by moving the domain to be destroyed to
    cpupool0 first and then remove it from this cpupool BEFORE deleting it from
    the domain_list. As cpupool0 is always active and a domain without any cpupool
    membership is implicitly regarded as belonging to cpupool0, this poses no
    problem.