From: Julien Grall <julien@xen.org>
To: Jim Fehlig <jfehlig@suse.com>, Jan Beulich <jbeulich@suse.com>
Cc: Hongyan Xia <hx242@xen.org>, Charles Arnold <CARNOLD@suse.com>,
Stefano Stabellini <sstabellini@kernel.org>, Wei Liu <wl@xen.org>,
Andrew Cooper <andrew.cooper3@citrix.com>,
Ian Jackson <ian.jackson@eu.citrix.com>,
George Dunlap <george.dunlap@citrix.com>,
xen-devel@lists.xenproject.org
Subject: Re: [PATCH] Revert "domctl: improve locking during domain destruction"
Date: Tue, 31 Mar 2020 11:31:30 +0100 [thread overview]
Message-ID: <94fc29a5-bfa4-4361-0654-789f2b80c89c@xen.org> (raw)
In-Reply-To: <547e509f-93ba-2bbf-f12d-21b9443e12e4@suse.com>
Hi Jim,
On 26/03/2020 16:55, Jim Fehlig wrote:
> On 3/25/20 1:11 AM, Jan Beulich wrote:
>> On 24.03.2020 19:39, Julien Grall wrote:
>>> On 24/03/2020 16:13, Jan Beulich wrote:
>>>> On 24.03.2020 16:21, Hongyan Xia wrote:
>>>>> From: Hongyan Xia <hongyxia@amazon.com>
>>>>> In contrast,
>>>>> after dropping that commit, parallel domain destructions will just
>>>>> fail
>>>>> to take the domctl lock, creating a hypercall continuation and backing
>>>>> off immediately, allowing the thread that holds the lock to destroy a
>>>>> domain much more quickly and allowing backed-off threads to process
>>>>> events and irqs.
>>>>>
>>>>> On a 144-core server with 4TiB of memory, destroying 32 guests (each
>>>>> with 4 vcpus and 122GiB memory) simultaneously takes:
>>>>>
>>>>> before the revert: 29 minutes
>>>>> after the revert: 6 minutes
>>>>
>>>> This wants comparing against numbers demonstrating the bad effects of
>>>> the global domctl lock. Iirc they were quite a bit higher than 6 min,
>>>> perhaps depending on guest properties.
>>>
>>> Your original commit message doesn't contain any clue in which
>>> cases the domctl lock was an issue. So please provide information
>>> on the setups you think it will make it worse.
>>
>> I did never observe the issue myself - let's see whether one of the SUSE
>> people possibly involved in this back then recall (or have further
>> pointers; Jim, Charles?), or whether any of the (partly former) Citrix
>> folks do. My vague recollection is that the issue was the tool stack as
>> a whole stalling for far too long in particular when destroying very
>> large guests.
>
> I too only have a vague memory of the issue but do recall shutting down
> large guests (e.g. 500GB) taking a long time and blocking other
> toolstack operations. I haven't checked on the behavior in quite some
> time though.
It might be worth checking how toolstack operations (such as domain
creating) is affected by the revert. @Hongyan would you be able to test it?
>
>> One important aspect not discussed in the commit message
>> at all is that holding the domctl lock block basically _all_ tool stack
>> operations (including e.g. creation of new guests), whereas the new
>> issue attempted to be addressed is limited to just domain cleanup.
>
> I more vaguely recall shutting down the host taking a *long* time when
> dom0 had large amounts of memory, e.g. when it had all host memory (no
> dom0_mem= setting and autoballooning enabled).
AFAIK, we never relinquish memory from dom0. So I am not sure how a
large amount of memory in Dom0 would affect the host shutting down.
Cheers,
--
Julien Grall
next prev parent reply other threads:[~2020-03-31 10:31 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-24 15:21 [Xen-devel] [PATCH] Revert "domctl: improve locking during domain destruction" Hongyan Xia
2020-03-24 16:13 ` Jan Beulich
2020-03-24 18:39 ` Julien Grall
2020-03-25 7:11 ` Jan Beulich
2020-03-26 14:39 ` Hongyan Xia
2020-03-26 16:55 ` Jim Fehlig
2020-03-31 10:31 ` Julien Grall [this message]
2020-03-24 18:40 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=94fc29a5-bfa4-4361-0654-789f2b80c89c@xen.org \
--to=julien@xen.org \
--cc=CARNOLD@suse.com \
--cc=andrew.cooper3@citrix.com \
--cc=george.dunlap@citrix.com \
--cc=hx242@xen.org \
--cc=ian.jackson@eu.citrix.com \
--cc=jbeulich@suse.com \
--cc=jfehlig@suse.com \
--cc=sstabellini@kernel.org \
--cc=wl@xen.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).