From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 031F7C433E0 for ; Sat, 13 Feb 2021 18:22:27 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 684E864DDC for ; Sat, 13 Feb 2021 18:22:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 684E864DDC Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=m5p.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.84690.158773 (Exim 4.92) (envelope-from ) id 1lAzYb-0004Q5-Gl; Sat, 13 Feb 2021 18:22:13 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 84690.158773; Sat, 13 Feb 2021 18:22:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lAzYb-0004Py-DH; Sat, 13 Feb 2021 18:22:13 +0000 Received: by outflank-mailman (input) for mailman id 84690; Sat, 13 Feb 2021 18:22:12 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lAzYa-0004Pt-L0 for xen-devel@lists.xenproject.org; Sat, 13 Feb 2021 18:22:12 +0000 Received: from mailhost.m5p.com (unknown [74.104.188.4]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f0840c19-aaf9-4ba0-9d37-cb3a1ed926bf; Sat, 13 Feb 2021 18:22:11 +0000 (UTC) Received: from m5p.com (mailhost.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:f7]) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPS id 11DILu80096894 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 13 Feb 2021 13:22:02 -0500 (EST) (envelope-from ehem@m5p.com) Received: (from ehem@localhost) by m5p.com (8.16.1/8.15.2/Submit) id 11DILuWU096893; Sat, 13 Feb 2021 10:21:56 -0800 (PST) (envelope-from ehem) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f0840c19-aaf9-4ba0-9d37-cb3a1ed926bf Date: Sat, 13 Feb 2021 10:21:56 -0800 From: Elliott Mitchell To: Maximilian Engelhardt Cc: xen-devel@lists.xenproject.org, pkg-xen-devel@lists.alioth.debian.org Subject: Re: [BUG] Linux pvh vm not getting destroyed on shutdown Message-ID: References: <2195346.r5JaYcbZso@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2195346.r5JaYcbZso@localhost> On Sat, Feb 13, 2021 at 04:36:24PM +0100, Maximilian Engelhardt wrote: > after a recent upgrade of one of our test systems to Debian Bullseye we > noticed an issue where on shutdown of a pvh vm the vm was not destroyed by xen > automatically. It could still be destroyed by manually issuing a 'xl destroy > $vm' command. Usually I would expect such an issue to show on the Debian bug database before xen-devel. In particular as this is a behavior change with security updates, there is a good chance this isn't attributable to the Xen Project. Additionally the Xen Project's support window is rather narrow. I've been observing the same (or similar) issue for a bit too. > Here are some things I noticed while trying to debug this issue: > > * It happens on a Debian buster dom0 as well as on a bullseye dom0 I stick with stable on non-development machines, so I can't say anything to this. > * It seems to only affect pvh vms. I've observed it with pv and hvm VMs as well. > * shutdown from the pvgrub menu ("c" -> "halt") does work Woah! That is quite the observation. Since I had a handy opportunity I tried this and this reproduces for me. > * the vm seems to shut down normal, the last lines in the console are: I agree with this. Everything appears typical until the last moment. > * issuing a reboot instead of a shutdown does work fine. I disagree with this. I'm seeing the issue occur with restart attempts too. > * The issue started with Debian kernel 5.8.3+1~exp1 running in the vm, Debian > kernel 5.7.17-1 does not show the issue. I think the first kernel update during which I saw the issue was around linux-image-4.19.0-12-amd64 or linux-image-4.19.0-13-amd64. I think the last security update to the Xen packages was in a similar timeframe though. Rate this portion as unreliable though. I can definitely state this occurs with Debian's linux-image-4.19.0-13-amd64 and kernels built from corresponding source, this may have shown earlier. > * setting vcpus equal to maxvcpus does *not* show the hang. I haven't tried things related to this, so I can't comment on this part. Fresh observation. During a similar timeframe I started noticing VM creation leaving a `xl create` process behind. I had discovered this process could be freely killed without appearing to effect the VM and had thus been doing so (memory in a lean Dom0 is precious). While typing this I realized there was another scenario I needed to try. Turns out if I boot PV GRUB and get to its command-line (press 'c'), then get away from the VM console, kill the `xl create` process, return to the console and type "halt". This results in a hung VM. Are you perhaps either killing the `xl create` process for effected VMs, or migrating the VM and thus splitting the `xl create` process from the effected VMs? This seems more a Debian issue than a Xen Project issue right now. -- (\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/) \BS ( | ehem+sigmsg@m5p.com PGP 87145445 | ) / \_CS\ | _____ -O #include O- _____ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445