From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF155C433DB for ; Thu, 18 Feb 2021 05:21:49 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5A4D664E33 for ; Thu, 18 Feb 2021 05:21:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5A4D664E33 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=zededa.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.86484.162421 (Exim 4.92) (envelope-from ) id 1lCbkt-0004FC-38; Thu, 18 Feb 2021 05:21:35 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 86484.162421; Thu, 18 Feb 2021 05:21:35 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lCbks-0004F5-Vy; Thu, 18 Feb 2021 05:21:34 +0000 Received: by outflank-mailman (input) for mailman id 86484; Thu, 18 Feb 2021 05:21:34 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lCbkr-0004Ey-Ki for xen-devel@lists.xenproject.org; Thu, 18 Feb 2021 05:21:34 +0000 Received: from mail-qv1-xf33.google.com (unknown [2607:f8b0:4864:20::f33]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f57f9047-4b19-4c3d-8dcf-675fab17a18f; Thu, 18 Feb 2021 05:21:32 +0000 (UTC) Received: by mail-qv1-xf33.google.com with SMTP id a1so401858qvd.13 for ; Wed, 17 Feb 2021 21:21:32 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f57f9047-4b19-4c3d-8dcf-675fab17a18f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zededa.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1aGb/J92Mb3UQqgHwhGKXT1nP4FiVjOT1qjd79sU3k4=; b=Pp1+Mt82q6+hCnorlaQaoh/Us1KLXpzFpd5sfB1DAqqSkNuDbNe5YM7A4iyVZm0dn/ nYFnxG1J7kiwI4wxJ/+KPSol6Nytn9gLNdpB/9h9qhk8X4yfMSaFLpSuiz/bGzmXNpMN K2eYP999AMzmFDkYOTMlSnlumLXtGM0CygPDQfoKcMVzqiSEmz93OptpDBuBgVxp3cMx NiNp06Np5JdL6Od/AqDQrIH+eJ+7x7k3+i9jmcFnpLtjQ2n6rAUd9Y6IY+aXkV2tcy/w qKkHcVoDHGUfdncujo1IlwBRsTJGdopsY4tS1VOgtKycMt/mKp3VkQmuxEU/cOoA1eCz 5m4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1aGb/J92Mb3UQqgHwhGKXT1nP4FiVjOT1qjd79sU3k4=; b=shEb7toKHTZwVuZc5apAlgcT0jWFSjP5EDqvSaqXtJkqPU7zjcGcUaKc/pzJlcs2d0 Cn+m5kUUKTkQ75nYaUPYOYzmua5onMirTy0sFtI/R50afOIlEuQNpBLcUhvwAoy9rGy8 RKp/35oSIR/z9ZGQF1+FlmY/IkxK+25LyDbyqWQMZT86lmpu7UMsJmpno+WGumiawA5b eF71o9SOeOcoNNKh8UYC8i22bHBKfZWHoB9OF9km5XgxCOw3Cwxp/Ltc5GqmbYKQwVVt B+Deu2x5cg2VFmNiav3CT+BEe36kpav+rdb0hli3SbHv638s/nU5Q5VYok3soclxsgqD 0h3A== X-Gm-Message-State: AOAM531xN+4hv3XKgbA99VmKpum2LRew1wkWssJNfsTt/gxKuitjIr0k bqzNDFQx4/XYED56dICOjsrfnQgZ3Lg2SNJ1W5MpUA== X-Google-Smtp-Source: ABdhPJw8B0Ch3knZDj5DCyDYJolQTNPl2596yRXZJe5rbaeDsacvV/YoEug1L9bakztRYWQmSqyjLCmo0CTSCxRZR0c= X-Received: by 2002:a0c:fa90:: with SMTP id o16mr2677614qvn.55.1613625692210; Wed, 17 Feb 2021 21:21:32 -0800 (PST) MIME-Version: 1.0 References: <45b8ef4c-6d36-e91b-ca1a-a82eeca5aaf5@suse.com> <49344e8d-5518-68c6-a417-68522a915e72@suse.com> In-Reply-To: <49344e8d-5518-68c6-a417-68522a915e72@suse.com> From: Roman Shaposhnik Date: Wed, 17 Feb 2021 21:21:21 -0800 Message-ID: Subject: Re: Linux DomU freezes and dies under heavy memory shuffling To: =?UTF-8?B?SsO8cmdlbiBHcm/Dnw==?= Cc: Stefano Stabellini , Xen-devel , Jan Beulich , Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Wei Liu , George Dunlap Content-Type: multipart/alternative; boundary="0000000000002f50b305bb95827a" --0000000000002f50b305bb95827a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 17, 2021 at 12:29 AM J=C3=BCrgen Gro=C3=9F wr= ote: > On 17.02.21 09:12, Roman Shaposhnik wrote: > > Hi J=C3=BCrgen, thanks for taking a look at this. A few comments below: > > > > On Tue, Feb 16, 2021 at 10:47 PM J=C3=BCrgen Gro=C3=9F wrote: > >> > >> On 16.02.21 21:34, Stefano Stabellini wrote: > >>> + x86 maintainers > >>> > >>> It looks like the tlbflush is getting stuck? > >> > >> I have seen this case multiple times on customer systems now, but > >> reproducing it reliably seems to be very hard. > > > > It is reliably reproducible under my workload but it take a long time > > (~3 days of the workload running in the lab). > > This is by far the best reproduction rate I have seen up to now. > > The next best reproducer seems to be a huge installation with several > hundred hosts and thousands of VMs with about 1 crash each week. > > > > >> I suspected fifo events to be blamed, but just yesterday I've been > >> informed of another case with fifo events disabled in the guest. > >> > >> One common pattern seems to be that up to now I have seen this effect > >> only on systems with Intel Gold cpus. Can it be confirmed to be true > >> in this case, too? > > > > I am pretty sure mine isn't -- I can get you full CPU specs if that's > useful. > > Just the output of "grep model /proc/cpuinfo" should be enough. > processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 77 model name : Intel(R) Atom(TM) CPU C2550 @ 2.40GHz stepping : 8 microcode : 0x12d cpu MHz : 1200.070 cache size : 1024 KB physical id : 0 siblings : 4 core id : 3 cpu cores : 4 apicid : 6 initial apicid : 6 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch cpuid_fault epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat md_clear vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest bugs : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only bogomips : 4800.19 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: > > > >> In case anybody has a reproducer (either in a guest or dom0) with a > >> setup where a diagnostic kernel can be used, I'd be _very_ interested! > > > > I can easily add things to Dom0 and DomU. Whether that will disrupt the > > experiment is, of course, another matter. Still please let me know what > > would be helpful to do. > > Is there a chance to switch to an upstream kernel in the guest? I'd like > to add some diagnostic code to the kernel and creating the patches will > be easier this way. > That's a bit tough -- the VM is based on stock Ubuntu and if I upgrade the kernel I'll have fiddle with a lot things to make workload functional again= . However, I can install debug kernel (from Ubuntu, etc. etc.) Of course, if patching the kernel is the only way to make progress -- lets try that -- please let me know. Thanks, Roman. --0000000000002f50b305bb95827a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Wed, Feb 17, 2021 at = 12:29 AM J=C3=BCrgen Gro=C3=9F <jgros= s@suse.com> wrote:
On 17.02.21 09:12, Roman Shaposhnik wrote:
> Hi J=C3=BCrgen, thanks for taking a look at this. A few comments below= :
>
> On Tue, Feb 16, 2021 at 10:47 PM J=C3=BCrgen Gro=C3=9F <jgross@suse.com> wrote: >>
>> On 16.02.21 21:34, Stefano Stabellini wrote:
>>> + x86 maintainers
>>>
>>> It looks like the tlbflush is getting stuck?
>>
>> I have seen this case multiple times on customer systems now, but<= br> >> reproducing it reliably seems to be very hard.
>
> It is reliably reproducible under my workload but it take a long time<= br> > (~3 days of the workload running in the lab).

This is by far the best reproduction rate I have seen up to now.

The next best reproducer seems to be a huge installation with several
hundred hosts and thousands of VMs with about 1 crash each week.

>
>> I suspected fifo events to be blamed, but just yesterday I've = been
>> informed of another case with fifo events disabled in the guest. >>
>> One common pattern seems to be that up to now I have seen this eff= ect
>> only on systems with Intel Gold cpus. Can it be confirmed to be tr= ue
>> in this case, too?
>
> I am pretty sure mine isn't -- I can get you full CPU specs if tha= t's useful.

Just the output of "grep model /proc/cpuinfo" should be enough.

processor : 3
vendor_id : GenuineIn= tel
cpu family : 6
model : 77
model name : Intel(R) Atom(TM= ) CPU =C2=A0C2550 =C2=A0@ 2.40GHz
stepping : 8
microcode : 0x12d<= /div>
cpu MHz : 1200.070
cache size : 1024 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 6
=
initial apicid : 6
fpu : yes
fpu_exception : yes
cpui= d level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 api= c sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht = tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nop= l xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cp= l vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_= timer aes rdrand lahf_lm 3dnowprefetch cpuid_fault epb pti ibrs ibpb stibp = tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat = md_clear
vmx flags : vnmi preemption_timer invvpid ept_x_only flexprio= rity tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs : cpu= _meltdown spectre_v1 spectre_v2 mds msbds_only
bogomips : 4800.19
clflush size : 64
cache_alignment : 64
address sizes : 36 bits p= hysical, 48 bits virtual
power management:
=C2=A0=
>
>> In case anybody has a reproducer (either in a guest or dom0) with = a
>> setup where a diagnostic kernel can be used, I'd be _very_ int= erested!
>
> I can easily add things to Dom0 and DomU. Whether that will disrupt th= e
> experiment is, of course, another matter. Still please let me know wha= t
> would be helpful to do.

Is there a chance to switch to an upstream kernel in the guest? I'd lik= e
to add some diagnostic code to the kernel and creating the patches will
be easier this way.

That's a bit to= ugh -- the VM is based on stock Ubuntu and if I upgrade the kernel I'll= have fiddle with a lot things to make workload functional again.

However, I can install debug kernel (from Ubuntu, etc. etc.= )

Of course, if patching the kernel is the only wa= y to make progress -- lets try that -- please let me know.

Thanks,
Roman.=C2=A0
--0000000000002f50b305bb95827a--