From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CFF7C2D0A3 for ; Sat, 24 Oct 2020 04:35:31 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 33EB320FC3 for ; Sat, 24 Oct 2020 04:35:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33EB320FC3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=m5p.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.11441.30349 (Exim 4.92) (envelope-from ) id 1kWBGd-0005Sq-Tu; Sat, 24 Oct 2020 04:34:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 11441.30349; Sat, 24 Oct 2020 04:34:59 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kWBGd-0005Sj-Qw; Sat, 24 Oct 2020 04:34:59 +0000 Received: by outflank-mailman (input) for mailman id 11441; Sat, 24 Oct 2020 04:34:58 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kWBGc-0005Se-Nu for xen-devel@lists.xenproject.org; Sat, 24 Oct 2020 04:34:58 +0000 Received: from mailhost.m5p.com (unknown [74.104.188.4]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 4b8c1b87-12b5-4c96-b5b6-a22715a1dad3; Sat, 24 Oct 2020 04:34:57 +0000 (UTC) Received: from m5p.com (mailhost.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:f7]) by mailhost.m5p.com (8.15.2/8.15.2) with ESMTPS id 09O4YlGZ097256 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 24 Oct 2020 00:34:53 -0400 (EDT) (envelope-from ehem@m5p.com) Received: (from ehem@localhost) by m5p.com (8.15.2/8.15.2/Submit) id 09O4Yk1e097255; Fri, 23 Oct 2020 21:34:46 -0700 (PDT) (envelope-from ehem) Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kWBGc-0005Se-Nu for xen-devel@lists.xenproject.org; Sat, 24 Oct 2020 04:34:58 +0000 X-Inumbo-ID: 4b8c1b87-12b5-4c96-b5b6-a22715a1dad3 Received: from mailhost.m5p.com (unknown [74.104.188.4]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 4b8c1b87-12b5-4c96-b5b6-a22715a1dad3; Sat, 24 Oct 2020 04:34:57 +0000 (UTC) Received: from m5p.com (mailhost.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:f7]) by mailhost.m5p.com (8.15.2/8.15.2) with ESMTPS id 09O4YlGZ097256 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 24 Oct 2020 00:34:53 -0400 (EDT) (envelope-from ehem@m5p.com) Received: (from ehem@localhost) by m5p.com (8.15.2/8.15.2/Submit) id 09O4Yk1e097255; Fri, 23 Oct 2020 21:34:46 -0700 (PDT) (envelope-from ehem) Date: Fri, 23 Oct 2020 21:34:46 -0700 From: Elliott Mitchell To: Stefano Stabellini Cc: Julien Grall , roman@zededa.com, xen-devel@lists.xenproject.org Subject: Re: Xen on RP4 Message-ID: <20201024043446.GA97167@mattapan.m5p.com> References: <20201012215751.GB89158@mattapan.m5p.com> <20201016003024.GA13290@mattapan.m5p.com> <23885c28-dee5-4e9a-dc43-6ccf19a94df6@xen.org> <20201022021655.GA74011@mattapan.m5p.com> <20201023005629.GA83870@mattapan.m5p.com> <20201023211941.GA90171@mattapan.m5p.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri, Oct 23, 2020 at 04:59:30PM -0700, Stefano Stabellini wrote: > On Fri, 23 Oct 2020, Elliott Mitchell wrote: > > Finally came up with one detail of a change I'd made in the right time > > frame to trigger this issue. As such I can now control this behavior and > > get it to occur or not. > > > > I have some suspicion my planned workload approach differs from others. > > > > During the runs where I was able to successfully boot a child domain, > > domain 0 had been allocated 512MB of memory. During the unsuccessful run > > where the above message occurred, domain 0 had been allocated 2GB of > > memory. This appears to reliably control the occurrence of this bug. > In your case, the allocation must fail, no_iotlb_memory is set, and I > expect you see this warning among the boot messages: > > Cannot allocate buffer > > Later during initialization swiotlb-xen comes in > (drivers/xen/swiotlb-xen.c:xen_swiotlb_init) and given that io_tlb_start > is != 0 it thinks the memory is ready to use when actually it is not. Then I look at more copies of `dmesg` logs and discover I did have one where that message occurred. Then start playing a bit more and a matching pattern emerges. dom0_mem=1023M => "software IO TLB: mapped [mem 0x2c000000-0x30000000] (64MB)" dom0_mem=1024M => "software IO TLB: Cannot allocate buffer" That looks suspicious. Really suspicious. I don't know how many bugs are combined together, nor where they are, but more data for you. I see a possibility Tianocore marks a smaller region of memory as being DMA-capable. This though is speculation. -- (\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/) \BS ( | ehem+sigmsg@m5p.com PGP 87145445 | ) / \_CS\ | _____ -O #include O- _____ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445