From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02AB0C433DF for ; Wed, 13 May 2020 19:51:51 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7FD5A205ED for ; Wed, 13 May 2020 19:51:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="J7HusWoQ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7FD5A205ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jYxPf-0006HT-62; Wed, 13 May 2020 19:51:31 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jYxPd-0006HO-QC for xen-devel@lists.xenproject.org; Wed, 13 May 2020 19:51:29 +0000 X-Inumbo-ID: 228f6ba4-9553-11ea-a401-12813bfff9fa Received: from mail.kernel.org (unknown [198.145.29.99]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 228f6ba4-9553-11ea-a401-12813bfff9fa; Wed, 13 May 2020 19:51:28 +0000 (UTC) Received: from localhost (c-67-164-102-47.hsd1.ca.comcast.net [67.164.102.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4EA27205ED; Wed, 13 May 2020 19:51:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1589399488; bh=q9pL9MyiDPkzpcxwhyoVVNHmRZJxL8DD6AP9lH+W12s=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=J7HusWoQ5Y+Ge5/d62uQu1IC/VMSsxA/jdUbWqcKoHAtUTTXmvlkay3w1N628GUFH jW5D8I/W+l47BKN32/ANeTJk1sCaj/JeLAxR8Tw72xsJjbCYwlDOXpYRbevZr9QvCF hTSITLg/iQHink6MqfgFvfx/NGc3X4l+ETO9m/Io= Date: Wed, 13 May 2020 12:51:26 -0700 (PDT) From: Stefano Stabellini X-X-Sender: sstabellini@sstabellini-ThinkPad-T480s To: Julien Grall Subject: Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU In-Reply-To: Message-ID: References: <9ee0fb6f-98df-d993-c42e-f47270bf2eaa@suse.com> <2187cd7c-4d48-986b-77d6-4428e8178404@oracle.com> <42253259-9663-67e8-117f-8ba631243585@xen.org> User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="8323329-1236722027-1589399488=:26167" X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: jgross@suse.com, Peng Fan , Stefano Stabellini , "minyard@acm.org" , roman@zededa.com, "jeff.kubascik@dornerworks.com" , Julien Grall , Nataliya Korovkina , "xen-devel@lists.xenproject.org" , boris.ostrovsky@oracle.com, Stefano Stabellini Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --8323329-1236722027-1589399488=:26167 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT On Wed, 13 May 2020, Julien Grall wrote: > On 13/05/2020 19:19, Julien Grall wrote: > > Hi, > > > > On 13/05/2020 16:11, Stefano Stabellini wrote: > > > On Wed, 13 May 2020, Julien Grall wrote: > > > > Hi, > > > > > > > > On 13/05/2020 01:33, Stefano Stabellini wrote: > > > > > I worked with Roman to do several more tests and here is an update on > > > > > the situation. We don't know why my patch didn't work when Boris' > > > > > patch > > > > > [1] worked.  Both of them should have worked the same way. > > > > > > > > > > Anyway, we continued with Boris patch to debug the new mmc error which > > > > > looks like this: > > > > > > > > > > [    3.084464] mmc0: unrecognised SCR structure version 15 > > > > > [    3.089176] mmc0: error -22 whilst initialising SD card > > > > > > > > > > I asked to add a lot of trancing, see attached diff. > > > > > > > > Please avoid attachment on mailing list and use pastebin for diff. > > > > > > > > > We discovered a bug > > > > > in xen_swiotlb_init: if io_tlb_start != 0 at the beginning of > > > > > xen_swiotlb_init, start_dma_addr is not set correctly. This oneline > > > > > patch fixes it: > > > > > > > > > > diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c > > > > > index 0a40ac332a4c..b75c43356eba 100644 > > > > > --- a/drivers/xen/swiotlb-xen.c > > > > > +++ b/drivers/xen/swiotlb-xen.c > > > > > @@ -191,6 +191,7 @@ int __ref xen_swiotlb_init(int verbose, bool > > > > > early) > > > > >            * IO TLB memory already allocated. Just use it. > > > > >            */ > > > > >           if (io_tlb_start != 0) { > > > > > +               start_dma_addr = io_tlb_start; > > > > >                   xen_io_tlb_start = phys_to_virt(io_tlb_start); > > > > >                   goto end; > > > > >           } > > > > > > > > > > Unfortunately it doesn't solve the mmc0 error. > > > > > > > > > > > > > > > As you might notice from the logs, none of the other interesting > > > > > printks > > > > > printed anything, which seems to mean that the memory allocated by > > > > > xen_swiotlb_alloc_coherent and mapped by xen_swiotlb_map_page should > > > > > be > > > > > just fine. > > > > > > > > > > I am starting to be out of ideas. Do you guys have any suggestions on > > > > > what could be the issue or what could be done to discover it? > > > > > > > > Sorry if my suggestions are going to be obvious, but I can't confirm > > > > whether > > > > this was already done: > > > >      1) Does the kernel boot on baremetal (i.e without Xen)? This should > > > > help > > > > to confirm whether the bug is Xen is related. > > > > > > Yes it boots > > > > > > >      2) Swiotlb should not be necessary for basic dom0 boot on Arm. Did > > > > you try > > > > to disable it? This should help to confirm whether swiotlb is the > > > > problem or > > > > not. > > > > > > It boots disabling swiotlb-xen > > > > Thank you for the answer! swiotlb-xen should basically be a NOP for dom0. So > > this suggests swiotlb is doing some transformation on the DMA address. > > > > I have an idea what may have gone wrong. AFAICT, xen-swiotlb seems to assume > > the DMA address space and CPU address space is the same. Is RPI using the > > same address space? > > Another question, is the DMA request bounced? If so, are we sure the bounce > buffer is in the first GB? Yes, it is. This is actually where we spent the last few days, and I found another little related bug in the initialization of the swiotlb-xen but now I am sure the memory is under 1GB (0x34000000-0x38000000) --8323329-1236722027-1589399488=:26167--