From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BADEC48BE5 for ; Wed, 16 Jun 2021 18:57:08 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9477611CE for ; Wed, 16 Jun 2021 18:57:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9477611CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LRsopZpdEygRvhdnflFqN/tI4T6U18h15ugOAFuaptk=; b=10ljBTJIUIbJdb fGTl5O9MPLwwNtzMUAvxpOQAL3UIRJuaQ68ddyXygdWGhZKl2gvksLT+V5nUiChT9AqcJtWrCKEef y+rxeUSZDX/mxcIa+2iw9UwQcz82X7B07+DHLI+07LiHWX4fgVUZOCFdXxjAfbsbNMXCVmdH65Uz4 ROWl3XUQUOUFft5zYujqfjICLw2wLDvsvkMqZ0wa/QBoW5GqY+XTJeg0LtjW/yQ+OLRFNlqYSfxvC pIDutcaoBwIzAzKa5SMYPcK5MYcbcluJ21qlYYgrl16j/M+694Cbmi1TF5hR8xBmWJORZ2xscLOxL xeozhJDkK4JJv8cr+DNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltahX-007Sva-46; Wed, 16 Jun 2021 18:55:47 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltahS-007Suy-SZ for linux-arm-kernel@lists.infradead.org; Wed, 16 Jun 2021 18:55:44 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id B29D9610A1; Wed, 16 Jun 2021 18:55:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623869741; bh=EmxHW5QxDjyC4LNq6FxnPmrtACPWW+fFr/Th8y3hIPU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=g4Cly83q4RO1qh1xfFqkWCrbOTbEbpRhnb8nYahwqN9I71U5IUHk+kqjOdPZTIiUA 5KrBfp2Z1VL3xUf77R5lZq8t0le+IoJV6gTh56KHZ/2j+FnYDBtyY90qX3ZrpNlwu8 9JcBxE8R0Dxq5Gx+V78Hith5Dyuuc9cF1lzY6Zs8Q03kbizpvKFP1B6Ei/iSvtKRlK A0Wt9N4zkVKQO9JVeME9XqjGUk37s0lCn5ZZMhaqsx8ErdmOz1LPaSiv9WpRO+vH3/ /mRWUXbiAKTCBzUaV9l7d3IM4dc3TgUMzJW2vuS2xaoJxrjT5LmHzddn7mT7/w2h2R yEH5CNFFb/icA== Date: Wed, 16 Jun 2021 19:55:36 +0100 From: Will Deacon To: Catalin Marinas Cc: Frank Li , Shenwei Wang , Han Xu , Nitin Garg , Jason Liu , "linux-arm-kernel@lists.infradead.org" , Zhi Li Subject: Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors Message-ID: <20210616185535.GA23483@willie-the-truck> References: <20210616184023.GG22433@arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210616184023.GG22433@arm.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210616_115542_987216_6BA8F169 X-CRM114-Status: GOOD ( 21.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jun 16, 2021 at 07:40:23PM +0100, Catalin Marinas wrote: > On Mon, Jun 14, 2021 at 10:41:38PM +0000, Frank Li wrote: > > commit 22ec71615d824f4f11d38d0e55a88d8956b7e45f > > Author: Will Deacon > > Date: Fri Jun 7 15:48:58 2019 +0100 > > > > arm64: io: Relax implicit barriers in default I/O accessors > > > > The arm64 implementation of the default I/O accessors requires barrier > > instructions to satisfy the memory ordering requirements documented in > > memory-barriers.txt [1], which are largely derived from the behaviour of > > I/O accesses on x86. > [...] > > If I added wmb() before xhci_ring_ep_doorbell, the problem gone. > > Writel include io_wmb, which map into dma_wmb(). > > > > 1. write ddr > > 2. writel > > 2a. io_wmb(), dmb(oshst) > > 2b, write usb register > > 3. usb dma read ddr. > > > > > > Internal bus fabric only guarantee the order for the same AXID. > > 1 write ddr may be slow. USB register get data before 1 because > > GPU occupy ddr now. So USB DMA start read from ddr and get old > > dma descriptor data and find not ready yet, then missed door > > bell. > > That's a complex topic, Will should have a better answer. I'll try some > thought exercise below introducing a hypothetical second CPU. It would also be helpful to know a bit more about the hardware: - What is the "internal bus fabric"? - Can you be more specific about the AxIDs? I can't tell how that correlates back to code running on the CPU. - Is the device cache coherent? - What memory types are used to map the DDR and the USB register on the CPU? (I got lost in the indirection) Also, do you know which part of the data appears to be stale when the device reads it? Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel