From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7755C48BE6 for ; Wed, 16 Jun 2021 18:42:14 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C53B613ED for ; Wed, 16 Jun 2021 18:42:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C53B613ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BRjENAwdnRWV15XcshxDkRergAghn454JO9KDtGl0h4=; b=PM5a9bhDLxvcLI v9U7S1t7lWud2hFwR5sDbg5S5svMgcFjm461O9TIk2wyafwWJPtBCOYyKO9It4PJzcfpI30hIFYlT XQ/7pVUy0XNHYLzb6IQWWncgQPGbzM6CWCLQQZfq8vwafk04jkTQfOdncXZZjgBqi2tpfqPBhqj10 EJ6rhm+ihKL4baaS7SmYdpGtKrkEOro0AgwsmcF+6p6F00+Pac/IZ98otNfbHBrYCIOfURxbOzbzJ 1Zfmw7cnj8mcn+6LyLcs4qRelx6DW/JbG20qMhYysvtaffa1zfj1PwxpP+uD6t22E2nvzfWYwkh/c pXc9EDgteAvEVmf2xHcQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltaSm-007RJ7-EL; Wed, 16 Jun 2021 18:40:32 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltaSi-007RIk-IN for linux-arm-kernel@lists.infradead.org; Wed, 16 Jun 2021 18:40:30 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 5A9DC613E2; Wed, 16 Jun 2021 18:40:26 +0000 (UTC) Date: Wed, 16 Jun 2021 19:40:23 +0100 From: Catalin Marinas To: Frank Li Cc: Will Deacon , Shenwei Wang , Han Xu , Nitin Garg , Jason Liu , "linux-arm-kernel@lists.infradead.org" , Zhi Li Subject: Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors Message-ID: <20210616184023.GG22433@arm.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210616_114028_667080_7828EDEA X-CRM114-Status: GOOD ( 22.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Jun 14, 2021 at 10:41:38PM +0000, Frank Li wrote: > commit 22ec71615d824f4f11d38d0e55a88d8956b7e45f > Author: Will Deacon > Date: Fri Jun 7 15:48:58 2019 +0100 > > arm64: io: Relax implicit barriers in default I/O accessors > > The arm64 implementation of the default I/O accessors requires barrier > instructions to satisfy the memory ordering requirements documented in > memory-barriers.txt [1], which are largely derived from the behaviour of > I/O accesses on x86. [...] > If I added wmb() before xhci_ring_ep_doorbell, the problem gone. > Writel include io_wmb, which map into dma_wmb(). > > 1. write ddr > 2. writel > 2a. io_wmb(), dmb(oshst) > 2b, write usb register > 3. usb dma read ddr. > > > Internal bus fabric only guarantee the order for the same AXID. > 1 write ddr may be slow. USB register get data before 1 because > GPU occupy ddr now. So USB DMA start read from ddr and get old > dma descriptor data and find not ready yet, then missed door > bell. That's a complex topic, Will should have a better answer. I'll try some thought exercise below introducing a hypothetical second CPU. >From Will's commit above w.r.t. other-multi-copy atomicity: 1. A write arriving at an endpoint shared between multiple CPUs is visible to all CPUs 2. A write that is visible to all CPUs is also visible to all other observers in the shareability domain So (1) would be the write to the USB device which is also an observer in the system (of the DDR writes). (2) refers to the write to the DDR. If we have CPU0 writing to DDR, followed by DMB and the write to the USB device, a CPU1 observing the write to the USB device would also observe the write to DDR (with a DMB between them). Since the USB device is an observer and the system is multi-copy atomic, the USB should also observe the CPU0 write to the DDR if CPU1 observed it. CPU1 can only observe the write to the USB device via an access to that USB device (e.g. a register read). Such access probably goes through some serialisation point and the DMB on CPU0 ensures that the prior write to DDR is visible. Now, a CPU1 read from the USB device cannot affect the DMA access that the USB device started to the DDR, so we can take it out of the equation. However, this means that the hardware should ensure such ordering USB DMA ordering otherwise it wouldn't be multi-copy atomic (or our understanding of it). Either the hardware doesn't match the memory model or our reasoning is incorrect (both are possible ;)). I wonder whether we can look at this in a different way: the USB device doing a "speculative" access to the DDR before the write to USB is globally observable. There isn't a way to fix it in the USB device since it does not observe the write to its register, so we are left with having to guarantee the completion of the write to the DDR before informing the USB about it. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel