From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 141A9C2B9F4 for ; Mon, 14 Jun 2021 22:44:31 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C390661078 for ; Mon, 14 Jun 2021 22:44:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C390661078 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nxp.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:CC :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=jnEDGyxhXO4r3qp8mdXvlgWcIDk7s7H6XhbkCTsf/Y0=; b=pANakxz3wiOAV3 YcNrCFFEMk2kY87J6/op5h2IuKMmTGDK2Z1KKhUxio27qawR9+fVaxvl9ZIrrZh+3E4yrSfWB05r5 lKPQAUhIJyEFnZK6a8WzqTMqSf54olFwfqMfRW+my5V4ZLz803Ysnp65f+Wl3fGapUiDGymegiQO/ uIWMjnSbLhHXQS3cehqoI/lK0L53wNB8qcABCQSOUxktCA8t0vZZZmNySrTMRa2KNAFLTOpheVvnh rb4F06ePNpOsrXljypNEI7iIgQgzRYN64EaYV6VIvnhLCxwCEkADUwA/aVxMpSqa0CQ9XZtio4RGZ PhOUjQzMjYXD1KFeDJkw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lsvH9-00GSbH-KR; Mon, 14 Jun 2021 22:41:47 +0000 Received: from mail-eopbgr70077.outbound.protection.outlook.com ([40.107.7.77] helo=EUR04-HE1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lsvH6-00GSZN-LT for linux-arm-kernel@lists.infradead.org; Mon, 14 Jun 2021 22:41:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GM8n3LRFzS+yPsLaBkgTV4MHyweWWjVppLyPnsuaxnarGFLI8s4AE3+yWyCN9pai1Vcg4expwZ+1oAGb3eOg7Ll6D4eHXO7BdVSQNLFHz2psbMTo2kNd19FsgHHWNvXIqUDP6/ivya7bHPgWAbb1GIM4dfZEE0X9wBJFZb5rHwVxZfyJ4iFbyJDiEqrhY8q5fFmUFRor7yGA9Jrr9mkCY3u6eBvW7IOeZLxkLtwKptCsQyx+44KBU20Z6msOFYG1HwMqLTbDNF0C3bBOjsUq4lMfuVexrjNdTiJsXmaCtUg/WLXScgV7ri8QSNyg16BfOXpY0BD5KaKorj8DXylLCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0AqDBf00nSXKtbl+/1FOZT/Dl5OxwL3c1IR1hKkIn2s=; b=CRLMCit6qs7jybg1uUn42n0Y7iKt6PRMRsLDybM+ahaveiLq3F8lHh0B/jlSLV+UClzg/YBhsISr341HJYdxYjl7joMSOTJaGeks13Efx2S/rnohhzPnO5/sU9MDd+oINIxd2GXhSVxjfpxidHkLx6GgqeXI5rJhiAodhnPwUveXG30Ee1dbfsOjxqBdQ57TqMy42ROOVOawNnY6zAdSJhKO0vglD+tn2N2a+UrRiV1S54eeSkWPfJASL5EhlGi4KEXfgId4X0UOfj1OHQKWGfXnqI9f1EpGLIYc22ku8MQtYFPIy5XB8jpa1qZfmslAi5aUvV8N0YHiSXL4UDZlmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0AqDBf00nSXKtbl+/1FOZT/Dl5OxwL3c1IR1hKkIn2s=; b=C3e53T+wg/+tNhXNxz4SWyHHj9pjgA4H8hTkVvQ39s2HoCJ2adlcPFFyfNusgYWs0Emc5l4obvfS34noX4rZRX3kEPDRag6tLSGa7AEO4YpS2QDwnZ01u+mfLM/ULTZAg766ClvQZTPDNg8/n7sCueo1qdzWJCNWru4cJ1oc1Ts= Received: from AS8PR04MB8500.eurprd04.prod.outlook.com (2603:10a6:20b:343::14) by AS8PR04MB8499.eurprd04.prod.outlook.com (2603:10a6:20b:342::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Mon, 14 Jun 2021 22:41:38 +0000 Received: from AS8PR04MB8500.eurprd04.prod.outlook.com ([fe80::2ca8:978f:5800:3e9a]) by AS8PR04MB8500.eurprd04.prod.outlook.com ([fe80::2ca8:978f:5800:3e9a%7]) with mapi id 15.20.4219.025; Mon, 14 Jun 2021 22:41:38 +0000 From: Frank Li To: Will Deacon CC: Shenwei Wang , Han Xu , Nitin Garg , Jason Liu , "linux-arm-kernel@lists.infradead.org" , Zhi Li Subject: The problem about arm64: io: Relax implicit barriers in default I/O accessors Thread-Topic: The problem about arm64: io: Relax implicit barriers in default I/O accessors Thread-Index: Addhawkczavy5sh9TS2ivC8EbHYnQA== Date: Mon, 14 Jun 2021 22:41:38 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=nxp.com; x-originating-ip: [72.177.87.193] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 527f03de-cb6a-485c-1cf2-08d92f859273 x-ms-traffictypediagnostic: AS8PR04MB8499: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2958; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: aKcq4+idpnUQVWx0IzGgY0JdVd3dyvRPe1fT3yRVwS1/nnx/vmB5fZv9zak4Gj4D7WhULYMxMZweugan33TA9iUA+szbJoFaoSktwmyeMA26yVMAd3cr+yoVM6c8ir0DpcMoRiFKvzf8Qx5Lv37QWaoFLplH9lyxpcCFIdMGi/gzYRjQVGOqYwj7tF/OJH4dGv8oqDQBsnhqqh9O/N1Id3OIn08hJCwKqY2XY522WrwIQdJpy32R6S37b17p3zyWCyy9MmbDakX41C/6qpNzwYsNyKuqY84Rda45FOgSbA/qIKNnuxpIOtAHTJbwDQy23U8CJvg09+DB2740DmYrwYf6F0VTOwP/B0+uB7Y+8RS+tbp8t/WomW804IrAYRh7eLFPol9SXg+Pmw0v+hlEEGfhEGjtfSMrJNiZlKRvFb/OlCmhqTEkvp2Tu+E84Yp4BrmuKB68PCFmmgfq6x5Gila+KbhgIkRTqkuXwQu5dzKxmYd6kpoXpsaoolhRT0KWgskPcgLQv0Jq1dCWRLvUwcl4YPSah+GgtnT6up137o9SVNtehgY+foQLTQ2l94qQ/Jnjia9rU8VtBGKJHgHXgTVsSuEXknR7/w6PZXcL9aQ= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR04MB8500.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(136003)(39860400002)(366004)(396003)(346002)(38100700002)(186003)(83380400001)(8936002)(26005)(54906003)(33656002)(55016002)(478600001)(8676002)(9686003)(76116006)(66476007)(66556008)(66946007)(66446008)(64756008)(71200400001)(52536014)(5660300002)(6916009)(2906002)(44832011)(316002)(6506007)(122000001)(4326008)(7696005)(86362001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?1qi0ZAT96xaFbeupLXACiTXQxVkn+U6h9YFsybsXojpR1ckNDsWQuNJhzy6F?= =?us-ascii?Q?6xjofM5xF+diBPmXHTMqPYRxeCDq7rEP55aMiamM93s80+a7ru6cU/htbW/C?= =?us-ascii?Q?iEd9gcUerr9lDob2qiGm05t5Ce/tASeRlVSwjVtmt27xgtiHuOo5SARHKGeI?= =?us-ascii?Q?1AQ78SyHcuJgvedzobAC1CtGcU0cSrPAjV+pcPbnqG/0pdTNXEjQLEb7iMTR?= =?us-ascii?Q?nblCA1plYuy4UIqMTJbOr4wm3Ul+m2fmmG8wNjUkcFj65azndquBl+qnlSG3?= =?us-ascii?Q?1ogkLsNhf80Sc2syU29p7lO7kM8GFtSclMsmlAwQAbAkTpyPhxvjlAFXDC73?= =?us-ascii?Q?ktPqer/+0IgjZcbA/VFU67vo6/V+A5Q7+eivmaH8cqJVTXlELdehMytLc4PA?= =?us-ascii?Q?BKSVFs0r7FI+zEogehtChdITxITWf6IXmaWmsc8BhfNayqmV/e8JFALNs+P0?= =?us-ascii?Q?dUiqACUwrTkmsZj4lVTTNPxhh0d/yaS2jiMY1cJpvWgM2Qm9XZUPMuA9ZPWA?= =?us-ascii?Q?2xcR2QYilz6X+uUhD1ChwUDInULXEqTYTv+EZLZ8Q3OyNoZljK0Y0eezPKb1?= =?us-ascii?Q?hIPp/zQQOxLhqjXPYriHaLxqmLU8Yd1V5XIjLMy/XX7OWurAHxqv5IZPYdJv?= =?us-ascii?Q?nGifcqHeiR0dQBBqjVpF4dx72X3+hj1gfvZ3vIDpxde1v4B60KF3WN68CIuA?= =?us-ascii?Q?Et0QnwwDfTq14mwJPTfdv9+oNQFbMIvU2p9MquOeqAcTNwuI1jm7HDmSaxo0?= =?us-ascii?Q?V7+MPv+BvlJHeQQ9HU76lOfLonG+OPQEppuumQsCMkh628F2GkD9IsKPKvkG?= =?us-ascii?Q?q7RyVi8/q2yckC+n267d9sr6iJI1D4UdGuXmjuQJu5ftJrLlLnpx9juglL/T?= =?us-ascii?Q?JMa7+ssxDv9WXuMVCF8tWqS690PC8Djz3MOQ/jwVY/sq3SOoQqvnL2yIr7xy?= =?us-ascii?Q?hh7uO+dPyW0iJ0dTJY994oonnH0bvQWQ4hi4egniW+hz6GLVhMrFdHwlWa4E?= =?us-ascii?Q?Otc9DF3NTIoNzKopXOPbosW2GgXYPMJf5IrJz5AQf7u+FEAJp5r4MA4dXkP/?= =?us-ascii?Q?sN1+53H+JBQ/oy1+/2HTaVHmVJQG3wjTXKJIef2kQjjo9ADKz2Xm6CpmfoBH?= =?us-ascii?Q?mao2fNRkaJtJY6o+zQgHVi1vYSSl1cg45bsG+81SFzSni4M8vPoYQwliq/HC?= =?us-ascii?Q?R95SAPNfpiWTNMLd6w8Y45RxaJOVeIrKQ6s+Clgr0gHOHuX/fcnMn+uE+93k?= =?us-ascii?Q?+H3WHNCqo2iQR90dypCHVOePCMCOOgonzM8+iNSjSd9h7nXsSvKV+pU22VCh?= =?us-ascii?Q?ggY9yVAkJ66Al0FNXJnMxi5x?= MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AS8PR04MB8500.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 527f03de-cb6a-485c-1cf2-08d92f859273 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Jun 2021 22:41:38.3042 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: lfS/9kqRL9Gf7OFq/122ThYDOL8pFPOXI4rADlanL7ZW7KC+QiizmoktdyqrtUM7Ga4xf5RgXINNCl59xGkjMg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8499 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210614_154144_788855_4E34B646 X-CRM114-Status: GOOD ( 12.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Will Deacon: Our a test case is failure at 8QM platform(arm64). USB transfer failure if run with GPU stress test. I found it related with your below change. commit 22ec71615d824f4f11d38d0e55a88d8956b7e45f Author: Will Deacon Date: Fri Jun 7 15:48:58 2019 +0100 arm64: io: Relax implicit barriers in default I/O accessors The arm64 implementation of the default I/O accessors requires barrier instructions to satisfy the memory ordering requirements documented in memory-barriers.txt [1], which are largely derived from the behaviour of I/O accesses on x86. drivers/usb/host/xhci-ring.c static void giveback_first_trb(struct xhci_hcd *xhci, int slot_id, unsigned int ep_index, unsigned int stream_id, int start_cycle, struct xhci_generic_trb *start_trb) { /* * Pass all the TRBs to the hardware at once and make sure this write * isn't reordered. */ wmb(); if (start_cycle) start_trb->field[3] |= cpu_to_le32(start_cycle); else start_trb->field[3] &= cpu_to_le32(~TRB_CYCLE); xhci_ring_ep_doorbell(xhci, slot_id, ep_index, stream_id); } If I added wmb() before xhci_ring_ep_doorbell, the problem gone. Writel include io_wmb, which map into dma_wmb(). 1. write ddr 2. writel 2a. io_wmb(), dmb(oshst) 2b, write usb register 3. usb dma read ddr. Internal bus fabric only guarantee the order for the same AXID. 1 write ddr may be slow. USB register get data before 1 because GPU occupy ddr now. So USB DMA start read from ddr and get old dma descriptor data and find not ready yet, then missed door bell. If do 2-3 times doorbell, problem also gone. So I think dmb(oshst) is not enough for writel. A writeX() by the CPU to the peripheral will first wait for the completion of all prior CPU writes to memory. For example, this ensures that writes by the CPU to an outbound DMA buffer allocated by dma_alloc_coherent() will be visible to a DMA engine when the CPU writes to its MMIO control register to trigger the transfer. Best regards Frank Li _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel