From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53415C48BE8 for ; Wed, 16 Jun 2021 16:31:19 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EEE861245 for ; Wed, 16 Jun 2021 16:31:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EEE861245 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nxp.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kB8EIFfaA0TeGgNon2zCPxG5YjeDhWGxTCc6koTS+G0=; b=tYFVWEND7z32YC 9gs7Ji3rk0NfxjSSPM4BlOpTY8aaDroGGCLXBJ2+sUTyYarZqY33gNpeW12IY80rTDOzA7dRH8Aqw ojl8cwibrTxvHwMWyWzbEOzjQsEoGWZfjwUm/YWnvFbKmVQrm11EIju7pTX56OWLHcS7Kjr8uznXd o5dvRluemUARogImedTo9HY8yVHUcM1TYTLZEWvinwH5PtLt8J5SibyzKeYbE55MYV3iwCwzHBLGp CHjjN7ZxVCwn+JgQAUVo/n50aARiDaqmInLBIsaWNB1Xl8Js+LSWQrI8kkBFkXi2AteI2lMemd7PZ euPEG8D5HrWXLU1kRC1Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltYQI-007811-PJ; Wed, 16 Jun 2021 16:29:51 +0000 Received: from mail-eopbgr60069.outbound.protection.outlook.com ([40.107.6.69] helo=EUR04-DB3-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltYQB-0077zY-OF for linux-arm-kernel@lists.infradead.org; Wed, 16 Jun 2021 16:29:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g0VHMMOy3ADjS6zldY8JBFXV+o2OdDXqWoUVMwZ6WOKmSbS+f+V9m88xrHlCN5KEjIvhuxy++8WwqW66PdK5eDRMsx2++6wGQOPH82tJ02vfcmOVVUR7L/Vnw3E0W0a0eM2wHVK12nTF/dGTWKVW+YXLkzF6W6DE8DyqQb8QFm2kA8QHHoHk7BbqJgA20hnZp6MBXDyxr8RdmusJg6dlG6oxR1z6f/KPFqkrwkkwZTMQWq6YcF00hxaufZl7MHgQrshapihKIIbT88Es3a5b0dGrOfn/GtWszvyp1EObjPFo+qNHX3RlUhJKEt0jbOFFrGouuoe4u5BS/2wmzJ27OQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ctxEm7wX7jsWLGzv6n7kLTchXTrKzzRHEB6JI0FX0qg=; b=cOfC7AQwM1uBSw06dEBz3dHJDg6kbNRMOwEYDXGOm+uiisQuIuHJfOGDgkY/YAYuBiFzJG0uzOD4PaESRORzbaexRbHx14VAqfVTeT4MK/aogZg+mYrl8zNQGY+4l9TguY04kmwbB8zdXh+zgyToz2NFyN1u3F/PN+sGLWV+LWLE7MRHHjR7VKRr9Q2QTed234KBe7ZrIK+uEcYOYVLPcl/JdHfjFZHsskDO3HDhcF9fN1BG2QnDAXYdttURFA8WjGdKCbQKMr5eNgT/V6Qf6HfdcFBmmKw9/KuYuar8Gh5M4l9s8YVZHT0VyGtmqDqX4Vf3ynV1SXrt6kG5LjtONg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ctxEm7wX7jsWLGzv6n7kLTchXTrKzzRHEB6JI0FX0qg=; b=lKbJa9V8ibgjzOez21eaCOtDy91oEa62Wb1JIaOHrxOgrbtrZNSFpUd7Xnn6L1PXuWgUOF8BGx6RShPg0h7mqOwsBDSTRcv0/yQ8skE0K80hWNQ6edTNPQ6ZBYIniGiRwWLyNGmD3H+3hcJQy4BqaFuSoElfa8UOAYIJyasI5gw= Received: from AS8PR04MB8500.eurprd04.prod.outlook.com (2603:10a6:20b:343::14) by AS8PR04MB8817.eurprd04.prod.outlook.com (2603:10a6:20b:42c::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.19; Wed, 16 Jun 2021 16:29:41 +0000 Received: from AS8PR04MB8500.eurprd04.prod.outlook.com ([fe80::2ca8:978f:5800:3e9a]) by AS8PR04MB8500.eurprd04.prod.outlook.com ([fe80::2ca8:978f:5800:3e9a%7]) with mapi id 15.20.4242.018; Wed, 16 Jun 2021 16:29:41 +0000 From: Frank Li To: Will Deacon , Catalin Marinas CC: Shenwei Wang , Han Xu , Nitin Garg , Jason Liu , "linux-arm-kernel@lists.infradead.org" , Zhi Li Subject: RE: The problem about arm64: io: Relax implicit barriers in default I/O accessors Thread-Topic: The problem about arm64: io: Relax implicit barriers in default I/O accessors Thread-Index: Addhawkczavy5sh9TS2ivC8EbHYnQABYXRPAAAAPeEA= Date: Wed, 16 Jun 2021 16:29:41 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=nxp.com; x-originating-ip: [72.177.87.193] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 74dc1c31-75fb-4036-6dc6-08d930e3f136 x-ms-traffictypediagnostic: AS8PR04MB8817: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2958; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: e6f6sL3/EfGPTfW1NU59/Uq2jSvRzgZM/+2I9aJfIHL4WRB/ri5nTE83uWAYeIAwjyxA5xxuxU9RB/zXaGKpG6ToTG3OvKlx9lwBdW+C1RxFHi58n4aTxraC+KMjvp2PK9kYzAkEs2mWtrkjhXGgEBNpFnteCT8WgfptUCC1CVouIyhJlR2KHiGUxZ9kTuU7Us7mMD0nyHy0RObyVKcYtKshGWltkN2j2IOlCLyeP2LzxwQL2MvNx++8bGwD4epjsqDHynD0J6xfbR1RSeGTnGDz7rAN0J3gTG2LvCNi/j/Lgoa/AHi5NVZyRGzUmotCQ0G6l8SJQfBamIY1yzcR0A+9NIQTUx19osPdGwsXNaCaBVK7rsD5mTmidLJboWbOncG0V4CcFvK9AO9ug0Of0AsYhMd5cirfwVW3+J+kdtagK/qojdwZngUWNt7VkwM2YfjKnbcg+rCZXCidcL3Bn+cjUcczPE4UyR9cXhIp6Wv0Z/81xBS45jrnKHjzzXO0pqLwEjPcJy7fUA+HW0vUAHo6/U2vIsfnqXs5jMU2roT36uLg0YYoOKdXHs4juoWtA4x0K0tzPVc0vAQcJR3wMiHt8PreI5aJ591n4izkK8E= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR04MB8500.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(346002)(366004)(39850400004)(396003)(376002)(478600001)(5660300002)(2940100002)(66946007)(4326008)(71200400001)(66476007)(64756008)(66556008)(66446008)(83380400001)(7696005)(38100700002)(86362001)(54906003)(53546011)(6506007)(110136005)(26005)(9686003)(8676002)(55016002)(44832011)(33656002)(122000001)(2906002)(316002)(52536014)(76116006)(8936002)(186003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?0ttq/SU2JL5WVPzZuK3NX4gneR06gyFwboDGpU5NAg90TQDhPxSrr9NI3awI?= =?us-ascii?Q?I2wQBr31puKi8aSXrzd3xoy+xnRiGIk1o24AN7cNLp5J3catxsRv1Fd38psd?= =?us-ascii?Q?hJw5evL1WLCC+C+xj+MYdm9RtimiPJccRRL/dQNFhv8658CREegzLHGpTlRk?= =?us-ascii?Q?Db5pAuwGGnXwlgReR+lnR635pKpXsu2gGfWaB1/hW9lg07FNzmXRpuknEqh4?= =?us-ascii?Q?l0Tei5FIcuQb3W9WBJJjKhy7j0/U88L+//Cj1l+0QMV0HivtHHk8ISYn4Acg?= =?us-ascii?Q?dti6P0/j+2FGZByluQrxapBPZ/Cs//GkAsjt9NvaPClsX3ZcfcNlP2TkIpAi?= =?us-ascii?Q?2Ds+sLtaM7frPf3cCjCeeGZuyiTuTpezDWByWk6/8H0W8C6b8jxAyTbuuS3o?= =?us-ascii?Q?XZW907URnfC1QOYkEOUmXlN+IMolXSrHgJumYp9SLPhPLkN+ShxSrNrbaSiA?= =?us-ascii?Q?FDY3h8eeXe2R+yXOl3DeGzFqW9r53g1PbUrYgAx/kpZpKzAfJ4dF1bWoIEBq?= =?us-ascii?Q?VG9VkCbLLgZ+27h9btIApvEalFOoFyVoFmpHMgglnoWpGOlSXpWv7HbOl7ge?= =?us-ascii?Q?QcO+Afu/30JKS+GvoW0+gQEo1VOIQRMJeO4HFDp95xIFSsm0Q9KRWaVV8Xm3?= =?us-ascii?Q?itIvOo/BHHcN8PpxSMoGsowSwU2GskklLTfpqVSD5k5k5vliDxdnmZc6kuju?= =?us-ascii?Q?MXnI50qMhO+VAI/ybVEn8o4ywQZZo5dJj2Xq297K6SPVN9xs2wMbuGSUM2g6?= =?us-ascii?Q?YoS/sqg2SoqxD4PGSVqZsgiLQJWG+OJ06WaWn9DssuGsqD2MQUdB97Sl+MwC?= =?us-ascii?Q?/hmeozWNZ1vGV4i3JwksV4+hX5SaK3ZGZv6tcQopJFVutkp5tgC1Ule07e0R?= =?us-ascii?Q?c2Ok40oPfK7gq+vxd2sBsQcIp2YuhKCNkkBvyx0bFtyTIdlHKm970BIthqzO?= =?us-ascii?Q?qDkEwg86BGOoFViatFqFE13X7DxeYHL2Roh8cfU9St1AeZnlsVkOOVMB75CW?= =?us-ascii?Q?3HlKrueKLW4esASJJp3MvA4aTOegPZaT3nrhXsljG9KAh46OydnXyddxN3xz?= =?us-ascii?Q?FPtRJRYmg39YsfXf9CUz4Z5YCgdcVRRWHJZItqPUkg/x6z2BgcQrib0Tj680?= =?us-ascii?Q?378u8m+mNTIjgdpXeteKRvkSjCZBSmvehQVx243+Yj66lnvyAUZvre35lhqI?= =?us-ascii?Q?Sji8LVX6to17jKAgYxl6h+H1CAtu+zq02LLhP725dHv/7ZMSCebQ0G5M2Lu0?= =?us-ascii?Q?2lEbb2TThPfE7M3n0odG9kj1BCiZ+qVKoAjRMLJze8wi7egUbdkCMQAdr4po?= =?us-ascii?Q?avV1Muy7aWsFL4P0E9AE37Dp?= MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AS8PR04MB8500.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 74dc1c31-75fb-4036-6dc6-08d930e3f136 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jun 2021 16:29:41.0513 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: by6pRTs9ya9GtGuAJq2ZoCUrHOHc0vMRIGRZpUpBwuKpk9GeMcCDnOHMNzlhx8zlnz8hEPkNIUmW92kGkOTTgw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8817 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210616_092943_877377_70E0DF5F X-CRM114-Status: GOOD ( 28.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org > > > -----Original Message----- > > From: Frank Li > > Sent: Monday, June 14, 2021 5:42 PM > > To: Will Deacon > > Cc: Shenwei Wang ; Han Xu ; > > Nitin Garg ; Jason Liu ; > linux- > > arm-kernel@lists.infradead.org; Zhi Li > > Subject: The problem about arm64: io: Relax implicit barriers in default I/O > > accessors > > Added Catalin. [Frank Li] sorry, corrected catalin's address > > > > > Will Deacon: > > > > Our a test case is failure at 8QM platform(arm64). USB transfer > > failure if run with GPU stress test. > > I found it related with your below change. > > > > commit 22ec71615d824f4f11d38d0e55a88d8956b7e45f > > Author: Will Deacon > > Date: Fri Jun 7 15:48:58 2019 +0100 > > > > arm64: io: Relax implicit barriers in default I/O accessors > > > > The arm64 implementation of the default I/O accessors requires barrier > > instructions to satisfy the memory ordering requirements documented in > > memory-barriers.txt [1], which are largely derived from the behaviour of > > I/O accesses on x86. > > > > drivers/usb/host/xhci-ring.c > > > > static void giveback_first_trb(struct xhci_hcd *xhci, int slot_id, > > unsigned int ep_index, unsigned int stream_id, int start_cycle, > > struct xhci_generic_trb *start_trb) > > { > > /* > > * Pass all the TRBs to the hardware at once and make sure this write > > * isn't reordered. > > */ > > wmb(); > > if (start_cycle) > > start_trb->field[3] |= cpu_to_le32(start_cycle); > > else > > start_trb->field[3] &= cpu_to_le32(~TRB_CYCLE); > > xhci_ring_ep_doorbell(xhci, slot_id, ep_index, stream_id); > > } > > > > If I added wmb() before xhci_ring_ep_doorbell, the problem gone. > > Writel include io_wmb, which map into dma_wmb(). > > > > 1. write ddr > > 2. writel > > 2a. io_wmb(), dmb(oshst) > > 2b, write usb register > > 3. usb dma read ddr. > > > > > > Internal bus fabric only guarantee the order for the same AXID. 1 > > write ddr may be slow. USB register get data before 1 because GPU occupy > > ddr now. So USB DMA start read from ddr and get old dma descriptor data > > and find not ready yet, then missed door bell. > > > > If do 2-3 times doorbell, problem also gone. > > > > So I think dmb(oshst) is not enough for writel. > > > > A writeX() by the CPU to the peripheral will first wait for the > > completion of all prior CPU writes to memory. For example, this > ensures > > that writes by the CPU to an outbound DMA buffer allocated by > > dma_alloc_coherent() will be visible to a DMA engine when the CPU > > writes > > to its MMIO control register to trigger the transfer. > > > > > > Best regards > > Frank Li _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel