From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30D61C48BE5 for ; Wed, 16 Jun 2021 16:29:38 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F2FB760BD3 for ; Wed, 16 Jun 2021 16:29:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2FB760BD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nxp.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PrHJLfIGxiUT+ptKoo2LD6lIGJ3jQhHn1eg3WpdUerc=; b=5BjNBpHQyhNJrc zh0+t0/W6zDlaVDVb74Iz+50a9YCND5tte0xvOTEwHz2eLpEBU0h4bXQyjL4vRD+ZKHjKr8bsH5Uf X/XewQsWMNcjBAc3nRRXj5BhHVzOMW3jUm/aXnRIfKn8H6pUXsymKprBiA+K47IXIHj/ZkpBfGj9t ZOpQ7Vyw79M6EGs3P/75hVlNCO6qq3xDMPNQ3g1KePzumgfxcX40tOr0FnMB6aTfirmy9ExaYs2L4 cY3REWGnDZZsuLQH+jtpPk7iqqDByKxm5oRJk7cFYlfaw5cNRoUx3BQ9/wLM00LOLPwRKjzks5aBx h0ajAnh4iFrU+3DOj6nw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltYOe-0077c8-73; Wed, 16 Jun 2021 16:28:08 +0000 Received: from mail-eopbgr30073.outbound.protection.outlook.com ([40.107.3.73] helo=EUR03-AM5-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltYOa-0077bO-FU for linux-arm-kernel@lists.infradead.org; Wed, 16 Jun 2021 16:28:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kRAvGCIafuYJPE4ZQ3nUiOpqbH4TH5SJ+1JAUYoHAkkOUko0niTbV8JJKV2eIyVw1wbSQyg+9mlIi/VYTOa2cQix2n8Chvlo5mZM9rwOI10ycbgTvG6hIbTO5YlBucyVdZkroOrC+6H8cWwiCBgIx6cM8Zd8wQhGlF1qsJ3uVstqboBNFwFu3w0/XwP0tEKLsY0EtAFFjg2Qngk/6n2qxFQYmOqZXDLthkVbCA5vzfni0mYv06mXAlWNI1zXJL9s3QaOoNhWaqtlTJfY9xq8a3IHs3oqlP1rl3IymBXhfSi/5Pcd79gVnmAUDzLEDJwbZP4X+vSxO/B3fCNEm3IHsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WXj4iCfxToBzg9w5k7RVQ1El5TxeEY3DZmNYaXqPxtc=; b=EM7VCCiKVD/T1GRW7Gf8EsfQwyqeN0dqPOg01PXMP7FEexur8Y9GRiNVWoXB32BD19z52brnHMSE7vGa2cvx4bHgcGTcS9d5Z8UuNmFrAW5081EVpxUrYnZvy0keaXit7i2xqIHaFfCTLr95OuZWGh6lfl4NvTwVg8ewXzmDbGEDkJSjXbqmEG4Ain1G5pdrpZWbjR7r8qDnNJVGwXlJfU0p5sq5zBD1owAoWg62DpFKfvy/e8H+5i4dZWlUWGv/n9vm2RiSH+kWDOxzmlE4TiuLu38EMCqTUIBQHqD1FO3GtONfi7mVwkDeFpOC9tzpqruUqLaPdIWzCkwfCtUdzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WXj4iCfxToBzg9w5k7RVQ1El5TxeEY3DZmNYaXqPxtc=; b=VsadXOV5dAbF6uBqy24XgZG72i9gpCJZWMOjHuUjTz30O8Zk5wdZL9pczclbYawdIkhSVfdobRWGt3+3Q953M38PjDfJK8xjpYNozur2T74rYVVeBInHNDjFfSs9qoyrMD6XrvY9wjE6P6RXlpMee0P8vrEvQKC2oHhNjN7kCFk= Received: from AS8PR04MB8500.eurprd04.prod.outlook.com (2603:10a6:20b:343::14) by AS8PR04MB8657.eurprd04.prod.outlook.com (2603:10a6:20b:428::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.18; Wed, 16 Jun 2021 16:27:59 +0000 Received: from AS8PR04MB8500.eurprd04.prod.outlook.com ([fe80::2ca8:978f:5800:3e9a]) by AS8PR04MB8500.eurprd04.prod.outlook.com ([fe80::2ca8:978f:5800:3e9a%7]) with mapi id 15.20.4242.018; Wed, 16 Jun 2021 16:27:59 +0000 From: Frank Li To: Will Deacon , =?gb2312?B?o7tjYXRhbGluLm1hcmluYXNAYXJtLmNvbQ==?= <ï¼›catalin.marinas@arm.com> CC: Shenwei Wang , Han Xu , Nitin Garg , Jason Liu , "linux-arm-kernel@lists.infradead.org" , Zhi Li Subject: RE: The problem about arm64: io: Relax implicit barriers in default I/O accessors Thread-Topic: The problem about arm64: io: Relax implicit barriers in default I/O accessors Thread-Index: Addhawkczavy5sh9TS2ivC8EbHYnQABYXRPA Date: Wed, 16 Jun 2021 16:27:59 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=nxp.com; x-originating-ip: [72.177.87.193] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3280e936-01e9-4420-20a8-08d930e3b4c9 x-ms-traffictypediagnostic: AS8PR04MB8657: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2958; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: JPtcumibTkalgu7Fr+X5xJ8uSwf16jfwfdlwuLcPv1dxwkOzPIoRUVDnZzVLgFprDVokzQLgnX0FVlt3gz5h5JEs8KQpUYNIllihw57E8UQD+1p1sou2N5dI33H4fGZwuuj2WgZ/RpbYRFgeLx1j99Vnwe1Rc28KjVW+B9W/e+dXKj0V5Pf5P7P9ndkpo+YreZsYcw80B56ScJGpkg99rnKUhmKNmdPVcb4OCxS4Lt/q0ID7JJhZVOPJTY3GHIzX7HAwTZAJIgGtIFEE7ftYe2DLVrVmoabsDAmgafg+3j+hRWf0Nq045uomgb6zrYbfxIAJmElpl3tRnrvQnb8av9mtNsTAK3dThb3oU47Ls+8Y7riaPaXHVZjdGH75cSlP+Sa0Zd8tcV60A+zpFcgbVsrzhzu9mL5WDgG4bFgfaQbJAOTzSwhCHByS1+D5Vq3Hnn/E0jXE3L8mrV01lMTf7n1JIqlsh7EQ8TDdj1XqxJZC1xCvpaMjw6ZBD9G3jUKOip4Qar2r/EOaZimP83tUTf6fMo9KNVNxKqOmEQcdgix+WjKnuKlzkFck+MJnDxo/gKwiDR2XqaizSYQN2DmiFzozcN2DGtwUUh1Wt+/yz+g= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR04MB8500.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(366004)(396003)(346002)(376002)(39850400004)(33656002)(110136005)(83380400001)(9686003)(186003)(55016002)(54906003)(38100700002)(122000001)(71200400001)(6506007)(316002)(53546011)(26005)(52536014)(5660300002)(66476007)(8936002)(76116006)(8676002)(4326008)(64756008)(66446008)(2906002)(86362001)(478600001)(66946007)(66556008)(44832011)(7696005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?gb2312?B?MG5va29wM0VQbXRuSitvOVhjenIyZllEMEI4bjBSOFdNcUVwNWY0dGh6OU1x?= =?gb2312?B?MFpqOE9oYitobVhDbTdVYjI5UDdZWXVrbmNoUzJHN1JNUWY0WVZBeHJ3Szk1?= =?gb2312?B?TVdVQnhJODdtUWVIVlBOMERwT0g3SFNaMGdkMHNZZ2FkMERJd2FISjlGSHA5?= =?gb2312?B?RGdSczN2OWhyL2s2RlBMckIxamQ1U003UkhXSE5PQkpzQ0FiMTc4aGdQaGtE?= =?gb2312?B?VERYUkI0S0lBQ3FPQ3RLZWpwaE0xcEtIUm15NXFoNFlwRGtCNjQ1WVNidXFJ?= =?gb2312?B?c3Zkbzh1ZnVvak5zcUQ3UUxMdDd5MHFNZlAwdmpadHlEd0xSWjh5dURKQWt2?= =?gb2312?B?OVdXWEgyYTM1U1V1Rmgwbk55c0p0akZtRnpJUFdVTHlKSmNCY3I2WlpmWHZN?= =?gb2312?B?QmRJM2RwaDAvZndXbDdnbzBDbzJDVGZIUEdVYVR6eXZVVTNXeXZqZG9UUDVt?= =?gb2312?B?K0RjNnFJcmRGUFZERTduRUdaYUlKRVhlemxCREVCc21mWlhkS0F2NVNjalZu?= =?gb2312?B?OXN5Y1FzUERFeWJPV1J0RCsraTM1cHZqVVJYaThvV0Q4TmZXNm1iSmtwTGky?= =?gb2312?B?T2Q4NGs4T0kzYjBSOVNmemhZTUpoYWhYM2svZWVteFR4RmE3RndyZDlyZ2tt?= =?gb2312?B?dDlSb09KRDlSR2F5SVM2RmZISGJZVmRkZnhGUGdkVU9GcldhMGZBeXc3cUI4?= =?gb2312?B?UzhjWTRjSmNtVk9CV2Q0UGpYdVM4aTFTbHUzU1VFTDMzUFRKNlJnZ1dGbW1B?= =?gb2312?B?Y3k5ekk5MUk2WElzMXlFWnRLbW5ud1NoMUEwc2szbFpCUEExMStKakhUU0xM?= =?gb2312?B?QjhMQklHZlFUVHBXVW9aeUVyYjZyMlFROXEzUFIzQ0ZCYkVMbnYrejlZZkhx?= =?gb2312?B?Q3NWTUpoU1RwYmpjejNTTDBKd2k3d1luWi9uMldQVGo0dFB2VG1MVVdvbTcz?= =?gb2312?B?NHQxNStIQ0NtdklQN01MaHZCTGdFdXovZyt2dzhJRWoxeXlaS0dxS0c4dTc2?= =?gb2312?B?ZTgvam03a2NDUXdXN2ZYdnJURnpGRTUzV2tQR2FiNzNkMkYzMHVQMVhvaHBo?= =?gb2312?B?c21GMnU2bmt3Q2ZPckpuUE1wUHhtSXBOK3IxM1lNR1g1dVd0cEFXMHgzR0RR?= =?gb2312?B?VExPWHZoSGdYeDQ4R25kRjdtaEVRQTNkNFh5blU3ZmxjZmdMVldyeFU4M2FB?= =?gb2312?B?U25MUlZDamoyK3VmS1JuOFFUZTJRaUxuZDZtUmNZV3NVVHg5RkwrUFUxKzdY?= =?gb2312?B?SUU0VnRKOTZzNHFRa044UGJzRzhDUEZTbHVLRExkdUFvZWNCWFg3OEdzbktL?= =?gb2312?B?MHVCY3FDWU5tY3FLS084MVZNTnVsbklDV0hsMUwwL2lHRDlPWU81V2tvY1Zh?= =?gb2312?B?TDBsOHNTVXNnU2cxUzd5ZUk3aG41NzFCQjBCR1dhOWE2NVZScExORkR3MzBz?= =?gb2312?B?dUFBQnNRN0paSGY1Qzh6V2xYNzlvSEtLai9qbExpVVJMNm0wU1krVysvY1Rx?= =?gb2312?B?MUUyYnZsZlVpSkRiWXpJeEZnY0duSlZaaG5TN2hUVGtpL29oZmppdmZhV25X?= =?gb2312?B?M3dNVTZZVnd5aStQK002bXk0YmEyeEowSWhMZjFrbTlsMjVobmEvNHlOdDY4?= =?gb2312?B?YktBeVlQckhFbzZDR3FQWjBtTDFRREdsdmVxbGxqV2FvTUxGcXdTR2kzeGti?= =?gb2312?B?c1ZyNG1WVmp0SHJsNFZKaWF6cjdudGphS1RSVG42M1l0OHU0WDFPVzJpNFVS?= =?gb2312?Q?U+vxStQooamYZRH521waAB4BxrtaUaFarydF0ZQ?= MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AS8PR04MB8500.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3280e936-01e9-4420-20a8-08d930e3b4c9 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jun 2021 16:27:59.7205 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DZzH0XimRJ7wE8YOt9aK2OCyraWX/sO7fMPrZTpyF1NeFLm7Ld7wquKpAXoQAK71Zd984Gsr50XYqO8PWfeErQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8657 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210616_092804_655545_19CDC474 X-CRM114-Status: GOOD ( 23.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org > -----Original Message----- > From: Frank Li > Sent: Monday, June 14, 2021 5:42 PM > To: Will Deacon > Cc: Shenwei Wang ; Han Xu ; > Nitin Garg ; Jason Liu ; linux- > arm-kernel@lists.infradead.org; Zhi Li > Subject: The problem about arm64: io: Relax implicit barriers in default I/O > accessors Added Catalin. > > Will Deacon: > > Our a test case is failure at 8QM platform(arm64). USB transfer > failure if run with GPU stress test. > I found it related with your below change. > > commit 22ec71615d824f4f11d38d0e55a88d8956b7e45f > Author: Will Deacon > Date: Fri Jun 7 15:48:58 2019 +0100 > > arm64: io: Relax implicit barriers in default I/O accessors > > The arm64 implementation of the default I/O accessors requires barrier > instructions to satisfy the memory ordering requirements documented in > memory-barriers.txt [1], which are largely derived from the behaviour of > I/O accesses on x86. > > drivers/usb/host/xhci-ring.c > > static void giveback_first_trb(struct xhci_hcd *xhci, int slot_id, > unsigned int ep_index, unsigned int stream_id, int start_cycle, > struct xhci_generic_trb *start_trb) > { > /* > * Pass all the TRBs to the hardware at once and make sure this write > * isn't reordered. > */ > wmb(); > if (start_cycle) > start_trb->field[3] |= cpu_to_le32(start_cycle); > else > start_trb->field[3] &= cpu_to_le32(~TRB_CYCLE); > xhci_ring_ep_doorbell(xhci, slot_id, ep_index, stream_id); > } > > If I added wmb() before xhci_ring_ep_doorbell, the problem gone. > Writel include io_wmb, which map into dma_wmb(). > > 1. write ddr > 2. writel > 2a. io_wmb(), dmb(oshst) > 2b, write usb register > 3. usb dma read ddr. > > > Internal bus fabric only guarantee the order for the same AXID. 1 > write ddr may be slow. USB register get data before 1 because GPU occupy > ddr now. So USB DMA start read from ddr and get old dma descriptor data > and find not ready yet, then missed door bell. > > If do 2-3 times doorbell, problem also gone. > > So I think dmb(oshst) is not enough for writel. > > A writeX() by the CPU to the peripheral will first wait for the > completion of all prior CPU writes to memory. For example, this ensures > that writes by the CPU to an outbound DMA buffer allocated by > dma_alloc_coherent() will be visible to a DMA engine when the CPU > writes > to its MMIO control register to trigger the transfer. > > > Best regards > Frank Li _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel