From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F850C433E1 for ; Fri, 14 Aug 2020 08:46:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 135F7207DA for ; Fri, 14 Aug 2020 08:46:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="FwCxKGg1"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=nxp.com header.i=@nxp.com header.b="X06XmTAx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 135F7207DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nxp.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MRWYMQgO6gtshQUpCrdQ7XzDZwN8hw5Kx1qHPnG7nOw=; b=FwCxKGg1YeBjK4R/BDzmPs8Eb V1Orrp8lh+X6Wy+b/huRlHqZHkbLeF0VWN/EobNHEjoutcyekyCDmpSnhjDlxfDJzTt/BIYtPhCsu gMGHmH7P4qPhSN6gK6axsG/pqpIs/vXBJEKefKWhx/0p3tyLV1vlp5AlMs+s3/8DNlZFwGoXe69ww WkguTn/Gdu+pkxdMsyVhi2eCkPoDfdtln1PkpRvQ02ggB8C2P55Jy3GPsD2R/oRhqVuUI1P+SZB5Q xiGWWkawAhOumZYqAlUE5+DNuz6YNNCpaZcjMiSeEgCO+J+RhtNO3fUUUmtcdVrexD+1zFcmjwJbc mXDj6Hxug==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k6VL2-0001Qz-I4; Fri, 14 Aug 2020 08:45:24 +0000 Received: from mail-he1eur02on062f.outbound.protection.outlook.com ([2a01:111:f400:fe05::62f] helo=EUR02-HE1-obe.outbound.protection.outlook.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k6VL0-0001QP-3H for linux-arm-kernel@lists.infradead.org; Fri, 14 Aug 2020 08:45:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TKHKFqHc3/g5pAeee1raTaoWB+4ady3OQIWnL26c2dVm5r/RzZuFJgripcbb/+dOjasYhAkDw7wJ5vyztn0UvpKU0AMToNH3HBdQBb8ZR0gqItL7RIxuFu7QCwYEJL0EXZjUTIkKWDIFeQ8wEMdMON9WU8+sSTKzIDL1WR90UtlFXdElch45IexzS5MLdtcYwyztkxzcYC+dVYoYh+hOVeMbPBtcNWcHP9RfwaMB+p6yzdQod/8IFk6M9W0bluzAjke/uZPMy6/NHVkBUPBXGbtUUvDLfYPxCtwzsmDMj9oAkI3VO12//w0VoUX0SZ5zjMj/8vB8y606WElqf6DQKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OT7gVHa83ywzYJSFGQ/gYZyrQfv4p5BGZSmbyQCrhS4=; b=WjMZnAzs7B8dsFeXa8OnlAMkexVG4fJ2EfUsGtIOPoU6Im42S2+Lf96diA5bo/6i3Mckpp+7TMdixkdl3EsHqEyPOSs8J52DiELkqKX/XD8zOkiGCuFoqE29k0ulzCKMa/Ojr827h4VrcTMNl2qvc/PgAVTRhEtrB9MYLM/NRGqbExFLZ7JNvf7yOeYPCb4s+xub3NSmarBZIE0/F5LLTMesF0KSjkqd+PKY/G2YwjaApl+n2MAC1lmO/MqySx594lEWnOJqqFZzXDjywEiRmlRgHHPOTrTMvxTYj3JEdJyolLnRSY2ACKP0nG8aIq/qHULvHT6TI1/+26/UhWSEew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OT7gVHa83ywzYJSFGQ/gYZyrQfv4p5BGZSmbyQCrhS4=; b=X06XmTAxiAj9NxDHaIFd0qsIxymmjHLk9QtCyN+bNC/6ea8UUUNu+oMgUstBZD+RJ07ORuue7r41um+1nrtxsvpZ5MSokR8vvtptTSDxAKrPoqmuM/u+9xXRbJqxUIq18wAm1aNiURToFLJJH4Z9aOO32J2RH8BQZQ1rpSG6uBc= Received: from VE1PR04MB6638.eurprd04.prod.outlook.com (2603:10a6:803:119::15) by VI1PR04MB5502.eurprd04.prod.outlook.com (2603:10a6:803:c9::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3283.15; Fri, 14 Aug 2020 08:45:18 +0000 Received: from VE1PR04MB6638.eurprd04.prod.outlook.com ([fe80::ad7f:d95a:5413:a950]) by VE1PR04MB6638.eurprd04.prod.outlook.com ([fe80::ad7f:d95a:5413:a950%3]) with mapi id 15.20.3283.020; Fri, 14 Aug 2020 08:45:17 +0000 From: Robin Gong To: Richard Leitner , "dmaengine@vger.kernel.org" , "alsa-devel@alsa-project.org" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" Subject: RE: pcm|dmaengine|imx-sdma race condition on i.MX6 Thread-Topic: pcm|dmaengine|imx-sdma race condition on i.MX6 Thread-Index: AQHWcWQbD9eqMlwY2U27XyctTVPmVak3Rd/w Date: Fri, 14 Aug 2020 08:45:17 +0000 Message-ID: References: <20200813112258.GA327172@pcleri> In-Reply-To: <20200813112258.GA327172@pcleri> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: skidata.com; dkim=none (message not signed) header.d=none;skidata.com; dmarc=none action=none header.from=nxp.com; x-originating-ip: [119.31.174.67] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 7990903c-ebe7-49c1-0208-08d8402e5f06 x-ms-traffictypediagnostic: VI1PR04MB5502: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: rJLHQuDk+RXddueGGDO64rqAdXRSzEoMPnAcpi69T9uM4MRaTM65JDWLdU9O9fc69CI6My6VlFpqLpMd7wwRjwCHWMpU4GcPDUU6vDrKr+INDIZcQRpOGHzRZg7F9rOd9ctRScasCM1RVMTF1JgyDa98Bt3ScYYXCP14zvL+ba2o08tH0fHJ/9VdfbsDQrZtlYIRUPXMO8dWV5Nex1O38j07aOLL2g7LLmZStFskEcHe0gjabFp5cEuZqvZV7iuTZZUk10jB6g4pRJiTFoxznC256unSYD1PpBkB6UbDZZIrwuCiBXwJ7UktRfSVFsybSWhFlW+G5MSAZenJ6bnekDJFUaK6ZuTwp5/PyyrwLhct3tO7plnIVhXCfBGiTSyIpLXuD3NWaLxaWKeTLCeHnQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VE1PR04MB6638.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(39860400002)(366004)(136003)(346002)(376002)(2906002)(8676002)(7416002)(316002)(110136005)(8936002)(71200400001)(33656002)(54906003)(6506007)(55016002)(26005)(7696005)(186003)(9686003)(66476007)(4326008)(86362001)(966005)(478600001)(76116006)(66946007)(66556008)(45080400002)(52536014)(5660300002)(83380400001)(83080400001)(64756008)(66446008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: k2OhU4goLA1npSdTfmq/W8qnlwelM1WGwVX+XeN1W/P1US1o6Y4Jl0w1W8kAUoI+O8hLbz0sLdkk9kE51M4TB/q24oJZnGKVdJyRRQe4yTXCYuULNhNdc6/M2hgTdXDQ6TDjbusQHuXnFM95D71BPfYdiQMPKGTpJ/+E/0WTgfGYbbafR2gEM8SWLSztkwlCGhEhpzEZcItm4XRev3CQ8xI4HyTWEDXDfM29O9SKr2DDPN0rbvC6+lWy8hon9aJ9l6FmQGRuofAELLCsrNuTHHtGfrToEC81W56GfBSGndRr4aVf73R5rDIO63wNaZogNNwpb8s8GhT2IBfOzbP5SM8N6xnc/jBkjvwGTsCbZiBNSfKiic+p0lh591GohnVCUbvc/s/CNE+A2e4S4cmOfVLE0ohkbR2B4sOVv79GDwX8aZbCYrkgfKHbfT5z88g/+SLHFFmd2PbBQvSDW0HSz00DLmXHkhEmskfeLrhz8bFfBeVGGRuXfifie8Vi33drWRaaucQDYEeaIE6Q2qVUO6eJQP2O85rUV2aeeQXc8Yny0gwEFI/k0rerKBTX4GTZ5gV8J9izyIMzYfwwFkagOI3LPZUomX+A+5BF4vTIs55oZe2g1uFVkCN3Q7gA3s9QAAiK2jOQW1VRjX/ySSrnHg== MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VE1PR04MB6638.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7990903c-ebe7-49c1-0208-08d8402e5f06 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Aug 2020 08:45:17.8318 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: OQzQH/uOEsV0ohV4aLU07OGYFTxWp/SopCQE+PSZfJVN1DUguIAnzhoYIoa4UIROIdepE2XdwL21rQz2LbjOiQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB5502 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200814_044522_262614_7D9043DF X-CRM114-Status: GOOD ( 28.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "timur@kernel.org" , "nicoleotsuka@gmail.com" , "vkoul@kernel.org" , dl-linux-imx , "kernel@pengutronix.de" , "dan.j.williams@intel.com" , "shawnguo@kernel.org" , Benjamin Bara Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2020/08/13 19:23: Richard Leitner wrote: > Hi, > we've found a race condition with the PCM on the i.MX6 which results in an > -EIO for the SNDRV_PCM_IOCTL_READI_FRAMES ioctl after an -EPIPE (XRUN). > > A possible reproduction may look like the following reduced call graph during a > PCM capture: > > us -> ioctl(SNDRV_PCM_IOCTL_READI_FRAMES) > - wait_for_avail() > - schedule_timeout() > -> snd_pcm_update_hw_ptr0() > - snd_pcm_update_state: EPIPE (XRUN) > - sdma_disable_channel_async() # get's scheduled away due to sleep us > <- ioctl(SNDRV_PCM_IOCTL_READI_FRAMES) returns -EPIPE us -> > ioctl(SNDRV_PCM_IOCTL_PREPARE) # as reaction to the EPIPE (XRUN) us -> > ioctl(SNDRV_PCM_IOCTL_READI_FRAMES) # next try to capture frames > - sdma_prep_dma_cyclic() > - sdma_load_context() # not loaded as context_loaded is 1 > - wait_for_avail() > - schedule_timeout() > # now the sdma_channel_terminate_work() comes back and sets # > context_loaded = false and frees in vchan_dma_desc_free_list(). > us <- ioctl returns -EIO (capture write error (DMA or IRQ trouble?)) Seems the write error caused by context_loaded not set to false before next transfer start? If yes, please have a try with the 03/04 of the below patch set, anyway, could you post your failure log? https://lkml.org/lkml/2020/8/11/111 > > > What we have found out, based on our understanding: > The dmaengine docu states that a dmaengine_terminate_async() must be > followed by a dmaengine_synchronize(). > However, in the pcm_dmaengine.c, only dmaengine_terminate_async() is > called (for performance reasons and because it might be called from an > interrupt handler). > > In our tests, we saw that the user-space immediately calls > ioctl(SNDRV_PCM_IOCTL_PREPARE) as a handler for the happened xrun > (previous ioctl(SNDRV_PCM_IOCTL_READI_FRAMES) returns with -EPIPE). In > our case (imx-sdma.c), the terminate really happens asynchronously with a > worker thread which is not awaited/synchronized by the > ioctl(SNDRV_PCM_IOCTL_PREPARE) call. > > Since the syscall immediately enters an atomic context > (snd_pcm_stream_lock_irq()), we are not able to flush the work of the > termination worker from within the DMA context. This leads to an > unterminated DMA getting re-initialized and then terminated. > > On the i.MX6 platform the problem is (if I got it correctly) that the > sdma_channel_terminate_work() called after the -EPIPE gets scheduled away > (for the 1-2ms sleep [1]). During that time the userspace already sends in the > ioctl(SNDRV_PCM_IOCTL_PREPARE) and > ioctl(SNDRV_PCM_IOCTL_READI_FRAMES). > As none of them are anyhow synchronized to the terminate_worker the > vchan_dma_desc_free_list() [2] and "sdmac->context_loaded = false;" [3] are > executed during the wait_for_avail() [4] of the > ioctl(SNDRV_PCM_IOCTL_READI_FRAMES). > > To make sure we identified the problem correctly we've tested to add a > "dmaengine_synchronize()" before the snd_pcm_prepare() in [5]. This fixed the > race condition in all our tests. (Before we were able to reproduce it in 100% of > the test runs). > > Based on our understanding, there are two different points to ensure the > termination: > Either ensure that the termination is finished within the previous > SNDRV_PCM_IOCTL_READI_FRAMES call (inside the DMA context) or finishing > it in the SNDRV_PCM_IOCTL_PREPARE call (and all other applicable ioclts) > before entering the atomic context (from the PCM context). > > We initially thought about implementing the first approach, basically splitting > up the dma_device terminate_all operation into a sync > (busy-wait) and a async one. This would align the operations with the > DMAengine interface and would enable a sync termination variant from atomic > contexts. > However, we saw that the dma_free_attrs() function has a WARN_ON on irqs > disabled, which would be the case for the sync variant. > Side note: We found this issue on the current v5.4.y LTS branch, but it also > affects v5.8.y. > > Any feedback or pointers how we may fix the problem are warmly welcome! > If anything is unclear please just ask :-) > > regards; > Richard Leitner > Benjamin Bara > > [1]https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir. > bootlin.com%2Flinux%2Fv5.8%2Fsource%2Fdrivers%2Fdma%2Fimx-sdma.c%23 > L1066&data=02%7C01%7Cyibin.gong%40nxp.com%7C79ad115b01ef453f7 > e7408d83f7b3c4d%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637 > 329145824068928&sdata=D9F%2FRUG27xv9nv8J1KtrLtld2eaI6gsXiWIAIgk > Avjw%3D&reserved=0 > [2]https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir. > bootlin.com%2Flinux%2Fv5.8%2Fsource%2Fdrivers%2Fdma%2Fimx-sdma.c%23 > L1071&data=02%7C01%7Cyibin.gong%40nxp.com%7C79ad115b01ef453f7 > e7408d83f7b3c4d%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637 > 329145824068928&sdata=0EKDVgzOZzL7TpX4ykhqjvpz5ryUHUpWw7frRe > cksBU%3D&reserved=0 > [3]https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir. > bootlin.com%2Flinux%2Fv5.8%2Fsource%2Fdrivers%2Fdma%2Fimx-sdma.c%23 > L1072&data=02%7C01%7Cyibin.gong%40nxp.com%7C79ad115b01ef453f7 > e7408d83f7b3c4d%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637 > 329145824068928&sdata=aIhatvb1ocQqyYCVFEg71LgJlRBoVusbDFPIxnte > PuY%3D&reserved=0 > [4]https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir. > bootlin.com%2Flinux%2Fv5.8%2Fsource%2Fsound%2Fcore%2Fpcm_lib.c%23L1 > 825&data=02%7C01%7Cyibin.gong%40nxp.com%7C79ad115b01ef453f7e > 7408d83f7b3c4d%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6373 > 29145824073919&sdata=y0Udbd%2FKGaVgqLrcp6fNOlMlFCGHCMfojkpp > B4HzUuE%3D&reserved=0 > [5]https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir. > bootlin.com%2Flinux%2Fv5.8%2Fsource%2Fsound%2Fcore%2Fpcm_native.c%2 > 3L3226&data=02%7C01%7Cyibin.gong%40nxp.com%7C79ad115b01ef453f > 7e7408d83f7b3c4d%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63 > 7329145824073919&sdata=ch3BQ5DDGU5HWXqIZSvUeFnBoRoP%2BMM > HEpnk8mIfWj8%3D&reserved=0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel