From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22EFCC47087 for ; Fri, 28 May 2021 10:03:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E606661358 for ; Fri, 28 May 2021 10:03:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235793AbhE1KFY (ORCPT ); Fri, 28 May 2021 06:05:24 -0400 Received: from ni.piap.pl ([195.187.100.5]:47424 "EHLO ni.piap.pl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236134AbhE1KEt (ORCPT ); Fri, 28 May 2021 06:04:49 -0400 Received: from t19.piap.pl (OSB1819.piap.pl [10.0.9.19]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ni.piap.pl (Postfix) with ESMTPSA id 4422C44423C; Fri, 28 May 2021 12:02:53 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.11.0 ni.piap.pl 4422C44423C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=piap.pl; s=mail; t=1622196173; bh=rBrr8No9YxW9GxRteQ/5coCNcDbxv/wQgs4N3978vXk=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=eGE0MnbetMS2eEZUzbFmwD29gGEtxpxbnwuSvMjeolTvuLLMFVnp+hInW8hzTHE6p YdQB50+FRr5kEX89uXgyDrd6Mg5QPvhWoYvwleaswpUw6sB8b91SK7sC47OoBYsgtu R/+jsPq0kEwjbKUttSO6MR2NLQJhnyJCsfEcX1zE= From: =?utf-8?Q?Krzysztof_Ha=C5=82asa?= To: "Russell King (Oracle)" Cc: linux-arm-kernel , lkml Subject: Re: Data corruption on i.MX6 IPU in arm_copy_from_user() References: <20210526100843.GD30436@shell.armlinux.org.uk> <20210526131853.GE30436@shell.armlinux.org.uk> Sender: khalasa@piap.pl Date: Fri, 28 May 2021 12:02:52 +0200 In-Reply-To: <20210526131853.GE30436@shell.armlinux.org.uk> (Russell King's message of "Wed, 26 May 2021 14:18:53 +0100") Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-KLMS-Rule-ID: 4 X-KLMS-Message-Action: skipped X-KLMS-AntiSpam-Status: not scanned, whitelist X-KLMS-AntiPhishing: not scanned, whitelist X-KLMS-AntiVirus: Kaspersky Security for Linux Mail Server, version 8.0.3.30, not scanned, whitelist Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org "Russell King (Oracle)" writes: > In any case, looking at the architecture reference manual, LDM is > permitted on device and strongly ordered mappings, and the memory > subsystem is required to decompose it into a series of 32-bit accesses. > So, it sounds to me like there could be a hardware bug in the buses/IPU > causing this. It seems so. I modified the kernel IPU module a bit, initialized a bunch of IPU registers to known values (1..0xD). Results (from 1 to 13 IPU registers) obtained with different instructions: readl(13 consecutive registers): CSI =3D 1 2 3 4 5 6 7 8 9 A B C D 1 =3D register #0 and so on - readl() results are obviously correct. LDM1: 1 (not corrupted) LDM2: 1 3 LDM3: 1 3 4 LDM4: 2 3 4 4 LDM5: 1 3 4 5 6 LDM6: 1 3 4 5 6 7 LDM7: 1 3 4 5 6 7 8 LDM8: 2 3 4 5 6 7 8 8 LDM9: 1 3 4 5 6 7 8 9 A LDM10: 1 3 4 5 6 7 8 9 A B LDM11: 1 3 4 5 6 7 8 9 A B C LDM12: 1 3 4 5 6 7 8 9 A B C D The last one uses: ldm r4, {r1, r2, r3, r4, r5, r6, r7, r8, r9, sl, fp, ip}. I haven't tested more than 12 registers in one kernel LDMIA instruction. The results don't depend on the address offset (adding 4, 8 or 12 to the address doesn't change anything). The arm_copy_from_user() is a specific case of the same corruption. It uses a number of PLDs and 8-register LDMIAs (and then possibly LDRs which don't fail). Each LDMIA ("LDM8") returns again: LMD8: 2 3 4 5 6 7 8 8 (the same with subsequent LDMIAs: 10 11 12 13 14 15 16 16 and so on). Summary: it appears all 64-bit and longer LDMIA instructions fail. The first or the second 32-bit access is skipped (possibly somewhere between AXI and IPU). In case of 4- and 8-register LDMs, the first (#0) value is skipped, otherwise, it's the second (#1) value. Now the PLDs ring a bell: "ERR003730 ARM: 743623=E2=80=94Bad interaction between a minimum of seven P= LDs and one Non-Cacheable LDM can lead to a deadlock". Looking at the disassembly I can count 6 PLDs (the first two seem to be the same, though I don't claim I understand this (source) .s code). Also this problem happens with IPU and not other devices, so I think it's not related to this erratum after all. size_t arm_copy_from_user(void *to, const void *from, size_t n) ... for n =3D 32 =3D 8 * 4 bytes: 2c: subs r2, r2, #4 ; =3D 28 30: blt e4 ; NOP 34: ands ip, r0, #3 ; r0 =3D destination 38: pld [r1] 3c: bne 108 ; NOP 40: ands ip, r1, #3 ; r1 =3D address in IPU 44: bne 138 ; NOP 48: subs r2, r2, #28 4c: push {r5, r6, r7, r8} 50: blt 88 ; NOP 54: pld [r1] ; duplicate PLD? 58: subs r2, r2, #0x60 5c: pld [r1, #28] 60: blt 70 64: pld [r1, #0x3c] 68: pld [r1, #0x5c] 6c: pld [r1, #0x7c] 70: ldm r1!, {r3, r4, r5, r6, r7, r8, ip, lr} ; <<<<< fails I also wonder if STMs may have similar problems - will check. --=20 Krzysztof Ha=C5=82asa Sie=C4=87 Badawcza =C5=81ukasiewicz Przemys=C5=82owy Instytut Automatyki i Pomiar=C3=B3w PIAP Al. Jerozolimskie 202, 02-486 Warszawa From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DC6CC4708C for ; Fri, 28 May 2021 10:51:27 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 492D9613B5 for ; Fri, 28 May 2021 10:51:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 492D9613B5 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=piap.pl Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:In-Reply-To: Date:References:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OwoNT4ECXq8k8W/yuUM8DQNQ5GUakLNDBLW40oCAxHQ=; b=mwBBxSgM0RXgOD QToBeMoEhXCuaTZNjRQy0ZyrbpQjRmooll3uJB9hg9hLuZrx80UZ+rjc0WCteU1rbmg3CKwmBoDQq 4luZiofUZuWvAz09dllaVOwrC/LLd3MKDhDG9IHxLPgvtog5h8lcaFZWQmYfw6EKwqfvD09+0kLkG kYyn65nKF5FK1TjVlqPT3nvjcLsQdd79wNTv9zY42/DmvNSbaRid+CGy4lBl9p9Hqdj/34yDYJA8C dL98WI7ZpFYqJs8L9o/f9Jpdtflko79MIddI2rwWg0dgMf49yYrJfEkl0+K20Zm8XPeu6ihdVJ4KU 3nCgDCRCC8g0kqUv6Q2w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lma16-00EX5B-HW; Fri, 28 May 2021 10:47:03 +0000 Received: from ni.piap.pl ([195.187.100.5]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lmZKa-00ED1u-Km for linux-arm-kernel@lists.infradead.org; Fri, 28 May 2021 10:03:08 +0000 Received: from t19.piap.pl (OSB1819.piap.pl [10.0.9.19]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ni.piap.pl (Postfix) with ESMTPSA id 4422C44423C; Fri, 28 May 2021 12:02:53 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.11.0 ni.piap.pl 4422C44423C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=piap.pl; s=mail; t=1622196173; bh=rBrr8No9YxW9GxRteQ/5coCNcDbxv/wQgs4N3978vXk=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=eGE0MnbetMS2eEZUzbFmwD29gGEtxpxbnwuSvMjeolTvuLLMFVnp+hInW8hzTHE6p YdQB50+FRr5kEX89uXgyDrd6Mg5QPvhWoYvwleaswpUw6sB8b91SK7sC47OoBYsgtu R/+jsPq0kEwjbKUttSO6MR2NLQJhnyJCsfEcX1zE= From: =?utf-8?Q?Krzysztof_Ha=C5=82asa?= To: "Russell King (Oracle)" Cc: linux-arm-kernel , lkml Subject: Re: Data corruption on i.MX6 IPU in arm_copy_from_user() References: <20210526100843.GD30436@shell.armlinux.org.uk> <20210526131853.GE30436@shell.armlinux.org.uk> Date: Fri, 28 May 2021 12:02:52 +0200 In-Reply-To: <20210526131853.GE30436@shell.armlinux.org.uk> (Russell King's message of "Wed, 26 May 2021 14:18:53 +0100") Message-ID: MIME-Version: 1.0 X-KLMS-Rule-ID: 4 X-KLMS-Message-Action: skipped X-KLMS-AntiSpam-Status: not scanned, whitelist X-KLMS-AntiPhishing: not scanned, whitelist X-KLMS-AntiVirus: Kaspersky Security for Linux Mail Server, version 8.0.3.30, not scanned, whitelist X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210528_030305_045324_8DEE75AF X-CRM114-Status: GOOD ( 17.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org IlJ1c3NlbGwgS2luZyAoT3JhY2xlKSIgPGxpbnV4QGFybWxpbnV4Lm9yZy51az4gd3JpdGVzOgoK PiBJbiBhbnkgY2FzZSwgbG9va2luZyBhdCB0aGUgYXJjaGl0ZWN0dXJlIHJlZmVyZW5jZSBtYW51 YWwsIExETSBpcwo+IHBlcm1pdHRlZCBvbiBkZXZpY2UgYW5kIHN0cm9uZ2x5IG9yZGVyZWQgbWFw cGluZ3MsIGFuZCB0aGUgbWVtb3J5Cj4gc3Vic3lzdGVtIGlzIHJlcXVpcmVkIHRvIGRlY29tcG9z ZSBpdCBpbnRvIGEgc2VyaWVzIG9mIDMyLWJpdCBhY2Nlc3Nlcy4KPiBTbywgaXQgc291bmRzIHRv IG1lIGxpa2UgdGhlcmUgY291bGQgYmUgYSBoYXJkd2FyZSBidWcgaW4gdGhlIGJ1c2VzL0lQVQo+ IGNhdXNpbmcgdGhpcy4KCkl0IHNlZW1zIHNvLgoKSSBtb2RpZmllZCB0aGUga2VybmVsIElQVSBt b2R1bGUgYSBiaXQsIGluaXRpYWxpemVkIGEgYnVuY2ggb2YgSVBVCnJlZ2lzdGVycyB0byBrbm93 biB2YWx1ZXMgKDEuLjB4RCkuIFJlc3VsdHMgKGZyb20gMSB0byAxMyBJUFUKcmVnaXN0ZXJzKSBv YnRhaW5lZCB3aXRoIGRpZmZlcmVudCBpbnN0cnVjdGlvbnM6CgpyZWFkbCgxMyBjb25zZWN1dGl2 ZSByZWdpc3RlcnMpOiBDU0kgPSAxIDIgMyA0IDUgNiA3IDggOSBBIEIgQyBECjEgPSByZWdpc3Rl ciAjMCBhbmQgc28gb24gLSByZWFkbCgpIHJlc3VsdHMgYXJlIG9idmlvdXNseSBjb3JyZWN0LgoK TERNMTogIDEgKG5vdCBjb3JydXB0ZWQpCkxETTI6ICAxIDMKTERNMzogIDEgMyA0CkxETTQ6ICAy IDMgNCA0CkxETTU6ICAxIDMgNCA1IDYKTERNNjogIDEgMyA0IDUgNiA3CkxETTc6ICAxIDMgNCA1 IDYgNyA4CkxETTg6ICAyIDMgNCA1IDYgNyA4IDgKTERNOTogIDEgMyA0IDUgNiA3IDggOSBBCkxE TTEwOiAxIDMgNCA1IDYgNyA4IDkgQSBCCkxETTExOiAxIDMgNCA1IDYgNyA4IDkgQSBCIEMKTERN MTI6IDEgMyA0IDUgNiA3IDggOSBBIEIgQyBECgpUaGUgbGFzdCBvbmUgdXNlczoKICAgICAgICBs ZG0gcjQsIHtyMSwgcjIsIHIzLCByNCwgcjUsIHI2LCByNywgcjgsIHI5LCBzbCwgZnAsIGlwfS4K CkkgaGF2ZW4ndCB0ZXN0ZWQgbW9yZSB0aGFuIDEyIHJlZ2lzdGVycyBpbiBvbmUga2VybmVsIExE TUlBIGluc3RydWN0aW9uLgoKVGhlIHJlc3VsdHMgZG9uJ3QgZGVwZW5kIG9uIHRoZSBhZGRyZXNz IG9mZnNldCAoYWRkaW5nIDQsIDggb3IgMTIgdG8gdGhlCmFkZHJlc3MgZG9lc24ndCBjaGFuZ2Ug YW55dGhpbmcpLgoKVGhlIGFybV9jb3B5X2Zyb21fdXNlcigpIGlzIGEgc3BlY2lmaWMgY2FzZSBv ZiB0aGUgc2FtZSBjb3JydXB0aW9uLiBJdAp1c2VzIGEgbnVtYmVyIG9mIFBMRHMgYW5kIDgtcmVn aXN0ZXIgTERNSUFzIChhbmQgdGhlbiBwb3NzaWJseSBMRFJzCndoaWNoIGRvbid0IGZhaWwpLiBF YWNoIExETUlBICgiTERNOCIpIHJldHVybnMgYWdhaW46CkxNRDg6ICAyIDMgNCA1IDYgNyA4IDgK KHRoZSBzYW1lIHdpdGggc3Vic2VxdWVudCBMRE1JQXM6IDEwIDExIDEyIDEzIDE0IDE1IDE2IDE2 IGFuZCBzbyBvbikuCgpTdW1tYXJ5OiBpdCBhcHBlYXJzIGFsbCA2NC1iaXQgYW5kIGxvbmdlciBM RE1JQSBpbnN0cnVjdGlvbnMgZmFpbC4gVGhlCmZpcnN0IG9yIHRoZSBzZWNvbmQgMzItYml0IGFj Y2VzcyBpcyBza2lwcGVkIChwb3NzaWJseSBzb21ld2hlcmUgYmV0d2VlbgpBWEkgYW5kIElQVSku IEluIGNhc2Ugb2YgNC0gYW5kIDgtcmVnaXN0ZXIgTERNcywgdGhlIGZpcnN0ICgjMCkgdmFsdWUg aXMKc2tpcHBlZCwgb3RoZXJ3aXNlLCBpdCdzIHRoZSBzZWNvbmQgKCMxKSB2YWx1ZS4KCgpOb3cg dGhlIFBMRHMgcmluZyBhIGJlbGw6CiJFUlIwMDM3MzAgQVJNOiA3NDM2MjPigJRCYWQgaW50ZXJh Y3Rpb24gYmV0d2VlbiBhIG1pbmltdW0gb2Ygc2V2ZW4gUExEcwphbmQgb25lIE5vbi1DYWNoZWFi bGUgTERNIGNhbiBsZWFkIHRvIGEgZGVhZGxvY2siLiBMb29raW5nIGF0IHRoZQpkaXNhc3NlbWJs eSBJIGNhbiBjb3VudCA2IFBMRHMgKHRoZSBmaXJzdCB0d28gc2VlbSB0byBiZSB0aGUgc2FtZSwK dGhvdWdoIEkgZG9uJ3QgY2xhaW0gSSB1bmRlcnN0YW5kIHRoaXMgKHNvdXJjZSkgLnMgY29kZSku IEFsc28gdGhpcwpwcm9ibGVtIGhhcHBlbnMgd2l0aCBJUFUgYW5kIG5vdCBvdGhlciBkZXZpY2Vz LCBzbyBJIHRoaW5rIGl0J3Mgbm90CnJlbGF0ZWQgdG8gdGhpcyBlcnJhdHVtIGFmdGVyIGFsbC4K CgpzaXplX3QgYXJtX2NvcHlfZnJvbV91c2VyKHZvaWQgKnRvLCBjb25zdCB2b2lkICpmcm9tLCBz aXplX3QgbikKLi4uIGZvciBuID0gMzIgPSA4ICogNCBieXRlczoKMmM6IHN1YnMgcjIsIHIyLCAj NCAgICAgOyA9IDI4CjMwOiBibHQgIGU0ICAgICAgICAgICAgIDsgTk9QCjM0OiBhbmRzIGlwLCBy MCwgIzMgICAgIDsgcjAgPSBkZXN0aW5hdGlvbgozODogcGxkICBbcjFdCjNjOiBibmUgIDEwOCAg ICAgICAgICAgIDsgTk9QCjQwOiBhbmRzIGlwLCByMSwgIzMgICAgIDsgcjEgPSBhZGRyZXNzIGlu IElQVQo0NDogYm5lICAxMzggICAgICAgICAgICA7IE5PUAo0ODogc3VicyByMiwgcjIsICMyOAo0 YzogcHVzaCB7cjUsIHI2LCByNywgcjh9CjUwOiBibHQgIDg4ICAgICAgICAgICAgIDsgTk9QCjU0 OiBwbGQgIFtyMV0gICAgICAgICAgIDsgZHVwbGljYXRlIFBMRD8KNTg6IHN1YnMgcjIsIHIyLCAj MHg2MAo1YzogcGxkICBbcjEsICMyOF0KNjA6IGJsdCAgNzAKNjQ6IHBsZCAgW3IxLCAjMHgzY10K Njg6IHBsZCAgW3IxLCAjMHg1Y10KNmM6IHBsZCAgW3IxLCAjMHg3Y10KNzA6IGxkbSAgcjEhLCB7 cjMsIHI0LCByNSwgcjYsIHI3LCByOCwgaXAsIGxyfSA7IDw8PDw8IGZhaWxzCgpJIGFsc28gd29u ZGVyIGlmIFNUTXMgbWF5IGhhdmUgc2ltaWxhciBwcm9ibGVtcyAtIHdpbGwgY2hlY2suCi0tIApL cnp5c3p0b2YgSGHFgmFzYQoKU2llxIcgQmFkYXdjemEgxYF1a2FzaWV3aWN6ClByemVteXPFgm93 eSBJbnN0eXR1dCBBdXRvbWF0eWtpIGkgUG9taWFyw7N3IFBJQVAKQWwuIEplcm96b2xpbXNraWUg MjAyLCAwMi00ODYgV2Fyc3phd2EKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJu ZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFu L2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK