From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01ED5C4320A for ; Tue, 24 Aug 2021 18:06:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E1DB611C8 for ; Tue, 24 Aug 2021 18:06:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8E1DB611C8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 324796B0071; Tue, 24 Aug 2021 14:06:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D48A8D0001; Tue, 24 Aug 2021 14:06:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19C736B0073; Tue, 24 Aug 2021 14:06:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0174.hostedemail.com [216.40.44.174]) by kanga.kvack.org (Postfix) with ESMTP id E5DD26B0071 for ; Tue, 24 Aug 2021 14:06:10 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8809C181AEF00 for ; Tue, 24 Aug 2021 18:06:10 +0000 (UTC) X-FDA: 78510753300.25.C9ADF26 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id E33FA300010D for ; Tue, 24 Aug 2021 18:06:09 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EEE71D6E; Tue, 24 Aug 2021 11:06:08 -0700 (PDT) Received: from [10.57.15.112] (unknown [10.57.15.112]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8D5BB3F766; Tue, 24 Aug 2021 11:06:07 -0700 (PDT) Subject: Re: [BUG 5.14] arm64/mm: dma memory mapping fails (in some cases) To: Catalin Marinas , Alex Bee Cc: Will Deacon , Andrew Morton , Anshuman Khandual , Linux Kernel Mailing List , linux-mm@kvack.org, Linux ARM , Mike Rapoport References: <20210824173741.GC623@arm.com> From: Robin Murphy Message-ID: <77eb6abd-4369-eb8f-e323-cf4e6f2ffce5@arm.com> Date: Tue, 24 Aug 2021 19:06:01 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <20210824173741.GC623@arm.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E33FA300010D X-Stat-Signature: na7519dct6targ1oz1k1afsszffojajk X-HE-Tag: 1629828369-172898 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2021-08-24 18:37, Catalin Marinas wrote: > Hi Alex, >=20 > Thanks for the report. >=20 > On Tue, Aug 24, 2021 at 03:40:47PM +0200, Alex Bee wrote: >> it seems there is a regression in arm64 memory mapping in 5.14, since = it >> fails on Rockchip RK3328 when the pl330 dmac tries to map with: >> >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.921909] ----= --------[ cut here ]------------ >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.921940] WARN= ING: CPU: 2 PID: 373 at kernel/dma/mapping.c:235 dma_map_resource+0x68/0x= c0 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.921973] Modu= les linked in: spi_rockchip(+) fuse >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.921996] CPU:= 2 PID: 373 Comm: systemd-udevd Not tainted 5.14.0-rc7 #1 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922004] Hard= ware name: Pine64 Rock64 (DT) >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922011] psta= te: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=3D--) >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922018] pc := dma_map_resource+0x68/0xc0 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922026] lr := pl330_prep_slave_fifo+0x78/0xd0 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922040] sp := ffff800012102ae0 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922043] x29:= ffff800012102ae0 x28: ffff000005c94800 x27: 0000000000000000 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922056] x26:= ffff000000566bd0 x25: 0000000000000001 x24: 0000000000000001 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922067] x23:= 0000000000000002 x22: ffff000000628c00 x21: 0000000000000001 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922078] x20:= ffff000000566bd0 x19: 0000000000000001 x18: 0000000000000000 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922089] x17:= 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922100] x14:= 0000000000000277 x13: 0000000000000001 x12: 0000000000000000 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922111] x11:= 0000000000000001 x10: 00000000000008e0 x9 : ffff800012102a80 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922123] x8 := ffff000000d14b80 x7 : ffff0000fe7b12f0 x6 : ffff0000fe7b1100 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922134] x5 := fffffc000000000f x4 : 0000000000000000 x3 : 0000000000000001 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922145] x2 := 0000000000000001 x1 : 00000000ff190800 x0 : ffff000000628c00 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922158] Call= trace: >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922163]=C3=AF= =C2=BF=C2=BD dma_map_resource+0x68/0xc0 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922173]=C3=AF= =C2=BF=C2=BD pl330_prep_slave_sg+0x58/0x220 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922181]=C3=AF= =C2=BF=C2=BD rockchip_spi_prepare_dma+0xd8/0x2c0 [spi_rockchip] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 8.922208]=C3=AF= =C2=BF=C2=BD rockchip_spi_transfer_one+0x294/0x3d8 [spi_rockchip] > [...] >> Note: This does not relate to the spi driver - when disabling this dev= ice in >> the device tree it fails for any other (i2s, for instance) which uses = dma. >> Commenting out the failing check at [1], however, helps and the mappin= g >> works again. >=20 > Do you know which address dma_map_resource() is trying to map (maybe > add some printk())? It's not supposed to map RAM, hence the warning. > Random guess, the address is 0xff190800 (based on the x1 above but the > regs might as well be mangled). Yup, that fits the signature of dma_map_resource(), and would indeed be=20 right in the middle of the SPI peripheral on RK3328. FWIW the comment about RAM there is a little inaccurate, but the point=20 remains that anything which *is* backed by a page should probably be=20 handled by dma_map_page() if at all. >> I tried to follow the recent changes for arm64 mm which could relate t= o the >> check failing at [1] and reverting >> =C3=AF=C2=BF=C2=BD commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_= VALID") >> helps and makes it work again, but I'm 100% uncertain if that commit i= s >> really the culprit. >> >> Note, that the firmware (legacy u-boot) injects memory configuration i= n the >> device tree as follows: >> >> /memreserve/=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0x0= 0000000fcefc000 0x000000000000d000; >> / { >> .. >> =C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD compatible =3D = "pine64,rock64\0rockchip,rk3328"; >> .. >> =C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD memory { >> =C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD =C3=AF=C2=BF=C2= =BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD reg =3D <0x00 0x200000 0x00 0xfee= 00000 0x00 0x00 0x00 0x00>; >> =C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD =C3=AF=C2=BF=C2= =BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD device_type =3D "memory"; >> =C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD }; >> >> .. >> } >=20 > Either pfn_valid() gets confused in 5.14 or something is wrong with the > DT. I have a suspicion it's the former since reverting the above commit > makes it disappear. >=20 >> So: there is a "hole" in the mappable memory and reading the commit me= ssage >> of >> =C3=AF=C2=BF=C2=BD commit a7d9f306ba70 ("arm64: drop pfn_valid_within(= ) and simplify >> pfn_valid()") >> suggests, there was a change for that case recently. >=20 > I think the change from the arm64 pfn_valid() to the generic one is > avoiding the call to memblock_is_memory(). I wonder whether pfn_valid() > returns true just because we have a struct page available but the memor= y > may have been reserved. Either way I think something's gone pretty badly wrong if Linux ends up=20 thinking that an MMIO region beyond the bounds of any possible RAM=20 should be page-backed :/ Robin. >=20 > Cc'ing Mike R. >=20 >> I also noticed there is a diff in the kernel log regarding memory init= up >> until 5.13.12 it says >> >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Zone= ranges: >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3= =AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD [mem 0x00000000002000= 00-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA32=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3= =AF=C2=BF=C2=BD empty >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD Normal=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD= empty >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Mova= ble zone start for each node >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Earl= y memory node ranges >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD node=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0= : [mem 0x0000000000200000-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Init= mem setup node 0 [mem 0x0000000000200000-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] On n= ode 0 totalpages: 1043968 >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA zone: 16312 pages used for memmap >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA zone: 0 pages reserved >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA zone: 1043968 pages, LIFO batch:63 >> >> In contrary in 5.14-rc7 it says: >> >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Zone= ranges: >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3= =AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD [mem 0x00000000002000= 00-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA32=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3= =AF=C2=BF=C2=BD empty >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD Normal=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD= empty >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Mova= ble zone start for each node >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Earl= y memory node ranges >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD node=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0= : [mem 0x0000000000200000-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Init= mem setup node 0 [mem 0x0000000000200000-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] On n= ode 0, zone DMA: 512 pages in unavailable ranges >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] On n= ode 0, zone DMA: 4096 pages in unavailable ranges >> >> (note the "unavailable ranges") >> I'm uncertain again here, if that diff is expected behavior because of= those >> recent mm changes for arm64. >> >> After reverting >> =C3=AF=C2=BF=C2=BD commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_= VALID") >> the log changes to >> >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Zone= ranges: >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3= =AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD [mem 0x00000000002000= 00-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD DMA32=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3= =AF=C2=BF=C2=BD empty >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD Normal=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD= empty >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Mova= ble zone start for each node >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Earl= y memory node ranges >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000]=C3=AF= =C2=BF=C2=BD=C3=AF=C2=BF=C2=BD node=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0= : [mem 0x0000000000200000-0x00000000feffffff] >> [=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD=C3=AF=C2=BF=C2=BD 0.000000] Init= mem setup node 0 [mem >> 0x0000000000200000-0x00000000feffffff] >> >> (no DMA zones here) >> >> As you might have noticed I have _zero_ clue about memory mapping and = dma >> subsystem - so let me know if there is any more information needed for= that >> and thanks for your help. >=20 > Adding Robin as well, he has a better clue than us on DMA ;). >=20 >> Alex >> >> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git= /tree/kernel/dma/mapping.c?id=3De22ce8eb631bdc47a4a4ea7ecf4e4ba499db4f93#= n235 >=20