From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BAEDC4338F for ; Wed, 25 Aug 2021 10:59:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7CE961212 for ; Wed, 25 Aug 2021 10:59:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C7CE961212 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 25C6A900002; Wed, 25 Aug 2021 06:59:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 20A726B0071; Wed, 25 Aug 2021 06:59:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F990900002; Wed, 25 Aug 2021 06:59:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id EA4BB6B006C for ; Wed, 25 Aug 2021 06:58:59 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9E62F250CD for ; Wed, 25 Aug 2021 10:58:59 +0000 (UTC) X-FDA: 78513305598.40.5ACD644 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf16.hostedemail.com (Postfix) with ESMTP id 2C894F000099 for ; Wed, 25 Aug 2021 10:58:59 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8415C31B; Wed, 25 Aug 2021 03:58:58 -0700 (PDT) Received: from [10.57.15.112] (unknown [10.57.15.112]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D502F3F766; Wed, 25 Aug 2021 03:58:56 -0700 (PDT) Subject: Re: [BUG 5.14] arm64/mm: dma memory mapping fails (in some cases) To: David Hildenbrand , Catalin Marinas Cc: Mike Rapoport , Alex Bee , Will Deacon , Andrew Morton , Anshuman Khandual , Linux Kernel Mailing List , linux-mm@kvack.org, Linux ARM , Christoph Hellwig References: <20210824173741.GC623@arm.com> <0908ce39-7e30-91fa-68ef-11620f9596ae@arm.com> <60a11eba-2910-3b5f-ef96-97d4556c1596@redhat.com> <20210825102044.GA3420@arm.com> From: Robin Murphy Message-ID: Date: Wed, 25 Aug 2021 11:58:51 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB X-Rspamd-Queue-Id: 2C894F000099 Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf16.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com X-Rspamd-Server: rspam01 X-Stat-Signature: rr6dy6ghkhordftk45cxp9sq9pb15abm X-HE-Tag: 1629889139-193625 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2021-08-25 11:38, David Hildenbrand wrote: > On 25.08.21 12:20, Catalin Marinas wrote: >> + hch >> >> On Tue, Aug 24, 2021 at 08:59:22PM +0200, David Hildenbrand wrote: >>> On 24.08.21 20:46, Robin Murphy wrote: >>>> On 2021-08-24 19:28, Mike Rapoport wrote: >>>>> On Tue, Aug 24, 2021 at 06:37:41PM +0100, Catalin Marinas wrote: >>>>>> On Tue, Aug 24, 2021 at 03:40:47PM +0200, Alex Bee wrote: >>>>>>> it seems there is a regression in arm64 memory mapping in 5.14,=20 >>>>>>> since it >>>>>>> fails on Rockchip RK3328 when the pl330 dmac tries to map with: >>>>>>> >>>>>>> =C2=A0 ------------[ cut here ]------------ >>>>>>> =C2=A0 WARNING: CPU: 2 PID: 373 at kernel/dma/mapping.c:235=20 >>>>>>> dma_map_resource+0x68/0xc0 >>>>>>> =C2=A0 Modules linked in: spi_rockchip(+) fuse >>>>>>> =C2=A0 CPU: 2 PID: 373 Comm: systemd-udevd Not tainted 5.14.0-rc7= #1 >>>>>>> =C2=A0 Hardware name: Pine64 Rock64 (DT) >>>>>>> =C2=A0 pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=3D--) >>>>>>> =C2=A0 pc : dma_map_resource+0x68/0xc0 >>>>>>> =C2=A0 lr : pl330_prep_slave_fifo+0x78/0xd0 >>>>>>> =C2=A0 sp : ffff800012102ae0 >>>>>>> =C2=A0 x29: ffff800012102ae0 x28: ffff000005c94800 x27: 000000000= 0000000 >>>>>>> =C2=A0 x26: ffff000000566bd0 x25: 0000000000000001 x24: 000000000= 0000001 >>>>>>> =C2=A0 x23: 0000000000000002 x22: ffff000000628c00 x21: 000000000= 0000001 >>>>>>> =C2=A0 x20: ffff000000566bd0 x19: 0000000000000001 x18: 000000000= 0000000 >>>>>>> =C2=A0 x17: 0000000000000000 x16: 0000000000000000 x15: 000000000= 0000000 >>>>>>> =C2=A0 x14: 0000000000000277 x13: 0000000000000001 x12: 000000000= 0000000 >>>>>>> =C2=A0 x11: 0000000000000001 x10: 00000000000008e0 x9 : ffff80001= 2102a80 >>>>>>> =C2=A0 x8 : ffff000000d14b80 x7 : ffff0000fe7b12f0 x6 : ffff0000f= e7b1100 >>>>>>> =C2=A0 x5 : fffffc000000000f x4 : 0000000000000000 x3 : 000000000= 0000001 >>>>>>> =C2=A0 x2 : 0000000000000001 x1 : 00000000ff190800 x0 : ffff00000= 0628c00 >>>>>>> =C2=A0 Call trace: >>>>>>> =C2=A0=C2=A0=C2=A0 dma_map_resource+0x68/0xc0 >>>>>>> =C2=A0=C2=A0=C2=A0 pl330_prep_slave_sg+0x58/0x220 >>>>>>> =C2=A0=C2=A0=C2=A0 rockchip_spi_prepare_dma+0xd8/0x2c0 [spi_rockc= hip] >>>>>>> =C2=A0=C2=A0=C2=A0 rockchip_spi_transfer_one+0x294/0x3d8 [spi_roc= kchip] >>>>>> [...] >>>>>>> Note: This does not relate to the spi driver - when disabling=20 >>>>>>> this device in >>>>>>> the device tree it fails for any other (i2s, for instance) which=20 >>>>>>> uses dma. >>>>>>> Commenting out the failing check at [1], however, helps and the=20 >>>>>>> mapping >>>>>>> works again. >>>>> >>>>>> Do you know which address dma_map_resource() is trying to map (may= be >>>>>> add some printk())? It's not supposed to map RAM, hence the warnin= g. >>>>>> Random guess, the address is 0xff190800 (based on the x1 above but= =20 >>>>>> the >>>>>> regs might as well be mangled). >>>>> >>>>> 0xff190800 will cause this warning for sure. It has a memory map,=20 >>>>> but it is >>>>> not RAM so old version of pfn_valid() would return 0 and the new on= e >>>>> returns 1. >>>> >>>> How does that happen, though? It's not a memory address, and it's no= t >>>> even within the bounds of anywhere there should or could be memory.=20 >>>> This >>>> SoC has a simple memory map - everything from 0 to 0xfeffffff goes t= o >>>> the DRAM controller (which may not all be populated, and may have=20 >>>> pieces >>>> carved out by secure firmware), while 0xff000000-0xffffffff is MMIO.= =20 >>>> Why >>>> do we have pages (or at least the assumption of pages) for somewhere >>>> which by all rights should not have them? >>> >>> Simple: we allocate the vmemmap for whole sections (e.g., 128 MiB) to= =20 >>> avoid >>> any such hacks. If there is a memory hole, it gets a memmap as well. Urgh, apologies for being slow. This case isn't a memory hole as such,=20 but I failed to consider the *ends* of memory not being section-aligned=20 and leading to an overlap anyway. >>> Tricking pfn_valid() into returning "false" where we actually have a=20 >>> memmap >>> only makes it look like there is no memmap; but there is one, and >>> it's PG_reserved. >> >> I can see the documentation for pfn_valid() does not claim anything mo= re >> than the presence of an memmap entry. But I wonder whether the confusi= on >> is wider-spread than just the DMA code. At a quick grep, try_ram_remap= () >> assumes __va() can be used on pfn_valid(), though I suspect it relies = on >> the calling function to check that the resource was RAM. The arm64 >> kern_addr_valid() returns true based on pfn_valid() and kcore.c uses >> standard memcpy on it, which wouldn't work for I/O (should we change >> this check to pfn_is_map_memory() for arm64?). >> >=20 > kern_addr_valid() checks that there is a direct map entry, and that the= =20 > mapped address has a valid mmap. (copied from x86-64) >=20 > Would you expect to have a direct map for memory holes and similar (IOW= ,=20 > !System RAM)? Probably - we can have no-map regions for firmware-reserved RAM which I=20 believe end up as PG_reserved if they're small enough, for the same=20 reasoning as this case. >>>>>> Either pfn_valid() gets confused in 5.14 or something is wrong=20 >>>>>> with the >>>>>> DT. I have a suspicion it's the former since reverting the above=20 >>>>>> commit >>>>>> makes it disappear. >>>>> >>>>> I think pfn_valid() actually behaves as expected but the caller is=20 >>>>> wrong >>>>> because pfn_valid !=3D RAM (this applies btw to !arm64 as well). >>>>> >>>>> =C2=A0=C2=A0=C2=A0=C2=A0/* Don't allow RAM to be mapped */ >>>>> =C2=A0=C2=A0=C2=A0=C2=A0if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_ad= dr)))) >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return DMA_MAPPING_ERROR= ; >>>>> >>>>> Alex, can you please try this patch: >>>> >>>> That will certainly paper over the issue, but it's avoiding the=20 >>>> question >>>> of what went wrong with the memory map in the first place. The comme= nt >>>> is indeed a bit inaccurate, but ultimately dma_map_resource() exists= =20 >>>> for >>>> addresses that would be wrong to pass to dma_map_page(), so I believ= e >>>> pfn_valid() is still the correct check. >>> >>> If we want to check for RAM, pfn_valid() would be wrong. If we want=20 >>> to check >>> for "is there a memmap, for whatever lives or does not live there", >>> pfn_valid() is the right check. >> >> So what should the DMA code use instead? Last time we needed something >> similar, the recommendation was to use pfn_to_online_page(). Mike is >> suggesting memblock_is_memory(). >=20 > We use pfn_to_online_page() when we want to know if it's system RAM and= =20 > that the memmap actually contains something sane (-> memmap content has= =20 > a well defined state). >=20 > You can have offline memory blocks where pfn_to_online_page() would=20 > return "false", memblock_is_memory() would return "true". IOW, there is= =20 > a memmap, it's System RAM, but the memmap is stale and not trustworthy. That's fine - offline memory is doubly-invalid to map as an MMIO resource= :) > If you want to make sure no System RAM (online/offline/...) will get=20 > mapped, memblock_is_memory() should be the right thing to use. I recall= =20 > that x86 traverse the resource tree instead to exclude system ram=20 > regions similarly. I'm thinking that "pfn_valid(pfn) && !PageReserved(pfn_to_page(pfn))"=20 might be the closest thing to what I'd like to express - does that seem=20 sensible at all? The one thing I'm not quite sure about is the=20 interaction with P2P mappings of ZONE_DEVICE, but that's also true of=20 the previous behaviour, and I'm not aware that the usage model has been=20 fully nailed down yet, so I suspect we have some wiggle room. At worst,=20 false negatives in certain situations wouldn't be the end of the world,=20 since this is merely a sanity check for something which is expected to=20 be a no-op the vast majority of the time, so being unobtrusive is more=20 important than being exhaustive. Thanks, Robin. >> Given how later we are in the -rc cycle, I suggest we revert Anshuman'= s >> commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID") and try to >> assess the implications in 5.15 (the patch doesn't seem to have the >> arm64 maintainers' ack anyway ;)). >=20