From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84E92C4338F for ; Tue, 24 Aug 2021 20:07:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0FD3C6135F for ; Tue, 24 Aug 2021 20:07:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0FD3C6135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 813326B006C; Tue, 24 Aug 2021 16:07:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C43A6B0071; Tue, 24 Aug 2021 16:07:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68B486B0072; Tue, 24 Aug 2021 16:07:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id 4AD966B006C for ; Tue, 24 Aug 2021 16:07:18 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BA68F211DC for ; Tue, 24 Aug 2021 20:07:17 +0000 (UTC) X-FDA: 78511058514.39.812CA42 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by imf26.hostedemail.com (Postfix) with ESMTP id 7861520019E2 for ; Tue, 24 Aug 2021 20:07:17 +0000 (UTC) Received: by mail-wm1-f52.google.com with SMTP id c8-20020a7bc008000000b002e6e462e95fso2690839wmb.2 for ; Tue, 24 Aug 2021 13:07:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=6x3j+OkRkyw9DV1B0EyAjBYSEA+AgCw5VMEZerH9gnk=; b=Cvsad9b3iy4PH2YjdA8xKFIwdztiL250XDd8wKeL25XYR6HdO44T1TXlyZ52MWcXPb LdO0OHh36NfmRmQxNtZmS4nt1SqbTMqQozcl2YxqnJwYqB7hb480YFZx1urY2inIAbpR nsOQiV07Ra3dt0tcRL479zXZZtEKrv57th/GvKa2rquaULzZ1eNgXwnsQ+mkGYrS0PUS KbCoRtSF82JzOX5sNCOkdUzRdYWpuq9S3VKYQ/Pj7Z23Ec5aw6V7ABi+LPQtIAnRfbhW Z8SNvdPqtqb8QBRbuWtjMfCmEhWBr71gZNVu41/Mjp8i8I83P86hzzJZTc3t+2Shv4K+ t28Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=6x3j+OkRkyw9DV1B0EyAjBYSEA+AgCw5VMEZerH9gnk=; b=fLSk2yaNoZAU7xsnoiEc7fnWvJP4u9Ze/S67PFQC112o32W5tagQnrvtNR87aCUEcP qN7jWxPUYvQiPfNI1/hAMLEHflEouzZ/7oTHYqjojXF0bhStI2YKTN8fMpMHatrCSpgx GbnQekf+b6xGpO4pHJGwcPR3WYDDEOqLEkVoxpT2QtCzQwCnkK2OhiPdR5pUb9YFY9xj Tns1iV8DLliOlkajYY+4AIHCP0s9fm1rBu00WL8+SdAfql7S1eeiNKbzIpmu/S16oBtO Z4poHDD7yX2u0r6Bh7QUTgZ/Mgq6GyoIQo7iBNcTUj+Mr9QLpt5F7WuFsIT4KlPqORfK YBwA== X-Gm-Message-State: AOAM531lb5toT1j3nORUNLadnkJYqky2v55VYbVpIrgwWadnIqKzjYQO zVbsFYLkvR/ghP2ZDOYPEw== X-Google-Smtp-Source: ABdhPJwsILYp3MZ7+BibFh27UDBGLt61Duv98lC0rb5IvSdcJpLuFiNyuej4/q2kV4060R4QURr2bg== X-Received: by 2002:a05:600c:898:: with SMTP id l24mr5289108wmp.90.1629835636270; Tue, 24 Aug 2021 13:07:16 -0700 (PDT) Received: from ?IPv6:2a02:810b:f40:4200:65a1:1952:1d23:2e04? ([2a02:810b:f40:4200:65a1:1952:1d23:2e04]) by smtp.gmail.com with ESMTPSA id a18sm3015005wmg.43.2021.08.24.13.07.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 24 Aug 2021 13:07:15 -0700 (PDT) Subject: Re: [BUG 5.14] arm64/mm: dma memory mapping fails (in some cases) To: Catalin Marinas Cc: Will Deacon , Andrew Morton , Anshuman Khandual , Linux Kernel Mailing List , linux-mm@kvack.org, Linux ARM , Mike Rapoport , Robin Murphy References: <20210824173741.GC623@arm.com> From: Alex Bee Message-ID: <72d211b4-fe40-cc88-e831-9257ccad48af@gmail.com> Date: Tue, 24 Aug 2021 22:07:14 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210824173741.GC623@arm.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Rspamd-Queue-Id: 7861520019E2 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=Cvsad9b3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of knaerzche@gmail.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=knaerzche@gmail.com X-Rspamd-Server: rspam01 X-Stat-Signature: sp7se5dx1cgkh8ee1c9bfon7m4nwcupy X-HE-Tag: 1629835637-76024 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Catalin, thanks for your quick reply. Am 24.08.21 um 19:37 schrieb Catalin Marinas: > Hi Alex, > > Thanks for the report. > > On Tue, Aug 24, 2021 at 03:40:47PM +0200, Alex Bee wrote: >> it seems there is a regression in arm64 memory mapping in 5.14, since = it >> fails on Rockchip RK3328 when the pl330 dmac tries to map with: >> >> [=C2=A0=C2=A0=C2=A0 8.921909] ------------[ cut here ]------------ >> [=C2=A0=C2=A0=C2=A0 8.921940] WARNING: CPU: 2 PID: 373 at kernel/dma/m= apping.c:235 dma_map_resource+0x68/0xc0 >> [=C2=A0=C2=A0=C2=A0 8.921973] Modules linked in: spi_rockchip(+) fuse >> [=C2=A0=C2=A0=C2=A0 8.921996] CPU: 2 PID: 373 Comm: systemd-udevd Not = tainted 5.14.0-rc7 #1 >> [=C2=A0=C2=A0=C2=A0 8.922004] Hardware name: Pine64 Rock64 (DT) >> [=C2=A0=C2=A0=C2=A0 8.922011] pstate: 80000005 (Nzcv daif -PAN -UAO -T= CO BTYPE=3D--) >> [=C2=A0=C2=A0=C2=A0 8.922018] pc : dma_map_resource+0x68/0xc0 >> [=C2=A0=C2=A0=C2=A0 8.922026] lr : pl330_prep_slave_fifo+0x78/0xd0 >> [=C2=A0=C2=A0=C2=A0 8.922040] sp : ffff800012102ae0 >> [=C2=A0=C2=A0=C2=A0 8.922043] x29: ffff800012102ae0 x28: ffff000005c94= 800 x27: 0000000000000000 >> [=C2=A0=C2=A0=C2=A0 8.922056] x26: ffff000000566bd0 x25: 0000000000000= 001 x24: 0000000000000001 >> [=C2=A0=C2=A0=C2=A0 8.922067] x23: 0000000000000002 x22: ffff000000628= c00 x21: 0000000000000001 >> [=C2=A0=C2=A0=C2=A0 8.922078] x20: ffff000000566bd0 x19: 0000000000000= 001 x18: 0000000000000000 >> [=C2=A0=C2=A0=C2=A0 8.922089] x17: 0000000000000000 x16: 0000000000000= 000 x15: 0000000000000000 >> [=C2=A0=C2=A0=C2=A0 8.922100] x14: 0000000000000277 x13: 0000000000000= 001 x12: 0000000000000000 >> [=C2=A0=C2=A0=C2=A0 8.922111] x11: 0000000000000001 x10: 0000000000000= 8e0 x9 : ffff800012102a80 >> [=C2=A0=C2=A0=C2=A0 8.922123] x8 : ffff000000d14b80 x7 : ffff0000fe7b1= 2f0 x6 : ffff0000fe7b1100 >> [=C2=A0=C2=A0=C2=A0 8.922134] x5 : fffffc000000000f x4 : 0000000000000= 000 x3 : 0000000000000001 >> [=C2=A0=C2=A0=C2=A0 8.922145] x2 : 0000000000000001 x1 : 00000000ff190= 800 x0 : ffff000000628c00 >> [=C2=A0=C2=A0=C2=A0 8.922158] Call trace: >> [=C2=A0=C2=A0=C2=A0 8.922163]=C2=A0 dma_map_resource+0x68/0xc0 >> [=C2=A0=C2=A0=C2=A0 8.922173]=C2=A0 pl330_prep_slave_sg+0x58/0x220 >> [=C2=A0=C2=A0=C2=A0 8.922181]=C2=A0 rockchip_spi_prepare_dma+0xd8/0x2c= 0 [spi_rockchip] >> [=C2=A0=C2=A0=C2=A0 8.922208]=C2=A0 rockchip_spi_transfer_one+0x294/0x= 3d8 [spi_rockchip] > [...] >> Note: This does not relate to the spi driver - when disabling this dev= ice in >> the device tree it fails for any other (i2s, for instance) which uses = dma. >> Commenting out the failing check at [1], however, helps and the mappin= g >> works again. > Do you know which address dma_map_resource() is trying to map (maybe > add some printk())? It's not supposed to map RAM, hence the warning. > Random guess, the address is 0xff190800 (based on the x1 above but the > regs might as well be mangled). Confirmed: [..] [=C2=A0=C2=A0=C2=A0 8.353879] dma_map_resource Failed to map address 0xff= 190800 [..] Alex > >> I tried to follow the recent changes for arm64 mm which could relate t= o the >> check failing at [1] and reverting >> =C2=A0 commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID") >> helps and makes it work again, but I'm 100% uncertain if that commit i= s >> really the culprit. >> >> Note, that the firmware (legacy u-boot) injects memory configuration i= n the >> device tree as follows: >> >> /memreserve/=C2=A0=C2=A0=C2=A0 0x00000000fcefc000 0x000000000000d000; >> / { >> .. >> =C2=A0=C2=A0=C2=A0 compatible =3D "pine64,rock64\0rockchip,rk3328"; >> .. >> =C2=A0=C2=A0=C2=A0 memory { >> =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 reg =3D <0x00 0x200000 0x00 0xf= ee00000 0x00 0x00 0x00 0x00>; >> =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 device_type =3D "memory"; >> =C2=A0=C2=A0=C2=A0 }; >> >> .. >> } > Either pfn_valid() gets confused in 5.14 or something is wrong with the > DT. I have a suspicion it's the former since reverting the above commit > makes it disappear. > >> So: there is a "hole" in the mappable memory and reading the commit me= ssage >> of >> =C2=A0 commit a7d9f306ba70 ("arm64: drop pfn_valid_within() and simpl= ify >> pfn_valid()") >> suggests, there was a change for that case recently. > I think the change from the arm64 pfn_valid() to the generic one is > avoiding the call to memblock_is_memory(). I wonder whether pfn_valid() > returns true just because we have a struct page available but the memor= y > may have been reserved. > > Cc'ing Mike R. > >> I also noticed there is a diff in the kernel log regarding memory init= up >> until 5.13.12 it says >> >> [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 [mem 0x0000000000200000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 empt= y >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty >> [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node >> [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x0= 000000000200000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem 0x000000000020= 0000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000] On node 0 totalpages: 1043968 >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 16312 pages used f= or memmap >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 0 pages reserved >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 1043968 pages, LIF= O batch:63 >> >> In contrary in 5.14-rc7 it says: >> >> [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 [mem 0x0000000000200000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 empt= y >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty >> [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node >> [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x0= 000000000200000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem 0x000000000020= 0000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000] On node 0, zone DMA: 512 pages in unavai= lable ranges >> [=C2=A0=C2=A0=C2=A0 0.000000] On node 0, zone DMA: 4096 pages in unava= ilable ranges >> >> (note the "unavailable ranges") >> I'm uncertain again here, if that diff is expected behavior because of= those >> recent mm changes for arm64. >> >> After reverting >> =C2=A0 commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID") >> the log changes to >> >> [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 [mem 0x0000000000200000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 empt= y >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty >> [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node >> [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges >> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x0= 000000000200000-0x00000000feffffff] >> [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem >> 0x0000000000200000-0x00000000feffffff] >> >> (no DMA zones here) >> >> As you might have noticed I have _zero_ clue about memory mapping and = dma >> subsystem - so let me know if there is any more information needed for= that >> and thanks for your help. > Adding Robin as well, he has a better clue than us on DMA ;). > >> Alex >> >> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git= /tree/kernel/dma/mapping.c?id=3De22ce8eb631bdc47a4a4ea7ecf4e4ba499db4f93#= n235