From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63872C4338F for ; Tue, 24 Aug 2021 13:40:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EBF086113B for ; Tue, 24 Aug 2021 13:40:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org EBF086113B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5ACA08D0002; Tue, 24 Aug 2021 09:40:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55E9A8D0001; Tue, 24 Aug 2021 09:40:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 425518D0002; Tue, 24 Aug 2021 09:40:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id 2654E8D0001 for ; Tue, 24 Aug 2021 09:40:51 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id ADE531C998 for ; Tue, 24 Aug 2021 13:40:50 +0000 (UTC) X-FDA: 78510084660.27.6804FC5 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by imf29.hostedemail.com (Postfix) with ESMTP id 6F08B9000249 for ; Tue, 24 Aug 2021 13:40:50 +0000 (UTC) Received: by mail-wm1-f50.google.com with SMTP id o39-20020a05600c512700b002e74638b567so2353572wms.2 for ; Tue, 24 Aug 2021 06:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:cc:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=WM83eOC6aODiMJf3m8T4eHvsH6CKQKlde1ED7pGGDB0=; b=SpNyimnl22BG9jTYMQjLk4887eYjBCYvrLwpD/rW43xYP5opD5HLpAXiCz+WL7sNh/ dwK8RND9PDQV06k3mxdbJvxa4d6a7klS3mtlmU3dlcBKiJ/QdrbUwbuGBzU2F6TZNtgN RAwVWvV3+S+QS63WYlW81cmlDaFgHMGis2sE6GVfwGLLDtRtTg7QvsrzzEPDtOz40Ee1 /S/vPxIlY1FkDuiIIs8QjxkGap62oM74bzM0lnsVZbDAlZoHcklUWhU0XvojJjt06svS 02K9D550ikzrptMl3DHn4u/s+yvEV7Un2DoFqRdwtAsZGNv2InrWoYRdeapK3LWkQ4Oy MBtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:cc:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=WM83eOC6aODiMJf3m8T4eHvsH6CKQKlde1ED7pGGDB0=; b=fRhpptHReWKxYk/ZjaL2dFN+GJ7Af4NLLqXcMq5wR6Is+TyOwNFg/KYVX4/yeTVxNG 8ruBfx2WyZcNZpOLEXqRGeIjl+P159gW0N9cj+WQYmhL2+3HZllqx8M3aPLKLXlvsyoT VjyUzyi1UrZZMyWAeTwMiU1XSm56bSUgiIvCMX+YCuV0nuoPfxEfTQLsgb+Dn4CFWrom 7BLixlejH8bFrwHdfxWHEFnbtQBz+m7I4HpbKcoy6+DtBQ1wdcxtPo+do1ZH1sjahWqc pkRuiCE6IuqfwIzNBQSNgJMeduvl5ZW4J0uKQWb1ju2rulrdiTGUxqb4Ks6nGHsy1NZC hzsA== X-Gm-Message-State: AOAM531fDSTqxEjLHCvx+D1tipVS02+PiR9nUk0B2VJjV+/B6GCcqzHZ l0JHHaHJypetYZYhBRmIyQ== X-Google-Smtp-Source: ABdhPJyIDcRyaK/nn0CULlwObzImyM8VRidpal6jvcOzm8xLT3+IVW5K7rvdul2RT37hmhphMATr+Q== X-Received: by 2002:a1c:1904:: with SMTP id 4mr4145985wmz.93.1629812449159; Tue, 24 Aug 2021 06:40:49 -0700 (PDT) Received: from [192.168.200.23] (ip5b434083.dynamic.kabel-deutschland.de. [91.67.64.131]) by smtp.gmail.com with ESMTPSA id z126sm2660793wmc.11.2021.08.24.06.40.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 24 Aug 2021 06:40:48 -0700 (PDT) To: Catalin Marinas , Will Deacon , Andrew Morton , Anshuman Khandual , Andrew Morton Cc: Linux Kernel Mailing List , linux-mm@kvack.org, Linux ARM From: Alex Bee Subject: [BUG 5.14] arm64/mm: dma memory mapping fails (in some cases) Message-ID: Date: Tue, 24 Aug 2021 15:40:47 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=SpNyimnl; spf=pass (imf29.hostedemail.com: domain of knaerzche@gmail.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=knaerzche@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: f7jjyq1szib654kfk779trccrxe1bxzr X-Rspamd-Queue-Id: 6F08B9000249 X-Rspamd-Server: rspam04 X-HE-Tag: 1629812450-222935 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi list(s), it seems there is a regression in arm64 memory mapping in 5.14, since it=20 fails on Rockchip RK3328 when the pl330 dmac tries to map with: [=C2=A0=C2=A0=C2=A0 8.921909] ------------[ cut here ]------------ [=C2=A0=C2=A0=C2=A0 8.921940] WARNING: CPU: 2 PID: 373 at kernel/dma/mapp= ing.c:235=20 dma_map_resource+0x68/0xc0 [=C2=A0=C2=A0=C2=A0 8.921973] Modules linked in: spi_rockchip(+) fuse [=C2=A0=C2=A0=C2=A0 8.921996] CPU: 2 PID: 373 Comm: systemd-udevd Not tai= nted=20 5.14.0-rc7 #1 [=C2=A0=C2=A0=C2=A0 8.922004] Hardware name: Pine64 Rock64 (DT) [=C2=A0=C2=A0=C2=A0 8.922011] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO = BTYPE=3D--) [=C2=A0=C2=A0=C2=A0 8.922018] pc : dma_map_resource+0x68/0xc0 [=C2=A0=C2=A0=C2=A0 8.922026] lr : pl330_prep_slave_fifo+0x78/0xd0 [=C2=A0=C2=A0=C2=A0 8.922040] sp : ffff800012102ae0 [=C2=A0=C2=A0=C2=A0 8.922043] x29: ffff800012102ae0 x28: ffff000005c94800= x27:=20 0000000000000000 [=C2=A0=C2=A0=C2=A0 8.922056] x26: ffff000000566bd0 x25: 0000000000000001= x24:=20 0000000000000001 [=C2=A0=C2=A0=C2=A0 8.922067] x23: 0000000000000002 x22: ffff000000628c00= x21:=20 0000000000000001 [=C2=A0=C2=A0=C2=A0 8.922078] x20: ffff000000566bd0 x19: 0000000000000001= x18:=20 0000000000000000 [=C2=A0=C2=A0=C2=A0 8.922089] x17: 0000000000000000 x16: 0000000000000000= x15:=20 0000000000000000 [=C2=A0=C2=A0=C2=A0 8.922100] x14: 0000000000000277 x13: 0000000000000001= x12:=20 0000000000000000 [=C2=A0=C2=A0=C2=A0 8.922111] x11: 0000000000000001 x10: 00000000000008e0= x9 :=20 ffff800012102a80 [=C2=A0=C2=A0=C2=A0 8.922123] x8 : ffff000000d14b80 x7 : ffff0000fe7b12f0= x6 :=20 ffff0000fe7b1100 [=C2=A0=C2=A0=C2=A0 8.922134] x5 : fffffc000000000f x4 : 0000000000000000= x3 :=20 0000000000000001 [=C2=A0=C2=A0=C2=A0 8.922145] x2 : 0000000000000001 x1 : 00000000ff190800= x0 :=20 ffff000000628c00 [=C2=A0=C2=A0=C2=A0 8.922158] Call trace: [=C2=A0=C2=A0=C2=A0 8.922163]=C2=A0 dma_map_resource+0x68/0xc0 [=C2=A0=C2=A0=C2=A0 8.922173]=C2=A0 pl330_prep_slave_sg+0x58/0x220 [=C2=A0=C2=A0=C2=A0 8.922181]=C2=A0 rockchip_spi_prepare_dma+0xd8/0x2c0 [= spi_rockchip] [=C2=A0=C2=A0=C2=A0 8.922208]=C2=A0 rockchip_spi_transfer_one+0x294/0x3d8= [spi_rockchip] [=C2=A0=C2=A0=C2=A0 8.922220]=C2=A0 spi_transfer_one_message+0x284/0x57c [=C2=A0=C2=A0=C2=A0 8.922232]=C2=A0 __spi_pump_messages+0x3dc/0x650 [=C2=A0=C2=A0=C2=A0 8.922240]=C2=A0 __spi_sync+0x3e4/0x48c [=C2=A0=C2=A0=C2=A0 8.922247]=C2=A0 spi_sync+0x30/0x54 [=C2=A0=C2=A0=C2=A0 8.922253]=C2=A0 spi_mem_exec_op+0x264/0x444 [=C2=A0=C2=A0=C2=A0 8.922260]=C2=A0 spi_nor_spimem_read_data+0x148/0x160 [=C2=A0=C2=A0=C2=A0 8.922269]=C2=A0 spi_nor_read_data+0x30/0x40 [=C2=A0=C2=A0=C2=A0 8.922276]=C2=A0 spi_nor_read_sfdp+0x74/0xe4 [=C2=A0=C2=A0=C2=A0 8.922285]=C2=A0 spi_nor_parse_sfdp+0x1d0/0x1130 [=C2=A0=C2=A0=C2=A0 8.922293]=C2=A0 spi_nor_sfdp_init_params+0x3c/0x90 [=C2=A0=C2=A0=C2=A0 8.922304]=C2=A0 spi_nor_scan+0x7b4/0xacc [=C2=A0=C2=A0=C2=A0 8.922311]=C2=A0 spi_nor_probe+0x94/0x2d0 [=C2=A0=C2=A0=C2=A0 8.922317]=C2=A0 spi_mem_probe+0x6c/0xb0 [=C2=A0=C2=A0=C2=A0 8.922325]=C2=A0 spi_probe+0x84/0xe4 [=C2=A0=C2=A0=C2=A0 8.922335]=C2=A0 really_probe+0xb4/0x45c [=C2=A0=C2=A0=C2=A0 8.922349]=C2=A0 __driver_probe_device+0x114/0x190 [=C2=A0=C2=A0=C2=A0 8.922358]=C2=A0 driver_probe_device+0x40/0x100 [=C2=A0=C2=A0=C2=A0 8.922367]=C2=A0 __device_attach_driver+0x98/0x130 [=C2=A0=C2=A0=C2=A0 8.922375]=C2=A0 bus_for_each_drv+0x78/0xd0 [=C2=A0=C2=A0=C2=A0 8.922383]=C2=A0 __device_attach+0xdc/0x1c0 [=C2=A0=C2=A0=C2=A0 8.922391]=C2=A0 device_initial_probe+0x14/0x20 [=C2=A0=C2=A0=C2=A0 8.922400]=C2=A0 bus_probe_device+0x98/0xa0 [=C2=A0=C2=A0=C2=A0 8.922408]=C2=A0 device_add+0x36c/0x8c0 [=C2=A0=C2=A0=C2=A0 8.922416]=C2=A0 __spi_add_device+0x74/0x170 [=C2=A0=C2=A0=C2=A0 8.922423]=C2=A0 spi_add_device+0x64/0xa4 [=C2=A0=C2=A0=C2=A0 8.922429]=C2=A0 of_register_spi_device+0x21c/0x36c [=C2=A0=C2=A0=C2=A0 8.922436]=C2=A0 spi_register_controller+0x5e0/0x834 [=C2=A0=C2=A0=C2=A0 8.922443]=C2=A0 devm_spi_register_controller+0x24/0x8= 0 [=C2=A0=C2=A0=C2=A0 8.922450]=C2=A0 rockchip_spi_probe+0x434/0x5b0 [spi_r= ockchip] [=C2=A0=C2=A0=C2=A0 8.922468]=C2=A0 platform_probe+0x68/0xe0 [=C2=A0=C2=A0=C2=A0 8.922478]=C2=A0 really_probe+0xb4/0x45c [=C2=A0=C2=A0=C2=A0 8.922487]=C2=A0 __driver_probe_device+0x114/0x190 [=C2=A0=C2=A0=C2=A0 8.922497]=C2=A0 driver_probe_device+0x40/0x100 [=C2=A0=C2=A0=C2=A0 8.922507]=C2=A0 __driver_attach+0xcc/0x1e0 [=C2=A0=C2=A0=C2=A0 8.922516]=C2=A0 bus_for_each_dev+0x70/0xd0 [=C2=A0=C2=A0=C2=A0 8.922524]=C2=A0 driver_attach+0x24/0x30 [=C2=A0=C2=A0=C2=A0 8.922533]=C2=A0 bus_add_driver+0x140/0x234 [=C2=A0=C2=A0=C2=A0 8.922540]=C2=A0 driver_register+0x78/0x130 [=C2=A0=C2=A0=C2=A0 8.922547]=C2=A0 __platform_driver_register+0x28/0x34 [=C2=A0=C2=A0=C2=A0 8.922554]=C2=A0 rockchip_spi_driver_init+0x20/0x1000 = [spi_rockchip] [=C2=A0=C2=A0=C2=A0 8.922566]=C2=A0 do_one_initcall+0x50/0x1b0 [=C2=A0=C2=A0=C2=A0 8.922579]=C2=A0 do_init_module+0x54/0x250 [=C2=A0=C2=A0=C2=A0 8.922589]=C2=A0 load_module+0x2230/0x285c [=C2=A0=C2=A0=C2=A0 8.922597]=C2=A0 __do_sys_finit_module+0xbc/0x12c [=C2=A0=C2=A0=C2=A0 8.922605]=C2=A0 __arm64_sys_finit_module+0x24/0x30 [=C2=A0=C2=A0=C2=A0 8.922613]=C2=A0 invoke_syscall+0x48/0x114 [=C2=A0=C2=A0=C2=A0 8.922625]=C2=A0 el0_svc_common+0x40/0xfc [=C2=A0=C2=A0=C2=A0 8.922632]=C2=A0 do_el0_svc_compat+0x20/0x50 [=C2=A0=C2=A0=C2=A0 8.922640]=C2=A0 el0_svc_compat+0x2c/0x54 [=C2=A0=C2=A0=C2=A0 8.922652]=C2=A0 el0t_32_sync_handler+0x90/0x140 [=C2=A0=C2=A0=C2=A0 8.922660]=C2=A0 el0t_32_sync+0x19c/0x1a0 [=C2=A0=C2=A0=C2=A0 8.922669] ---[ end trace 2245d8ba23a1d75c ]--- Note: This does not relate to the spi driver - when disabling this=20 device in the device tree it fails for any other (i2s, for instance)=20 which uses dma. Commenting out the failing check at [1], however, helps and the mapping=20 works again. I tried to follow the recent changes for arm64 mm which could relate to=20 the check failing at [1] and reverting =C2=A0 commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID") helps and makes it work again, but I'm 100% uncertain if that commit is=20 really the culprit. Note, that the firmware (legacy u-boot) injects memory configuration in=20 the device tree as follows: /memreserve/=C2=A0=C2=A0=C2=A0 0x00000000fcefc000 0x000000000000d000; / { .. =C2=A0=C2=A0=C2=A0 compatible =3D "pine64,rock64\0rockchip,rk3328"; .. =C2=A0=C2=A0=C2=A0 memory { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 reg =3D <0x00 0x200000 0x00 0xfee0= 0000 0x00 0x00 0x00 0x00>; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 device_type =3D "memory"; =C2=A0=C2=A0=C2=A0 }; .. } So: there is a "hole" in the mappable memory and reading the commit=20 message of =C2=A0 commit a7d9f306ba70 ("arm64: drop pfn_valid_within() and simplify= =20 pfn_valid()") suggests, there was a change for that case recently. I also noticed there is a diff in the kernel log regarding memory init=20 up until 5.13.12 it says [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= [mem 0x0000000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 empty [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x0000= 000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem=20 0x0000000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000] On node 0 totalpages: 1043968 [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 16312 pages used for = memmap [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 0 pages reserved [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 1043968 pages, LIFO b= atch:63 In contrary in 5.14-rc7 it says: [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= [mem 0x0000000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 empty [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x0000= 000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem=20 0x0000000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000] On node 0, zone DMA: 512 pages in unavailab= le ranges [=C2=A0=C2=A0=C2=A0 0.000000] On node 0, zone DMA: 4096 pages in unavaila= ble ranges (note the "unavailable ranges") I'm uncertain again here, if that diff is expected behavior because of=20 those recent mm changes for arm64. After reverting =C2=A0 commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID") the log changes to [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= [mem 0x0000000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 empty [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x0000= 000000200000-0x00000000feffffff] [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem=20 0x0000000000200000-0x00000000feffffff] (no DMA zones here) As you might have noticed I have _zero_ clue about memory mapping and=20 dma subsystem - so let me know if there is any more information needed=20 for that and thanks for your help. Alex [1]=20 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/k= ernel/dma/mapping.c?id=3De22ce8eb631bdc47a4a4ea7ecf4e4ba499db4f93#n235