From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A531FC47428 for ; Wed, 7 Oct 2020 16:44:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1E2E82193E for ; Wed, 7 Oct 2020 16:44:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="E4wIl1kh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1E2E82193E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EF2108E0001; Wed, 7 Oct 2020 12:44:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E58C7900002; Wed, 7 Oct 2020 12:44:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB6C38E0003; Wed, 7 Oct 2020 12:44:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0161.hostedemail.com [216.40.44.161]) by kanga.kvack.org (Postfix) with ESMTP id 8226A8E0001 for ; Wed, 7 Oct 2020 12:44:45 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 2890B8249980 for ; Wed, 7 Oct 2020 16:44:45 +0000 (UTC) X-FDA: 77345703330.28.cook38_07160fd271d0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 03BDD6C05 for ; Wed, 7 Oct 2020 16:44:44 +0000 (UTC) X-HE-Tag: cook38_07160fd271d0 X-Filterd-Recvd-Size: 8124 Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Wed, 7 Oct 2020 16:44:44 +0000 (UTC) Received: by mail-wr1-f66.google.com with SMTP id n18so2989363wrs.5 for ; Wed, 07 Oct 2020 09:44:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=y0cYv7pt2ajdPKJ71Sud+gUJyoyQWkHY61omgO6gUCQ=; b=E4wIl1khATPAWsmoahDSq5XJ9XB3/O17Fju2KtWFpZcnpDunS6aB2uzf/IAoSlz+9+ 5VxZXcG5teVG2C1MloXLK4xiaqUArGcDx8l8ZuU+9gaO8lbpEVnn6vpAnmLx4qTlAd8x TdyxsIClr7xEEKfXI/m8EFXclIzKOCjXa7FyM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=y0cYv7pt2ajdPKJ71Sud+gUJyoyQWkHY61omgO6gUCQ=; b=L4dZaYt53YtspP8aFJ17EHOw3r3Qjgf0zyij5jiT1MURUf4KirhKihOiyC8tyQUP0D n+9d1tK31nN2876QPLiWslQSroLhdzR0/9apS7jlVD8kzndMkBNhf+UeAUryxoMfV6gz GUHlbcls3VVsOept2LJyrsWdCkju3tH2jXnD8hp9ec6CQMkiYuQaq8PGkAZlicLzFHJU vTIXkag8/UOG3E+uGLuEUwHLyCcRwENOobP8xrqSztWVfczsIrk/Z23LS4XyPcV7cVoO M1Uo5xrLtXPF5Wu8aSl9E4/Tm8L6AONtTHtJp33FBPyCtqO4NHr0JBjN3g39q5wwWveC M1pA== X-Gm-Message-State: AOAM533kPyo/b+xhY/rQpwraI7vUEIjMAKJbUU+NkMEETqNZrQfpOnIf 0tAiKsQg0SgjvENcEJQrR/UfSA== X-Google-Smtp-Source: ABdhPJyOhUNMEwwCQ+AEh8GM5Lra9AvYxZp9hFKsj4jVU6Oiy0Fq4b4Qf4lRUIggnlkhrBK3gQ/vuQ== X-Received: by 2002:a5d:51ca:: with SMTP id n10mr4539798wrv.222.1602089083290; Wed, 07 Oct 2020 09:44:43 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id z191sm3332280wme.40.2020.10.07.09.44.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Oct 2020 09:44:42 -0700 (PDT) From: Daniel Vetter To: DRI Development , LKML Cc: kvm@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-samsung-soc@vger.kernel.org, linux-media@vger.kernel.org, linux-s390@vger.kernel.org, Daniel Vetter , Jason Gunthorpe , Dan Williams , Kees Cook , Rik van Riel , Benjamin Herrensmidt , Dave Airlie , Hugh Dickins , Andrew Morton , John Hubbard , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Jan Kara , Daniel Vetter Subject: [PATCH 07/13] mm: close race in generic_access_phys Date: Wed, 7 Oct 2020 18:44:20 +0200 Message-Id: <20201007164426.1812530-8-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201007164426.1812530-1-daniel.vetter@ffwll.ch> References: <20201007164426.1812530-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Way back it was a reasonable assumptions that iomem mappings never change the pfn range they point at. But this has changed: - gpu drivers dynamically manage their memory nowadays, invalidating ptes with unmap_mapping_range when buffers get moved - contiguous dma allocations have moved from dedicated carvetouts to cma regions. This means if we miss the unmap the pfn might contain pagecache or anon memory (well anything allocated with GFP_MOVEABLE) - even /dev/mem now invalidates mappings when the kernel requests that iomem region when CONFIG_IO_STRICT_DEVMEM is set, see 3234ac664a87 ("/dev/mem: Revoke mappings when a driver claims the region") Accessing pfns obtained from ptes without holding all the locks is therefore no longer a good idea. Fix this. Since ioremap might need to manipulate pagetables too we need to drop the pt lock and have a retry loop if we raced. While at it, also add kerneldoc and improve the comment for the vma_ops->access function. It's for accessing, not for moving the memory from iomem to system memory, as the old comment seemed to suggest. References: 28b2ee20c7cb ("access_process_vm device memory infrastructure= ") Cc: Jason Gunthorpe Cc: Dan Williams Cc: Kees Cook Cc: Rik van Riel Cc: Benjamin Herrensmidt Cc: Dave Airlie Cc: Hugh Dickins Cc: Andrew Morton Cc: John Hubbard Cc: J=C3=A9r=C3=B4me Glisse Cc: Jan Kara Cc: Dan Williams Cc: linux-mm@kvack.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-samsung-soc@vger.kernel.org Cc: linux-media@vger.kernel.org Signed-off-by: Daniel Vetter --- include/linux/mm.h | 3 ++- mm/memory.c | 44 ++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 44 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index acd60fbf1a5a..2a16631c1fda 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -566,7 +566,8 @@ struct vm_operations_struct { vm_fault_t (*pfn_mkwrite)(struct vm_fault *vmf); =20 /* called by access_process_vm when get_user_pages() fails, typically - * for use by special VMAs that can switch between memory and hardware + * for use by special VMAs. See also generic_access_phys() for a generi= c + * implementation useful for any iomem mapping. */ int (*access)(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index fcfc4ca36eba..8d467e23b44e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4873,28 +4873,68 @@ int follow_phys(struct vm_area_struct *vma, return ret; } =20 +/** + * generic_access_phys - generic implementation for iomem mmap access + * @vma: the vma to access + * @addr: userspace addres, not relative offset within @vma + * @buf: buffer to read/write + * @len: length of transfer + * @write: set to FOLL_WRITE when writing, otherwise reading + * + * This is a generic implementation for &vm_operations_struct.access for= an + * iomem mapping. This callback is used by access_process_vm() when the = @vma is + * not page based. + */ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write) { resource_size_t phys_addr; unsigned long prot =3D 0; void __iomem *maddr; + pte_t *ptep, pte; + spinlock_t *ptl; int offset =3D addr & (PAGE_SIZE-1); + int ret =3D -EINVAL; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + return -EINVAL; + +retry: + if (follow_pte(vma->vm_mm, addr, &ptep, &ptl)) + return -EINVAL; + pte =3D *ptep; + pte_unmap_unlock(ptep, ptl); =20 - if (follow_phys(vma, addr, write, &prot, &phys_addr)) + prot =3D pgprot_val(pte_pgprot(pte)); + phys_addr =3D (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; + + if ((write & FOLL_WRITE) && !pte_write(pte)) return -EINVAL; =20 maddr =3D ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; =20 + if (follow_pte(vma->vm_mm, addr, &ptep, &ptl)) + goto out_unmap; + + if (pte_same(pte, *ptep)) { + pte_unmap_unlock(ptep, ptl); + iounmap(maddr); + + goto retry; + } + if (write) memcpy_toio(maddr + offset, buf, len); else memcpy_fromio(buf, maddr + offset, len); + ret =3D len; + pte_unmap_unlock(ptep, ptl); +out_unmap: iounmap(maddr); =20 - return len; + return ret; } EXPORT_SYMBOL_GPL(generic_access_phys); #endif --=20 2.28.0