From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E42FC4361B for ; Tue, 15 Dec 2020 03:12:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 04E3822473 for ; Tue, 15 Dec 2020 03:12:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 04E3822473 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8C0F08D0069; Mon, 14 Dec 2020 22:12:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 876F18D001C; Mon, 14 Dec 2020 22:12:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 787668D0069; Mon, 14 Dec 2020 22:12:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id 6320E8D001C for ; Mon, 14 Dec 2020 22:12:58 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 35C758249980 for ; Tue, 15 Dec 2020 03:12:58 +0000 (UTC) X-FDA: 77594044836.16.ice38_5f024ff27420 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 13550100E690B for ; Tue, 15 Dec 2020 03:12:58 +0000 (UTC) X-HE-Tag: ice38_5f024ff27420 X-Filterd-Recvd-Size: 3264 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Tue, 15 Dec 2020 03:12:57 +0000 (UTC) Date: Mon, 14 Dec 2020 19:12:55 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1608001976; bh=t/bd141POhUnPd5FDmkqapbaJU1Xc9Ym6ihwFUMCibY=; h=From:To:Subject:In-Reply-To:From; b=I2Zlzkaqko1P6T3drrrG0Eh9//8KTqiOZ8JtuPHqVLDpbiM5T0olaVAYFhizWBF6F 1eRnB9EUP4ecbMu2yvTOvS4IX7RSYSB4WiEx/TSk5UP1SuRhsZMqXxWMBzoijLg26r YOLDH0J+Xahz4iUy7rsY00VEY/ud9/Df6Xn3BECo= From: Andrew Morton To: akpm@linux-foundation.org, apopple@nvidia.com, hch@lst.de, jgg@nvidia.com, jglisse@redhat.com, jhubbard@nvidia.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, rcampbell@nvidia.com, torvalds@linux-foundation.org Subject: [patch 164/200] mm/migrate.c: optimize migrate_vma_pages() mmu notifier Message-ID: <20201215031255.JOIcQ-YOL%akpm@linux-foundation.org> In-Reply-To: <20201214190237.a17b70ae14f129e2dca3d204@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Ralph Campbell Subject: mm/migrate.c: optimize migrate_vma_pages() mmu notifier When migrating a zero page or pte_none() anonymous page to device private memory, migrate_vma_setup() will initialize the src[] array with a NULL PFN. This lets the device driver allocate device private memory and clear it instead of DMAing a page of zeros over the device bus. Since the source page didn't exist at the time, no struct page was locked nor a migration PTE inserted into the CPU page tables. The actual PTE insertion happens in migrate_vma_pages() when it tries to insert the device private struct page PTE into the CPU page tables. migrate_vma_pages() has to call the mmu notifiers again since another device could fault on the same page before the page table locks are acquired. Allow device drivers to optimize the invalidation similar to migrate_vma_setup() by calling mmu_notifier_range_init() which sets struct mmu_notifier_range event type to MMU_NOTIFY_MIGRATE and the migrate_pgmap_owner field. Link: https://lkml.kernel.org/r/20201021191335.10916-1-rcampbell@nvidia.com Signed-off-by: Ralph Campbell Cc: Jerome Glisse Cc: John Hubbard Cc: Alistair Popple Cc: Christoph Hellwig Cc: Jason Gunthorpe Signed-off-by: Andrew Morton --- mm/migrate.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) --- a/mm/migrate.c~mm-optimize-migrate_vma_pages-mmu-notifier +++ a/mm/migrate.c @@ -3001,11 +3001,10 @@ void migrate_vma_pages(struct migrate_vm if (!notified) { notified = true; - mmu_notifier_range_init(&range, - MMU_NOTIFY_CLEAR, 0, - NULL, - migrate->vma->vm_mm, - addr, migrate->end); + mmu_notifier_range_init_migrate(&range, 0, + migrate->vma, migrate->vma->vm_mm, + addr, migrate->end, + migrate->pgmap_owner); mmu_notifier_invalidate_range_start(&range); } migrate_vma_insert_page(migrate, addr, newpage, _