From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17ACAC43217 for ; Mon, 16 May 2022 23:15:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348931AbiEPXPA (ORCPT ); Mon, 16 May 2022 19:15:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240651AbiEPXO7 (ORCPT ); Mon, 16 May 2022 19:14:59 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2046.outbound.protection.outlook.com [40.107.94.46]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A45646657; Mon, 16 May 2022 16:14:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J8204WaykwvU9xVipS+7G75cXko5lWKglR5WMFPvoCAQXdnfAVczGrSnvjEhIn36a+idza9CRjhIaQ/v4brfRxrW0idKZNyly1TF7Ke/k8DqOjdaelNQafrhhHZ47sG4/DpsIYWypFUImWgFWuyLiMIgF0DVrkb5ctn+JhGvsZ12LAIkPavVFpxqCkzsZviHy8JNMsGDZ/kJb4A5ux2V+2DZeRVUfKDVG04PJAr5o5dBg8JwKtAG7DX1exGXAvIdYwKr13uUhCQdTOWaB10UgH9b9WtFI2B84XWtivkzdiGpyXIbTff50efZK8IhNBLZ0BKuu6cC3GeT11kHTLag2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=taq+2xQIp+kKBI3hn+kqMlxsyIimhQ9rSg1pxQwJdz8=; b=TLwcxQKVxp7noYLmR+Ot4XVG3f5XvcjLYjg7RZRodqp/D/G182ScC6v6Kr1BM25uGnVb3rWFvFAfb07NPrcFfad6ejswXwWOjCPlK4jVO8QMHQeZl+cXEIaxoqGJGWGh9sQkrKLFdeMIG5pqpRCnPJR4EqUMaboEM33GUfl1UcIOqt8d3SWiOZD66R64QDKX3LL/8YFXjmA59uBpPwsr20UUrmebRZ8Fo40wpVfLMKZdkyjSxo/SJKjH5bwuJQTBGeJ/bybubNzWEn6mIw7uyf8VkjTGwXUj3cFmbD7KAqjVK3IyUqclvnS/J47eT4+UyeG636JbBeGL4XHKtBbrLg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=nvidia.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=taq+2xQIp+kKBI3hn+kqMlxsyIimhQ9rSg1pxQwJdz8=; b=mOfj5pHQPG/TjNuvfamcK3ede5cMBE86Qf0lU/A+04/qCLNpXw2VWnE815N7LyUYtCYxyZVF91f5aiuMkGtAJUSKVLd5WZbDdaro6SHlqs4h/jK7G4J4KDvAKCG/nLYjxYgKr9Td0uIue5TkIIZ0PL6q4hNXcfjFVLVD0Fta0xc= Received: from BN9PR03CA0175.namprd03.prod.outlook.com (2603:10b6:408:f4::30) by DM4PR12MB6495.namprd12.prod.outlook.com (2603:10b6:8:bc::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5250.18; Mon, 16 May 2022 23:14:56 +0000 Received: from BN8NAM11FT013.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f4:cafe::71) by BN9PR03CA0175.outlook.office365.com (2603:10b6:408:f4::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5250.13 via Frontend Transport; Mon, 16 May 2022 23:14:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT013.mail.protection.outlook.com (10.13.176.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5250.13 via Frontend Transport; Mon, 16 May 2022 23:14:56 +0000 Received: from alex-MS-7B09.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 16 May 2022 18:14:54 -0500 From: Alex Sierra To: CC: , , , , , , , , , , , , Subject: [PATCH v2 00/13] Add MEMORY_DEVICE_COHERENT for coherent device memory mapping Date: Mon, 16 May 2022 18:14:19 -0500 Message-ID: <20220516231432.20200-1-alex.sierra@amd.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ee60e979-7d4e-4453-3e3f-08da3791e409 X-MS-TrafficTypeDiagnostic: DM4PR12MB6495:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: c+3qghOWtZ/96lKs8jNs1PDc6ISoCgaIgpw4118ynrx827vqPKC4+sbSOhVVBISv7VnqvMgbpT2/mO5uZfsOxwbCdYyQK4jTtzWLT7ABPCRj4FGDiYLmg2Fb0OcG9V42sBfeebTTMQScJgP6CXqcgvCNPr553+T9DtCCiFESBw/tQTmot7uN3lUcpQ6/MefMWuwAlvbG2ZHvl0uZnzSg9jTSl60rb0xMS/3u1uspfxSbuFf62wPLjmWj5nmxcWXLjMuB4YIJsNJtCiYD3z4G2Fauzqt+uqdHhObjASSH53ZCAfZZ+ofqr3v84eOGy+fJlPElp2ZSl3fval54VpaLGgM3cjcMtlURztaDWf2WdJTOyOe/nzZAOm5pJGW4AlLKY1WsLRIz+YhmAbWkCMEitf0wcq3HHMxiMl90JwLOqJeODzEDhnTyIdTKInDEj8fXxGd1bzuSf89iw1W0MHuanUwwtTf63w9wzAluX53XHkxFc8c3SrGdA//HDbWC8jcY5N2bc3xgzQfdar+2XHUVn3PvcnC0NMecwQ4ZSXNgEgMP2L7VrqWEXrFbqrV8nFHm8b2hIRG6YV6JDkBE8SE/UNKiK6TfWas7hAwxR+ZH8zlwck8obm5UkvtE0l/DCYJYIvhmvhsRUboWGXsmfq7HkInbiG2gLKZ+pnGydhWUj2wR8dmtbtnI0DXDCy3S4ccf7nIx5GrRtOdiDcT4YgzsIw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(36860700001)(40460700003)(356005)(26005)(6916009)(54906003)(36756003)(4326008)(1076003)(86362001)(81166007)(7416002)(70586007)(70206006)(316002)(44832011)(8936002)(508600001)(186003)(8676002)(16526019)(2906002)(7696005)(47076005)(336012)(426003)(6666004)(2616005)(83380400001)(5660300002)(82310400005)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2022 23:14:56.1318 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ee60e979-7d4e-4453-3e3f-08da3791e409 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT013.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6495 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org This is our MEMORY_DEVICE_COHERENT patch series rebased and updated for current 5.18-rc7. Changes since the last version: - Fixed problems with migration during long-term pinning in get_user_pages - Open coded vm_normal_lru_pages as suggested in previous code review - Update hmm_gup_test with more get_user_pages calls, include hmm_cow_in_device in hmm-test. This patch series introduces MEMORY_DEVICE_COHERENT, a type of memory owned by a device that can be mapped into CPU page tables like MEMORY_DEVICE_GENERIC and can also be migrated like MEMORY_DEVICE_PRIVATE. This patch series is mostly self-contained except for a few places where it needs to update other subsystems to handle the new memory type. System stability and performance are not affected according to our ongoing testing, including xfstests. How it works: The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver registers the memory with devmap as MEMORY_DEVICE_COHERENT using devm_memremap_pages. The initial user for this hardware page migration capability is the Frontier supercomputer project. This functionality is not AMD-specific. We expect other GPU vendors to find this functionality useful, and possibly other hardware types in the future. Our test nodes in the lab are similar to the Frontier configuration, with .5 TB of system memory plus 256 GB of device memory split across 4 GPUs, all in a single coherent address space. Page migration is expected to improve application efficiency significantly. We will report empirical results as they become available. Coherent device type pages at gup are now migrated back to system memory if they are being pinned long-term (FOLL_LONGTERM). The reason is, that long-term pinning would interfere with the device memory manager owning the device-coherent pages (e.g. evictions in TTM). These series incorporate Alistair Popple patches to do this migration from pin_user_pages() calls. hmm_gup_test has been added to hmm-test to test different get user pages calls. This series includes handling of device-managed anonymous pages returned by vm_normal_pages. Although they behave like normal pages for purposes of mapping in CPU page tables and for COW, they do not support LRU lists, NUMA migration or THP. We also introduced a FOLL_LRU flag that adds the same behaviour to follow_page and related APIs, to allow callers to specify that they expect to put pages on an LRU list. v2: - Rebase to latest 5.18-rc7. - Drop patch "mm: add device coherent checker to remove migration pte" and modify try_to_migrate_one, to let DEVICE_COHERENT pages fall through to normal page path. Based on Alistair Popple's comment. - Fix comment formatting. - Reword comment in vm_normal_page about pte_devmap(). - Merge "drm/amdkfd: coherent type as sys mem on migration to ram" to "drm/amdkfd: add SPM support for SVM". Alex Sierra (11): mm: add zone device coherent type memory support mm: add device coherent vma selection for memory migration drm/amdkfd: add SPM support for SVM lib: test_hmm add ioctl to get zone device type lib: test_hmm add module param for zone device type lib: add support for device coherent type in test_hmm tools: update hmm-test to support device coherent type tools: update test_hmm script to support SP config mm: handling Non-LRU pages returned by vm_normal_pages tools: add hmm gup tests for device coherent type tools: add selftests to hmm for COW in device memory Alistair Popple (2): mm: remove the vma check in migrate_vma_setup() mm/gup: migrate device coherent pages when pinning instead of failing drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 33 ++- fs/proc/task_mmu.c | 2 +- include/linux/memremap.h | 19 ++ include/linux/migrate.h | 1 + include/linux/mm.h | 3 +- lib/test_hmm.c | 349 +++++++++++++++++------ lib/test_hmm_uapi.h | 22 +- mm/gup.c | 49 +++- mm/huge_memory.c | 2 +- mm/internal.h | 1 + mm/khugepaged.c | 9 +- mm/ksm.c | 6 +- mm/madvise.c | 4 +- mm/memcontrol.c | 7 +- mm/memory-failure.c | 8 +- mm/memory.c | 9 +- mm/mempolicy.c | 2 +- mm/memremap.c | 10 + mm/migrate.c | 4 +- mm/migrate_device.c | 115 ++++++-- mm/mlock.c | 2 +- mm/mprotect.c | 2 +- mm/rmap.c | 5 +- tools/testing/selftests/vm/hmm-tests.c | 308 ++++++++++++++++++-- tools/testing/selftests/vm/test_hmm.sh | 24 +- 25 files changed, 814 insertions(+), 182 deletions(-) -- 2.32.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8A62C4332F for ; Mon, 16 May 2022 23:15:00 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BE2B810E1CF; Mon, 16 May 2022 23:14:59 +0000 (UTC) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2086.outbound.protection.outlook.com [40.107.94.86]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2A8CE10E196; Mon, 16 May 2022 23:14:58 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J8204WaykwvU9xVipS+7G75cXko5lWKglR5WMFPvoCAQXdnfAVczGrSnvjEhIn36a+idza9CRjhIaQ/v4brfRxrW0idKZNyly1TF7Ke/k8DqOjdaelNQafrhhHZ47sG4/DpsIYWypFUImWgFWuyLiMIgF0DVrkb5ctn+JhGvsZ12LAIkPavVFpxqCkzsZviHy8JNMsGDZ/kJb4A5ux2V+2DZeRVUfKDVG04PJAr5o5dBg8JwKtAG7DX1exGXAvIdYwKr13uUhCQdTOWaB10UgH9b9WtFI2B84XWtivkzdiGpyXIbTff50efZK8IhNBLZ0BKuu6cC3GeT11kHTLag2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=taq+2xQIp+kKBI3hn+kqMlxsyIimhQ9rSg1pxQwJdz8=; b=TLwcxQKVxp7noYLmR+Ot4XVG3f5XvcjLYjg7RZRodqp/D/G182ScC6v6Kr1BM25uGnVb3rWFvFAfb07NPrcFfad6ejswXwWOjCPlK4jVO8QMHQeZl+cXEIaxoqGJGWGh9sQkrKLFdeMIG5pqpRCnPJR4EqUMaboEM33GUfl1UcIOqt8d3SWiOZD66R64QDKX3LL/8YFXjmA59uBpPwsr20UUrmebRZ8Fo40wpVfLMKZdkyjSxo/SJKjH5bwuJQTBGeJ/bybubNzWEn6mIw7uyf8VkjTGwXUj3cFmbD7KAqjVK3IyUqclvnS/J47eT4+UyeG636JbBeGL4XHKtBbrLg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=nvidia.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=taq+2xQIp+kKBI3hn+kqMlxsyIimhQ9rSg1pxQwJdz8=; b=mOfj5pHQPG/TjNuvfamcK3ede5cMBE86Qf0lU/A+04/qCLNpXw2VWnE815N7LyUYtCYxyZVF91f5aiuMkGtAJUSKVLd5WZbDdaro6SHlqs4h/jK7G4J4KDvAKCG/nLYjxYgKr9Td0uIue5TkIIZ0PL6q4hNXcfjFVLVD0Fta0xc= Received: from BN9PR03CA0175.namprd03.prod.outlook.com (2603:10b6:408:f4::30) by DM4PR12MB6495.namprd12.prod.outlook.com (2603:10b6:8:bc::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5250.18; Mon, 16 May 2022 23:14:56 +0000 Received: from BN8NAM11FT013.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f4:cafe::71) by BN9PR03CA0175.outlook.office365.com (2603:10b6:408:f4::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5250.13 via Frontend Transport; Mon, 16 May 2022 23:14:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT013.mail.protection.outlook.com (10.13.176.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5250.13 via Frontend Transport; Mon, 16 May 2022 23:14:56 +0000 Received: from alex-MS-7B09.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 16 May 2022 18:14:54 -0500 From: Alex Sierra To: Subject: [PATCH v2 00/13] Add MEMORY_DEVICE_COHERENT for coherent device memory mapping Date: Mon, 16 May 2022 18:14:19 -0500 Message-ID: <20220516231432.20200-1-alex.sierra@amd.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ee60e979-7d4e-4453-3e3f-08da3791e409 X-MS-TrafficTypeDiagnostic: DM4PR12MB6495:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: c+3qghOWtZ/96lKs8jNs1PDc6ISoCgaIgpw4118ynrx827vqPKC4+sbSOhVVBISv7VnqvMgbpT2/mO5uZfsOxwbCdYyQK4jTtzWLT7ABPCRj4FGDiYLmg2Fb0OcG9V42sBfeebTTMQScJgP6CXqcgvCNPr553+T9DtCCiFESBw/tQTmot7uN3lUcpQ6/MefMWuwAlvbG2ZHvl0uZnzSg9jTSl60rb0xMS/3u1uspfxSbuFf62wPLjmWj5nmxcWXLjMuB4YIJsNJtCiYD3z4G2Fauzqt+uqdHhObjASSH53ZCAfZZ+ofqr3v84eOGy+fJlPElp2ZSl3fval54VpaLGgM3cjcMtlURztaDWf2WdJTOyOe/nzZAOm5pJGW4AlLKY1WsLRIz+YhmAbWkCMEitf0wcq3HHMxiMl90JwLOqJeODzEDhnTyIdTKInDEj8fXxGd1bzuSf89iw1W0MHuanUwwtTf63w9wzAluX53XHkxFc8c3SrGdA//HDbWC8jcY5N2bc3xgzQfdar+2XHUVn3PvcnC0NMecwQ4ZSXNgEgMP2L7VrqWEXrFbqrV8nFHm8b2hIRG6YV6JDkBE8SE/UNKiK6TfWas7hAwxR+ZH8zlwck8obm5UkvtE0l/DCYJYIvhmvhsRUboWGXsmfq7HkInbiG2gLKZ+pnGydhWUj2wR8dmtbtnI0DXDCy3S4ccf7nIx5GrRtOdiDcT4YgzsIw== X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(36860700001)(40460700003)(356005)(26005)(6916009)(54906003)(36756003)(4326008)(1076003)(86362001)(81166007)(7416002)(70586007)(70206006)(316002)(44832011)(8936002)(508600001)(186003)(8676002)(16526019)(2906002)(7696005)(47076005)(336012)(426003)(6666004)(2616005)(83380400001)(5660300002)(82310400005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2022 23:14:56.1318 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ee60e979-7d4e-4453-3e3f-08da3791e409 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT013.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6495 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: rcampbell@nvidia.com, willy@infradead.org, david@redhat.com, Felix.Kuehling@amd.com, apopple@nvidia.com, amd-gfx@lists.freedesktop.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, jglisse@redhat.com, dri-devel@lists.freedesktop.org, akpm@linux-foundation.org, linux-ext4@vger.kernel.org, hch@lst.de Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This is our MEMORY_DEVICE_COHERENT patch series rebased and updated for current 5.18-rc7. Changes since the last version: - Fixed problems with migration during long-term pinning in get_user_pages - Open coded vm_normal_lru_pages as suggested in previous code review - Update hmm_gup_test with more get_user_pages calls, include hmm_cow_in_device in hmm-test. This patch series introduces MEMORY_DEVICE_COHERENT, a type of memory owned by a device that can be mapped into CPU page tables like MEMORY_DEVICE_GENERIC and can also be migrated like MEMORY_DEVICE_PRIVATE. This patch series is mostly self-contained except for a few places where it needs to update other subsystems to handle the new memory type. System stability and performance are not affected according to our ongoing testing, including xfstests. How it works: The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver registers the memory with devmap as MEMORY_DEVICE_COHERENT using devm_memremap_pages. The initial user for this hardware page migration capability is the Frontier supercomputer project. This functionality is not AMD-specific. We expect other GPU vendors to find this functionality useful, and possibly other hardware types in the future. Our test nodes in the lab are similar to the Frontier configuration, with .5 TB of system memory plus 256 GB of device memory split across 4 GPUs, all in a single coherent address space. Page migration is expected to improve application efficiency significantly. We will report empirical results as they become available. Coherent device type pages at gup are now migrated back to system memory if they are being pinned long-term (FOLL_LONGTERM). The reason is, that long-term pinning would interfere with the device memory manager owning the device-coherent pages (e.g. evictions in TTM). These series incorporate Alistair Popple patches to do this migration from pin_user_pages() calls. hmm_gup_test has been added to hmm-test to test different get user pages calls. This series includes handling of device-managed anonymous pages returned by vm_normal_pages. Although they behave like normal pages for purposes of mapping in CPU page tables and for COW, they do not support LRU lists, NUMA migration or THP. We also introduced a FOLL_LRU flag that adds the same behaviour to follow_page and related APIs, to allow callers to specify that they expect to put pages on an LRU list. v2: - Rebase to latest 5.18-rc7. - Drop patch "mm: add device coherent checker to remove migration pte" and modify try_to_migrate_one, to let DEVICE_COHERENT pages fall through to normal page path. Based on Alistair Popple's comment. - Fix comment formatting. - Reword comment in vm_normal_page about pte_devmap(). - Merge "drm/amdkfd: coherent type as sys mem on migration to ram" to "drm/amdkfd: add SPM support for SVM". Alex Sierra (11): mm: add zone device coherent type memory support mm: add device coherent vma selection for memory migration drm/amdkfd: add SPM support for SVM lib: test_hmm add ioctl to get zone device type lib: test_hmm add module param for zone device type lib: add support for device coherent type in test_hmm tools: update hmm-test to support device coherent type tools: update test_hmm script to support SP config mm: handling Non-LRU pages returned by vm_normal_pages tools: add hmm gup tests for device coherent type tools: add selftests to hmm for COW in device memory Alistair Popple (2): mm: remove the vma check in migrate_vma_setup() mm/gup: migrate device coherent pages when pinning instead of failing drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 33 ++- fs/proc/task_mmu.c | 2 +- include/linux/memremap.h | 19 ++ include/linux/migrate.h | 1 + include/linux/mm.h | 3 +- lib/test_hmm.c | 349 +++++++++++++++++------ lib/test_hmm_uapi.h | 22 +- mm/gup.c | 49 +++- mm/huge_memory.c | 2 +- mm/internal.h | 1 + mm/khugepaged.c | 9 +- mm/ksm.c | 6 +- mm/madvise.c | 4 +- mm/memcontrol.c | 7 +- mm/memory-failure.c | 8 +- mm/memory.c | 9 +- mm/mempolicy.c | 2 +- mm/memremap.c | 10 + mm/migrate.c | 4 +- mm/migrate_device.c | 115 ++++++-- mm/mlock.c | 2 +- mm/mprotect.c | 2 +- mm/rmap.c | 5 +- tools/testing/selftests/vm/hmm-tests.c | 308 ++++++++++++++++++-- tools/testing/selftests/vm/test_hmm.sh | 24 +- 25 files changed, 814 insertions(+), 182 deletions(-) -- 2.32.0