From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A7A9C4708F for ; Sat, 29 May 2021 22:51:27 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F347F610FA for ; Sat, 29 May 2021 22:51:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F347F610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9DF616E0D4; Sat, 29 May 2021 22:51:25 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2062.outbound.protection.outlook.com [40.107.243.62]) by gabe.freedesktop.org (Postfix) with ESMTPS id 008F46E0D4 for ; Sat, 29 May 2021 22:51:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fmrVO2tlNbksZgKPA7j7e8gYlmltK0u/lSb2k5pczbUEQslbBRXs6madlz2SgD/XvGjWpmk08d52apuvnJb13Fw5MMdEpBHFmivaSSmjUIEWA2Lsmz5GByrI1LB5RMQK05s42+HHpB0of3HrQvvZ0+6nsS/SjuzCzRFRlzbcbYE8r/iIWWgJP6qNQnOWqU+/V7xR+NqaQlGeDgTrgZgJdx/QDhtUVLckYKkoAue/Ui9X1Aug5EJte6ncUlXz6E+I0b1uaw2eSE7woIo8e+zGzm+gx0Kla3FP2x3AkkC7tGcBLrSV1AAm/O5iNfV0E4WSJbSeiVgnzEVkRVtKSUtolg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fT5Na1QXDN6EU8m/414Xd3gAylEUlMa29No2v8pYtzc=; b=h/iCp/JqwhkaJIgZAOlbHSVdMwzk1T8CZ13y5dZXV3mgyDUex9GtZvqE7h0vkZIbautlrxI5/9PnCBshtITGEZAOK66B1uURwaWrkamqSk247MF0X+hXbRYngCz5f77vEdy4NeF2tEmb0RINSrkYX/Uc+0oZ4ItSN57pc98a4QbDUf29efHz0SL5tc+dtcjr3199oTbPqV76K40DTIMy/23mT/lyCCQgFZOUTSlyX5u0tg7UbrMzkBTkm+Y4d/0CIy7BqtPL2QnacQWTBcA/bg0mt1v9vW2WvqQaBC84reeDGYrA3ig39aeOSVjbpatJED3Y4wla1pLik+uk3gywvw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=amd.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fT5Na1QXDN6EU8m/414Xd3gAylEUlMa29No2v8pYtzc=; b=aaxtyig5BupTVCCdJ37SBMPNBQ9uyWXwya8uJ1equkVePHR6o+fdgbo57nENa3PILg81gBnfMp4CBVMfw2FYrW3XIVj2V3kxCIDfkF6u5vRMvNa8Z/SiOhC/TpadETVADp2u58eLGhxk3uFPOSYKVEho+qe5ybK6QluOpgXwlFo= Received: from MWHPR1701CA0006.namprd17.prod.outlook.com (2603:10b6:301:14::16) by DM4PR12MB5037.namprd12.prod.outlook.com (2603:10b6:5:388::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4173.24; Sat, 29 May 2021 22:51:19 +0000 Received: from CO1NAM11FT058.eop-nam11.prod.protection.outlook.com (2603:10b6:301:14:cafe::a7) by MWHPR1701CA0006.outlook.office365.com (2603:10b6:301:14::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4173.20 via Frontend Transport; Sat, 29 May 2021 22:51:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; lists.freedesktop.org; dkim=none (message not signed) header.d=none;lists.freedesktop.org; dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1NAM11FT058.mail.protection.outlook.com (10.13.174.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4150.30 via Frontend Transport; Sat, 29 May 2021 22:51:18 +0000 Received: from eric-HP-EliteBook-745-G4.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4; Sat, 29 May 2021 17:51:17 -0500 From: Eric Huang To: Subject: [PATCH 2/2] drm/amdkfd: optimize memory mapping latency Date: Sat, 29 May 2021 18:51:01 -0400 Message-ID: <20210529225101.459466-2-jinhuieric.huang@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210529225101.459466-1-jinhuieric.huang@amd.com> References: <20210529225101.459466-1-jinhuieric.huang@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b718cbae-e55d-4bd4-86ce-08d922f445f2 X-MS-TrafficTypeDiagnostic: DM4PR12MB5037: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dwbH1yrc08TihaUn8ismhzq1wu/0S5UikrMmsbiv7LgupTgQJMhyPTLA42X/Oj0I9ZvmFMrQuAWTxhmmg72vkgf1avosootF924Kia3yVsLpnalaUxQ3lLOB6iLtdEPLwkdD1FnLQVx62SJppjmVgdwgdkSc9q2Acn/2YTI1dRPN3JEXlrAPZtq2du9/AemeXPWthSB83KUlAyLsaZ3cEZlQ7SQ0FI3YdXu6XqfF3hMZZmKORIF8F3YF1M8gctGhlX5gzXQA1cu7tAd5BNFdNOiMu9M55FlidC2SdM6zj2wVRpVcE5/+BrzORDT/GRX/CIPITSJqvibI6x4tALhmjG8iIYwMfRg0Tkc5rnteT4ThURcGUUZlIZ+u0W4hgemXto790Q/yDJPxrBFGi/EhY4ON/xMKLcv6CQqCwUoHW6BDQRnz1YCqAQvrujfMw6Enf/PcBT1DaGwbZH03rh9A6V/oP9rKjY+WhSk4e8hlQmGcYfHwpzlYYZLCzjq1JB8UMhTY7z6qbS7Manr4783pdgi4GEb/lHTyfCXEZ6dx82w+6M/atSrGs5NPFJd/FnPN2P+aYVcydZL2QPFTky4Lbjg4M9xYbIsSpoCeb1NeiPOf7R+60utAVi/s+gW9EIB1G7vPkdO6TNfPBdmdUSiILC20hrXSX1h62qfks56OdlhOu8GNu5X82yxK/Ut4HZMi X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(4636009)(39860400002)(376002)(396003)(136003)(346002)(36840700001)(46966006)(5660300002)(36756003)(26005)(81166007)(82310400003)(336012)(6916009)(7696005)(16526019)(36860700001)(2616005)(70586007)(186003)(426003)(4326008)(86362001)(83380400001)(82740400003)(2906002)(8936002)(478600001)(47076005)(6666004)(70206006)(1076003)(316002)(8676002)(356005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 May 2021 22:51:18.8833 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b718cbae-e55d-4bd4-86ce-08d922f445f2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT058.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5037 X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eric Huang Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" 1. conditionally flush TLBs after map. 2. add heavy weight TLBs flushing after unmap. Signed-off-by: Eric Huang --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 1 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 24 +++++++++++-------- .../drm/amd/amdkfd/kfd_device_queue_manager.c | 6 ++--- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 4 ++-- 8 files changed, 25 insertions(+), 17 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h index 2560977760b3..997258c24ef2 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h @@ -85,6 +85,7 @@ struct kgd_mem { bool aql_queue; bool is_imported; + bool table_freed; }; /* KFD Memory Eviction */ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index 385c33675227..8ac0d849fd3f 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -1132,6 +1132,8 @@ static int update_gpuvm_pte(struct kgd_mem *mem, return ret; } + mem->table_freed = bo_va->table_freed; + return amdgpu_sync_fence(sync, bo_va->last_pt_update); } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h index 36e7f088d4ee..0e0f27f779cd 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h @@ -87,6 +87,7 @@ struct amdgpu_bo_va { bool cleared; bool is_xgmi; + bool table_freed; }; struct amdgpu_bo { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 95b94c95adac..ff3eb8395017 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -1885,7 +1885,7 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va, resv, mapping->start, mapping->last, update_flags, mapping->offset, mem, - pages_addr, last_update, NULL, + pages_addr, last_update, &bo_va->table_freed, vram_base_offset); if (r) return r; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 960913a35ee4..c45ccd1d03c0 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -1658,16 +1658,18 @@ static int kfd_ioctl_map_memory_to_gpu(struct file *filep, } /* Flush TLBs after waiting for the page table updates to complete */ - for (i = 0; i < args->n_devices; i++) { - peer = kfd_device_by_id(devices_arr[i]); - if (WARN_ON_ONCE(!peer)) - continue; - peer_pdd = kfd_get_process_device_data(peer, p); - if (WARN_ON_ONCE(!peer_pdd)) - continue; - if (!amdgpu_read_lock(peer->ddev, true)) { - kfd_flush_tlb(peer_pdd); - amdgpu_read_unlock(peer->ddev); + if (((struct kgd_mem *)mem)->table_freed) { + for (i = 0; i < args->n_devices; i++) { + peer = kfd_device_by_id(devices_arr[i]); + if (WARN_ON_ONCE(!peer)) + continue; + peer_pdd = kfd_get_process_device_data(peer, p); + if (WARN_ON_ONCE(!peer_pdd)) + continue; + if (!amdgpu_read_lock(peer->ddev, true)) { + kfd_flush_tlb(peer_pdd, TLB_FLUSH_LEGACY); + amdgpu_read_unlock(peer->ddev); + } } } @@ -1766,6 +1768,8 @@ static int kfd_ioctl_unmap_memory_from_gpu(struct file *filep, amdgpu_read_unlock(peer->ddev); goto unmap_memory_from_gpu_failed; } + ((struct kgd_mem *)mem)->table_freed = false; + kfd_flush_tlb(peer_pdd, TLB_FLUSH_HEAVYWEIGHT); amdgpu_read_unlock(peer->ddev); args->n_success = i+1; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index c1bea1f7627b..a4920bc5cfbc 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -278,7 +278,7 @@ static int allocate_vmid(struct device_queue_manager *dqm, qpd->vmid, qpd->page_table_base); /* invalidate the VM context after pasid and vmid mapping is set up */ - kfd_flush_tlb(qpd_to_pdd(qpd)); + kfd_flush_tlb(qpd_to_pdd(qpd), TLB_FLUSH_LEGACY); if (dqm->dev->kfd2kgd->set_scratch_backing_va) dqm->dev->kfd2kgd->set_scratch_backing_va(dqm->dev->kgd, @@ -314,7 +314,7 @@ static void deallocate_vmid(struct device_queue_manager *dqm, if (flush_texture_cache_nocpsch(q->device, qpd)) pr_err("Failed to flush TC\n"); - kfd_flush_tlb(qpd_to_pdd(qpd)); + kfd_flush_tlb(qpd_to_pdd(qpd), TLB_FLUSH_LEGACY); /* Release the vmid mapping */ set_pasid_vmid_mapping(dqm, 0, qpd->vmid); @@ -885,7 +885,7 @@ static int restore_process_queues_nocpsch(struct device_queue_manager *dqm, dqm->dev->kgd, qpd->vmid, qpd->page_table_base); - kfd_flush_tlb(pdd); + kfd_flush_tlb(pdd, TLB_FLUSH_LEGACY); } /* Take a safe reference to the mm_struct, which may otherwise diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index ecdd5e782b81..edce3ecf207d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -1338,7 +1338,7 @@ void kfd_signal_reset_event(struct kfd_dev *dev); void kfd_signal_poison_consumed_event(struct kfd_dev *dev, u32 pasid); -void kfd_flush_tlb(struct kfd_process_device *pdd); +void kfd_flush_tlb(struct kfd_process_device *pdd, enum TLB_FLUSH_TYPE type); int dbgdev_wave_reset_wavefronts(struct kfd_dev *dev, struct kfd_process *p); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 4ab9da288f90..a03373743a3d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -2161,7 +2161,7 @@ int kfd_reserved_mem_mmap(struct kfd_dev *dev, struct kfd_process *process, KFD_CWSR_TBA_TMA_SIZE, vma->vm_page_prot); } -void kfd_flush_tlb(struct kfd_process_device *pdd) +void kfd_flush_tlb(struct kfd_process_device *pdd, enum TLB_FLUSH_TYPE type) { struct kfd_dev *dev = pdd->dev; @@ -2174,7 +2174,7 @@ void kfd_flush_tlb(struct kfd_process_device *pdd) pdd->qpd.vmid); } else { amdgpu_amdkfd_flush_gpu_tlb_pasid(dev->kgd, - pdd->process->pasid, TLB_FLUSH_LEGACY); + pdd->process->pasid, type); } } -- 2.25.1 _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx