From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751914AbdLNBJD (ORCPT ); Wed, 13 Dec 2017 20:09:03 -0500 Received: from mga11.intel.com ([192.55.52.93]:12940 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751832AbdLNBJB (ORCPT ); Wed, 13 Dec 2017 20:09:01 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.45,399,1508828400"; d="scan'208";a="2612202" From: Lu Baolu To: Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" , Alex Williamson , Joerg Roedel , David Woodhouse Cc: iommu@lists.linux-foundation.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Ashok Raj , Dave Hansen , Huang Ying , CQ Tang , Andy Lutomirski , Rik van Riel , Kees Cook , Andrew Morton , Michal Hocko , "Paul E . McKenney" , Vegard Nossum , Jean-Phillipe Brucker , Lu Baolu Subject: [PATCH 2/2] iommu/vt-d: Register kernel MMU notifier to manage IOTLB/DEVTLB Date: Thu, 14 Dec 2017 09:02:46 +0800 Message-Id: <1513213366-22594-3-git-send-email-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1513213366-22594-1-git-send-email-baolu.lu@linux.intel.com> References: <1513213366-22594-1-git-send-email-baolu.lu@linux.intel.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Ashok Raj When a kernel client calls intel_svm_bind_mm() and gets a valid supervisor PASID, the memory mapping of init_mm will be shared between CPUs and device. IOMMU has to track the changes to this memory mapping, and get notified whenever a TLB flush is needed. Otherwise, the device TLB will be stale compared to that on the cpu for kernel mappings. This is similar to what have been done for user space registrations via mmu_notifier_register() api's. To: Alex Williamson To: linux-kernel@vger.kernel.org To: Joerg Roedel Cc: Ashok Raj Cc: Dave Hansen Cc: Huang Ying Cc: CQ Tang Cc: Thomas Gleixner Cc: Ingo Molnar Cc: H. Peter Anvin Cc: Andy Lutomirski Cc: Rik van Riel Cc: Kees Cook Cc: Andrew Morton Cc: Michal Hocko Cc: Paul E. McKenney Cc: Vegard Nossum Cc: x86@kernel.org Cc: linux-mm@kvack.org Cc: iommu@lists.linux-foundation.org Cc: David Woodhouse CC: Jean-Phillipe Brucker Signed-off-by: Ashok Raj Signed-off-by: Lu Baolu --- drivers/iommu/intel-svm.c | 27 +++++++++++++++++++++++++-- include/linux/intel-iommu.h | 5 ++++- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c index ed1cf7c..1456092 100644 --- a/drivers/iommu/intel-svm.c +++ b/drivers/iommu/intel-svm.c @@ -283,6 +283,24 @@ static const struct mmu_notifier_ops intel_mmuops = { static DEFINE_MUTEX(pasid_mutex); +static int intel_init_mm_inval_range(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct kernel_mmu_address_range *range; + struct intel_svm *svm = container_of(nb, struct intel_svm, init_mm_nb); + unsigned long start, end; + + if (action == KERNEL_MMU_INVALIDATE_RANGE) { + range = data; + start = range->start; + end = range->end; + + intel_flush_svm_range(svm, start, + (end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0, 0); + } + return 0; +} + int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ops *ops) { struct intel_iommu *iommu = intel_svm_device_to_iommu(dev); @@ -382,12 +400,12 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ goto out; } svm->pasid = ret; - svm->notifier.ops = &intel_mmuops; svm->mm = mm; svm->flags = flags; INIT_LIST_HEAD_RCU(&svm->devs); ret = -ENOMEM; if (mm) { + svm->notifier.ops = &intel_mmuops; ret = mmu_notifier_register(&svm->notifier, mm); if (ret) { idr_remove(&svm->iommu->pasid_idr, svm->pasid); @@ -396,8 +414,11 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ goto out; } iommu->pasid_table[svm->pasid].val = (u64)__pa(mm->pgd) | 1; - } else + } else { + svm->init_mm_nb.notifier_call = intel_init_mm_inval_range; + kernel_mmu_notifier_register(&svm->init_mm_nb); iommu->pasid_table[svm->pasid].val = (u64)__pa(init_mm.pgd) | 1 | (1ULL << 11); + } wmb(); /* In caching mode, we still have to flush with PASID 0 when * a PASID table entry becomes present. Not entirely clear @@ -464,6 +485,8 @@ int intel_svm_unbind_mm(struct device *dev, int pasid) idr_remove(&svm->iommu->pasid_idr, svm->pasid); if (svm->mm) mmu_notifier_unregister(&svm->notifier, svm->mm); + else + kernel_mmu_notifier_unregister(&svm->init_mm_nb); /* We mandate that no page faults may be outstanding * for the PASID when intel_svm_unbind_mm() is called. diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h index f3274d9..5cf83db 100644 --- a/include/linux/intel-iommu.h +++ b/include/linux/intel-iommu.h @@ -478,7 +478,10 @@ struct intel_svm_dev { }; struct intel_svm { - struct mmu_notifier notifier; + union { + struct mmu_notifier notifier; + struct notifier_block init_mm_nb; + }; struct mm_struct *mm; struct intel_iommu *iommu; int flags; -- 2.7.4 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-f71.google.com (mail-pg0-f71.google.com [74.125.83.71]) by kanga.kvack.org (Postfix) with ESMTP id AC4D36B025E for ; Wed, 13 Dec 2017 20:09:02 -0500 (EST) Received: by mail-pg0-f71.google.com with SMTP id x24so2693403pgv.5 for ; Wed, 13 Dec 2017 17:09:02 -0800 (PST) Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id i11si2062131pgf.430.2017.12.13.17.09.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Dec 2017 17:09:01 -0800 (PST) From: Lu Baolu Subject: [PATCH 2/2] iommu/vt-d: Register kernel MMU notifier to manage IOTLB/DEVTLB Date: Thu, 14 Dec 2017 09:02:46 +0800 Message-Id: <1513213366-22594-3-git-send-email-baolu.lu@linux.intel.com> In-Reply-To: <1513213366-22594-1-git-send-email-baolu.lu@linux.intel.com> References: <1513213366-22594-1-git-send-email-baolu.lu@linux.intel.com> Sender: owner-linux-mm@kvack.org List-ID: To: Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" , Alex Williamson , Joerg Roedel , David Woodhouse Cc: iommu@lists.linux-foundation.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Ashok Raj , Dave Hansen , Huang Ying , CQ Tang , Andy Lutomirski , Rik van Riel , Kees Cook , Andrew Morton , Michal Hocko , "Paul E . McKenney" , Vegard Nossum , Jean-Phillipe Brucker , Lu Baolu From: Ashok Raj When a kernel client calls intel_svm_bind_mm() and gets a valid supervisor PASID, the memory mapping of init_mm will be shared between CPUs and device. IOMMU has to track the changes to this memory mapping, and get notified whenever a TLB flush is needed. Otherwise, the device TLB will be stale compared to that on the cpu for kernel mappings. This is similar to what have been done for user space registrations via mmu_notifier_register() api's. To: Alex Williamson To: linux-kernel@vger.kernel.org To: Joerg Roedel Cc: Ashok Raj Cc: Dave Hansen Cc: Huang Ying Cc: CQ Tang Cc: Thomas Gleixner Cc: Ingo Molnar Cc: H. Peter Anvin Cc: Andy Lutomirski Cc: Rik van Riel Cc: Kees Cook Cc: Andrew Morton Cc: Michal Hocko Cc: Paul E. McKenney Cc: Vegard Nossum Cc: x86@kernel.org Cc: linux-mm@kvack.org Cc: iommu@lists.linux-foundation.org Cc: David Woodhouse CC: Jean-Phillipe Brucker Signed-off-by: Ashok Raj Signed-off-by: Lu Baolu --- drivers/iommu/intel-svm.c | 27 +++++++++++++++++++++++++-- include/linux/intel-iommu.h | 5 ++++- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c index ed1cf7c..1456092 100644 --- a/drivers/iommu/intel-svm.c +++ b/drivers/iommu/intel-svm.c @@ -283,6 +283,24 @@ static const struct mmu_notifier_ops intel_mmuops = { static DEFINE_MUTEX(pasid_mutex); +static int intel_init_mm_inval_range(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct kernel_mmu_address_range *range; + struct intel_svm *svm = container_of(nb, struct intel_svm, init_mm_nb); + unsigned long start, end; + + if (action == KERNEL_MMU_INVALIDATE_RANGE) { + range = data; + start = range->start; + end = range->end; + + intel_flush_svm_range(svm, start, + (end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0, 0); + } + return 0; +} + int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ops *ops) { struct intel_iommu *iommu = intel_svm_device_to_iommu(dev); @@ -382,12 +400,12 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ goto out; } svm->pasid = ret; - svm->notifier.ops = &intel_mmuops; svm->mm = mm; svm->flags = flags; INIT_LIST_HEAD_RCU(&svm->devs); ret = -ENOMEM; if (mm) { + svm->notifier.ops = &intel_mmuops; ret = mmu_notifier_register(&svm->notifier, mm); if (ret) { idr_remove(&svm->iommu->pasid_idr, svm->pasid); @@ -396,8 +414,11 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ goto out; } iommu->pasid_table[svm->pasid].val = (u64)__pa(mm->pgd) | 1; - } else + } else { + svm->init_mm_nb.notifier_call = intel_init_mm_inval_range; + kernel_mmu_notifier_register(&svm->init_mm_nb); iommu->pasid_table[svm->pasid].val = (u64)__pa(init_mm.pgd) | 1 | (1ULL << 11); + } wmb(); /* In caching mode, we still have to flush with PASID 0 when * a PASID table entry becomes present. Not entirely clear @@ -464,6 +485,8 @@ int intel_svm_unbind_mm(struct device *dev, int pasid) idr_remove(&svm->iommu->pasid_idr, svm->pasid); if (svm->mm) mmu_notifier_unregister(&svm->notifier, svm->mm); + else + kernel_mmu_notifier_unregister(&svm->init_mm_nb); /* We mandate that no page faults may be outstanding * for the PASID when intel_svm_unbind_mm() is called. diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h index f3274d9..5cf83db 100644 --- a/include/linux/intel-iommu.h +++ b/include/linux/intel-iommu.h @@ -478,7 +478,10 @@ struct intel_svm_dev { }; struct intel_svm { - struct mmu_notifier notifier; + union { + struct mmu_notifier notifier; + struct notifier_block init_mm_nb; + }; struct mm_struct *mm; struct intel_iommu *iommu; int flags; -- 2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lu Baolu Subject: [PATCH 2/2] iommu/vt-d: Register kernel MMU notifier to manage IOTLB/DEVTLB Date: Thu, 14 Dec 2017 09:02:46 +0800 Message-ID: <1513213366-22594-3-git-send-email-baolu.lu@linux.intel.com> References: <1513213366-22594-1-git-send-email-baolu.lu@linux.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1513213366-22594-1-git-send-email-baolu.lu-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org To: Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" , Alex Williamson , Joerg Roedel , David Woodhouse Cc: Rik van Riel , Michal Hocko , Dave Hansen , x86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org, Vegard Nossum , Andy Lutomirski , Huang Ying , Andrew Morton , "Paul E . McKenney" , Kees Cook List-Id: iommu@lists.linux-foundation.org From: Ashok Raj When a kernel client calls intel_svm_bind_mm() and gets a valid supervisor PASID, the memory mapping of init_mm will be shared between CPUs and device. IOMMU has to track the changes to this memory mapping, and get notified whenever a TLB flush is needed. Otherwise, the device TLB will be stale compared to that on the cpu for kernel mappings. This is similar to what have been done for user space registrations via mmu_notifier_register() api's. To: Alex Williamson To: linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Joerg Roedel Cc: Ashok Raj Cc: Dave Hansen Cc: Huang Ying Cc: CQ Tang Cc: Thomas Gleixner Cc: Ingo Molnar Cc: H. Peter Anvin Cc: Andy Lutomirski Cc: Rik van Riel Cc: Kees Cook Cc: Andrew Morton Cc: Michal Hocko Cc: Paul E. McKenney Cc: Vegard Nossum Cc: x86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org Cc: linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Cc: David Woodhouse CC: Jean-Phillipe Brucker Signed-off-by: Ashok Raj Signed-off-by: Lu Baolu --- drivers/iommu/intel-svm.c | 27 +++++++++++++++++++++++++-- include/linux/intel-iommu.h | 5 ++++- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c index ed1cf7c..1456092 100644 --- a/drivers/iommu/intel-svm.c +++ b/drivers/iommu/intel-svm.c @@ -283,6 +283,24 @@ static const struct mmu_notifier_ops intel_mmuops = { static DEFINE_MUTEX(pasid_mutex); +static int intel_init_mm_inval_range(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct kernel_mmu_address_range *range; + struct intel_svm *svm = container_of(nb, struct intel_svm, init_mm_nb); + unsigned long start, end; + + if (action == KERNEL_MMU_INVALIDATE_RANGE) { + range = data; + start = range->start; + end = range->end; + + intel_flush_svm_range(svm, start, + (end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0, 0); + } + return 0; +} + int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ops *ops) { struct intel_iommu *iommu = intel_svm_device_to_iommu(dev); @@ -382,12 +400,12 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ goto out; } svm->pasid = ret; - svm->notifier.ops = &intel_mmuops; svm->mm = mm; svm->flags = flags; INIT_LIST_HEAD_RCU(&svm->devs); ret = -ENOMEM; if (mm) { + svm->notifier.ops = &intel_mmuops; ret = mmu_notifier_register(&svm->notifier, mm); if (ret) { idr_remove(&svm->iommu->pasid_idr, svm->pasid); @@ -396,8 +414,11 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ goto out; } iommu->pasid_table[svm->pasid].val = (u64)__pa(mm->pgd) | 1; - } else + } else { + svm->init_mm_nb.notifier_call = intel_init_mm_inval_range; + kernel_mmu_notifier_register(&svm->init_mm_nb); iommu->pasid_table[svm->pasid].val = (u64)__pa(init_mm.pgd) | 1 | (1ULL << 11); + } wmb(); /* In caching mode, we still have to flush with PASID 0 when * a PASID table entry becomes present. Not entirely clear @@ -464,6 +485,8 @@ int intel_svm_unbind_mm(struct device *dev, int pasid) idr_remove(&svm->iommu->pasid_idr, svm->pasid); if (svm->mm) mmu_notifier_unregister(&svm->notifier, svm->mm); + else + kernel_mmu_notifier_unregister(&svm->init_mm_nb); /* We mandate that no page faults may be outstanding * for the PASID when intel_svm_unbind_mm() is called. diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h index f3274d9..5cf83db 100644 --- a/include/linux/intel-iommu.h +++ b/include/linux/intel-iommu.h @@ -478,7 +478,10 @@ struct intel_svm_dev { }; struct intel_svm { - struct mmu_notifier notifier; + union { + struct mmu_notifier notifier; + struct notifier_block init_mm_nb; + }; struct mm_struct *mm; struct intel_iommu *iommu; int flags; -- 2.7.4