[PATCH 0/4] Patches to support ring0 SVM and devtlb

[PATCH 0/4] Patches to support ring0 SVM and devtlb

[Ashok Raj]

Hi

Sorry for resending.. iommu list email was mistyped :-(

The first 2 patches in the series fix some simple bugs in Intel vt-d driver.
The 3rd patch Adds support for kmem notify required to support ring0 SVM.
4th patch uses the hooks to perform device tlb invalidations.

Ashok Raj (3):
  iommu/vt-d: IOMMU Page Request needs to check if address is canonical.
  iommu/vt-d: Avoid calling virt_to_phys() on null pointer
  iommu/vt-d: Hooks to invalidate iotlb/devtlb when using supervisor
    PASID's.

Huang Ying (1):
  mm: Add kernel MMU notifier to manage remote TLB

 arch/x86/include/asm/tlbflush.h |  1 +
 arch/x86/mm/tlb.c               |  1 +
 drivers/iommu/intel-iommu.c     |  3 ++-
 drivers/iommu/intel-svm.c       | 43 +++++++++++++++++++++++++++++++++++++++--
 include/linux/intel-iommu.h     |  5 ++++-
 include/linux/mmu_notifier.h    | 33 +++++++++++++++++++++++++++++++
 mm/mmu_notifier.c               | 25 ++++++++++++++++++++++++
 7 files changed, 107 insertions(+), 4 deletions(-)

-- 
2.7.4

[PATCH 1/4] iommu/vt-d: IOMMU Page Request needs to check if address is canonical.

[Ashok Raj]

Page Request from devices that support device-tlb would request translation
to pre-cache them in device to avoid overhead of IOMMU lookups.

IOMMU needs to check for canonicallity of the address before performing
page-fault processing.

To: Joerg Roedel <joro@8bytes.org>
To: linux-kernel@vger.kernel.org>
Cc: iommu@lists.linux-foundation.org
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Jacob Pan <jacob.jun.pan@intel.com>
Cc: Ashok Raj <ashok.raj@intel.com>

Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Reported-by: Sudeep Dutt <sudeep.dutt@intel.com>
---
 drivers/iommu/intel-svm.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c
index f167c0d..0c9f077 100644
--- a/drivers/iommu/intel-svm.c
+++ b/drivers/iommu/intel-svm.c
@@ -24,6 +24,7 @@
 #include <linux/pci-ats.h>
 #include <linux/dmar.h>
 #include <linux/interrupt.h>
+#include <asm/page.h>
 
 static irqreturn_t prq_event_thread(int irq, void *d);
 
@@ -555,6 +556,14 @@ static bool access_error(struct vm_area_struct *vma, struct page_req_dsc *req)
 	return (requested & ~vma->vm_flags) != 0;
 }
 
+static bool is_canonical_address(u64 addr)
+{
+	int shift = 64 - (__VIRTUAL_MASK_SHIFT + 1);
+	long saddr = (long) addr;
+
+	return (((saddr << shift) >> shift) == saddr);
+}
+
 static irqreturn_t prq_event_thread(int irq, void *d)
 {
 	struct intel_iommu *iommu = d;
@@ -612,6 +621,11 @@ static irqreturn_t prq_event_thread(int irq, void *d)
 		/* If the mm is already defunct, don't handle faults. */
 		if (!mmget_not_zero(svm->mm))
 			goto bad_req;
+
+		/* If address is not canonical, return invalid response */
+		if (!is_canonical_address(address))
+			goto bad_req;
+
 		down_read(&svm->mm->mmap_sem);
 		vma = find_extend_vma(svm->mm, address);
 		if (!vma || address < vma->vm_start)
-- 
2.7.4

[PATCH 2/4] iommu/vt-d: Avoid calling virt_to_phys() on null pointer

[Ashok Raj]

New kernels with debug show panic() from __phys_addr() checks. Avoid
calling virt_to_phys()  when pasid_state_tbl pointer is null

To: Joerg Roedel <joro@8bytes.org>
To: linux-kernel@vger.kernel.org>
Cc: iommu@lists.linux-foundation.org
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Jacob Pan <jacob.jun.pan@intel.com>
Cc: Ashok Raj <ashok.raj@intel.com>

Signed-off-by: Ashok Raj <ashok.raj@intel.com>
---
 drivers/iommu/intel-iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 687f18f..5c6118d 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -5341,7 +5341,8 @@ int intel_iommu_enable_pasid(struct intel_iommu *iommu, struct intel_svm_dev *sd
 	sdev->sid = PCI_DEVID(info->bus, info->devfn);
 
 	if (!(ctx_lo & CONTEXT_PASIDE)) {
-		context[1].hi = (u64)virt_to_phys(iommu->pasid_state_table);
+		if (iommu->pasid_state_table)
+			context[1].hi = (u64)virt_to_phys(iommu->pasid_state_table);
 		context[1].lo = (u64)virt_to_phys(iommu->pasid_table) |
 			intel_iommu_get_pts(iommu);
 
-- 
2.7.4

[PATCH 3/4] mm: Add kernel MMU notifier to manage remote TLB

[Ashok Raj]

From: Huang Ying <ying.huang@intel.com>

Shared Virtual Memory (SVM) devices have TLBs that cache entries from
the CPU's page tables.  We need SVM device drivers to flush them at
the same time that we flush the CPU TLBs.  We can use the existing MMU
notifiers for userspace updates, but we lack a mechanism to get
notified when kernel page tables are updated.

To implement the MMU notification mechanism for the kernel address
space, a kernel MMU notifier chain is defined, and will be called when
the CPU TLB is flushed for the kernel address space.  The IOMMU SVM
driver can register on the notifier chain to flush the device TLBs
when necessary.

To: linux-kernel@vger.kernel.org
To: Joerg Roedel <joro@8bytes.org>
Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: CQ Tang <cq.tang@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Vegard Nossum <vegard.nossum@oracle.com>
Cc: x86@kernel.org
Cc: linux-mm@kvack.org
Cc: iommu@lists.linux-foundation.org
Cc: David Woodhouse <dwmw2@infradead.org>
CC: Jean-Phillipe Brucker <jean-philippe.brucker@arm.com>
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
---
 arch/x86/include/asm/tlbflush.h |  1 +
 arch/x86/mm/tlb.c               |  1 +
 include/linux/mmu_notifier.h    | 33 +++++++++++++++++++++++++++++++++
 mm/mmu_notifier.c               | 25 +++++++++++++++++++++++++
 4 files changed, 60 insertions(+)

diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 50ea348..f5fd0b8 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -3,6 +3,7 @@
 
 #include <linux/mm.h>
 #include <linux/sched.h>
+#include <linux/mmu_notifier.h>
 
 #include <asm/processor.h>
 #include <asm/cpufeature.h>
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 014d07a..6dea8e9 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -314,6 +314,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 		info.end = end;
 		on_each_cpu(do_kernel_range_flush, &info, 1);
 	}
+	kernel_mmu_notifier_invalidate_range(start, end);
 }
 
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h
index c91b3bc..4a96089 100644
--- a/include/linux/mmu_notifier.h
+++ b/include/linux/mmu_notifier.h
@@ -418,6 +418,25 @@ extern void mmu_notifier_call_srcu(struct rcu_head *rcu,
 				   void (*func)(struct rcu_head *rcu));
 extern void mmu_notifier_synchronize(void);
 
+struct kernel_mmu_address_range {
+	unsigned long start;
+	unsigned long end;
+};
+
+/*
+ * Before the virtual address range managed by kernel (vmalloc/kmap)
+ * is reused, That is, remapped to the new physical addresses, the
+ * kernel MMU notifier will be called with KERNEL_MMU_INVALIDATE_RANGE
+ * and struct kernel_mmu_address_range as parameters.  This is used to
+ * manage the remote TLB.
+ */
+#define KERNEL_MMU_INVALIDATE_RANGE		1
+extern int kernel_mmu_notifier_register(struct notifier_block *nb);
+extern int kernel_mmu_notifier_unregister(struct notifier_block *nb);
+
+extern int kernel_mmu_notifier_invalidate_range(unsigned long start,
+						unsigned long end);
+
 #else /* CONFIG_MMU_NOTIFIER */
 
 static inline void mmu_notifier_release(struct mm_struct *mm)
@@ -479,6 +498,20 @@ static inline void mmu_notifier_mm_destroy(struct mm_struct *mm)
 #define pudp_huge_clear_flush_notify pudp_huge_clear_flush
 #define set_pte_at_notify set_pte_at
 
+static inline int kernel_mmu_notifier_register(struct notifier_block *nb)
+{
+	return 0;
+}
+
+static inline int kernel_mmu_notifier_unregister(struct notifier_block *nb)
+{
+	return 0;
+}
+
+static inline void kernel_mmu_notifier_invalidate_range(unsigned long start,
+							unsigned long end)
+{
+}
 #endif /* CONFIG_MMU_NOTIFIER */
 
 #endif /* _LINUX_MMU_NOTIFIER_H */
diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c
index 54ca545..a919038 100644
--- a/mm/mmu_notifier.c
+++ b/mm/mmu_notifier.c
@@ -400,3 +400,28 @@ void mmu_notifier_unregister_no_release(struct mmu_notifier *mn,
 	mmdrop(mm);
 }
 EXPORT_SYMBOL_GPL(mmu_notifier_unregister_no_release);
+
+static ATOMIC_NOTIFIER_HEAD(kernel_mmu_notifier_list);
+
+int kernel_mmu_notifier_register(struct notifier_block *nb)
+{
+	return atomic_notifier_chain_register(&kernel_mmu_notifier_list, nb);
+}
+
+int kernel_mmu_notifier_unregister(struct notifier_block *nb)
+{
+	return atomic_notifier_chain_unregister(&kernel_mmu_notifier_list, nb);
+}
+
+int kernel_mmu_notifier_invalidate_range(unsigned long start,
+					 unsigned long end)
+{
+	struct kernel_mmu_address_range range = {
+		.start	= start,
+		.end	= end,
+	};
+
+	return atomic_notifier_call_chain(&kernel_mmu_notifier_list,
+					  KERNEL_MMU_INVALIDATE_RANGE,
+					  &range);
+}
-- 
2.7.4

[PATCH 4/4] iommu/vt-d: Hooks to invalidate iotlb/devtlb when using supervisor PASID's.

[Ashok Raj]

When a kernel client uses intel_svm_bind_mm() and requests a supervisor
PASID, IOMMU needs to track changes to these addresses. Otherwise the device
tlb will be stale compared to what's on the cpu for kernel mappings. This
is similar to what's done for user space registrations via
mmu_notifier_register() api's.

To: linux-kernel@vger.kernel.org
To: Joerg Roedel <joro@8bytes.org>
Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc:	Huang Ying <ying.huang@intel.com>
Cc: CQ Tang <cq.tang@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Vegard Nossum <vegard.nossum@oracle.com>
Cc: x86@kernel.org
Cc: linux-mm@kvack.org
Cc: iommu@lists.linux-foundation.org
Cc: David Woodhouse <dwmw2@infradead.org>
CC: Jean-Phillipe Brucker <jean-philippe.brucker@arm.com>

Signed-off-by: Ashok Raj <ashok.raj@intel.com>
---
 drivers/iommu/intel-svm.c   | 29 +++++++++++++++++++++++++++--
 include/linux/intel-iommu.h |  5 ++++-
 2 files changed, 31 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c
index 0c9f077..1758814 100644
--- a/drivers/iommu/intel-svm.c
+++ b/drivers/iommu/intel-svm.c
@@ -292,6 +292,26 @@ static const struct mmu_notifier_ops intel_mmuops = {
 
 static DEFINE_MUTEX(pasid_mutex);
 
+static int intel_init_mm_inval_range(struct notifier_block *nb,
+	unsigned long action, void *data)
+{
+	struct kernel_mmu_address_range *range;
+	struct intel_svm *svm = container_of(nb, struct intel_svm, init_mm_nb);
+	unsigned long start, end;
+	struct intel_iommu *iommu;
+
+	if (action == KERNEL_MMU_INVALIDATE_RANGE) {
+		range = data;
+		start = range->start;
+		end = range->end;
+		iommu = svm->iommu;
+
+		intel_flush_svm_range(svm, start,
+			(end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0, 0);
+	}
+	return 0;
+}
+
 int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ops *ops)
 {
 	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
@@ -391,12 +411,12 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_
 			goto out;
 		}
 		svm->pasid = ret;
-		svm->notifier.ops = &intel_mmuops;
 		svm->mm = mm;
 		svm->flags = flags;
 		INIT_LIST_HEAD_RCU(&svm->devs);
 		ret = -ENOMEM;
 		if (mm) {
+			svm->notifier.ops = &intel_mmuops;
 			ret = mmu_notifier_register(&svm->notifier, mm);
 			if (ret) {
 				idr_remove(&svm->iommu->pasid_idr, svm->pasid);
@@ -405,8 +425,11 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_
 				goto out;
 			}
 			iommu->pasid_table[svm->pasid].val = (u64)__pa(mm->pgd) | 1;
-		} else
+		} else {
+			svm->init_mm_nb.notifier_call = intel_init_mm_inval_range;
+			kernel_mmu_notifier_register(&svm->init_mm_nb);
 			iommu->pasid_table[svm->pasid].val = (u64)__pa(init_mm.pgd) | 1 | (1ULL << 11);
+		}
 		wmb();
 		/* In caching mode, we still have to flush with PASID 0 when
 		 * a PASID table entry becomes present. Not entirely clear
@@ -471,6 +494,8 @@ int intel_svm_unbind_mm(struct device *dev, int pasid)
 					idr_remove(&svm->iommu->pasid_idr, svm->pasid);
 					if (svm->mm)
 						mmu_notifier_unregister(&svm->notifier, svm->mm);
+					else
+						kernel_mmu_notifier_unregister(&svm->init_mm_nb);
 
 					/* We mandate that no page faults may be outstanding
 					 * for the PASID when intel_svm_unbind_mm() is called.
diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h
index 485a5b4..d6019b4 100644
--- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h
@@ -477,7 +477,10 @@ struct intel_svm_dev {
 };
 
 struct intel_svm {
-	struct mmu_notifier notifier;
+	union {
+		struct mmu_notifier notifier;
+		struct notifier_block init_mm_nb;
+	};
 	struct mm_struct *mm;
 	struct intel_iommu *iommu;
 	int flags;
-- 
2.7.4

Re: [PATCH 0/4] Patches to support ring0 SVM and devtlb

[Raj, Ashok]

Hi Joerg

I haven't received any update to this patchset.. 

Could you help get this merged through your tree? we have tested this
series internally.

Cheers,
Ashok

On Tue, Aug 08, 2017 at 01:29:26PM -0700, Ashok Raj wrote:
> Hi
> 
> Sorry for resending.. iommu list email was mistyped :-(
> 
> The first 2 patches in the series fix some simple bugs in Intel vt-d driver.
> The 3rd patch Adds support for kmem notify required to support ring0 SVM.
> 4th patch uses the hooks to perform device tlb invalidations.
> 
> Ashok Raj (3):
>   iommu/vt-d: IOMMU Page Request needs to check if address is canonical.
>   iommu/vt-d: Avoid calling virt_to_phys() on null pointer
>   iommu/vt-d: Hooks to invalidate iotlb/devtlb when using supervisor
>     PASID's.
> 
> Huang Ying (1):
>   mm: Add kernel MMU notifier to manage remote TLB
> 
>  arch/x86/include/asm/tlbflush.h |  1 +
>  arch/x86/mm/tlb.c               |  1 +
>  drivers/iommu/intel-iommu.c     |  3 ++-
>  drivers/iommu/intel-svm.c       | 43 +++++++++++++++++++++++++++++++++++++++--
>  include/linux/intel-iommu.h     |  5 ++++-
>  include/linux/mmu_notifier.h    | 33 +++++++++++++++++++++++++++++++
>  mm/mmu_notifier.c               | 25 ++++++++++++++++++++++++
>  7 files changed, 107 insertions(+), 4 deletions(-)
> 
> -- 
> 2.7.4
> 

Re: [PATCH 0/4] Patches to support ring0 SVM and devtlb

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 287 bytes --]

On Tue, 2017-08-15 at 07:59 -0700, Raj, Ashok wrote:
> Hi Joerg
> 
> I haven't received any update to this patchset.. 
> 
> Could you help get this merged through your tree? we have tested this
> series internally.

For all four:

Acked-by: David Woodhouse <dwmw@amazon.co.uk>

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4938 bytes --]

Re: [PATCH 0/4] Patches to support ring0 SVM and devtlb

[Joerg Roedel]

Hi Ashok,

On Tue, Aug 15, 2017 at 07:59:29AM -0700, Raj, Ashok wrote:
> I haven't received any update to this patchset.. 
> 
> Could you help get this merged through your tree? we have tested this
> series internally.

The mm-notifier patch needs acks or rb from mm-developers before I can
merge this patch-set.


Regards,

	Joerg

Re: [PATCH 0/4] Patches to support ring0 SVM and devtlb

[Raj, Ashok]

Thanks Joerg

On Fri, Aug 18, 2017 at 11:35:21AM +0200, Joerg Roedel wrote:
> Hi Ashok,
> 
> On Tue, Aug 15, 2017 at 07:59:29AM -0700, Raj, Ashok wrote:
> > I haven't received any update to this patchset.. 
> > 
> > Could you help get this merged through your tree? we have tested this
> > series internally.
> 
> The mm-notifier patch needs acks or rb from mm-developers before I can
> merge this patch-set.

someone from the mm list.. could you help ack/rb this patch?

Cheers,
Ashok

[PATCH 4/4] iommu/vt-d: Hooks to invalidate iotlb/devtlb when using supervisor PASID's.

[Ashok Raj]

When a kernel client uses intel_svm_bind_mm() and requests a supervisor
PASID, IOMMU needs to track changes to these addresses. Otherwise the device
tlb will be stale compared to what's on the cpu for kernel mappings. This
is similar to what's done for user space registrations via
mmu_notifier_register() api's.

To: linux-kernel@vger.kernel.org
To: Joerg Roedel <joro@8bytes.org>
Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc:	Huang Ying <ying.huang@intel.com>
Cc: CQ Tang <cq.tang@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Vegard Nossum <vegard.nossum@oracle.com>
Cc: x86@kernel.org
Cc: linux-mm@kvack.org
Cc: iommu@lists-foundation.org
Cc: David Woodhouse <dwmw2@infradead.org>
CC: Jean-Phillipe Brucker <jean-philippe.brucker@arm.com>

Signed-off-by: Ashok Raj <ashok.raj@intel.com>
---
 drivers/iommu/intel-svm.c   | 29 +++++++++++++++++++++++++++--
 include/linux/intel-iommu.h |  5 ++++-
 2 files changed, 31 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c
index 0c9f077..1758814 100644
--- a/drivers/iommu/intel-svm.c
+++ b/drivers/iommu/intel-svm.c
@@ -292,6 +292,26 @@ static const struct mmu_notifier_ops intel_mmuops = {
 
 static DEFINE_MUTEX(pasid_mutex);
 
+static int intel_init_mm_inval_range(struct notifier_block *nb,
+	unsigned long action, void *data)
+{
+	struct kernel_mmu_address_range *range;
+	struct intel_svm *svm = container_of(nb, struct intel_svm, init_mm_nb);
+	unsigned long start, end;
+	struct intel_iommu *iommu;
+
+	if (action == KERNEL_MMU_INVALIDATE_RANGE) {
+		range = data;
+		start = range->start;
+		end = range->end;
+		iommu = svm->iommu;
+
+		intel_flush_svm_range(svm, start,
+			(end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0, 0);
+	}
+	return 0;
+}
+
 int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ops *ops)
 {
 	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
@@ -391,12 +411,12 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_
 			goto out;
 		}
 		svm->pasid = ret;
-		svm->notifier.ops = &intel_mmuops;
 		svm->mm = mm;
 		svm->flags = flags;
 		INIT_LIST_HEAD_RCU(&svm->devs);
 		ret = -ENOMEM;
 		if (mm) {
+			svm->notifier.ops = &intel_mmuops;
 			ret = mmu_notifier_register(&svm->notifier, mm);
 			if (ret) {
 				idr_remove(&svm->iommu->pasid_idr, svm->pasid);
@@ -405,8 +425,11 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_
 				goto out;
 			}
 			iommu->pasid_table[svm->pasid].val = (u64)__pa(mm->pgd) | 1;
-		} else
+		} else {
+			svm->init_mm_nb.notifier_call = intel_init_mm_inval_range;
+			kernel_mmu_notifier_register(&svm->init_mm_nb);
 			iommu->pasid_table[svm->pasid].val = (u64)__pa(init_mm.pgd) | 1 | (1ULL << 11);
+		}
 		wmb();
 		/* In caching mode, we still have to flush with PASID 0 when
 		 * a PASID table entry becomes present. Not entirely clear
@@ -471,6 +494,8 @@ int intel_svm_unbind_mm(struct device *dev, int pasid)
 					idr_remove(&svm->iommu->pasid_idr, svm->pasid);
 					if (svm->mm)
 						mmu_notifier_unregister(&svm->notifier, svm->mm);
+					else
+						kernel_mmu_notifier_unregister(&svm->init_mm_nb);
 
 					/* We mandate that no page faults may be outstanding
 					 * for the PASID when intel_svm_unbind_mm() is called.
diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h
index 485a5b4..d6019b4 100644
--- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h
@@ -477,7 +477,10 @@ struct intel_svm_dev {
 };
 
 struct intel_svm {
-	struct mmu_notifier notifier;
+	union {
+		struct mmu_notifier notifier;
+		struct notifier_block init_mm_nb;
+	};
 	struct mm_struct *mm;
 	struct intel_iommu *iommu;
 	int flags;
-- 
2.7.4