* [PATCH] mm: add account_locked_vm utility function
@ 2019-05-03 20:16 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-03 20:16 UTC (permalink / raw)
To: akpm
Cc: Daniel Jordan, Alan Tull, Alexey Kardashevskiy, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper. Standardize the debug prints, which vary
slightly. Error codes stay the same, so user-visible behavior does too.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Cc: Alan Tull <atull@kernel.org>
Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
welcome.
Andrew, this one patch replaces these six from [1]:
mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
That series converts locked_vm to an atomic, but on closer inspection causes at
least one accounting race in mremap, and fixing it just for this type
conversion came with too much ugly in the core mm to justify, especially when
the right long-term fix is making these drivers use pinned_vm instead.
Christophe's suggestion of cmpxchg[2] does prevent the races he
mentioned for pinned_vm, but others would still remain. In perf_mmap
and the hfi1 driver, pinned_vm is checked against the rlimit racily and
then later increased when the pinned_vm originally read may have gone
stale. Any fixes for that, that I could think of, seem about as good as
what's there now, so I left it. I have a patch that uses cmpxchg with
pinned_vm if others feel strongly that the aforementioned races should
be fixed.
Daniel
[1] https://lore.kernel.org/linux-mm/20190402204158.27582-1-daniel.m.jordan@oracle.com/
[2] https://lore.kernel.org/linux-mm/964bd5b0-f1e5-7bf0-5c58-18e75c550841@c-s.fr/
arch/powerpc/kvm/book3s_64_vio.c | 44 +++---------------------
arch/powerpc/mm/mmu_context_iommu.c | 41 +++-------------------
drivers/fpga/dfl-afu-dma-region.c | 53 +++--------------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 52 +++++-----------------------
drivers/vfio/vfio_iommu_type1.c | 23 ++++---------
include/linux/mm.h | 19 +++++++++++
mm/util.c | 48 ++++++++++++++++++++++++++
7 files changed, 94 insertions(+), 186 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index f02b04973710..f7d37fa6003a 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -277,7 +241,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -303,7 +267,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -359,7 +323,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
index 8330f135294f..9e7001a70570 100644
--- a/arch/powerpc/mm/mmu_context_iommu.c
+++ b/arch/powerpc/mm/mmu_context_iommu.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -215,7 +182,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -315,7 +282,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index e18a786fc943..059438e17193 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 6b64e45a5269..d39a1b830d82 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,49 +35,13 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
+static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
+ bool inc)
{
- long ret = 0, locked, lock_limit;
-
if (WARN_ON_ONCE(!mm))
return -EPERM;
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
+ return account_locked_vm(mm, npages, inc);
}
/*
@@ -336,7 +301,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = tce_account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +320,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ tce_account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -658,7 +623,8 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = tce_account_locked_vm(container->mm, table_size >> PAGE_SHIFT,
+ true);
if (ret)
return ret;
@@ -677,7 +643,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ tce_account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index d0f731c9920a..15ac76171ccd 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
return -ESRCH; /* process exited */
ret = down_write_killable(&mm->mmap_sem);
- if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
+ if (ret)
+ goto out;
- if (!ret)
- mm->locked_vm += npage;
-
- up_write(&mm->mmap_sem);
- }
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
+ up_write(&mm->mmap_sem);
+out:
if (async)
mmput(mm);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6b10c21630f5..7134e55ca23f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1521,6 +1521,25 @@ static inline long get_user_pages_longterm(unsigned long start,
int get_user_pages_fast(unsigned long start, int nr_pages, int write,
struct page **pages);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
+static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
+ bool inc)
+{
+ int ret;
+
+ if (pages == 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index 43a2984bccaa..552302665bc2 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -6,6 +6,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -346,6 +347,53 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * 0 if @mm is NULL (can happen for example if the task is exiting)
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+out:
+ pr_debug("%s: [%d] %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: 37624b58542fb9f2d9a70e6ea006ef8a5f66c30b
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH] mm: add account_locked_vm utility function
@ 2019-05-03 20:16 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-03 20:16 UTC (permalink / raw)
To: akpm
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
Daniel Jordan, linux-mm, Alex Williamson, Jason Gunthorpe,
Moritz Fischer, Steve Sistare, Christoph Lameter, linuxppc-dev,
Wu Hao
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper. Standardize the debug prints, which vary
slightly. Error codes stay the same, so user-visible behavior does too.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Cc: Alan Tull <atull@kernel.org>
Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
welcome.
Andrew, this one patch replaces these six from [1]:
mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
That series converts locked_vm to an atomic, but on closer inspection causes at
least one accounting race in mremap, and fixing it just for this type
conversion came with too much ugly in the core mm to justify, especially when
the right long-term fix is making these drivers use pinned_vm instead.
Christophe's suggestion of cmpxchg[2] does prevent the races he
mentioned for pinned_vm, but others would still remain. In perf_mmap
and the hfi1 driver, pinned_vm is checked against the rlimit racily and
then later increased when the pinned_vm originally read may have gone
stale. Any fixes for that, that I could think of, seem about as good as
what's there now, so I left it. I have a patch that uses cmpxchg with
pinned_vm if others feel strongly that the aforementioned races should
be fixed.
Daniel
[1] https://lore.kernel.org/linux-mm/20190402204158.27582-1-daniel.m.jordan@oracle.com/
[2] https://lore.kernel.org/linux-mm/964bd5b0-f1e5-7bf0-5c58-18e75c550841@c-s.fr/
arch/powerpc/kvm/book3s_64_vio.c | 44 +++---------------------
arch/powerpc/mm/mmu_context_iommu.c | 41 +++-------------------
drivers/fpga/dfl-afu-dma-region.c | 53 +++--------------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 52 +++++-----------------------
drivers/vfio/vfio_iommu_type1.c | 23 ++++---------
include/linux/mm.h | 19 +++++++++++
mm/util.c | 48 ++++++++++++++++++++++++++
7 files changed, 94 insertions(+), 186 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index f02b04973710..f7d37fa6003a 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -277,7 +241,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -303,7 +267,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -359,7 +323,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
index 8330f135294f..9e7001a70570 100644
--- a/arch/powerpc/mm/mmu_context_iommu.c
+++ b/arch/powerpc/mm/mmu_context_iommu.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -215,7 +182,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -315,7 +282,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index e18a786fc943..059438e17193 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 6b64e45a5269..d39a1b830d82 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,49 +35,13 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
+static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
+ bool inc)
{
- long ret = 0, locked, lock_limit;
-
if (WARN_ON_ONCE(!mm))
return -EPERM;
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
+ return account_locked_vm(mm, npages, inc);
}
/*
@@ -336,7 +301,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = tce_account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +320,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ tce_account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -658,7 +623,8 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = tce_account_locked_vm(container->mm, table_size >> PAGE_SHIFT,
+ true);
if (ret)
return ret;
@@ -677,7 +643,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ tce_account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index d0f731c9920a..15ac76171ccd 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
return -ESRCH; /* process exited */
ret = down_write_killable(&mm->mmap_sem);
- if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
+ if (ret)
+ goto out;
- if (!ret)
- mm->locked_vm += npage;
-
- up_write(&mm->mmap_sem);
- }
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
+ up_write(&mm->mmap_sem);
+out:
if (async)
mmput(mm);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6b10c21630f5..7134e55ca23f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1521,6 +1521,25 @@ static inline long get_user_pages_longterm(unsigned long start,
int get_user_pages_fast(unsigned long start, int nr_pages, int write,
struct page **pages);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
+static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
+ bool inc)
+{
+ int ret;
+
+ if (pages == 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index 43a2984bccaa..552302665bc2 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -6,6 +6,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -346,6 +347,53 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * 0 if @mm is NULL (can happen for example if the task is exiting)
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+out:
+ pr_debug("%s: [%d] %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: 37624b58542fb9f2d9a70e6ea006ef8a5f66c30b
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH] mm: add account_locked_vm utility function
@ 2019-05-03 20:16 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-03 20:16 UTC (permalink / raw)
To: akpm
Cc: Daniel Jordan, Alan Tull, Alexey Kardashevskiy, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper. Standardize the debug prints, which vary
slightly. Error codes stay the same, so user-visible behavior does too.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Cc: Alan Tull <atull@kernel.org>
Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
welcome.
Andrew, this one patch replaces these six from [1]:
mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
That series converts locked_vm to an atomic, but on closer inspection causes at
least one accounting race in mremap, and fixing it just for this type
conversion came with too much ugly in the core mm to justify, especially when
the right long-term fix is making these drivers use pinned_vm instead.
Christophe's suggestion of cmpxchg[2] does prevent the races he
mentioned for pinned_vm, but others would still remain. In perf_mmap
and the hfi1 driver, pinned_vm is checked against the rlimit racily and
then later increased when the pinned_vm originally read may have gone
stale. Any fixes for that, that I could think of, seem about as good as
what's there now, so I left it. I have a patch that uses cmpxchg with
pinned_vm if others feel strongly that the aforementioned races should
be fixed.
Daniel
[1] https://lore.kernel.org/linux-mm/20190402204158.27582-1-daniel.m.jordan@oracle.com/
[2] https://lore.kernel.org/linux-mm/964bd5b0-f1e5-7bf0-5c58-18e75c550841@c-s.fr/
arch/powerpc/kvm/book3s_64_vio.c | 44 +++---------------------
arch/powerpc/mm/mmu_context_iommu.c | 41 +++-------------------
drivers/fpga/dfl-afu-dma-region.c | 53 +++--------------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 52 +++++-----------------------
drivers/vfio/vfio_iommu_type1.c | 23 ++++---------
include/linux/mm.h | 19 +++++++++++
mm/util.c | 48 ++++++++++++++++++++++++++
7 files changed, 94 insertions(+), 186 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index f02b04973710..f7d37fa6003a 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -277,7 +241,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -303,7 +267,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -359,7 +323,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
index 8330f135294f..9e7001a70570 100644
--- a/arch/powerpc/mm/mmu_context_iommu.c
+++ b/arch/powerpc/mm/mmu_context_iommu.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa = MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -215,7 +182,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -315,7 +282,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index e18a786fc943..059438e17193 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 6b64e45a5269..d39a1b830d82 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,49 +35,13 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
+static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
+ bool inc)
{
- long ret = 0, locked, lock_limit;
-
if (WARN_ON_ONCE(!mm))
return -EPERM;
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
+ return account_locked_vm(mm, npages, inc);
}
/*
@@ -336,7 +301,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = tce_account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +320,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ tce_account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -658,7 +623,8 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = tce_account_locked_vm(container->mm, table_size >> PAGE_SHIFT,
+ true);
if (ret)
return ret;
@@ -677,7 +643,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ tce_account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index d0f731c9920a..15ac76171ccd 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
return -ESRCH; /* process exited */
ret = down_write_killable(&mm->mmap_sem);
- if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
+ if (ret)
+ goto out;
- if (!ret)
- mm->locked_vm += npage;
-
- up_write(&mm->mmap_sem);
- }
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
+ up_write(&mm->mmap_sem);
+out:
if (async)
mmput(mm);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6b10c21630f5..7134e55ca23f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1521,6 +1521,25 @@ static inline long get_user_pages_longterm(unsigned long start,
int get_user_pages_fast(unsigned long start, int nr_pages, int write,
struct page **pages);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
+static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
+ bool inc)
+{
+ int ret;
+
+ if (pages = 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index 43a2984bccaa..552302665bc2 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -6,6 +6,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -346,6 +347,53 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * 0 if @mm is NULL (can happen for example if the task is exiting)
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+out:
+ pr_debug("%s: [%d] %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: 37624b58542fb9f2d9a70e6ea006ef8a5f66c30b
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
2019-05-03 20:16 ` Daniel Jordan
(?)
@ 2019-05-03 23:28 ` Jason Gunthorpe
-1 siblings, 0 replies; 42+ messages in thread
From: Jason Gunthorpe @ 2019-05-03 23:28 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alan Tull, Alexey Kardashevskiy, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On Fri, May 03, 2019 at 01:16:30PM -0700, Daniel Jordan wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly. Error codes stay the same, so user-visible behavior does too.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
>
> Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
> welcome.
>
> Andrew, this one patch replaces these six from [1]:
>
> mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
>
> That series converts locked_vm to an atomic, but on closer inspection causes at
> least one accounting race in mremap, and fixing it just for this type
> conversion came with too much ugly in the core mm to justify, especially when
> the right long-term fix is making these drivers use pinned_vm instead.
Did we ever decide what to do here? Should all these drivers be
switched to pinned_vm anyhow?
Jason
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-03 23:28 ` Jason Gunthorpe
0 siblings, 0 replies; 42+ messages in thread
From: Jason Gunthorpe @ 2019-05-03 23:28 UTC (permalink / raw)
To: Daniel Jordan
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
linux-mm, Alex Williamson, Moritz Fischer, Steve Sistare, akpm,
linuxppc-dev, Christoph Lameter, Wu Hao
On Fri, May 03, 2019 at 01:16:30PM -0700, Daniel Jordan wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly. Error codes stay the same, so user-visible behavior does too.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
>
> Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
> welcome.
>
> Andrew, this one patch replaces these six from [1]:
>
> mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
>
> That series converts locked_vm to an atomic, but on closer inspection causes at
> least one accounting race in mremap, and fixing it just for this type
> conversion came with too much ugly in the core mm to justify, especially when
> the right long-term fix is making these drivers use pinned_vm instead.
Did we ever decide what to do here? Should all these drivers be
switched to pinned_vm anyhow?
Jason
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-03 23:28 ` Jason Gunthorpe
0 siblings, 0 replies; 42+ messages in thread
From: Jason Gunthorpe @ 2019-05-03 23:28 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alan Tull, Alexey Kardashevskiy, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On Fri, May 03, 2019 at 01:16:30PM -0700, Daniel Jordan wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly. Error codes stay the same, so user-visible behavior does too.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
>
> Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
> welcome.
>
> Andrew, this one patch replaces these six from [1]:
>
> mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
>
> That series converts locked_vm to an atomic, but on closer inspection causes at
> least one accounting race in mremap, and fixing it just for this type
> conversion came with too much ugly in the core mm to justify, especially when
> the right long-term fix is making these drivers use pinned_vm instead.
Did we ever decide what to do here? Should all these drivers be
switched to pinned_vm anyhow?
Jason
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
2019-05-03 23:28 ` Jason Gunthorpe
(?)
@ 2019-05-07 3:09 ` Daniel Jordan
-1 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-07 3:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Daniel Jordan, akpm, Alan Tull, Alexey Kardashevskiy,
Alex Williamson, Benjamin Herrenschmidt, Christoph Lameter,
Christophe Leroy, Davidlohr Bueso, Mark Rutland,
Michael Ellerman, Moritz Fischer, Paul Mackerras, Steve Sistare,
Wu Hao, linux-mm, kvm, kvm-ppc, linuxppc-dev, linux-fpga,
linux-kernel
On Fri, May 03, 2019 at 11:28:22PM +0000, Jason Gunthorpe wrote:
> On Fri, May 03, 2019 at 01:16:30PM -0700, Daniel Jordan wrote:
> > Andrew, this one patch replaces these six from [1]:
> >
> > mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> > vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> >
> > That series converts locked_vm to an atomic, but on closer inspection causes at
> > least one accounting race in mremap, and fixing it just for this type
> > conversion came with too much ugly in the core mm to justify, especially when
> > the right long-term fix is making these drivers use pinned_vm instead.
>
> Did we ever decide what to do here? Should all these drivers be
> switched to pinned_vm anyhow?
Well, there were the concerns about switching in [1]. Alex, is there an
example of an application or library that would break or be exploitable? If
there were particular worries (qemu for vfio type1, for example), perhaps some
coordinated changes across the kernel and userspace would be possible,
especially given the amount of effort it's likely going to take to get the
locked_vm/pinned_vm accounting sorted out.
[1] https://lore.kernel.org/linux-mm/20190213130330.76ef1987@w520.home/
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-07 3:09 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-07 3:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
Daniel Jordan, linux-mm, Alex Williamson, Moritz Fischer,
Steve Sistare, akpm, linuxppc-dev, Christoph Lameter, Wu Hao
On Fri, May 03, 2019 at 11:28:22PM +0000, Jason Gunthorpe wrote:
> On Fri, May 03, 2019 at 01:16:30PM -0700, Daniel Jordan wrote:
> > Andrew, this one patch replaces these six from [1]:
> >
> > mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> > vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> >
> > That series converts locked_vm to an atomic, but on closer inspection causes at
> > least one accounting race in mremap, and fixing it just for this type
> > conversion came with too much ugly in the core mm to justify, especially when
> > the right long-term fix is making these drivers use pinned_vm instead.
>
> Did we ever decide what to do here? Should all these drivers be
> switched to pinned_vm anyhow?
Well, there were the concerns about switching in [1]. Alex, is there an
example of an application or library that would break or be exploitable? If
there were particular worries (qemu for vfio type1, for example), perhaps some
coordinated changes across the kernel and userspace would be possible,
especially given the amount of effort it's likely going to take to get the
locked_vm/pinned_vm accounting sorted out.
[1] https://lore.kernel.org/linux-mm/20190213130330.76ef1987@w520.home/
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-07 3:09 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-07 3:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Daniel Jordan, akpm, Alan Tull, Alexey Kardashevskiy,
Alex Williamson, Benjamin Herrenschmidt, Christoph Lameter,
Christophe Leroy, Davidlohr Bueso, Mark Rutland,
Michael Ellerman, Moritz Fischer, Paul Mackerras, Steve Sistare,
Wu Hao, linux-mm, kvm, kvm-ppc, linuxppc-dev, linux-fpga,
linux-kernel
On Fri, May 03, 2019 at 11:28:22PM +0000, Jason Gunthorpe wrote:
> On Fri, May 03, 2019 at 01:16:30PM -0700, Daniel Jordan wrote:
> > Andrew, this one patch replaces these six from [1]:
> >
> > mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> > vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> > powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> >
> > That series converts locked_vm to an atomic, but on closer inspection causes at
> > least one accounting race in mremap, and fixing it just for this type
> > conversion came with too much ugly in the core mm to justify, especially when
> > the right long-term fix is making these drivers use pinned_vm instead.
>
> Did we ever decide what to do here? Should all these drivers be
> switched to pinned_vm anyhow?
Well, there were the concerns about switching in [1]. Alex, is there an
example of an application or library that would break or be exploitable? If
there were particular worries (qemu for vfio type1, for example), perhaps some
coordinated changes across the kernel and userspace would be possible,
especially given the amount of effort it's likely going to take to get the
locked_vm/pinned_vm accounting sorted out.
[1] https://lore.kernel.org/linux-mm/20190213130330.76ef1987@w520.home/
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
2019-05-03 20:16 ` Daniel Jordan
(?)
@ 2019-05-20 6:19 ` Alexey Kardashevskiy
-1 siblings, 0 replies; 42+ messages in thread
From: Alexey Kardashevskiy @ 2019-05-20 6:19 UTC (permalink / raw)
To: Daniel Jordan, akpm
Cc: Alan Tull, Alex Williamson, Benjamin Herrenschmidt,
Christoph Lameter, Christophe Leroy, Davidlohr Bueso,
Jason Gunthorpe, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On 04/05/2019 06:16, Daniel Jordan wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly.
And I rather liked that prints were different and tell precisely which
one of three each printk is.
I commented below but in general this seems working.
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Error codes stay the same, so user-visible behavior does too.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>
> Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
> welcome.
>
> Andrew, this one patch replaces these six from [1]:
>
> mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
>
> That series converts locked_vm to an atomic, but on closer inspection causes at
> least one accounting race in mremap, and fixing it just for this type
> conversion came with too much ugly in the core mm to justify, especially when
> the right long-term fix is making these drivers use pinned_vm instead.
>
> Christophe's suggestion of cmpxchg[2] does prevent the races he
> mentioned for pinned_vm, but others would still remain. In perf_mmap
> and the hfi1 driver, pinned_vm is checked against the rlimit racily and
> then later increased when the pinned_vm originally read may have gone
> stale. Any fixes for that, that I could think of, seem about as good as
> what's there now, so I left it. I have a patch that uses cmpxchg with
> pinned_vm if others feel strongly that the aforementioned races should
> be fixed.
>
> Daniel
>
> [1] https://lore.kernel.org/linux-mm/20190402204158.27582-1-daniel.m.jordan@oracle.com/
> [2] https://lore.kernel.org/linux-mm/964bd5b0-f1e5-7bf0-5c58-18e75c550841@c-s.fr/
>
> arch/powerpc/kvm/book3s_64_vio.c | 44 +++---------------------
> arch/powerpc/mm/mmu_context_iommu.c | 41 +++-------------------
> drivers/fpga/dfl-afu-dma-region.c | 53 +++--------------------------
> drivers/vfio/vfio_iommu_spapr_tce.c | 52 +++++-----------------------
> drivers/vfio/vfio_iommu_type1.c | 23 ++++---------
> include/linux/mm.h | 19 +++++++++++
> mm/util.c | 48 ++++++++++++++++++++++++++
> 7 files changed, 94 insertions(+), 186 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
> index f02b04973710..f7d37fa6003a 100644
> --- a/arch/powerpc/kvm/book3s_64_vio.c
> +++ b/arch/powerpc/kvm/book3s_64_vio.c
> @@ -30,6 +30,7 @@
> #include <linux/anon_inodes.h>
> #include <linux/iommu.h>
> #include <linux/file.h>
> +#include <linux/mm.h>
>
> #include <asm/kvm_ppc.h>
> #include <asm/kvm_book3s.h>
> @@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
> return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
> }
>
> -static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
> -{
> - long ret = 0;
> -
> - if (!current || !current->mm)
> - return ret; /* process exited */
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (inc) {
> - unsigned long locked, lock_limit;
> -
> - locked = current->mm->locked_vm + stt_pages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += stt_pages;
> - } else {
> - if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
> - stt_pages = current->mm->locked_vm;
> -
> - current->mm->locked_vm -= stt_pages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
> - inc ? '+' : '-',
> - stt_pages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
> {
> struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
> @@ -277,7 +241,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
>
> kvm_put_kvm(stt->kvm);
>
> - kvmppc_account_memlimit(
> + account_locked_vm(current->mm,
> kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
> call_rcu(&stt->rcu, release_spapr_tce_table);
>
> @@ -303,7 +267,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
> return -EINVAL;
>
> npages = kvmppc_tce_pages(size);
> - ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
> + ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
> if (ret)
> return ret;
>
> @@ -359,7 +323,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
>
> kfree(stt);
> fail_acct:
> - kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
> + account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
> return ret;
> }
>
> diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
> index 8330f135294f..9e7001a70570 100644
> --- a/arch/powerpc/mm/mmu_context_iommu.c
> +++ b/arch/powerpc/mm/mmu_context_iommu.c
> @@ -19,6 +19,7 @@
> #include <linux/hugetlb.h>
> #include <linux/swap.h>
> #include <linux/sizes.h>
> +#include <linux/mm.h>
> #include <asm/mmu_context.h>
> #include <asm/pte-walk.h>
> #include <linux/mm_inline.h>
> @@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
> u64 dev_hpa; /* Device memory base address */
> };
>
> -static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
> - unsigned long npages, bool incr)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> -
> - if (incr) {
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
> - current ? current->pid : 0,
> - incr ? '+' : '-',
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> bool mm_iommu_preregistered(struct mm_struct *mm)
> {
> return !list_empty(&mm->context.iommu_group_mem_list);
> @@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> unsigned long entry, chunk;
>
> if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
> - ret = mm_iommu_adjust_locked_vm(mm, entries, true);
> + ret = account_locked_vm(mm, entries, true);
> if (ret)
> return ret;
>
> @@ -215,7 +182,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> kfree(mem);
>
> unlock_exit:
> - mm_iommu_adjust_locked_vm(mm, locked_entries, false);
> + account_locked_vm(mm, locked_entries, false);
>
> return ret;
> }
> @@ -315,7 +282,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
> unlock_exit:
> mutex_unlock(&mem_list_mutex);
>
> - mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
> + account_locked_vm(mm, unlock_entries, false);
>
> return ret;
> }
> diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
> index e18a786fc943..059438e17193 100644
> --- a/drivers/fpga/dfl-afu-dma-region.c
> +++ b/drivers/fpga/dfl-afu-dma-region.c
> @@ -12,6 +12,7 @@
> #include <linux/dma-mapping.h>
> #include <linux/sched/signal.h>
> #include <linux/uaccess.h>
> +#include <linux/mm.h>
>
> #include "dfl-afu.h"
>
> @@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
> afu->dma_regions = RB_ROOT;
> }
>
> -/**
> - * afu_dma_adjust_locked_vm - adjust locked memory
> - * @dev: port device
> - * @npages: number of pages
> - * @incr: increase or decrease locked memory
> - *
> - * Increase or decrease the locked memory size with npages input.
> - *
> - * Return 0 on success.
> - * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
> - */
> -static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> -{
> - unsigned long locked, lock_limit;
> - int ret = 0;
> -
> - /* the task is exiting. */
> - if (!current->mm)
> - return 0;
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (incr) {
> - locked = current->mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> - npages = current->mm->locked_vm;
> - current->mm->locked_vm -= npages;
> - }
> -
> - dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> - incr ? '+' : '-', npages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
> - ret ? "- exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> /**
> * afu_dma_pin_pages - pin pages of given dma memory region
> * @pdata: feature device platform data
> @@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> struct device *dev = &pdata->dev->dev;
> int ret, pinned;
>
> - ret = afu_dma_adjust_locked_vm(dev, npages, true);
> + ret = account_locked_vm(current->mm, npages, true);
> if (ret)
> return ret;
>
> @@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> free_pages:
> kfree(region->pages);
> unlock_vm:
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
> return ret;
> }
>
> @@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
>
> put_all_pages(region->pages, npages);
> kfree(region->pages);
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
>
> dev_dbg(dev, "%ld pages unpinned\n", npages);
> }
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 6b64e45a5269..d39a1b830d82 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -22,6 +22,7 @@
> #include <linux/vmalloc.h>
> #include <linux/sched/mm.h>
> #include <linux/sched/signal.h>
> +#include <linux/mm.h>
>
> #include <asm/iommu.h>
> #include <asm/tce.h>
> @@ -34,49 +35,13 @@
> static void tce_iommu_detach_group(void *iommu_data,
> struct iommu_group *iommu_group);
>
> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
> + bool inc)
> {
> - long ret = 0, locked, lock_limit;
> -
> if (WARN_ON_ONCE(!mm))
> return -EPERM;
If this WARN_ON is the only reason for having tce_account_locked_vm()
instead of calling account_locked_vm() directly, you can then ditch the
check as I have never ever seen this triggered.
>
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> -static void decrement_locked_vm(struct mm_struct *mm, long npages)
> -{
> - if (!mm || !npages)
> - return;
> -
> - down_write(&mm->mmap_sem);
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> + return account_locked_vm(mm, npages, inc);
> }
>
> /*
> @@ -336,7 +301,7 @@ static int tce_iommu_enable(struct tce_container *container)
> return ret;
>
> locked = table_group->tce32_size >> PAGE_SHIFT;
> - ret = try_increment_locked_vm(container->mm, locked);
> + ret = tce_account_locked_vm(container->mm, locked, true);
> if (ret)
> return ret;
>
> @@ -355,7 +320,7 @@ static void tce_iommu_disable(struct tce_container *container)
> container->enabled = false;
>
> BUG_ON(!container->mm);
> - decrement_locked_vm(container->mm, container->locked_pages);
> + tce_account_locked_vm(container->mm, container->locked_pages, false);
> }
>
> static void *tce_iommu_open(unsigned long arg)
> @@ -658,7 +623,8 @@ static long tce_iommu_create_table(struct tce_container *container,
> if (!table_size)
> return -EINVAL;
>
> - ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
> + ret = tce_account_locked_vm(container->mm, table_size >> PAGE_SHIFT,
> + true);
> if (ret)
> return ret;
>
> @@ -677,7 +643,7 @@ static void tce_iommu_free_table(struct tce_container *container,
> unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
>
> iommu_tce_table_put(tbl);
> - decrement_locked_vm(container->mm, pages);
> + tce_account_locked_vm(container->mm, pages, false);
> }
>
> static long tce_iommu_create_window(struct tce_container *container,
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index d0f731c9920a..15ac76171ccd 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
> return -ESRCH; /* process exited */
>
> ret = down_write_killable(&mm->mmap_sem);
> - if (!ret) {
> - if (npage > 0) {
> - if (!dma->lock_cap) {
> - unsigned long limit;
> -
> - limit = task_rlimit(dma->task,
> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (mm->locked_vm + npage > limit)
> - ret = -ENOMEM;
> - }
> - }
> + if (ret)
> + goto out;
A single "goto" to jump just 3 lines below seems unnecessary.
>
> - if (!ret)
> - mm->locked_vm += npage;
> -
> - up_write(&mm->mmap_sem);
> - }
> + ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
> + dma->lock_cap);
> + up_write(&mm->mmap_sem);
>
> +out:
> if (async)
> mmput(mm);
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 6b10c21630f5..7134e55ca23f 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1521,6 +1521,25 @@ static inline long get_user_pages_longterm(unsigned long start,
> int get_user_pages_fast(unsigned long start, int nr_pages, int write,
> struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index 43a2984bccaa..552302665bc2 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -6,6 +6,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -346,6 +347,53 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * 0 if @mm is NULL (can happen for example if the task is exiting)
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + locked_vm = mm->locked_vm;
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit) {
> + ret = -ENOMEM;
> + goto out;
> + }
> + }
Nit:
if (!ret)
and then you don't need "goto out".
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
Can go negative here. Not a huge deal but inaccurate imo.
> + }
> +
> +out:
> + pr_debug("%s: [%d] %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: 37624b58542fb9f2d9a70e6ea006ef8a5f66c30b
>
--
Alexey
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-20 6:19 ` Alexey Kardashevskiy
0 siblings, 0 replies; 42+ messages in thread
From: Alexey Kardashevskiy @ 2019-05-20 6:19 UTC (permalink / raw)
To: Daniel Jordan, akpm
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull, linux-fpga,
linux-kernel, kvm-ppc, linux-mm, Alex Williamson,
Jason Gunthorpe, Moritz Fischer, Steve Sistare,
Christoph Lameter, linuxppc-dev, Wu Hao
On 04/05/2019 06:16, Daniel Jordan wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly.
And I rather liked that prints were different and tell precisely which
one of three each printk is.
I commented below but in general this seems working.
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Error codes stay the same, so user-visible behavior does too.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>
> Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
> welcome.
>
> Andrew, this one patch replaces these six from [1]:
>
> mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
>
> That series converts locked_vm to an atomic, but on closer inspection causes at
> least one accounting race in mremap, and fixing it just for this type
> conversion came with too much ugly in the core mm to justify, especially when
> the right long-term fix is making these drivers use pinned_vm instead.
>
> Christophe's suggestion of cmpxchg[2] does prevent the races he
> mentioned for pinned_vm, but others would still remain. In perf_mmap
> and the hfi1 driver, pinned_vm is checked against the rlimit racily and
> then later increased when the pinned_vm originally read may have gone
> stale. Any fixes for that, that I could think of, seem about as good as
> what's there now, so I left it. I have a patch that uses cmpxchg with
> pinned_vm if others feel strongly that the aforementioned races should
> be fixed.
>
> Daniel
>
> [1] https://lore.kernel.org/linux-mm/20190402204158.27582-1-daniel.m.jordan@oracle.com/
> [2] https://lore.kernel.org/linux-mm/964bd5b0-f1e5-7bf0-5c58-18e75c550841@c-s.fr/
>
> arch/powerpc/kvm/book3s_64_vio.c | 44 +++---------------------
> arch/powerpc/mm/mmu_context_iommu.c | 41 +++-------------------
> drivers/fpga/dfl-afu-dma-region.c | 53 +++--------------------------
> drivers/vfio/vfio_iommu_spapr_tce.c | 52 +++++-----------------------
> drivers/vfio/vfio_iommu_type1.c | 23 ++++---------
> include/linux/mm.h | 19 +++++++++++
> mm/util.c | 48 ++++++++++++++++++++++++++
> 7 files changed, 94 insertions(+), 186 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
> index f02b04973710..f7d37fa6003a 100644
> --- a/arch/powerpc/kvm/book3s_64_vio.c
> +++ b/arch/powerpc/kvm/book3s_64_vio.c
> @@ -30,6 +30,7 @@
> #include <linux/anon_inodes.h>
> #include <linux/iommu.h>
> #include <linux/file.h>
> +#include <linux/mm.h>
>
> #include <asm/kvm_ppc.h>
> #include <asm/kvm_book3s.h>
> @@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
> return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
> }
>
> -static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
> -{
> - long ret = 0;
> -
> - if (!current || !current->mm)
> - return ret; /* process exited */
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (inc) {
> - unsigned long locked, lock_limit;
> -
> - locked = current->mm->locked_vm + stt_pages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += stt_pages;
> - } else {
> - if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
> - stt_pages = current->mm->locked_vm;
> -
> - current->mm->locked_vm -= stt_pages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
> - inc ? '+' : '-',
> - stt_pages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
> {
> struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
> @@ -277,7 +241,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
>
> kvm_put_kvm(stt->kvm);
>
> - kvmppc_account_memlimit(
> + account_locked_vm(current->mm,
> kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
> call_rcu(&stt->rcu, release_spapr_tce_table);
>
> @@ -303,7 +267,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
> return -EINVAL;
>
> npages = kvmppc_tce_pages(size);
> - ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
> + ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
> if (ret)
> return ret;
>
> @@ -359,7 +323,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
>
> kfree(stt);
> fail_acct:
> - kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
> + account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
> return ret;
> }
>
> diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
> index 8330f135294f..9e7001a70570 100644
> --- a/arch/powerpc/mm/mmu_context_iommu.c
> +++ b/arch/powerpc/mm/mmu_context_iommu.c
> @@ -19,6 +19,7 @@
> #include <linux/hugetlb.h>
> #include <linux/swap.h>
> #include <linux/sizes.h>
> +#include <linux/mm.h>
> #include <asm/mmu_context.h>
> #include <asm/pte-walk.h>
> #include <linux/mm_inline.h>
> @@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
> u64 dev_hpa; /* Device memory base address */
> };
>
> -static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
> - unsigned long npages, bool incr)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> -
> - if (incr) {
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
> - current ? current->pid : 0,
> - incr ? '+' : '-',
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> bool mm_iommu_preregistered(struct mm_struct *mm)
> {
> return !list_empty(&mm->context.iommu_group_mem_list);
> @@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> unsigned long entry, chunk;
>
> if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
> - ret = mm_iommu_adjust_locked_vm(mm, entries, true);
> + ret = account_locked_vm(mm, entries, true);
> if (ret)
> return ret;
>
> @@ -215,7 +182,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> kfree(mem);
>
> unlock_exit:
> - mm_iommu_adjust_locked_vm(mm, locked_entries, false);
> + account_locked_vm(mm, locked_entries, false);
>
> return ret;
> }
> @@ -315,7 +282,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
> unlock_exit:
> mutex_unlock(&mem_list_mutex);
>
> - mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
> + account_locked_vm(mm, unlock_entries, false);
>
> return ret;
> }
> diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
> index e18a786fc943..059438e17193 100644
> --- a/drivers/fpga/dfl-afu-dma-region.c
> +++ b/drivers/fpga/dfl-afu-dma-region.c
> @@ -12,6 +12,7 @@
> #include <linux/dma-mapping.h>
> #include <linux/sched/signal.h>
> #include <linux/uaccess.h>
> +#include <linux/mm.h>
>
> #include "dfl-afu.h"
>
> @@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
> afu->dma_regions = RB_ROOT;
> }
>
> -/**
> - * afu_dma_adjust_locked_vm - adjust locked memory
> - * @dev: port device
> - * @npages: number of pages
> - * @incr: increase or decrease locked memory
> - *
> - * Increase or decrease the locked memory size with npages input.
> - *
> - * Return 0 on success.
> - * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
> - */
> -static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> -{
> - unsigned long locked, lock_limit;
> - int ret = 0;
> -
> - /* the task is exiting. */
> - if (!current->mm)
> - return 0;
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (incr) {
> - locked = current->mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> - npages = current->mm->locked_vm;
> - current->mm->locked_vm -= npages;
> - }
> -
> - dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> - incr ? '+' : '-', npages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
> - ret ? "- exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> /**
> * afu_dma_pin_pages - pin pages of given dma memory region
> * @pdata: feature device platform data
> @@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> struct device *dev = &pdata->dev->dev;
> int ret, pinned;
>
> - ret = afu_dma_adjust_locked_vm(dev, npages, true);
> + ret = account_locked_vm(current->mm, npages, true);
> if (ret)
> return ret;
>
> @@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> free_pages:
> kfree(region->pages);
> unlock_vm:
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
> return ret;
> }
>
> @@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
>
> put_all_pages(region->pages, npages);
> kfree(region->pages);
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
>
> dev_dbg(dev, "%ld pages unpinned\n", npages);
> }
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 6b64e45a5269..d39a1b830d82 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -22,6 +22,7 @@
> #include <linux/vmalloc.h>
> #include <linux/sched/mm.h>
> #include <linux/sched/signal.h>
> +#include <linux/mm.h>
>
> #include <asm/iommu.h>
> #include <asm/tce.h>
> @@ -34,49 +35,13 @@
> static void tce_iommu_detach_group(void *iommu_data,
> struct iommu_group *iommu_group);
>
> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
> + bool inc)
> {
> - long ret = 0, locked, lock_limit;
> -
> if (WARN_ON_ONCE(!mm))
> return -EPERM;
If this WARN_ON is the only reason for having tce_account_locked_vm()
instead of calling account_locked_vm() directly, you can then ditch the
check as I have never ever seen this triggered.
>
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> -static void decrement_locked_vm(struct mm_struct *mm, long npages)
> -{
> - if (!mm || !npages)
> - return;
> -
> - down_write(&mm->mmap_sem);
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> + return account_locked_vm(mm, npages, inc);
> }
>
> /*
> @@ -336,7 +301,7 @@ static int tce_iommu_enable(struct tce_container *container)
> return ret;
>
> locked = table_group->tce32_size >> PAGE_SHIFT;
> - ret = try_increment_locked_vm(container->mm, locked);
> + ret = tce_account_locked_vm(container->mm, locked, true);
> if (ret)
> return ret;
>
> @@ -355,7 +320,7 @@ static void tce_iommu_disable(struct tce_container *container)
> container->enabled = false;
>
> BUG_ON(!container->mm);
> - decrement_locked_vm(container->mm, container->locked_pages);
> + tce_account_locked_vm(container->mm, container->locked_pages, false);
> }
>
> static void *tce_iommu_open(unsigned long arg)
> @@ -658,7 +623,8 @@ static long tce_iommu_create_table(struct tce_container *container,
> if (!table_size)
> return -EINVAL;
>
> - ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
> + ret = tce_account_locked_vm(container->mm, table_size >> PAGE_SHIFT,
> + true);
> if (ret)
> return ret;
>
> @@ -677,7 +643,7 @@ static void tce_iommu_free_table(struct tce_container *container,
> unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
>
> iommu_tce_table_put(tbl);
> - decrement_locked_vm(container->mm, pages);
> + tce_account_locked_vm(container->mm, pages, false);
> }
>
> static long tce_iommu_create_window(struct tce_container *container,
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index d0f731c9920a..15ac76171ccd 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
> return -ESRCH; /* process exited */
>
> ret = down_write_killable(&mm->mmap_sem);
> - if (!ret) {
> - if (npage > 0) {
> - if (!dma->lock_cap) {
> - unsigned long limit;
> -
> - limit = task_rlimit(dma->task,
> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (mm->locked_vm + npage > limit)
> - ret = -ENOMEM;
> - }
> - }
> + if (ret)
> + goto out;
A single "goto" to jump just 3 lines below seems unnecessary.
>
> - if (!ret)
> - mm->locked_vm += npage;
> -
> - up_write(&mm->mmap_sem);
> - }
> + ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
> + dma->lock_cap);
> + up_write(&mm->mmap_sem);
>
> +out:
> if (async)
> mmput(mm);
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 6b10c21630f5..7134e55ca23f 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1521,6 +1521,25 @@ static inline long get_user_pages_longterm(unsigned long start,
> int get_user_pages_fast(unsigned long start, int nr_pages, int write,
> struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index 43a2984bccaa..552302665bc2 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -6,6 +6,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -346,6 +347,53 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * 0 if @mm is NULL (can happen for example if the task is exiting)
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + locked_vm = mm->locked_vm;
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit) {
> + ret = -ENOMEM;
> + goto out;
> + }
> + }
Nit:
if (!ret)
and then you don't need "goto out".
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
Can go negative here. Not a huge deal but inaccurate imo.
> + }
> +
> +out:
> + pr_debug("%s: [%d] %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: 37624b58542fb9f2d9a70e6ea006ef8a5f66c30b
>
--
Alexey
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-20 6:19 ` Alexey Kardashevskiy
0 siblings, 0 replies; 42+ messages in thread
From: Alexey Kardashevskiy @ 2019-05-20 6:19 UTC (permalink / raw)
To: Daniel Jordan, akpm
Cc: Alan Tull, Alex Williamson, Benjamin Herrenschmidt,
Christoph Lameter, Christophe Leroy, Davidlohr Bueso,
Jason Gunthorpe, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On 04/05/2019 06:16, Daniel Jordan wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly.
And I rather liked that prints were different and tell precisely which
one of three each printk is.
I commented below but in general this seems working.
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Error codes stay the same, so user-visible behavior does too.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>
> Based on v5.1-rc7. Tested with the vfio type1 driver. Any feedback
> welcome.
>
> Andrew, this one patch replaces these six from [1]:
>
> mm-change-locked_vms-type-from-unsigned-long-to-atomic64_t.patch
> vfio-type1-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> vfio-spapr_tce-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> fpga-dlf-afu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> kvm-book3s-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
> powerpc-mmu-drop-mmap_sem-now-that-locked_vm-is-atomic.patch
>
> That series converts locked_vm to an atomic, but on closer inspection causes at
> least one accounting race in mremap, and fixing it just for this type
> conversion came with too much ugly in the core mm to justify, especially when
> the right long-term fix is making these drivers use pinned_vm instead.
>
> Christophe's suggestion of cmpxchg[2] does prevent the races he
> mentioned for pinned_vm, but others would still remain. In perf_mmap
> and the hfi1 driver, pinned_vm is checked against the rlimit racily and
> then later increased when the pinned_vm originally read may have gone
> stale. Any fixes for that, that I could think of, seem about as good as
> what's there now, so I left it. I have a patch that uses cmpxchg with
> pinned_vm if others feel strongly that the aforementioned races should
> be fixed.
>
> Daniel
>
> [1] https://lore.kernel.org/linux-mm/20190402204158.27582-1-daniel.m.jordan@oracle.com/
> [2] https://lore.kernel.org/linux-mm/964bd5b0-f1e5-7bf0-5c58-18e75c550841@c-s.fr/
>
> arch/powerpc/kvm/book3s_64_vio.c | 44 +++---------------------
> arch/powerpc/mm/mmu_context_iommu.c | 41 +++-------------------
> drivers/fpga/dfl-afu-dma-region.c | 53 +++--------------------------
> drivers/vfio/vfio_iommu_spapr_tce.c | 52 +++++-----------------------
> drivers/vfio/vfio_iommu_type1.c | 23 ++++---------
> include/linux/mm.h | 19 +++++++++++
> mm/util.c | 48 ++++++++++++++++++++++++++
> 7 files changed, 94 insertions(+), 186 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
> index f02b04973710..f7d37fa6003a 100644
> --- a/arch/powerpc/kvm/book3s_64_vio.c
> +++ b/arch/powerpc/kvm/book3s_64_vio.c
> @@ -30,6 +30,7 @@
> #include <linux/anon_inodes.h>
> #include <linux/iommu.h>
> #include <linux/file.h>
> +#include <linux/mm.h>
>
> #include <asm/kvm_ppc.h>
> #include <asm/kvm_book3s.h>
> @@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
> return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
> }
>
> -static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
> -{
> - long ret = 0;
> -
> - if (!current || !current->mm)
> - return ret; /* process exited */
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (inc) {
> - unsigned long locked, lock_limit;
> -
> - locked = current->mm->locked_vm + stt_pages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += stt_pages;
> - } else {
> - if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
> - stt_pages = current->mm->locked_vm;
> -
> - current->mm->locked_vm -= stt_pages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
> - inc ? '+' : '-',
> - stt_pages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
> {
> struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
> @@ -277,7 +241,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
>
> kvm_put_kvm(stt->kvm);
>
> - kvmppc_account_memlimit(
> + account_locked_vm(current->mm,
> kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
> call_rcu(&stt->rcu, release_spapr_tce_table);
>
> @@ -303,7 +267,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
> return -EINVAL;
>
> npages = kvmppc_tce_pages(size);
> - ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
> + ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
> if (ret)
> return ret;
>
> @@ -359,7 +323,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
>
> kfree(stt);
> fail_acct:
> - kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
> + account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
> return ret;
> }
>
> diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
> index 8330f135294f..9e7001a70570 100644
> --- a/arch/powerpc/mm/mmu_context_iommu.c
> +++ b/arch/powerpc/mm/mmu_context_iommu.c
> @@ -19,6 +19,7 @@
> #include <linux/hugetlb.h>
> #include <linux/swap.h>
> #include <linux/sizes.h>
> +#include <linux/mm.h>
> #include <asm/mmu_context.h>
> #include <asm/pte-walk.h>
> #include <linux/mm_inline.h>
> @@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
> u64 dev_hpa; /* Device memory base address */
> };
>
> -static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
> - unsigned long npages, bool incr)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> -
> - if (incr) {
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
> - current ? current->pid : 0,
> - incr ? '+' : '-',
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> bool mm_iommu_preregistered(struct mm_struct *mm)
> {
> return !list_empty(&mm->context.iommu_group_mem_list);
> @@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> unsigned long entry, chunk;
>
> if (dev_hpa = MM_IOMMU_TABLE_INVALID_HPA) {
> - ret = mm_iommu_adjust_locked_vm(mm, entries, true);
> + ret = account_locked_vm(mm, entries, true);
> if (ret)
> return ret;
>
> @@ -215,7 +182,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> kfree(mem);
>
> unlock_exit:
> - mm_iommu_adjust_locked_vm(mm, locked_entries, false);
> + account_locked_vm(mm, locked_entries, false);
>
> return ret;
> }
> @@ -315,7 +282,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
> unlock_exit:
> mutex_unlock(&mem_list_mutex);
>
> - mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
> + account_locked_vm(mm, unlock_entries, false);
>
> return ret;
> }
> diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
> index e18a786fc943..059438e17193 100644
> --- a/drivers/fpga/dfl-afu-dma-region.c
> +++ b/drivers/fpga/dfl-afu-dma-region.c
> @@ -12,6 +12,7 @@
> #include <linux/dma-mapping.h>
> #include <linux/sched/signal.h>
> #include <linux/uaccess.h>
> +#include <linux/mm.h>
>
> #include "dfl-afu.h"
>
> @@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
> afu->dma_regions = RB_ROOT;
> }
>
> -/**
> - * afu_dma_adjust_locked_vm - adjust locked memory
> - * @dev: port device
> - * @npages: number of pages
> - * @incr: increase or decrease locked memory
> - *
> - * Increase or decrease the locked memory size with npages input.
> - *
> - * Return 0 on success.
> - * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
> - */
> -static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> -{
> - unsigned long locked, lock_limit;
> - int ret = 0;
> -
> - /* the task is exiting. */
> - if (!current->mm)
> - return 0;
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (incr) {
> - locked = current->mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> - npages = current->mm->locked_vm;
> - current->mm->locked_vm -= npages;
> - }
> -
> - dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> - incr ? '+' : '-', npages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
> - ret ? "- exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> /**
> * afu_dma_pin_pages - pin pages of given dma memory region
> * @pdata: feature device platform data
> @@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> struct device *dev = &pdata->dev->dev;
> int ret, pinned;
>
> - ret = afu_dma_adjust_locked_vm(dev, npages, true);
> + ret = account_locked_vm(current->mm, npages, true);
> if (ret)
> return ret;
>
> @@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> free_pages:
> kfree(region->pages);
> unlock_vm:
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
> return ret;
> }
>
> @@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
>
> put_all_pages(region->pages, npages);
> kfree(region->pages);
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
>
> dev_dbg(dev, "%ld pages unpinned\n", npages);
> }
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 6b64e45a5269..d39a1b830d82 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -22,6 +22,7 @@
> #include <linux/vmalloc.h>
> #include <linux/sched/mm.h>
> #include <linux/sched/signal.h>
> +#include <linux/mm.h>
>
> #include <asm/iommu.h>
> #include <asm/tce.h>
> @@ -34,49 +35,13 @@
> static void tce_iommu_detach_group(void *iommu_data,
> struct iommu_group *iommu_group);
>
> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
> + bool inc)
> {
> - long ret = 0, locked, lock_limit;
> -
> if (WARN_ON_ONCE(!mm))
> return -EPERM;
If this WARN_ON is the only reason for having tce_account_locked_vm()
instead of calling account_locked_vm() directly, you can then ditch the
check as I have never ever seen this triggered.
>
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> -static void decrement_locked_vm(struct mm_struct *mm, long npages)
> -{
> - if (!mm || !npages)
> - return;
> -
> - down_write(&mm->mmap_sem);
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> + return account_locked_vm(mm, npages, inc);
> }
>
> /*
> @@ -336,7 +301,7 @@ static int tce_iommu_enable(struct tce_container *container)
> return ret;
>
> locked = table_group->tce32_size >> PAGE_SHIFT;
> - ret = try_increment_locked_vm(container->mm, locked);
> + ret = tce_account_locked_vm(container->mm, locked, true);
> if (ret)
> return ret;
>
> @@ -355,7 +320,7 @@ static void tce_iommu_disable(struct tce_container *container)
> container->enabled = false;
>
> BUG_ON(!container->mm);
> - decrement_locked_vm(container->mm, container->locked_pages);
> + tce_account_locked_vm(container->mm, container->locked_pages, false);
> }
>
> static void *tce_iommu_open(unsigned long arg)
> @@ -658,7 +623,8 @@ static long tce_iommu_create_table(struct tce_container *container,
> if (!table_size)
> return -EINVAL;
>
> - ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
> + ret = tce_account_locked_vm(container->mm, table_size >> PAGE_SHIFT,
> + true);
> if (ret)
> return ret;
>
> @@ -677,7 +643,7 @@ static void tce_iommu_free_table(struct tce_container *container,
> unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
>
> iommu_tce_table_put(tbl);
> - decrement_locked_vm(container->mm, pages);
> + tce_account_locked_vm(container->mm, pages, false);
> }
>
> static long tce_iommu_create_window(struct tce_container *container,
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index d0f731c9920a..15ac76171ccd 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
> return -ESRCH; /* process exited */
>
> ret = down_write_killable(&mm->mmap_sem);
> - if (!ret) {
> - if (npage > 0) {
> - if (!dma->lock_cap) {
> - unsigned long limit;
> -
> - limit = task_rlimit(dma->task,
> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (mm->locked_vm + npage > limit)
> - ret = -ENOMEM;
> - }
> - }
> + if (ret)
> + goto out;
A single "goto" to jump just 3 lines below seems unnecessary.
>
> - if (!ret)
> - mm->locked_vm += npage;
> -
> - up_write(&mm->mmap_sem);
> - }
> + ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
> + dma->lock_cap);
> + up_write(&mm->mmap_sem);
>
> +out:
> if (async)
> mmput(mm);
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 6b10c21630f5..7134e55ca23f 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1521,6 +1521,25 @@ static inline long get_user_pages_longterm(unsigned long start,
> int get_user_pages_fast(unsigned long start, int nr_pages, int write,
> struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages = 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index 43a2984bccaa..552302665bc2 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -6,6 +6,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -346,6 +347,53 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * 0 if @mm is NULL (can happen for example if the task is exiting)
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + locked_vm = mm->locked_vm;
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit) {
> + ret = -ENOMEM;
> + goto out;
> + }
> + }
Nit:
if (!ret)
and then you don't need "goto out".
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
Can go negative here. Not a huge deal but inaccurate imo.
> + }
> +
> +out:
> + pr_debug("%s: [%d] %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: 37624b58542fb9f2d9a70e6ea006ef8a5f66c30b
>
--
Alexey
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
2019-05-20 6:19 ` Alexey Kardashevskiy
(?)
@ 2019-05-20 15:30 ` Daniel Jordan
-1 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-20 15:30 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: Daniel Jordan, akpm, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Mon, May 20, 2019 at 04:19:34PM +1000, Alexey Kardashevskiy wrote:
> On 04/05/2019 06:16, Daniel Jordan wrote:
> > locked_vm accounting is done roughly the same way in five places, so
> > unify them in a helper. Standardize the debug prints, which vary
> > slightly.
>
> And I rather liked that prints were different and tell precisely which
> one of three each printk is.
I'm not following. One of three...callsites? But there were five callsites.
Anyway, I added a _RET_IP_ to the debug print so you can differentiate.
> I commented below but in general this seems working.
>
> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Thanks! And for the review as well.
> > diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> > index 6b64e45a5269..d39a1b830d82 100644
> > --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> > +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> > @@ -34,49 +35,13 @@
> > static void tce_iommu_detach_group(void *iommu_data,
> > struct iommu_group *iommu_group);
> >
> > -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> > +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
> > + bool inc)
> > {
> > - long ret = 0, locked, lock_limit;
> > -
> > if (WARN_ON_ONCE(!mm))
> > return -EPERM;
>
>
> If this WARN_ON is the only reason for having tce_account_locked_vm()
> instead of calling account_locked_vm() directly, you can then ditch the
> check as I have never ever seen this triggered.
Great, will do.
> > diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> > index d0f731c9920a..15ac76171ccd 100644
> > --- a/drivers/vfio/vfio_iommu_type1.c
> > +++ b/drivers/vfio/vfio_iommu_type1.c
> > @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
> > return -ESRCH; /* process exited */
> >
> > ret = down_write_killable(&mm->mmap_sem);
> > - if (!ret) {
> > - if (npage > 0) {
> > - if (!dma->lock_cap) {
> > - unsigned long limit;
> > -
> > - limit = task_rlimit(dma->task,
> > - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > -
> > - if (mm->locked_vm + npage > limit)
> > - ret = -ENOMEM;
> > - }
> > - }
> > + if (ret)
> > + goto out;
>
>
> A single "goto" to jump just 3 lines below seems unnecessary.
No strong preference here, I'll take out the goto.
> > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > + struct task_struct *task, bool bypass_rlim)
> > +{
> > + unsigned long locked_vm, limit;
> > + int ret = 0;
> > +
> > + locked_vm = mm->locked_vm;
> > + if (inc) {
> > + if (!bypass_rlim) {
> > + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > + if (locked_vm + pages > limit) {
> > + ret = -ENOMEM;
> > + goto out;
> > + }
> > + }
>
> Nit:
>
> if (!ret)
>
> and then you don't need "goto out".
Ok, sure.
> > + mm->locked_vm = locked_vm + pages;
> > + } else {
> > + WARN_ON_ONCE(pages > locked_vm);
> > + mm->locked_vm = locked_vm - pages;
>
>
> Can go negative here. Not a huge deal but inaccurate imo.
I hear you, but setting a negative value to zero, as we had done previously,
doesn't make much sense to me.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-20 15:30 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-20 15:30 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull, linux-fpga,
linux-kernel, kvm-ppc, Daniel Jordan, linux-mm, Alex Williamson,
Jason Gunthorpe, Moritz Fischer, Steve Sistare, akpm,
linuxppc-dev, Christoph Lameter, Wu Hao
On Mon, May 20, 2019 at 04:19:34PM +1000, Alexey Kardashevskiy wrote:
> On 04/05/2019 06:16, Daniel Jordan wrote:
> > locked_vm accounting is done roughly the same way in five places, so
> > unify them in a helper. Standardize the debug prints, which vary
> > slightly.
>
> And I rather liked that prints were different and tell precisely which
> one of three each printk is.
I'm not following. One of three...callsites? But there were five callsites.
Anyway, I added a _RET_IP_ to the debug print so you can differentiate.
> I commented below but in general this seems working.
>
> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Thanks! And for the review as well.
> > diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> > index 6b64e45a5269..d39a1b830d82 100644
> > --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> > +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> > @@ -34,49 +35,13 @@
> > static void tce_iommu_detach_group(void *iommu_data,
> > struct iommu_group *iommu_group);
> >
> > -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> > +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
> > + bool inc)
> > {
> > - long ret = 0, locked, lock_limit;
> > -
> > if (WARN_ON_ONCE(!mm))
> > return -EPERM;
>
>
> If this WARN_ON is the only reason for having tce_account_locked_vm()
> instead of calling account_locked_vm() directly, you can then ditch the
> check as I have never ever seen this triggered.
Great, will do.
> > diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> > index d0f731c9920a..15ac76171ccd 100644
> > --- a/drivers/vfio/vfio_iommu_type1.c
> > +++ b/drivers/vfio/vfio_iommu_type1.c
> > @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
> > return -ESRCH; /* process exited */
> >
> > ret = down_write_killable(&mm->mmap_sem);
> > - if (!ret) {
> > - if (npage > 0) {
> > - if (!dma->lock_cap) {
> > - unsigned long limit;
> > -
> > - limit = task_rlimit(dma->task,
> > - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > -
> > - if (mm->locked_vm + npage > limit)
> > - ret = -ENOMEM;
> > - }
> > - }
> > + if (ret)
> > + goto out;
>
>
> A single "goto" to jump just 3 lines below seems unnecessary.
No strong preference here, I'll take out the goto.
> > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > + struct task_struct *task, bool bypass_rlim)
> > +{
> > + unsigned long locked_vm, limit;
> > + int ret = 0;
> > +
> > + locked_vm = mm->locked_vm;
> > + if (inc) {
> > + if (!bypass_rlim) {
> > + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > + if (locked_vm + pages > limit) {
> > + ret = -ENOMEM;
> > + goto out;
> > + }
> > + }
>
> Nit:
>
> if (!ret)
>
> and then you don't need "goto out".
Ok, sure.
> > + mm->locked_vm = locked_vm + pages;
> > + } else {
> > + WARN_ON_ONCE(pages > locked_vm);
> > + mm->locked_vm = locked_vm - pages;
>
>
> Can go negative here. Not a huge deal but inaccurate imo.
I hear you, but setting a negative value to zero, as we had done previously,
doesn't make much sense to me.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-20 15:30 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-20 15:30 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: Daniel Jordan, akpm, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Mon, May 20, 2019 at 04:19:34PM +1000, Alexey Kardashevskiy wrote:
> On 04/05/2019 06:16, Daniel Jordan wrote:
> > locked_vm accounting is done roughly the same way in five places, so
> > unify them in a helper. Standardize the debug prints, which vary
> > slightly.
>
> And I rather liked that prints were different and tell precisely which
> one of three each printk is.
I'm not following. One of three...callsites? But there were five callsites.
Anyway, I added a _RET_IP_ to the debug print so you can differentiate.
> I commented below but in general this seems working.
>
> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Thanks! And for the review as well.
> > diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> > index 6b64e45a5269..d39a1b830d82 100644
> > --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> > +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> > @@ -34,49 +35,13 @@
> > static void tce_iommu_detach_group(void *iommu_data,
> > struct iommu_group *iommu_group);
> >
> > -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> > +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
> > + bool inc)
> > {
> > - long ret = 0, locked, lock_limit;
> > -
> > if (WARN_ON_ONCE(!mm))
> > return -EPERM;
>
>
> If this WARN_ON is the only reason for having tce_account_locked_vm()
> instead of calling account_locked_vm() directly, you can then ditch the
> check as I have never ever seen this triggered.
Great, will do.
> > diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> > index d0f731c9920a..15ac76171ccd 100644
> > --- a/drivers/vfio/vfio_iommu_type1.c
> > +++ b/drivers/vfio/vfio_iommu_type1.c
> > @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
> > return -ESRCH; /* process exited */
> >
> > ret = down_write_killable(&mm->mmap_sem);
> > - if (!ret) {
> > - if (npage > 0) {
> > - if (!dma->lock_cap) {
> > - unsigned long limit;
> > -
> > - limit = task_rlimit(dma->task,
> > - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > -
> > - if (mm->locked_vm + npage > limit)
> > - ret = -ENOMEM;
> > - }
> > - }
> > + if (ret)
> > + goto out;
>
>
> A single "goto" to jump just 3 lines below seems unnecessary.
No strong preference here, I'll take out the goto.
> > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > + struct task_struct *task, bool bypass_rlim)
> > +{
> > + unsigned long locked_vm, limit;
> > + int ret = 0;
> > +
> > + locked_vm = mm->locked_vm;
> > + if (inc) {
> > + if (!bypass_rlim) {
> > + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > + if (locked_vm + pages > limit) {
> > + ret = -ENOMEM;
> > + goto out;
> > + }
> > + }
>
> Nit:
>
> if (!ret)
>
> and then you don't need "goto out".
Ok, sure.
> > + mm->locked_vm = locked_vm + pages;
> > + } else {
> > + WARN_ON_ONCE(pages > locked_vm);
> > + mm->locked_vm = locked_vm - pages;
>
>
> Can go negative here. Not a huge deal but inaccurate imo.
I hear you, but setting a negative value to zero, as we had done previously,
doesn't make much sense to me.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
2019-05-20 15:30 ` Daniel Jordan
(?)
@ 2019-05-24 6:43 ` Alexey Kardashevskiy
-1 siblings, 0 replies; 42+ messages in thread
From: Alexey Kardashevskiy @ 2019-05-24 6:43 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alan Tull, Alex Williamson, Benjamin Herrenschmidt,
Christoph Lameter, Christophe Leroy, Davidlohr Bueso,
Jason Gunthorpe, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On 21/05/2019 01:30, Daniel Jordan wrote:
> On Mon, May 20, 2019 at 04:19:34PM +1000, Alexey Kardashevskiy wrote:
>> On 04/05/2019 06:16, Daniel Jordan wrote:
>>> locked_vm accounting is done roughly the same way in five places, so
>>> unify them in a helper. Standardize the debug prints, which vary
>>> slightly.
>>
>> And I rather liked that prints were different and tell precisely which
>> one of three each printk is.
>
> I'm not following. One of three...callsites? But there were five callsites.
Well, 3 of them are mine, I was referring to them :)
> Anyway, I added a _RET_IP_ to the debug print so you can differentiate.
I did not know that existed, cool!
>
>> I commented below but in general this seems working.
>>
>> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>
> Thanks! And for the review as well.
>
>>> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
>>> index 6b64e45a5269..d39a1b830d82 100644
>>> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
>>> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
>>> @@ -34,49 +35,13 @@
>>> static void tce_iommu_detach_group(void *iommu_data,
>>> struct iommu_group *iommu_group);
>>>
>>> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
>>> +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
>>> + bool inc)
>>> {
>>> - long ret = 0, locked, lock_limit;
>>> -
>>> if (WARN_ON_ONCE(!mm))
>>> return -EPERM;
>>
>>
>> If this WARN_ON is the only reason for having tce_account_locked_vm()
>> instead of calling account_locked_vm() directly, you can then ditch the
>> check as I have never ever seen this triggered.
>
> Great, will do.
>
>>> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
>>> index d0f731c9920a..15ac76171ccd 100644
>>> --- a/drivers/vfio/vfio_iommu_type1.c
>>> +++ b/drivers/vfio/vfio_iommu_type1.c
>>> @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
>>> return -ESRCH; /* process exited */
>>>
>>> ret = down_write_killable(&mm->mmap_sem);
>>> - if (!ret) {
>>> - if (npage > 0) {
>>> - if (!dma->lock_cap) {
>>> - unsigned long limit;
>>> -
>>> - limit = task_rlimit(dma->task,
>>> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>>> -
>>> - if (mm->locked_vm + npage > limit)
>>> - ret = -ENOMEM;
>>> - }
>>> - }
>>> + if (ret)
>>> + goto out;
>>
>>
>> A single "goto" to jump just 3 lines below seems unnecessary.
>
> No strong preference here, I'll take out the goto.
>
>>> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
>>> + struct task_struct *task, bool bypass_rlim)
>>> +{
>>> + unsigned long locked_vm, limit;
>>> + int ret = 0;
>>> +
>>> + locked_vm = mm->locked_vm;
>>> + if (inc) {
>>> + if (!bypass_rlim) {
>>> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>>> + if (locked_vm + pages > limit) {
>>> + ret = -ENOMEM;
>>> + goto out;
>>> + }
>>> + }
>>
>> Nit:
>>
>> if (!ret)
>>
>> and then you don't need "goto out".
>
> Ok, sure.
>
>>> + mm->locked_vm = locked_vm + pages;
>>> + } else {
>>> + WARN_ON_ONCE(pages > locked_vm);
>>> + mm->locked_vm = locked_vm - pages;
>>
>>
>> Can go negative here. Not a huge deal but inaccurate imo.
>
> I hear you, but setting a negative value to zero, as we had done previously,
> doesn't make much sense to me.
Ok then. I have not seen these WARN_ON for a very long time anyway.
--
Alexey
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-24 6:43 ` Alexey Kardashevskiy
0 siblings, 0 replies; 42+ messages in thread
From: Alexey Kardashevskiy @ 2019-05-24 6:43 UTC (permalink / raw)
To: Daniel Jordan
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull, linux-fpga,
linux-kernel, kvm-ppc, linux-mm, Alex Williamson,
Jason Gunthorpe, Moritz Fischer, Steve Sistare, akpm,
linuxppc-dev, Christoph Lameter, Wu Hao
On 21/05/2019 01:30, Daniel Jordan wrote:
> On Mon, May 20, 2019 at 04:19:34PM +1000, Alexey Kardashevskiy wrote:
>> On 04/05/2019 06:16, Daniel Jordan wrote:
>>> locked_vm accounting is done roughly the same way in five places, so
>>> unify them in a helper. Standardize the debug prints, which vary
>>> slightly.
>>
>> And I rather liked that prints were different and tell precisely which
>> one of three each printk is.
>
> I'm not following. One of three...callsites? But there were five callsites.
Well, 3 of them are mine, I was referring to them :)
> Anyway, I added a _RET_IP_ to the debug print so you can differentiate.
I did not know that existed, cool!
>
>> I commented below but in general this seems working.
>>
>> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>
> Thanks! And for the review as well.
>
>>> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
>>> index 6b64e45a5269..d39a1b830d82 100644
>>> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
>>> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
>>> @@ -34,49 +35,13 @@
>>> static void tce_iommu_detach_group(void *iommu_data,
>>> struct iommu_group *iommu_group);
>>>
>>> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
>>> +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
>>> + bool inc)
>>> {
>>> - long ret = 0, locked, lock_limit;
>>> -
>>> if (WARN_ON_ONCE(!mm))
>>> return -EPERM;
>>
>>
>> If this WARN_ON is the only reason for having tce_account_locked_vm()
>> instead of calling account_locked_vm() directly, you can then ditch the
>> check as I have never ever seen this triggered.
>
> Great, will do.
>
>>> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
>>> index d0f731c9920a..15ac76171ccd 100644
>>> --- a/drivers/vfio/vfio_iommu_type1.c
>>> +++ b/drivers/vfio/vfio_iommu_type1.c
>>> @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
>>> return -ESRCH; /* process exited */
>>>
>>> ret = down_write_killable(&mm->mmap_sem);
>>> - if (!ret) {
>>> - if (npage > 0) {
>>> - if (!dma->lock_cap) {
>>> - unsigned long limit;
>>> -
>>> - limit = task_rlimit(dma->task,
>>> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>>> -
>>> - if (mm->locked_vm + npage > limit)
>>> - ret = -ENOMEM;
>>> - }
>>> - }
>>> + if (ret)
>>> + goto out;
>>
>>
>> A single "goto" to jump just 3 lines below seems unnecessary.
>
> No strong preference here, I'll take out the goto.
>
>>> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
>>> + struct task_struct *task, bool bypass_rlim)
>>> +{
>>> + unsigned long locked_vm, limit;
>>> + int ret = 0;
>>> +
>>> + locked_vm = mm->locked_vm;
>>> + if (inc) {
>>> + if (!bypass_rlim) {
>>> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>>> + if (locked_vm + pages > limit) {
>>> + ret = -ENOMEM;
>>> + goto out;
>>> + }
>>> + }
>>
>> Nit:
>>
>> if (!ret)
>>
>> and then you don't need "goto out".
>
> Ok, sure.
>
>>> + mm->locked_vm = locked_vm + pages;
>>> + } else {
>>> + WARN_ON_ONCE(pages > locked_vm);
>>> + mm->locked_vm = locked_vm - pages;
>>
>>
>> Can go negative here. Not a huge deal but inaccurate imo.
>
> I hear you, but setting a negative value to zero, as we had done previously,
> doesn't make much sense to me.
Ok then. I have not seen these WARN_ON for a very long time anyway.
--
Alexey
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH] mm: add account_locked_vm utility function
@ 2019-05-24 6:43 ` Alexey Kardashevskiy
0 siblings, 0 replies; 42+ messages in thread
From: Alexey Kardashevskiy @ 2019-05-24 6:43 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alan Tull, Alex Williamson, Benjamin Herrenschmidt,
Christoph Lameter, Christophe Leroy, Davidlohr Bueso,
Jason Gunthorpe, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On 21/05/2019 01:30, Daniel Jordan wrote:
> On Mon, May 20, 2019 at 04:19:34PM +1000, Alexey Kardashevskiy wrote:
>> On 04/05/2019 06:16, Daniel Jordan wrote:
>>> locked_vm accounting is done roughly the same way in five places, so
>>> unify them in a helper. Standardize the debug prints, which vary
>>> slightly.
>>
>> And I rather liked that prints were different and tell precisely which
>> one of three each printk is.
>
> I'm not following. One of three...callsites? But there were five callsites.
Well, 3 of them are mine, I was referring to them :)
> Anyway, I added a _RET_IP_ to the debug print so you can differentiate.
I did not know that existed, cool!
>
>> I commented below but in general this seems working.
>>
>> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>
> Thanks! And for the review as well.
>
>>> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
>>> index 6b64e45a5269..d39a1b830d82 100644
>>> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
>>> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
>>> @@ -34,49 +35,13 @@
>>> static void tce_iommu_detach_group(void *iommu_data,
>>> struct iommu_group *iommu_group);
>>>
>>> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
>>> +static int tce_account_locked_vm(struct mm_struct *mm, unsigned long npages,
>>> + bool inc)
>>> {
>>> - long ret = 0, locked, lock_limit;
>>> -
>>> if (WARN_ON_ONCE(!mm))
>>> return -EPERM;
>>
>>
>> If this WARN_ON is the only reason for having tce_account_locked_vm()
>> instead of calling account_locked_vm() directly, you can then ditch the
>> check as I have never ever seen this triggered.
>
> Great, will do.
>
>>> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
>>> index d0f731c9920a..15ac76171ccd 100644
>>> --- a/drivers/vfio/vfio_iommu_type1.c
>>> +++ b/drivers/vfio/vfio_iommu_type1.c
>>> @@ -273,25 +273,14 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
>>> return -ESRCH; /* process exited */
>>>
>>> ret = down_write_killable(&mm->mmap_sem);
>>> - if (!ret) {
>>> - if (npage > 0) {
>>> - if (!dma->lock_cap) {
>>> - unsigned long limit;
>>> -
>>> - limit = task_rlimit(dma->task,
>>> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>>> -
>>> - if (mm->locked_vm + npage > limit)
>>> - ret = -ENOMEM;
>>> - }
>>> - }
>>> + if (ret)
>>> + goto out;
>>
>>
>> A single "goto" to jump just 3 lines below seems unnecessary.
>
> No strong preference here, I'll take out the goto.
>
>>> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
>>> + struct task_struct *task, bool bypass_rlim)
>>> +{
>>> + unsigned long locked_vm, limit;
>>> + int ret = 0;
>>> +
>>> + locked_vm = mm->locked_vm;
>>> + if (inc) {
>>> + if (!bypass_rlim) {
>>> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>>> + if (locked_vm + pages > limit) {
>>> + ret = -ENOMEM;
>>> + goto out;
>>> + }
>>> + }
>>
>> Nit:
>>
>> if (!ret)
>>
>> and then you don't need "goto out".
>
> Ok, sure.
>
>>> + mm->locked_vm = locked_vm + pages;
>>> + } else {
>>> + WARN_ON_ONCE(pages > locked_vm);
>>> + mm->locked_vm = locked_vm - pages;
>>
>>
>> Can go negative here. Not a huge deal but inaccurate imo.
>
> I hear you, but setting a negative value to zero, as we had done previously,
> doesn't make much sense to me.
Ok then. I have not seen these WARN_ON for a very long time anyway.
--
Alexey
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v2] mm: add account_locked_vm utility function
2019-05-24 6:43 ` Alexey Kardashevskiy
(?)
@ 2019-05-24 17:50 ` Daniel Jordan
-1 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-24 17:50 UTC (permalink / raw)
To: akpm
Cc: Daniel Jordan, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper. Standardize the debug prints, which vary
slightly, but include the helper's caller to disambiguate between
callsites.
Error codes stay the same, so user-visible behavior does too. The one
exception is that the -EPERM case in tce_account_locked_vm is removed
because Alexey has never seen it triggered.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alan Tull <atull@kernel.org>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Against v5.2-rc1.
v2:
- applied review comments from Alexey
- added _RET_IP_ to debug print to disambiguate callers
arch/powerpc/kvm/book3s_64_vio.c | 44 +++--------------------
arch/powerpc/mm/book3s64/iommu_api.c | 41 +++------------------
drivers/fpga/dfl-afu-dma-region.c | 53 +++------------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 54 +++-------------------------
drivers/vfio/vfio_iommu_type1.c | 17 ++-------
include/linux/mm.h | 19 ++++++++++
mm/util.c | 46 ++++++++++++++++++++++++
7 files changed, 84 insertions(+), 190 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 66270e07449a..768b645c7edf 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
index 5c521f3924a5..18d22eec0ebd 100644
--- a/arch/powerpc/mm/book3s64/iommu_api.c
+++ b/arch/powerpc/mm/book3s64/iommu_api.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index c438722bf4e1..0a532c602d8f 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 40ddc0c5f677..d06e8e291924 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,51 +35,6 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
-{
- long ret = 0, locked, lock_limit;
-
- if (WARN_ON_ONCE(!mm))
- return -EPERM;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-}
-
/*
* VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
*
@@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
if (ret)
return ret;
@@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 3ddc375e7063..bf449ace1676 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
ret = down_write_killable(&mm->mmap_sem);
if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
-
- if (!ret)
- mm->locked_vm += npage;
-
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
up_write(&mm->mmap_sem);
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0e8834ac32b7..72c1034d2ec7 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
int get_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
+static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
+ bool inc)
+{
+ int ret;
+
+ if (pages == 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index e2e4f8c3fa12..bd3bdf16a084 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -6,6 +6,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -346,6 +347,51 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * 0 if @mm is NULL (can happen for example if the task is exiting)
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit)
+ ret = -ENOMEM;
+ }
+ if (!ret)
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+ pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: a188339ca5a396acc588e5851ed7e19f66b0ebd9
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-24 17:50 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-24 17:50 UTC (permalink / raw)
To: akpm
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
Daniel Jordan, linux-mm, Alex Williamson, Jason Gunthorpe,
Moritz Fischer, Steve Sistare, Christoph Lameter, linuxppc-dev,
Wu Hao
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper. Standardize the debug prints, which vary
slightly, but include the helper's caller to disambiguate between
callsites.
Error codes stay the same, so user-visible behavior does too. The one
exception is that the -EPERM case in tce_account_locked_vm is removed
because Alexey has never seen it triggered.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alan Tull <atull@kernel.org>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Against v5.2-rc1.
v2:
- applied review comments from Alexey
- added _RET_IP_ to debug print to disambiguate callers
arch/powerpc/kvm/book3s_64_vio.c | 44 +++--------------------
arch/powerpc/mm/book3s64/iommu_api.c | 41 +++------------------
drivers/fpga/dfl-afu-dma-region.c | 53 +++------------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 54 +++-------------------------
drivers/vfio/vfio_iommu_type1.c | 17 ++-------
include/linux/mm.h | 19 ++++++++++
mm/util.c | 46 ++++++++++++++++++++++++
7 files changed, 84 insertions(+), 190 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 66270e07449a..768b645c7edf 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
index 5c521f3924a5..18d22eec0ebd 100644
--- a/arch/powerpc/mm/book3s64/iommu_api.c
+++ b/arch/powerpc/mm/book3s64/iommu_api.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index c438722bf4e1..0a532c602d8f 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 40ddc0c5f677..d06e8e291924 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,51 +35,6 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
-{
- long ret = 0, locked, lock_limit;
-
- if (WARN_ON_ONCE(!mm))
- return -EPERM;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-}
-
/*
* VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
*
@@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
if (ret)
return ret;
@@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 3ddc375e7063..bf449ace1676 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
ret = down_write_killable(&mm->mmap_sem);
if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
-
- if (!ret)
- mm->locked_vm += npage;
-
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
up_write(&mm->mmap_sem);
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0e8834ac32b7..72c1034d2ec7 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
int get_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
+static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
+ bool inc)
+{
+ int ret;
+
+ if (pages == 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index e2e4f8c3fa12..bd3bdf16a084 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -6,6 +6,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -346,6 +347,51 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * 0 if @mm is NULL (can happen for example if the task is exiting)
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit)
+ ret = -ENOMEM;
+ }
+ if (!ret)
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+ pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: a188339ca5a396acc588e5851ed7e19f66b0ebd9
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-24 17:50 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-24 17:50 UTC (permalink / raw)
To: akpm
Cc: Daniel Jordan, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper. Standardize the debug prints, which vary
slightly, but include the helper's caller to disambiguate between
callsites.
Error codes stay the same, so user-visible behavior does too. The one
exception is that the -EPERM case in tce_account_locked_vm is removed
because Alexey has never seen it triggered.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alan Tull <atull@kernel.org>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Against v5.2-rc1.
v2:
- applied review comments from Alexey
- added _RET_IP_ to debug print to disambiguate callers
arch/powerpc/kvm/book3s_64_vio.c | 44 +++--------------------
arch/powerpc/mm/book3s64/iommu_api.c | 41 +++------------------
drivers/fpga/dfl-afu-dma-region.c | 53 +++------------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 54 +++-------------------------
drivers/vfio/vfio_iommu_type1.c | 17 ++-------
include/linux/mm.h | 19 ++++++++++
mm/util.c | 46 ++++++++++++++++++++++++
7 files changed, 84 insertions(+), 190 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 66270e07449a..768b645c7edf 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
index 5c521f3924a5..18d22eec0ebd 100644
--- a/arch/powerpc/mm/book3s64/iommu_api.c
+++ b/arch/powerpc/mm/book3s64/iommu_api.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa = MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index c438722bf4e1..0a532c602d8f 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 40ddc0c5f677..d06e8e291924 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,51 +35,6 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
-{
- long ret = 0, locked, lock_limit;
-
- if (WARN_ON_ONCE(!mm))
- return -EPERM;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-}
-
/*
* VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
*
@@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
if (ret)
return ret;
@@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 3ddc375e7063..bf449ace1676 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
ret = down_write_killable(&mm->mmap_sem);
if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
-
- if (!ret)
- mm->locked_vm += npage;
-
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
up_write(&mm->mmap_sem);
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0e8834ac32b7..72c1034d2ec7 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
int get_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
+static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
+ bool inc)
+{
+ int ret;
+
+ if (pages = 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index e2e4f8c3fa12..bd3bdf16a084 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -6,6 +6,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -346,6 +347,51 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * 0 if @mm is NULL (can happen for example if the task is exiting)
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit)
+ ret = -ENOMEM;
+ }
+ if (!ret)
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+ pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: a188339ca5a396acc588e5851ed7e19f66b0ebd9
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
2019-05-24 17:50 ` Daniel Jordan
(?)
@ 2019-05-25 21:51 ` Andrew Morton
-1 siblings, 0 replies; 42+ messages in thread
From: Andrew Morton @ 2019-05-25 21:51 UTC (permalink / raw)
To: Daniel Jordan
Cc: Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly, but include the helper's caller to disambiguate between
> callsites.
>
> Error codes stay the same, so user-visible behavior does too. The one
> exception is that the -EPERM case in tce_account_locked_vm is removed
> because Alexey has never seen it triggered.
>
> ...
>
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
That's quite a mouthful for an inlined function. How about uninlining
the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
I wonder why it does down_write_killable and whether it really needs
to...
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-25 21:51 ` Andrew Morton
0 siblings, 0 replies; 42+ messages in thread
From: Andrew Morton @ 2019-05-25 21:51 UTC (permalink / raw)
To: Daniel Jordan
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
linux-mm, Alex Williamson, Jason Gunthorpe, Moritz Fischer,
Steve Sistare, Christoph Lameter, linuxppc-dev, Wu Hao
On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly, but include the helper's caller to disambiguate between
> callsites.
>
> Error codes stay the same, so user-visible behavior does too. The one
> exception is that the -EPERM case in tce_account_locked_vm is removed
> because Alexey has never seen it triggered.
>
> ...
>
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
That's quite a mouthful for an inlined function. How about uninlining
the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
I wonder why it does down_write_killable and whether it really needs
to...
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-25 21:51 ` Andrew Morton
0 siblings, 0 replies; 42+ messages in thread
From: Andrew Morton @ 2019-05-25 21:51 UTC (permalink / raw)
To: Daniel Jordan
Cc: Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper. Standardize the debug prints, which vary
> slightly, but include the helper's caller to disambiguate between
> callsites.
>
> Error codes stay the same, so user-visible behavior does too. The one
> exception is that the -EPERM case in tce_account_locked_vm is removed
> because Alexey has never seen it triggered.
>
> ...
>
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages = 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
That's quite a mouthful for an inlined function. How about uninlining
the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
I wonder why it does down_write_killable and whether it really needs
to...
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
2019-05-25 21:51 ` Andrew Morton
(?)
@ 2019-05-28 15:04 ` Daniel Jordan
-1 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-28 15:04 UTC (permalink / raw)
To: Andrew Morton
Cc: Daniel Jordan, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Sat, May 25, 2019 at 02:51:18PM -0700, Andrew Morton wrote:
> On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
>
> > locked_vm accounting is done roughly the same way in five places, so
> > unify them in a helper. Standardize the debug prints, which vary
> > slightly, but include the helper's caller to disambiguate between
> > callsites.
> >
> > Error codes stay the same, so user-visible behavior does too. The one
> > exception is that the -EPERM case in tce_account_locked_vm is removed
> > because Alexey has never seen it triggered.
> >
> > ...
> >
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> > int get_user_pages_fast(unsigned long start, int nr_pages,
> > unsigned int gup_flags, struct page **pages);
> >
> > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > + struct task_struct *task, bool bypass_rlim);
> > +
> > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > + bool inc)
> > +{
> > + int ret;
> > +
> > + if (pages == 0 || !mm)
> > + return 0;
> > +
> > + down_write(&mm->mmap_sem);
> > + ret = __account_locked_vm(mm, pages, inc, current,
> > + capable(CAP_IPC_LOCK));
> > + up_write(&mm->mmap_sem);
> > +
> > + return ret;
> > +}
>
> That's quite a mouthful for an inlined function. How about uninlining
> the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
> I wonder why it does down_write_killable and whether it really needs
> to...
Sure, I can uninline it. vfio changelogs don't show a particular reason for
_killable[1]. Maybe Alex has something to add. Otherwise I'll respin without
it since the simplification seems worth removing _killable.
[1] 0cfef2b7410b ("vfio/type1: Remove locked page accounting workqueue")
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-28 15:04 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-28 15:04 UTC (permalink / raw)
To: Andrew Morton
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
Daniel Jordan, linux-mm, Alex Williamson, Jason Gunthorpe,
Moritz Fischer, Steve Sistare, Christoph Lameter, linuxppc-dev,
Wu Hao
On Sat, May 25, 2019 at 02:51:18PM -0700, Andrew Morton wrote:
> On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
>
> > locked_vm accounting is done roughly the same way in five places, so
> > unify them in a helper. Standardize the debug prints, which vary
> > slightly, but include the helper's caller to disambiguate between
> > callsites.
> >
> > Error codes stay the same, so user-visible behavior does too. The one
> > exception is that the -EPERM case in tce_account_locked_vm is removed
> > because Alexey has never seen it triggered.
> >
> > ...
> >
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> > int get_user_pages_fast(unsigned long start, int nr_pages,
> > unsigned int gup_flags, struct page **pages);
> >
> > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > + struct task_struct *task, bool bypass_rlim);
> > +
> > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > + bool inc)
> > +{
> > + int ret;
> > +
> > + if (pages == 0 || !mm)
> > + return 0;
> > +
> > + down_write(&mm->mmap_sem);
> > + ret = __account_locked_vm(mm, pages, inc, current,
> > + capable(CAP_IPC_LOCK));
> > + up_write(&mm->mmap_sem);
> > +
> > + return ret;
> > +}
>
> That's quite a mouthful for an inlined function. How about uninlining
> the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
> I wonder why it does down_write_killable and whether it really needs
> to...
Sure, I can uninline it. vfio changelogs don't show a particular reason for
_killable[1]. Maybe Alex has something to add. Otherwise I'll respin without
it since the simplification seems worth removing _killable.
[1] 0cfef2b7410b ("vfio/type1: Remove locked page accounting workqueue")
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-28 15:04 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-28 15:04 UTC (permalink / raw)
To: Andrew Morton
Cc: Daniel Jordan, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Sat, May 25, 2019 at 02:51:18PM -0700, Andrew Morton wrote:
> On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
>
> > locked_vm accounting is done roughly the same way in five places, so
> > unify them in a helper. Standardize the debug prints, which vary
> > slightly, but include the helper's caller to disambiguate between
> > callsites.
> >
> > Error codes stay the same, so user-visible behavior does too. The one
> > exception is that the -EPERM case in tce_account_locked_vm is removed
> > because Alexey has never seen it triggered.
> >
> > ...
> >
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> > int get_user_pages_fast(unsigned long start, int nr_pages,
> > unsigned int gup_flags, struct page **pages);
> >
> > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > + struct task_struct *task, bool bypass_rlim);
> > +
> > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > + bool inc)
> > +{
> > + int ret;
> > +
> > + if (pages = 0 || !mm)
> > + return 0;
> > +
> > + down_write(&mm->mmap_sem);
> > + ret = __account_locked_vm(mm, pages, inc, current,
> > + capable(CAP_IPC_LOCK));
> > + up_write(&mm->mmap_sem);
> > +
> > + return ret;
> > +}
>
> That's quite a mouthful for an inlined function. How about uninlining
> the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
> I wonder why it does down_write_killable and whether it really needs
> to...
Sure, I can uninline it. vfio changelogs don't show a particular reason for
_killable[1]. Maybe Alex has something to add. Otherwise I'll respin without
it since the simplification seems worth removing _killable.
[1] 0cfef2b7410b ("vfio/type1: Remove locked page accounting workqueue")
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
2019-05-24 17:50 ` Daniel Jordan
(?)
@ 2019-05-29 18:05 ` Ira Weiny
-1 siblings, 0 replies; 42+ messages in thread
From: Ira Weiny @ 2019-05-29 18:05 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Fri, May 24, 2019 at 01:50:45PM -0400, Daniel Jordan wrote:
[snip]
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0e8834ac32b7..72c1034d2ec7 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index e2e4f8c3fa12..bd3bdf16a084 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -6,6 +6,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -346,6 +347,51 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
This kernel doc is wrong. You dereference mm straight away...
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * 0 if @mm is NULL (can happen for example if the task is exiting)
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + locked_vm = mm->locked_vm;
here...
Perhaps the comment was meant to document account_locked_vm()? Or should the
parameter checks be moved here?
Ira
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit)
> + ret = -ENOMEM;
> + }
> + if (!ret)
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
> + }
> +
> + pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
>
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: a188339ca5a396acc588e5851ed7e19f66b0ebd9
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-29 18:05 ` Ira Weiny
0 siblings, 0 replies; 42+ messages in thread
From: Ira Weiny @ 2019-05-29 18:05 UTC (permalink / raw)
To: Daniel Jordan
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
linux-mm, Alex Williamson, Jason Gunthorpe, Moritz Fischer,
Steve Sistare, akpm, linuxppc-dev, Christoph Lameter, Wu Hao
On Fri, May 24, 2019 at 01:50:45PM -0400, Daniel Jordan wrote:
[snip]
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0e8834ac32b7..72c1034d2ec7 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index e2e4f8c3fa12..bd3bdf16a084 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -6,6 +6,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -346,6 +347,51 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
This kernel doc is wrong. You dereference mm straight away...
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * 0 if @mm is NULL (can happen for example if the task is exiting)
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + locked_vm = mm->locked_vm;
here...
Perhaps the comment was meant to document account_locked_vm()? Or should the
parameter checks be moved here?
Ira
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit)
> + ret = -ENOMEM;
> + }
> + if (!ret)
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
> + }
> +
> + pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
>
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: a188339ca5a396acc588e5851ed7e19f66b0ebd9
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-29 18:05 ` Ira Weiny
0 siblings, 0 replies; 42+ messages in thread
From: Ira Weiny @ 2019-05-29 18:05 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Fri, May 24, 2019 at 01:50:45PM -0400, Daniel Jordan wrote:
[snip]
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0e8834ac32b7..72c1034d2ec7 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> + bool inc)
> +{
> + int ret;
> +
> + if (pages = 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index e2e4f8c3fa12..bd3bdf16a084 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -6,6 +6,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -346,6 +347,51 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
This kernel doc is wrong. You dereference mm straight away...
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * 0 if @mm is NULL (can happen for example if the task is exiting)
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + locked_vm = mm->locked_vm;
here...
Perhaps the comment was meant to document account_locked_vm()? Or should the
parameter checks be moved here?
Ira
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit)
> + ret = -ENOMEM;
> + }
> + if (!ret)
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
> + }
> +
> + pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
>
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: a188339ca5a396acc588e5851ed7e19f66b0ebd9
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
2019-05-29 18:05 ` Ira Weiny
(?)
@ 2019-05-29 18:35 ` Daniel Jordan
-1 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-29 18:35 UTC (permalink / raw)
To: Ira Weiny
Cc: Daniel Jordan, akpm, Alexey Kardashevskiy, Alan Tull,
Alex Williamson, Benjamin Herrenschmidt, Christoph Lameter,
Christophe Leroy, Davidlohr Bueso, Jason Gunthorpe, Mark Rutland,
Michael Ellerman, Moritz Fischer, Paul Mackerras, Steve Sistare,
Wu Hao, linux-mm, kvm, kvm-ppc, linuxppc-dev, linux-fpga,
linux-kernel
On Wed, May 29, 2019 at 11:05:48AM -0700, Ira Weiny wrote:
> On Fri, May 24, 2019 at 01:50:45PM -0400, Daniel Jordan wrote:
> > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > + bool inc)
> > +{
> > + int ret;
> > +
> > + if (pages == 0 || !mm)
> > + return 0;
> > +
> > + down_write(&mm->mmap_sem);
> > + ret = __account_locked_vm(mm, pages, inc, current,
> > + capable(CAP_IPC_LOCK));
> > + up_write(&mm->mmap_sem);
> > +
> > + return ret;
> > +}
> > +
...snip...
> > +/**
> > + * __account_locked_vm - account locked pages to an mm's locked_vm
> > + * @mm: mm to account against, may be NULL
>
> This kernel doc is wrong. You dereference mm straight away...
...snip...
> > +
> > + locked_vm = mm->locked_vm;
>
> here...
>
> Perhaps the comment was meant to document account_locked_vm()?
Yes, the comment got out of sync when I moved the !mm check outside
__account_locked_vm. Thanks for catching, will fix.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-29 18:35 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-29 18:35 UTC (permalink / raw)
To: Ira Weiny
Cc: Mark Rutland, kvm, Alexey Kardashevskiy, linux-fpga, linux-mm,
Steve Sistare, Christoph Lameter, Davidlohr Bueso, Daniel Jordan,
Jason Gunthorpe, Wu Hao, Alan Tull, kvm-ppc, Alex Williamson,
Moritz Fischer, linux-kernel, akpm, linuxppc-dev
On Wed, May 29, 2019 at 11:05:48AM -0700, Ira Weiny wrote:
> On Fri, May 24, 2019 at 01:50:45PM -0400, Daniel Jordan wrote:
> > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > + bool inc)
> > +{
> > + int ret;
> > +
> > + if (pages == 0 || !mm)
> > + return 0;
> > +
> > + down_write(&mm->mmap_sem);
> > + ret = __account_locked_vm(mm, pages, inc, current,
> > + capable(CAP_IPC_LOCK));
> > + up_write(&mm->mmap_sem);
> > +
> > + return ret;
> > +}
> > +
...snip...
> > +/**
> > + * __account_locked_vm - account locked pages to an mm's locked_vm
> > + * @mm: mm to account against, may be NULL
>
> This kernel doc is wrong. You dereference mm straight away...
...snip...
> > +
> > + locked_vm = mm->locked_vm;
>
> here...
>
> Perhaps the comment was meant to document account_locked_vm()?
Yes, the comment got out of sync when I moved the !mm check outside
__account_locked_vm. Thanks for catching, will fix.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-29 18:35 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-29 18:35 UTC (permalink / raw)
To: Ira Weiny
Cc: Daniel Jordan, akpm, Alexey Kardashevskiy, Alan Tull,
Alex Williamson, Benjamin Herrenschmidt, Christoph Lameter,
Christophe Leroy, Davidlohr Bueso, Jason Gunthorpe, Mark Rutland,
Michael Ellerman, Moritz Fischer, Paul Mackerras, Steve Sistare,
Wu Hao, linux-mm, kvm, kvm-ppc, linuxppc-dev, linux-fpga,
linux-kernel
On Wed, May 29, 2019 at 11:05:48AM -0700, Ira Weiny wrote:
> On Fri, May 24, 2019 at 01:50:45PM -0400, Daniel Jordan wrote:
> > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > + bool inc)
> > +{
> > + int ret;
> > +
> > + if (pages = 0 || !mm)
> > + return 0;
> > +
> > + down_write(&mm->mmap_sem);
> > + ret = __account_locked_vm(mm, pages, inc, current,
> > + capable(CAP_IPC_LOCK));
> > + up_write(&mm->mmap_sem);
> > +
> > + return ret;
> > +}
> > +
...snip...
> > +/**
> > + * __account_locked_vm - account locked pages to an mm's locked_vm
> > + * @mm: mm to account against, may be NULL
>
> This kernel doc is wrong. You dereference mm straight away...
...snip...
> > +
> > + locked_vm = mm->locked_vm;
>
> here...
>
> Perhaps the comment was meant to document account_locked_vm()?
Yes, the comment got out of sync when I moved the !mm check outside
__account_locked_vm. Thanks for catching, will fix.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
2019-05-28 15:04 ` Daniel Jordan
(?)
@ 2019-05-29 18:56 ` Alex Williamson
-1 siblings, 0 replies; 42+ messages in thread
From: Alex Williamson @ 2019-05-29 18:56 UTC (permalink / raw)
To: Daniel Jordan
Cc: Andrew Morton, Alexey Kardashevskiy, Alan Tull,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Tue, 28 May 2019 11:04:24 -0400
Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> On Sat, May 25, 2019 at 02:51:18PM -0700, Andrew Morton wrote:
> > On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> >
> > > locked_vm accounting is done roughly the same way in five places, so
> > > unify them in a helper. Standardize the debug prints, which vary
> > > slightly, but include the helper's caller to disambiguate between
> > > callsites.
> > >
> > > Error codes stay the same, so user-visible behavior does too. The one
> > > exception is that the -EPERM case in tce_account_locked_vm is removed
> > > because Alexey has never seen it triggered.
> > >
> > > ...
> > >
> > > --- a/include/linux/mm.h
> > > +++ b/include/linux/mm.h
> > > @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> > > int get_user_pages_fast(unsigned long start, int nr_pages,
> > > unsigned int gup_flags, struct page **pages);
> > >
> > > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > > + struct task_struct *task, bool bypass_rlim);
> > > +
> > > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > > + bool inc)
> > > +{
> > > + int ret;
> > > +
> > > + if (pages == 0 || !mm)
> > > + return 0;
> > > +
> > > + down_write(&mm->mmap_sem);
> > > + ret = __account_locked_vm(mm, pages, inc, current,
> > > + capable(CAP_IPC_LOCK));
> > > + up_write(&mm->mmap_sem);
> > > +
> > > + return ret;
> > > +}
> >
> > That's quite a mouthful for an inlined function. How about uninlining
> > the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
> > I wonder why it does down_write_killable and whether it really needs
> > to...
>
> Sure, I can uninline it. vfio changelogs don't show a particular reason for
> _killable[1]. Maybe Alex has something to add. Otherwise I'll respin without
> it since the simplification seems worth removing _killable.
>
> [1] 0cfef2b7410b ("vfio/type1: Remove locked page accounting workqueue")
A userspace vfio driver maps DMA via an ioctl through this path, so I
believe I used killable here just to be friendly that it could be
interrupted and we could fall out with an errno if it were stuck here.
No harm, no foul, the user's mapping is aborted and unwound. If we're
deadlocked or seriously contended on mmap_sem, maybe we're already in
trouble, but it seemed like a valid and low hanging use case for
killable. Thanks,
Alex
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-29 18:56 ` Alex Williamson
0 siblings, 0 replies; 42+ messages in thread
From: Alex Williamson @ 2019-05-29 18:56 UTC (permalink / raw)
To: Daniel Jordan
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linux-kernel, kvm-ppc,
linux-mm, Jason Gunthorpe, Moritz Fischer, Steve Sistare,
Andrew Morton, linuxppc-dev, Christoph Lameter, Wu Hao
On Tue, 28 May 2019 11:04:24 -0400
Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> On Sat, May 25, 2019 at 02:51:18PM -0700, Andrew Morton wrote:
> > On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> >
> > > locked_vm accounting is done roughly the same way in five places, so
> > > unify them in a helper. Standardize the debug prints, which vary
> > > slightly, but include the helper's caller to disambiguate between
> > > callsites.
> > >
> > > Error codes stay the same, so user-visible behavior does too. The one
> > > exception is that the -EPERM case in tce_account_locked_vm is removed
> > > because Alexey has never seen it triggered.
> > >
> > > ...
> > >
> > > --- a/include/linux/mm.h
> > > +++ b/include/linux/mm.h
> > > @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> > > int get_user_pages_fast(unsigned long start, int nr_pages,
> > > unsigned int gup_flags, struct page **pages);
> > >
> > > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > > + struct task_struct *task, bool bypass_rlim);
> > > +
> > > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > > + bool inc)
> > > +{
> > > + int ret;
> > > +
> > > + if (pages == 0 || !mm)
> > > + return 0;
> > > +
> > > + down_write(&mm->mmap_sem);
> > > + ret = __account_locked_vm(mm, pages, inc, current,
> > > + capable(CAP_IPC_LOCK));
> > > + up_write(&mm->mmap_sem);
> > > +
> > > + return ret;
> > > +}
> >
> > That's quite a mouthful for an inlined function. How about uninlining
> > the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
> > I wonder why it does down_write_killable and whether it really needs
> > to...
>
> Sure, I can uninline it. vfio changelogs don't show a particular reason for
> _killable[1]. Maybe Alex has something to add. Otherwise I'll respin without
> it since the simplification seems worth removing _killable.
>
> [1] 0cfef2b7410b ("vfio/type1: Remove locked page accounting workqueue")
A userspace vfio driver maps DMA via an ioctl through this path, so I
believe I used killable here just to be friendly that it could be
interrupted and we could fall out with an errno if it were stuck here.
No harm, no foul, the user's mapping is aborted and unwound. If we're
deadlocked or seriously contended on mmap_sem, maybe we're already in
trouble, but it seemed like a valid and low hanging use case for
killable. Thanks,
Alex
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v2] mm: add account_locked_vm utility function
@ 2019-05-29 18:56 ` Alex Williamson
0 siblings, 0 replies; 42+ messages in thread
From: Alex Williamson @ 2019-05-29 18:56 UTC (permalink / raw)
To: Daniel Jordan
Cc: Andrew Morton, Alexey Kardashevskiy, Alan Tull,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Jason Gunthorpe, Mark Rutland, Michael Ellerman,
Moritz Fischer, Paul Mackerras, Steve Sistare, Wu Hao, linux-mm,
kvm, kvm-ppc, linuxppc-dev, linux-fpga, linux-kernel
On Tue, 28 May 2019 11:04:24 -0400
Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> On Sat, May 25, 2019 at 02:51:18PM -0700, Andrew Morton wrote:
> > On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> >
> > > locked_vm accounting is done roughly the same way in five places, so
> > > unify them in a helper. Standardize the debug prints, which vary
> > > slightly, but include the helper's caller to disambiguate between
> > > callsites.
> > >
> > > Error codes stay the same, so user-visible behavior does too. The one
> > > exception is that the -EPERM case in tce_account_locked_vm is removed
> > > because Alexey has never seen it triggered.
> > >
> > > ...
> > >
> > > --- a/include/linux/mm.h
> > > +++ b/include/linux/mm.h
> > > @@ -1564,6 +1564,25 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> > > int get_user_pages_fast(unsigned long start, int nr_pages,
> > > unsigned int gup_flags, struct page **pages);
> > >
> > > +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> > > + struct task_struct *task, bool bypass_rlim);
> > > +
> > > +static inline int account_locked_vm(struct mm_struct *mm, unsigned long pages,
> > > + bool inc)
> > > +{
> > > + int ret;
> > > +
> > > + if (pages = 0 || !mm)
> > > + return 0;
> > > +
> > > + down_write(&mm->mmap_sem);
> > > + ret = __account_locked_vm(mm, pages, inc, current,
> > > + capable(CAP_IPC_LOCK));
> > > + up_write(&mm->mmap_sem);
> > > +
> > > + return ret;
> > > +}
> >
> > That's quite a mouthful for an inlined function. How about uninlining
> > the whole thing and fiddling drivers/vfio/vfio_iommu_type1.c to suit.
> > I wonder why it does down_write_killable and whether it really needs
> > to...
>
> Sure, I can uninline it. vfio changelogs don't show a particular reason for
> _killable[1]. Maybe Alex has something to add. Otherwise I'll respin without
> it since the simplification seems worth removing _killable.
>
> [1] 0cfef2b7410b ("vfio/type1: Remove locked page accounting workqueue")
A userspace vfio driver maps DMA via an ioctl through this path, so I
believe I used killable here just to be friendly that it could be
interrupted and we could fall out with an errno if it were stuck here.
No harm, no foul, the user's mapping is aborted and unwound. If we're
deadlocked or seriously contended on mmap_sem, maybe we're already in
trouble, but it seemed like a valid and low hanging use case for
killable. Thanks,
Alex
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v3] mm: add account_locked_vm utility function
2019-05-29 18:56 ` Alex Williamson
(?)
@ 2019-05-29 20:50 ` Daniel Jordan
-1 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-29 20:50 UTC (permalink / raw)
To: akpm
Cc: Daniel Jordan, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Ira Weiny, Jason Gunthorpe, Mark Rutland,
Michael Ellerman, Moritz Fischer, Paul Mackerras, Steve Sistare,
Wu Hao, linux-mm, kvm, kvm-ppc, linuxppc-dev, linux-fpga,
linux-kernel
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper.
Include the helper's caller in the debug print to distinguish between
callsites.
Error codes stay the same, so user-visible behavior does too. The one
exception is that the -EPERM case in tce_account_locked_vm is removed
because Alexey has never seen it triggered.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alan Tull <atull@kernel.org>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
v3:
- uninline account_locked_vm (Andrew)
- fix doc comment (Ira)
- retain down_write_killable in vfio type1 (Alex)
- leave Alexey's T-b since the code is the same aside from uninlining
- sanity tested with vfio type1, sanity-built on ppc
arch/powerpc/kvm/book3s_64_vio.c | 44 ++--------------
arch/powerpc/mm/book3s64/iommu_api.c | 41 ++-------------
drivers/fpga/dfl-afu-dma-region.c | 53 ++------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 54 ++------------------
drivers/vfio/vfio_iommu_type1.c | 17 +------
include/linux/mm.h | 4 ++
mm/util.c | 75 ++++++++++++++++++++++++++++
7 files changed, 98 insertions(+), 190 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 66270e07449a..768b645c7edf 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
index 5c521f3924a5..18d22eec0ebd 100644
--- a/arch/powerpc/mm/book3s64/iommu_api.c
+++ b/arch/powerpc/mm/book3s64/iommu_api.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index c438722bf4e1..0a532c602d8f 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 40ddc0c5f677..d06e8e291924 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,51 +35,6 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
-{
- long ret = 0, locked, lock_limit;
-
- if (WARN_ON_ONCE(!mm))
- return -EPERM;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-}
-
/*
* VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
*
@@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
if (ret)
return ret;
@@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 3ddc375e7063..bf449ace1676 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
ret = down_write_killable(&mm->mmap_sem);
if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
-
- if (!ret)
- mm->locked_vm += npage;
-
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
up_write(&mm->mmap_sem);
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0e8834ac32b7..95510f6fad45 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1564,6 +1564,10 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
int get_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
+int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index 91682a2090ee..cbbcc035b12b 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -7,6 +7,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -347,6 +348,80 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ lockdep_assert_held_exclusive(&mm->mmap_sem);
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit)
+ ret = -ENOMEM;
+ }
+ if (!ret)
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+ pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
+/**
+ * account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ *
+ * Assumes a non-NULL @mm is valid (i.e. at least one reference on it).
+ *
+ * Return:
+ * * 0 on success, or if mm is NULL
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc)
+{
+ int ret;
+
+ if (pages == 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: cd6c84d8f0cdc911df435bb075ba22ce3c605b07
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH v3] mm: add account_locked_vm utility function
@ 2019-05-29 20:50 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-29 20:50 UTC (permalink / raw)
To: akpm
Cc: Mark Rutland, kvm, Alexey Kardashevskiy, linux-fpga, linux-mm,
Steve Sistare, Christoph Lameter, Ira Weiny, Davidlohr Bueso,
Daniel Jordan, Jason Gunthorpe, Wu Hao, Alan Tull, kvm-ppc,
Alex Williamson, Moritz Fischer, linux-kernel, linuxppc-dev
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper.
Include the helper's caller in the debug print to distinguish between
callsites.
Error codes stay the same, so user-visible behavior does too. The one
exception is that the -EPERM case in tce_account_locked_vm is removed
because Alexey has never seen it triggered.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alan Tull <atull@kernel.org>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
v3:
- uninline account_locked_vm (Andrew)
- fix doc comment (Ira)
- retain down_write_killable in vfio type1 (Alex)
- leave Alexey's T-b since the code is the same aside from uninlining
- sanity tested with vfio type1, sanity-built on ppc
arch/powerpc/kvm/book3s_64_vio.c | 44 ++--------------
arch/powerpc/mm/book3s64/iommu_api.c | 41 ++-------------
drivers/fpga/dfl-afu-dma-region.c | 53 ++------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 54 ++------------------
drivers/vfio/vfio_iommu_type1.c | 17 +------
include/linux/mm.h | 4 ++
mm/util.c | 75 ++++++++++++++++++++++++++++
7 files changed, 98 insertions(+), 190 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 66270e07449a..768b645c7edf 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
index 5c521f3924a5..18d22eec0ebd 100644
--- a/arch/powerpc/mm/book3s64/iommu_api.c
+++ b/arch/powerpc/mm/book3s64/iommu_api.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index c438722bf4e1..0a532c602d8f 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 40ddc0c5f677..d06e8e291924 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,51 +35,6 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
-{
- long ret = 0, locked, lock_limit;
-
- if (WARN_ON_ONCE(!mm))
- return -EPERM;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-}
-
/*
* VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
*
@@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
if (ret)
return ret;
@@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 3ddc375e7063..bf449ace1676 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
ret = down_write_killable(&mm->mmap_sem);
if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
-
- if (!ret)
- mm->locked_vm += npage;
-
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
up_write(&mm->mmap_sem);
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0e8834ac32b7..95510f6fad45 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1564,6 +1564,10 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
int get_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
+int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index 91682a2090ee..cbbcc035b12b 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -7,6 +7,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -347,6 +348,80 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ lockdep_assert_held_exclusive(&mm->mmap_sem);
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit)
+ ret = -ENOMEM;
+ }
+ if (!ret)
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+ pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
+/**
+ * account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ *
+ * Assumes a non-NULL @mm is valid (i.e. at least one reference on it).
+ *
+ * Return:
+ * * 0 on success, or if mm is NULL
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc)
+{
+ int ret;
+
+ if (pages == 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: cd6c84d8f0cdc911df435bb075ba22ce3c605b07
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH v3] mm: add account_locked_vm utility function
@ 2019-05-29 20:50 ` Daniel Jordan
0 siblings, 0 replies; 42+ messages in thread
From: Daniel Jordan @ 2019-05-29 20:50 UTC (permalink / raw)
To: akpm
Cc: Daniel Jordan, Alexey Kardashevskiy, Alan Tull, Alex Williamson,
Benjamin Herrenschmidt, Christoph Lameter, Christophe Leroy,
Davidlohr Bueso, Ira Weiny, Jason Gunthorpe, Mark Rutland,
Michael Ellerman, Moritz Fischer, Paul Mackerras, Steve Sistare,
Wu Hao, linux-mm, kvm, kvm-ppc, linuxppc-dev, linux-fpga,
linux-kernel
locked_vm accounting is done roughly the same way in five places, so
unify them in a helper.
Include the helper's caller in the debug print to distinguish between
callsites.
Error codes stay the same, so user-visible behavior does too. The one
exception is that the -EPERM case in tce_account_locked_vm is removed
because Alexey has never seen it triggered.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Cc: Alan Tull <atull@kernel.org>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Moritz Fischer <mdf@kernel.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Steve Sistare <steven.sistare@oracle.com>
Cc: Wu Hao <hao.wu@intel.com>
Cc: linux-mm@kvack.org
Cc: kvm@vger.kernel.org
Cc: kvm-ppc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-fpga@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
v3:
- uninline account_locked_vm (Andrew)
- fix doc comment (Ira)
- retain down_write_killable in vfio type1 (Alex)
- leave Alexey's T-b since the code is the same aside from uninlining
- sanity tested with vfio type1, sanity-built on ppc
arch/powerpc/kvm/book3s_64_vio.c | 44 ++--------------
arch/powerpc/mm/book3s64/iommu_api.c | 41 ++-------------
drivers/fpga/dfl-afu-dma-region.c | 53 ++------------------
drivers/vfio/vfio_iommu_spapr_tce.c | 54 ++------------------
drivers/vfio/vfio_iommu_type1.c | 17 +------
include/linux/mm.h | 4 ++
mm/util.c | 75 ++++++++++++++++++++++++++++
7 files changed, 98 insertions(+), 190 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 66270e07449a..768b645c7edf 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -30,6 +30,7 @@
#include <linux/anon_inodes.h>
#include <linux/iommu.h>
#include <linux/file.h>
+#include <linux/mm.h>
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>
@@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
}
-static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
-{
- long ret = 0;
-
- if (!current || !current->mm)
- return ret; /* process exited */
-
- down_write(¤t->mm->mmap_sem);
-
- if (inc) {
- unsigned long locked, lock_limit;
-
- locked = current->mm->locked_vm + stt_pages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += stt_pages;
- } else {
- if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
- stt_pages = current->mm->locked_vm;
-
- current->mm->locked_vm -= stt_pages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
- inc ? '+' : '-',
- stt_pages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
{
struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
@@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
kvm_put_kvm(stt->kvm);
- kvmppc_account_memlimit(
+ account_locked_vm(current->mm,
kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
call_rcu(&stt->rcu, release_spapr_tce_table);
@@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
return -EINVAL;
npages = kvmppc_tce_pages(size);
- ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
+ ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
kfree(stt);
fail_acct:
- kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
+ account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
return ret;
}
diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
index 5c521f3924a5..18d22eec0ebd 100644
--- a/arch/powerpc/mm/book3s64/iommu_api.c
+++ b/arch/powerpc/mm/book3s64/iommu_api.c
@@ -19,6 +19,7 @@
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/sizes.h>
+#include <linux/mm.h>
#include <asm/mmu_context.h>
#include <asm/pte-walk.h>
#include <linux/mm_inline.h>
@@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
u64 dev_hpa; /* Device memory base address */
};
-static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
- unsigned long npages, bool incr)
-{
- long ret = 0, locked, lock_limit;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
-
- if (incr) {
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- }
-
- pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
- current ? current->pid : 0,
- incr ? '+' : '-',
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
bool mm_iommu_preregistered(struct mm_struct *mm)
{
return !list_empty(&mm->context.iommu_group_mem_list);
@@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
unsigned long entry, chunk;
if (dev_hpa = MM_IOMMU_TABLE_INVALID_HPA) {
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ ret = account_locked_vm(mm, entries, true);
if (ret)
return ret;
@@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
kfree(mem);
unlock_exit:
- mm_iommu_adjust_locked_vm(mm, locked_entries, false);
+ account_locked_vm(mm, locked_entries, false);
return ret;
}
@@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
unlock_exit:
mutex_unlock(&mem_list_mutex);
- mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
+ account_locked_vm(mm, unlock_entries, false);
return ret;
}
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
index c438722bf4e1..0a532c602d8f 100644
--- a/drivers/fpga/dfl-afu-dma-region.c
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -12,6 +12,7 @@
#include <linux/dma-mapping.h>
#include <linux/sched/signal.h>
#include <linux/uaccess.h>
+#include <linux/mm.h>
#include "dfl-afu.h"
@@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
afu->dma_regions = RB_ROOT;
}
-/**
- * afu_dma_adjust_locked_vm - adjust locked memory
- * @dev: port device
- * @npages: number of pages
- * @incr: increase or decrease locked memory
- *
- * Increase or decrease the locked memory size with npages input.
- *
- * Return 0 on success.
- * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
- */
-static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
-{
- unsigned long locked, lock_limit;
- int ret = 0;
-
- /* the task is exiting. */
- if (!current->mm)
- return 0;
-
- down_write(¤t->mm->mmap_sem);
-
- if (incr) {
- locked = current->mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- current->mm->locked_vm += npages;
- } else {
- if (WARN_ON_ONCE(npages > current->mm->locked_vm))
- npages = current->mm->locked_vm;
- current->mm->locked_vm -= npages;
- }
-
- dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
- incr ? '+' : '-', npages << PAGE_SHIFT,
- current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
- ret ? "- exceeded" : "");
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
/**
* afu_dma_pin_pages - pin pages of given dma memory region
* @pdata: feature device platform data
@@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
struct device *dev = &pdata->dev->dev;
int ret, pinned;
- ret = afu_dma_adjust_locked_vm(dev, npages, true);
+ ret = account_locked_vm(current->mm, npages, true);
if (ret)
return ret;
@@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
free_pages:
kfree(region->pages);
unlock_vm:
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
return ret;
}
@@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
put_all_pages(region->pages, npages);
kfree(region->pages);
- afu_dma_adjust_locked_vm(dev, npages, false);
+ account_locked_vm(current->mm, npages, false);
dev_dbg(dev, "%ld pages unpinned\n", npages);
}
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 40ddc0c5f677..d06e8e291924 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -22,6 +22,7 @@
#include <linux/vmalloc.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
+#include <linux/mm.h>
#include <asm/iommu.h>
#include <asm/tce.h>
@@ -34,51 +35,6 @@
static void tce_iommu_detach_group(void *iommu_data,
struct iommu_group *iommu_group);
-static long try_increment_locked_vm(struct mm_struct *mm, long npages)
-{
- long ret = 0, locked, lock_limit;
-
- if (WARN_ON_ONCE(!mm))
- return -EPERM;
-
- if (!npages)
- return 0;
-
- down_write(&mm->mmap_sem);
- locked = mm->locked_vm + npages;
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
- if (locked > lock_limit && !capable(CAP_IPC_LOCK))
- ret = -ENOMEM;
- else
- mm->locked_vm += npages;
-
- pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK),
- ret ? " - exceeded" : "");
-
- up_write(&mm->mmap_sem);
-
- return ret;
-}
-
-static void decrement_locked_vm(struct mm_struct *mm, long npages)
-{
- if (!mm || !npages)
- return;
-
- down_write(&mm->mmap_sem);
- if (WARN_ON_ONCE(npages > mm->locked_vm))
- npages = mm->locked_vm;
- mm->locked_vm -= npages;
- pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
- npages << PAGE_SHIFT,
- mm->locked_vm << PAGE_SHIFT,
- rlimit(RLIMIT_MEMLOCK));
- up_write(&mm->mmap_sem);
-}
-
/*
* VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
*
@@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
return ret;
locked = table_group->tce32_size >> PAGE_SHIFT;
- ret = try_increment_locked_vm(container->mm, locked);
+ ret = account_locked_vm(container->mm, locked, true);
if (ret)
return ret;
@@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
container->enabled = false;
BUG_ON(!container->mm);
- decrement_locked_vm(container->mm, container->locked_pages);
+ account_locked_vm(container->mm, container->locked_pages, false);
}
static void *tce_iommu_open(unsigned long arg)
@@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
if (!table_size)
return -EINVAL;
- ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
+ ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
if (ret)
return ret;
@@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
iommu_tce_table_put(tbl);
- decrement_locked_vm(container->mm, pages);
+ account_locked_vm(container->mm, pages, false);
}
static long tce_iommu_create_window(struct tce_container *container,
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 3ddc375e7063..bf449ace1676 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
ret = down_write_killable(&mm->mmap_sem);
if (!ret) {
- if (npage > 0) {
- if (!dma->lock_cap) {
- unsigned long limit;
-
- limit = task_rlimit(dma->task,
- RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (mm->locked_vm + npage > limit)
- ret = -ENOMEM;
- }
- }
-
- if (!ret)
- mm->locked_vm += npage;
-
+ ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
+ dma->lock_cap);
up_write(&mm->mmap_sem);
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0e8834ac32b7..95510f6fad45 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1564,6 +1564,10 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
int get_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
+int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc);
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim);
+
/* Container for pinned pfns / pages */
struct frame_vector {
unsigned int nr_allocated; /* Number of frames we have space for */
diff --git a/mm/util.c b/mm/util.c
index 91682a2090ee..cbbcc035b12b 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -7,6 +7,7 @@
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
#include <linux/sched/task_stack.h>
#include <linux/security.h>
#include <linux/swap.h>
@@ -347,6 +348,80 @@ int __weak get_user_pages_fast(unsigned long start,
}
EXPORT_SYMBOL_GPL(get_user_pages_fast);
+/**
+ * __account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ * @task: task used to check RLIMIT_MEMLOCK
+ * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
+ *
+ * Assumes @task and @mm are valid (i.e. at least one reference on each), and
+ * that mmap_sem is held as writer.
+ *
+ * Return:
+ * * 0 on success
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
+ struct task_struct *task, bool bypass_rlim)
+{
+ unsigned long locked_vm, limit;
+ int ret = 0;
+
+ lockdep_assert_held_exclusive(&mm->mmap_sem);
+
+ locked_vm = mm->locked_vm;
+ if (inc) {
+ if (!bypass_rlim) {
+ limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+ if (locked_vm + pages > limit)
+ ret = -ENOMEM;
+ }
+ if (!ret)
+ mm->locked_vm = locked_vm + pages;
+ } else {
+ WARN_ON_ONCE(pages > locked_vm);
+ mm->locked_vm = locked_vm - pages;
+ }
+
+ pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
+ (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
+ locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
+ ret ? " - exceeded" : "");
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(__account_locked_vm);
+
+/**
+ * account_locked_vm - account locked pages to an mm's locked_vm
+ * @mm: mm to account against, may be NULL
+ * @pages: number of pages to account
+ * @inc: %true if @pages should be considered positive, %false if not
+ *
+ * Assumes a non-NULL @mm is valid (i.e. at least one reference on it).
+ *
+ * Return:
+ * * 0 on success, or if mm is NULL
+ * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
+ */
+int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc)
+{
+ int ret;
+
+ if (pages = 0 || !mm)
+ return 0;
+
+ down_write(&mm->mmap_sem);
+ ret = __account_locked_vm(mm, pages, inc, current,
+ capable(CAP_IPC_LOCK));
+ up_write(&mm->mmap_sem);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(account_locked_vm);
+
unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
unsigned long len, unsigned long prot,
unsigned long flag, unsigned long pgoff)
base-commit: cd6c84d8f0cdc911df435bb075ba22ce3c605b07
--
2.21.0
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [PATCH v3] mm: add account_locked_vm utility function
2019-05-29 20:50 ` Daniel Jordan
(?)
@ 2019-06-03 19:15 ` Alex Williamson
-1 siblings, 0 replies; 42+ messages in thread
From: Alex Williamson @ 2019-06-03 19:15 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alexey Kardashevskiy, Alan Tull, Benjamin Herrenschmidt,
Christoph Lameter, Christophe Leroy, Davidlohr Bueso, Ira Weiny,
Jason Gunthorpe, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On Wed, 29 May 2019 16:50:19 -0400
Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper.
>
> Include the helper's caller in the debug print to distinguish between
> callsites.
>
> Error codes stay the same, so user-visible behavior does too. The one
> exception is that the -EPERM case in tce_account_locked_vm is removed
> because Alexey has never seen it triggered.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Ira Weiny <ira.weiny@intel.com>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
> v3:
> - uninline account_locked_vm (Andrew)
> - fix doc comment (Ira)
> - retain down_write_killable in vfio type1 (Alex)
> - leave Alexey's T-b since the code is the same aside from uninlining
> - sanity tested with vfio type1, sanity-built on ppc
>
> arch/powerpc/kvm/book3s_64_vio.c | 44 ++--------------
> arch/powerpc/mm/book3s64/iommu_api.c | 41 ++-------------
> drivers/fpga/dfl-afu-dma-region.c | 53 ++------------------
> drivers/vfio/vfio_iommu_spapr_tce.c | 54 ++------------------
> drivers/vfio/vfio_iommu_type1.c | 17 +------
> include/linux/mm.h | 4 ++
> mm/util.c | 75 ++++++++++++++++++++++++++++
> 7 files changed, 98 insertions(+), 190 deletions(-)
I tend to prefer adding a negative rather than converting to absolute
and passing a bool for inc/dec, but it all seems equivalent, so for
vfio parts
Acked-by: Alex Williamson <alex.williamson@redhat.com>
>
> diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
> index 66270e07449a..768b645c7edf 100644
> --- a/arch/powerpc/kvm/book3s_64_vio.c
> +++ b/arch/powerpc/kvm/book3s_64_vio.c
> @@ -30,6 +30,7 @@
> #include <linux/anon_inodes.h>
> #include <linux/iommu.h>
> #include <linux/file.h>
> +#include <linux/mm.h>
>
> #include <asm/kvm_ppc.h>
> #include <asm/kvm_book3s.h>
> @@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
> return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
> }
>
> -static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
> -{
> - long ret = 0;
> -
> - if (!current || !current->mm)
> - return ret; /* process exited */
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (inc) {
> - unsigned long locked, lock_limit;
> -
> - locked = current->mm->locked_vm + stt_pages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += stt_pages;
> - } else {
> - if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
> - stt_pages = current->mm->locked_vm;
> -
> - current->mm->locked_vm -= stt_pages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
> - inc ? '+' : '-',
> - stt_pages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
> {
> struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
> @@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
>
> kvm_put_kvm(stt->kvm);
>
> - kvmppc_account_memlimit(
> + account_locked_vm(current->mm,
> kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
> call_rcu(&stt->rcu, release_spapr_tce_table);
>
> @@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
> return -EINVAL;
>
> npages = kvmppc_tce_pages(size);
> - ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
> + ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
> if (ret)
> return ret;
>
> @@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
>
> kfree(stt);
> fail_acct:
> - kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
> + account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
> return ret;
> }
>
> diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
> index 5c521f3924a5..18d22eec0ebd 100644
> --- a/arch/powerpc/mm/book3s64/iommu_api.c
> +++ b/arch/powerpc/mm/book3s64/iommu_api.c
> @@ -19,6 +19,7 @@
> #include <linux/hugetlb.h>
> #include <linux/swap.h>
> #include <linux/sizes.h>
> +#include <linux/mm.h>
> #include <asm/mmu_context.h>
> #include <asm/pte-walk.h>
> #include <linux/mm_inline.h>
> @@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
> u64 dev_hpa; /* Device memory base address */
> };
>
> -static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
> - unsigned long npages, bool incr)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> -
> - if (incr) {
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
> - current ? current->pid : 0,
> - incr ? '+' : '-',
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> bool mm_iommu_preregistered(struct mm_struct *mm)
> {
> return !list_empty(&mm->context.iommu_group_mem_list);
> @@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> unsigned long entry, chunk;
>
> if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
> - ret = mm_iommu_adjust_locked_vm(mm, entries, true);
> + ret = account_locked_vm(mm, entries, true);
> if (ret)
> return ret;
>
> @@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> kfree(mem);
>
> unlock_exit:
> - mm_iommu_adjust_locked_vm(mm, locked_entries, false);
> + account_locked_vm(mm, locked_entries, false);
>
> return ret;
> }
> @@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
> unlock_exit:
> mutex_unlock(&mem_list_mutex);
>
> - mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
> + account_locked_vm(mm, unlock_entries, false);
>
> return ret;
> }
> diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
> index c438722bf4e1..0a532c602d8f 100644
> --- a/drivers/fpga/dfl-afu-dma-region.c
> +++ b/drivers/fpga/dfl-afu-dma-region.c
> @@ -12,6 +12,7 @@
> #include <linux/dma-mapping.h>
> #include <linux/sched/signal.h>
> #include <linux/uaccess.h>
> +#include <linux/mm.h>
>
> #include "dfl-afu.h"
>
> @@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
> afu->dma_regions = RB_ROOT;
> }
>
> -/**
> - * afu_dma_adjust_locked_vm - adjust locked memory
> - * @dev: port device
> - * @npages: number of pages
> - * @incr: increase or decrease locked memory
> - *
> - * Increase or decrease the locked memory size with npages input.
> - *
> - * Return 0 on success.
> - * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
> - */
> -static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> -{
> - unsigned long locked, lock_limit;
> - int ret = 0;
> -
> - /* the task is exiting. */
> - if (!current->mm)
> - return 0;
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (incr) {
> - locked = current->mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> - npages = current->mm->locked_vm;
> - current->mm->locked_vm -= npages;
> - }
> -
> - dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> - incr ? '+' : '-', npages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
> - ret ? "- exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> /**
> * afu_dma_pin_pages - pin pages of given dma memory region
> * @pdata: feature device platform data
> @@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> struct device *dev = &pdata->dev->dev;
> int ret, pinned;
>
> - ret = afu_dma_adjust_locked_vm(dev, npages, true);
> + ret = account_locked_vm(current->mm, npages, true);
> if (ret)
> return ret;
>
> @@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> free_pages:
> kfree(region->pages);
> unlock_vm:
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
> return ret;
> }
>
> @@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
>
> put_all_pages(region->pages, npages);
> kfree(region->pages);
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
>
> dev_dbg(dev, "%ld pages unpinned\n", npages);
> }
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 40ddc0c5f677..d06e8e291924 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -22,6 +22,7 @@
> #include <linux/vmalloc.h>
> #include <linux/sched/mm.h>
> #include <linux/sched/signal.h>
> +#include <linux/mm.h>
>
> #include <asm/iommu.h>
> #include <asm/tce.h>
> @@ -34,51 +35,6 @@
> static void tce_iommu_detach_group(void *iommu_data,
> struct iommu_group *iommu_group);
>
> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (WARN_ON_ONCE(!mm))
> - return -EPERM;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> -static void decrement_locked_vm(struct mm_struct *mm, long npages)
> -{
> - if (!mm || !npages)
> - return;
> -
> - down_write(&mm->mmap_sem);
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -}
> -
> /*
> * VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
> *
> @@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
> return ret;
>
> locked = table_group->tce32_size >> PAGE_SHIFT;
> - ret = try_increment_locked_vm(container->mm, locked);
> + ret = account_locked_vm(container->mm, locked, true);
> if (ret)
> return ret;
>
> @@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
> container->enabled = false;
>
> BUG_ON(!container->mm);
> - decrement_locked_vm(container->mm, container->locked_pages);
> + account_locked_vm(container->mm, container->locked_pages, false);
> }
>
> static void *tce_iommu_open(unsigned long arg)
> @@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
> if (!table_size)
> return -EINVAL;
>
> - ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
> + ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
> if (ret)
> return ret;
>
> @@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
> unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
>
> iommu_tce_table_put(tbl);
> - decrement_locked_vm(container->mm, pages);
> + account_locked_vm(container->mm, pages, false);
> }
>
> static long tce_iommu_create_window(struct tce_container *container,
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 3ddc375e7063..bf449ace1676 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
>
> ret = down_write_killable(&mm->mmap_sem);
> if (!ret) {
> - if (npage > 0) {
> - if (!dma->lock_cap) {
> - unsigned long limit;
> -
> - limit = task_rlimit(dma->task,
> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (mm->locked_vm + npage > limit)
> - ret = -ENOMEM;
> - }
> - }
> -
> - if (!ret)
> - mm->locked_vm += npage;
> -
> + ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
> + dma->lock_cap);
> up_write(&mm->mmap_sem);
> }
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0e8834ac32b7..95510f6fad45 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,10 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc);
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index 91682a2090ee..cbbcc035b12b 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -7,6 +7,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -347,6 +348,80 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + lockdep_assert_held_exclusive(&mm->mmap_sem);
> +
> + locked_vm = mm->locked_vm;
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit)
> + ret = -ENOMEM;
> + }
> + if (!ret)
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
> + }
> +
> + pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
> +
> +/**
> + * account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + *
> + * Assumes a non-NULL @mm is valid (i.e. at least one reference on it).
> + *
> + * Return:
> + * * 0 on success, or if mm is NULL
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(account_locked_vm);
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: cd6c84d8f0cdc911df435bb075ba22ce3c605b07
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3] mm: add account_locked_vm utility function
@ 2019-06-03 19:15 ` Alex Williamson
0 siblings, 0 replies; 42+ messages in thread
From: Alex Williamson @ 2019-06-03 19:15 UTC (permalink / raw)
To: Daniel Jordan
Cc: Mark Rutland, Davidlohr Bueso, kvm, Alan Tull,
Alexey Kardashevskiy, linux-fpga, linuxppc-dev, kvm-ppc,
linux-kernel, linux-mm, Jason Gunthorpe, Moritz Fischer,
Steve Sistare, akpm, Ira Weiny, Christoph Lameter, Wu Hao
On Wed, 29 May 2019 16:50:19 -0400
Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper.
>
> Include the helper's caller in the debug print to distinguish between
> callsites.
>
> Error codes stay the same, so user-visible behavior does too. The one
> exception is that the -EPERM case in tce_account_locked_vm is removed
> because Alexey has never seen it triggered.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Ira Weiny <ira.weiny@intel.com>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
> v3:
> - uninline account_locked_vm (Andrew)
> - fix doc comment (Ira)
> - retain down_write_killable in vfio type1 (Alex)
> - leave Alexey's T-b since the code is the same aside from uninlining
> - sanity tested with vfio type1, sanity-built on ppc
>
> arch/powerpc/kvm/book3s_64_vio.c | 44 ++--------------
> arch/powerpc/mm/book3s64/iommu_api.c | 41 ++-------------
> drivers/fpga/dfl-afu-dma-region.c | 53 ++------------------
> drivers/vfio/vfio_iommu_spapr_tce.c | 54 ++------------------
> drivers/vfio/vfio_iommu_type1.c | 17 +------
> include/linux/mm.h | 4 ++
> mm/util.c | 75 ++++++++++++++++++++++++++++
> 7 files changed, 98 insertions(+), 190 deletions(-)
I tend to prefer adding a negative rather than converting to absolute
and passing a bool for inc/dec, but it all seems equivalent, so for
vfio parts
Acked-by: Alex Williamson <alex.williamson@redhat.com>
>
> diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
> index 66270e07449a..768b645c7edf 100644
> --- a/arch/powerpc/kvm/book3s_64_vio.c
> +++ b/arch/powerpc/kvm/book3s_64_vio.c
> @@ -30,6 +30,7 @@
> #include <linux/anon_inodes.h>
> #include <linux/iommu.h>
> #include <linux/file.h>
> +#include <linux/mm.h>
>
> #include <asm/kvm_ppc.h>
> #include <asm/kvm_book3s.h>
> @@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
> return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
> }
>
> -static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
> -{
> - long ret = 0;
> -
> - if (!current || !current->mm)
> - return ret; /* process exited */
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (inc) {
> - unsigned long locked, lock_limit;
> -
> - locked = current->mm->locked_vm + stt_pages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += stt_pages;
> - } else {
> - if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
> - stt_pages = current->mm->locked_vm;
> -
> - current->mm->locked_vm -= stt_pages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
> - inc ? '+' : '-',
> - stt_pages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
> {
> struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
> @@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
>
> kvm_put_kvm(stt->kvm);
>
> - kvmppc_account_memlimit(
> + account_locked_vm(current->mm,
> kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
> call_rcu(&stt->rcu, release_spapr_tce_table);
>
> @@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
> return -EINVAL;
>
> npages = kvmppc_tce_pages(size);
> - ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
> + ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
> if (ret)
> return ret;
>
> @@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
>
> kfree(stt);
> fail_acct:
> - kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
> + account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
> return ret;
> }
>
> diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
> index 5c521f3924a5..18d22eec0ebd 100644
> --- a/arch/powerpc/mm/book3s64/iommu_api.c
> +++ b/arch/powerpc/mm/book3s64/iommu_api.c
> @@ -19,6 +19,7 @@
> #include <linux/hugetlb.h>
> #include <linux/swap.h>
> #include <linux/sizes.h>
> +#include <linux/mm.h>
> #include <asm/mmu_context.h>
> #include <asm/pte-walk.h>
> #include <linux/mm_inline.h>
> @@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
> u64 dev_hpa; /* Device memory base address */
> };
>
> -static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
> - unsigned long npages, bool incr)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> -
> - if (incr) {
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
> - current ? current->pid : 0,
> - incr ? '+' : '-',
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> bool mm_iommu_preregistered(struct mm_struct *mm)
> {
> return !list_empty(&mm->context.iommu_group_mem_list);
> @@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> unsigned long entry, chunk;
>
> if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
> - ret = mm_iommu_adjust_locked_vm(mm, entries, true);
> + ret = account_locked_vm(mm, entries, true);
> if (ret)
> return ret;
>
> @@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> kfree(mem);
>
> unlock_exit:
> - mm_iommu_adjust_locked_vm(mm, locked_entries, false);
> + account_locked_vm(mm, locked_entries, false);
>
> return ret;
> }
> @@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
> unlock_exit:
> mutex_unlock(&mem_list_mutex);
>
> - mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
> + account_locked_vm(mm, unlock_entries, false);
>
> return ret;
> }
> diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
> index c438722bf4e1..0a532c602d8f 100644
> --- a/drivers/fpga/dfl-afu-dma-region.c
> +++ b/drivers/fpga/dfl-afu-dma-region.c
> @@ -12,6 +12,7 @@
> #include <linux/dma-mapping.h>
> #include <linux/sched/signal.h>
> #include <linux/uaccess.h>
> +#include <linux/mm.h>
>
> #include "dfl-afu.h"
>
> @@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
> afu->dma_regions = RB_ROOT;
> }
>
> -/**
> - * afu_dma_adjust_locked_vm - adjust locked memory
> - * @dev: port device
> - * @npages: number of pages
> - * @incr: increase or decrease locked memory
> - *
> - * Increase or decrease the locked memory size with npages input.
> - *
> - * Return 0 on success.
> - * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
> - */
> -static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> -{
> - unsigned long locked, lock_limit;
> - int ret = 0;
> -
> - /* the task is exiting. */
> - if (!current->mm)
> - return 0;
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (incr) {
> - locked = current->mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> - npages = current->mm->locked_vm;
> - current->mm->locked_vm -= npages;
> - }
> -
> - dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> - incr ? '+' : '-', npages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
> - ret ? "- exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> /**
> * afu_dma_pin_pages - pin pages of given dma memory region
> * @pdata: feature device platform data
> @@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> struct device *dev = &pdata->dev->dev;
> int ret, pinned;
>
> - ret = afu_dma_adjust_locked_vm(dev, npages, true);
> + ret = account_locked_vm(current->mm, npages, true);
> if (ret)
> return ret;
>
> @@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> free_pages:
> kfree(region->pages);
> unlock_vm:
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
> return ret;
> }
>
> @@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
>
> put_all_pages(region->pages, npages);
> kfree(region->pages);
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
>
> dev_dbg(dev, "%ld pages unpinned\n", npages);
> }
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 40ddc0c5f677..d06e8e291924 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -22,6 +22,7 @@
> #include <linux/vmalloc.h>
> #include <linux/sched/mm.h>
> #include <linux/sched/signal.h>
> +#include <linux/mm.h>
>
> #include <asm/iommu.h>
> #include <asm/tce.h>
> @@ -34,51 +35,6 @@
> static void tce_iommu_detach_group(void *iommu_data,
> struct iommu_group *iommu_group);
>
> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (WARN_ON_ONCE(!mm))
> - return -EPERM;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> -static void decrement_locked_vm(struct mm_struct *mm, long npages)
> -{
> - if (!mm || !npages)
> - return;
> -
> - down_write(&mm->mmap_sem);
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -}
> -
> /*
> * VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
> *
> @@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
> return ret;
>
> locked = table_group->tce32_size >> PAGE_SHIFT;
> - ret = try_increment_locked_vm(container->mm, locked);
> + ret = account_locked_vm(container->mm, locked, true);
> if (ret)
> return ret;
>
> @@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
> container->enabled = false;
>
> BUG_ON(!container->mm);
> - decrement_locked_vm(container->mm, container->locked_pages);
> + account_locked_vm(container->mm, container->locked_pages, false);
> }
>
> static void *tce_iommu_open(unsigned long arg)
> @@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
> if (!table_size)
> return -EINVAL;
>
> - ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
> + ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
> if (ret)
> return ret;
>
> @@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
> unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
>
> iommu_tce_table_put(tbl);
> - decrement_locked_vm(container->mm, pages);
> + account_locked_vm(container->mm, pages, false);
> }
>
> static long tce_iommu_create_window(struct tce_container *container,
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 3ddc375e7063..bf449ace1676 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
>
> ret = down_write_killable(&mm->mmap_sem);
> if (!ret) {
> - if (npage > 0) {
> - if (!dma->lock_cap) {
> - unsigned long limit;
> -
> - limit = task_rlimit(dma->task,
> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (mm->locked_vm + npage > limit)
> - ret = -ENOMEM;
> - }
> - }
> -
> - if (!ret)
> - mm->locked_vm += npage;
> -
> + ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
> + dma->lock_cap);
> up_write(&mm->mmap_sem);
> }
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0e8834ac32b7..95510f6fad45 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,10 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc);
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index 91682a2090ee..cbbcc035b12b 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -7,6 +7,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -347,6 +348,80 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + lockdep_assert_held_exclusive(&mm->mmap_sem);
> +
> + locked_vm = mm->locked_vm;
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit)
> + ret = -ENOMEM;
> + }
> + if (!ret)
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
> + }
> +
> + pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
> +
> +/**
> + * account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + *
> + * Assumes a non-NULL @mm is valid (i.e. at least one reference on it).
> + *
> + * Return:
> + * * 0 on success, or if mm is NULL
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc)
> +{
> + int ret;
> +
> + if (pages == 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(account_locked_vm);
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: cd6c84d8f0cdc911df435bb075ba22ce3c605b07
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3] mm: add account_locked_vm utility function
@ 2019-06-03 19:15 ` Alex Williamson
0 siblings, 0 replies; 42+ messages in thread
From: Alex Williamson @ 2019-06-03 19:15 UTC (permalink / raw)
To: Daniel Jordan
Cc: akpm, Alexey Kardashevskiy, Alan Tull, Benjamin Herrenschmidt,
Christoph Lameter, Christophe Leroy, Davidlohr Bueso, Ira Weiny,
Jason Gunthorpe, Mark Rutland, Michael Ellerman, Moritz Fischer,
Paul Mackerras, Steve Sistare, Wu Hao, linux-mm, kvm, kvm-ppc,
linuxppc-dev, linux-fpga, linux-kernel
On Wed, 29 May 2019 16:50:19 -0400
Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> locked_vm accounting is done roughly the same way in five places, so
> unify them in a helper.
>
> Include the helper's caller in the debug print to distinguish between
> callsites.
>
> Error codes stay the same, so user-visible behavior does too. The one
> exception is that the -EPERM case in tce_account_locked_vm is removed
> because Alexey has never seen it triggered.
>
> Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Cc: Alan Tull <atull@kernel.org>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Ira Weiny <ira.weiny@intel.com>
> Cc: Jason Gunthorpe <jgg@mellanox.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Moritz Fischer <mdf@kernel.org>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: Steve Sistare <steven.sistare@oracle.com>
> Cc: Wu Hao <hao.wu@intel.com>
> Cc: linux-mm@kvack.org
> Cc: kvm@vger.kernel.org
> Cc: kvm-ppc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-fpga@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
> v3:
> - uninline account_locked_vm (Andrew)
> - fix doc comment (Ira)
> - retain down_write_killable in vfio type1 (Alex)
> - leave Alexey's T-b since the code is the same aside from uninlining
> - sanity tested with vfio type1, sanity-built on ppc
>
> arch/powerpc/kvm/book3s_64_vio.c | 44 ++--------------
> arch/powerpc/mm/book3s64/iommu_api.c | 41 ++-------------
> drivers/fpga/dfl-afu-dma-region.c | 53 ++------------------
> drivers/vfio/vfio_iommu_spapr_tce.c | 54 ++------------------
> drivers/vfio/vfio_iommu_type1.c | 17 +------
> include/linux/mm.h | 4 ++
> mm/util.c | 75 ++++++++++++++++++++++++++++
> 7 files changed, 98 insertions(+), 190 deletions(-)
I tend to prefer adding a negative rather than converting to absolute
and passing a bool for inc/dec, but it all seems equivalent, so for
vfio parts
Acked-by: Alex Williamson <alex.williamson@redhat.com>
>
> diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
> index 66270e07449a..768b645c7edf 100644
> --- a/arch/powerpc/kvm/book3s_64_vio.c
> +++ b/arch/powerpc/kvm/book3s_64_vio.c
> @@ -30,6 +30,7 @@
> #include <linux/anon_inodes.h>
> #include <linux/iommu.h>
> #include <linux/file.h>
> +#include <linux/mm.h>
>
> #include <asm/kvm_ppc.h>
> #include <asm/kvm_book3s.h>
> @@ -56,43 +57,6 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)
> return tce_pages + ALIGN(stt_bytes, PAGE_SIZE) / PAGE_SIZE;
> }
>
> -static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)
> -{
> - long ret = 0;
> -
> - if (!current || !current->mm)
> - return ret; /* process exited */
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (inc) {
> - unsigned long locked, lock_limit;
> -
> - locked = current->mm->locked_vm + stt_pages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += stt_pages;
> - } else {
> - if (WARN_ON_ONCE(stt_pages > current->mm->locked_vm))
> - stt_pages = current->mm->locked_vm;
> -
> - current->mm->locked_vm -= stt_pages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK KVM %c%ld %ld/%ld%s\n", current->pid,
> - inc ? '+' : '-',
> - stt_pages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> static void kvm_spapr_tce_iommu_table_free(struct rcu_head *head)
> {
> struct kvmppc_spapr_tce_iommu_table *stit = container_of(head,
> @@ -302,7 +266,7 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp)
>
> kvm_put_kvm(stt->kvm);
>
> - kvmppc_account_memlimit(
> + account_locked_vm(current->mm,
> kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false);
> call_rcu(&stt->rcu, release_spapr_tce_table);
>
> @@ -327,7 +291,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
> return -EINVAL;
>
> npages = kvmppc_tce_pages(size);
> - ret = kvmppc_account_memlimit(kvmppc_stt_pages(npages), true);
> + ret = account_locked_vm(current->mm, kvmppc_stt_pages(npages), true);
> if (ret)
> return ret;
>
> @@ -373,7 +337,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
>
> kfree(stt);
> fail_acct:
> - kvmppc_account_memlimit(kvmppc_stt_pages(npages), false);
> + account_locked_vm(current->mm, kvmppc_stt_pages(npages), false);
> return ret;
> }
>
> diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c
> index 5c521f3924a5..18d22eec0ebd 100644
> --- a/arch/powerpc/mm/book3s64/iommu_api.c
> +++ b/arch/powerpc/mm/book3s64/iommu_api.c
> @@ -19,6 +19,7 @@
> #include <linux/hugetlb.h>
> #include <linux/swap.h>
> #include <linux/sizes.h>
> +#include <linux/mm.h>
> #include <asm/mmu_context.h>
> #include <asm/pte-walk.h>
> #include <linux/mm_inline.h>
> @@ -51,40 +52,6 @@ struct mm_iommu_table_group_mem_t {
> u64 dev_hpa; /* Device memory base address */
> };
>
> -static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
> - unsigned long npages, bool incr)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> -
> - if (incr) {
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - }
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK HASH64 %c%ld %ld/%ld\n",
> - current ? current->pid : 0,
> - incr ? '+' : '-',
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> bool mm_iommu_preregistered(struct mm_struct *mm)
> {
> return !list_empty(&mm->context.iommu_group_mem_list);
> @@ -101,7 +68,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> unsigned long entry, chunk;
>
> if (dev_hpa = MM_IOMMU_TABLE_INVALID_HPA) {
> - ret = mm_iommu_adjust_locked_vm(mm, entries, true);
> + ret = account_locked_vm(mm, entries, true);
> if (ret)
> return ret;
>
> @@ -216,7 +183,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
> kfree(mem);
>
> unlock_exit:
> - mm_iommu_adjust_locked_vm(mm, locked_entries, false);
> + account_locked_vm(mm, locked_entries, false);
>
> return ret;
> }
> @@ -316,7 +283,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
> unlock_exit:
> mutex_unlock(&mem_list_mutex);
>
> - mm_iommu_adjust_locked_vm(mm, unlock_entries, false);
> + account_locked_vm(mm, unlock_entries, false);
>
> return ret;
> }
> diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
> index c438722bf4e1..0a532c602d8f 100644
> --- a/drivers/fpga/dfl-afu-dma-region.c
> +++ b/drivers/fpga/dfl-afu-dma-region.c
> @@ -12,6 +12,7 @@
> #include <linux/dma-mapping.h>
> #include <linux/sched/signal.h>
> #include <linux/uaccess.h>
> +#include <linux/mm.h>
>
> #include "dfl-afu.h"
>
> @@ -31,52 +32,6 @@ void afu_dma_region_init(struct dfl_feature_platform_data *pdata)
> afu->dma_regions = RB_ROOT;
> }
>
> -/**
> - * afu_dma_adjust_locked_vm - adjust locked memory
> - * @dev: port device
> - * @npages: number of pages
> - * @incr: increase or decrease locked memory
> - *
> - * Increase or decrease the locked memory size with npages input.
> - *
> - * Return 0 on success.
> - * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
> - */
> -static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> -{
> - unsigned long locked, lock_limit;
> - int ret = 0;
> -
> - /* the task is exiting. */
> - if (!current->mm)
> - return 0;
> -
> - down_write(¤t->mm->mmap_sem);
> -
> - if (incr) {
> - locked = current->mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - current->mm->locked_vm += npages;
> - } else {
> - if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> - npages = current->mm->locked_vm;
> - current->mm->locked_vm -= npages;
> - }
> -
> - dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> - incr ? '+' : '-', npages << PAGE_SHIFT,
> - current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
> - ret ? "- exceeded" : "");
> -
> - up_write(¤t->mm->mmap_sem);
> -
> - return ret;
> -}
> -
> /**
> * afu_dma_pin_pages - pin pages of given dma memory region
> * @pdata: feature device platform data
> @@ -92,7 +47,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> struct device *dev = &pdata->dev->dev;
> int ret, pinned;
>
> - ret = afu_dma_adjust_locked_vm(dev, npages, true);
> + ret = account_locked_vm(current->mm, npages, true);
> if (ret)
> return ret;
>
> @@ -121,7 +76,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata,
> free_pages:
> kfree(region->pages);
> unlock_vm:
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
> return ret;
> }
>
> @@ -141,7 +96,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata,
>
> put_all_pages(region->pages, npages);
> kfree(region->pages);
> - afu_dma_adjust_locked_vm(dev, npages, false);
> + account_locked_vm(current->mm, npages, false);
>
> dev_dbg(dev, "%ld pages unpinned\n", npages);
> }
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 40ddc0c5f677..d06e8e291924 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -22,6 +22,7 @@
> #include <linux/vmalloc.h>
> #include <linux/sched/mm.h>
> #include <linux/sched/signal.h>
> +#include <linux/mm.h>
>
> #include <asm/iommu.h>
> #include <asm/tce.h>
> @@ -34,51 +35,6 @@
> static void tce_iommu_detach_group(void *iommu_data,
> struct iommu_group *iommu_group);
>
> -static long try_increment_locked_vm(struct mm_struct *mm, long npages)
> -{
> - long ret = 0, locked, lock_limit;
> -
> - if (WARN_ON_ONCE(!mm))
> - return -EPERM;
> -
> - if (!npages)
> - return 0;
> -
> - down_write(&mm->mmap_sem);
> - locked = mm->locked_vm + npages;
> - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> - if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> - ret = -ENOMEM;
> - else
> - mm->locked_vm += npages;
> -
> - pr_debug("[%d] RLIMIT_MEMLOCK +%ld %ld/%ld%s\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK),
> - ret ? " - exceeded" : "");
> -
> - up_write(&mm->mmap_sem);
> -
> - return ret;
> -}
> -
> -static void decrement_locked_vm(struct mm_struct *mm, long npages)
> -{
> - if (!mm || !npages)
> - return;
> -
> - down_write(&mm->mmap_sem);
> - if (WARN_ON_ONCE(npages > mm->locked_vm))
> - npages = mm->locked_vm;
> - mm->locked_vm -= npages;
> - pr_debug("[%d] RLIMIT_MEMLOCK -%ld %ld/%ld\n", current->pid,
> - npages << PAGE_SHIFT,
> - mm->locked_vm << PAGE_SHIFT,
> - rlimit(RLIMIT_MEMLOCK));
> - up_write(&mm->mmap_sem);
> -}
> -
> /*
> * VFIO IOMMU fd for SPAPR_TCE IOMMU implementation
> *
> @@ -336,7 +292,7 @@ static int tce_iommu_enable(struct tce_container *container)
> return ret;
>
> locked = table_group->tce32_size >> PAGE_SHIFT;
> - ret = try_increment_locked_vm(container->mm, locked);
> + ret = account_locked_vm(container->mm, locked, true);
> if (ret)
> return ret;
>
> @@ -355,7 +311,7 @@ static void tce_iommu_disable(struct tce_container *container)
> container->enabled = false;
>
> BUG_ON(!container->mm);
> - decrement_locked_vm(container->mm, container->locked_pages);
> + account_locked_vm(container->mm, container->locked_pages, false);
> }
>
> static void *tce_iommu_open(unsigned long arg)
> @@ -659,7 +615,7 @@ static long tce_iommu_create_table(struct tce_container *container,
> if (!table_size)
> return -EINVAL;
>
> - ret = try_increment_locked_vm(container->mm, table_size >> PAGE_SHIFT);
> + ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true);
> if (ret)
> return ret;
>
> @@ -678,7 +634,7 @@ static void tce_iommu_free_table(struct tce_container *container,
> unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT;
>
> iommu_tce_table_put(tbl);
> - decrement_locked_vm(container->mm, pages);
> + account_locked_vm(container->mm, pages, false);
> }
>
> static long tce_iommu_create_window(struct tce_container *container,
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 3ddc375e7063..bf449ace1676 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -275,21 +275,8 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)
>
> ret = down_write_killable(&mm->mmap_sem);
> if (!ret) {
> - if (npage > 0) {
> - if (!dma->lock_cap) {
> - unsigned long limit;
> -
> - limit = task_rlimit(dma->task,
> - RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> -
> - if (mm->locked_vm + npage > limit)
> - ret = -ENOMEM;
> - }
> - }
> -
> - if (!ret)
> - mm->locked_vm += npage;
> -
> + ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task,
> + dma->lock_cap);
> up_write(&mm->mmap_sem);
> }
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0e8834ac32b7..95510f6fad45 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1564,6 +1564,10 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
> int get_user_pages_fast(unsigned long start, int nr_pages,
> unsigned int gup_flags, struct page **pages);
>
> +int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc);
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim);
> +
> /* Container for pinned pfns / pages */
> struct frame_vector {
> unsigned int nr_allocated; /* Number of frames we have space for */
> diff --git a/mm/util.c b/mm/util.c
> index 91682a2090ee..cbbcc035b12b 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -7,6 +7,7 @@
> #include <linux/err.h>
> #include <linux/sched.h>
> #include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> #include <linux/sched/task_stack.h>
> #include <linux/security.h>
> #include <linux/swap.h>
> @@ -347,6 +348,80 @@ int __weak get_user_pages_fast(unsigned long start,
> }
> EXPORT_SYMBOL_GPL(get_user_pages_fast);
>
> +/**
> + * __account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + * @task: task used to check RLIMIT_MEMLOCK
> + * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped
> + *
> + * Assumes @task and @mm are valid (i.e. at least one reference on each), and
> + * that mmap_sem is held as writer.
> + *
> + * Return:
> + * * 0 on success
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc,
> + struct task_struct *task, bool bypass_rlim)
> +{
> + unsigned long locked_vm, limit;
> + int ret = 0;
> +
> + lockdep_assert_held_exclusive(&mm->mmap_sem);
> +
> + locked_vm = mm->locked_vm;
> + if (inc) {
> + if (!bypass_rlim) {
> + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> + if (locked_vm + pages > limit)
> + ret = -ENOMEM;
> + }
> + if (!ret)
> + mm->locked_vm = locked_vm + pages;
> + } else {
> + WARN_ON_ONCE(pages > locked_vm);
> + mm->locked_vm = locked_vm - pages;
> + }
> +
> + pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid,
> + (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT,
> + locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK),
> + ret ? " - exceeded" : "");
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(__account_locked_vm);
> +
> +/**
> + * account_locked_vm - account locked pages to an mm's locked_vm
> + * @mm: mm to account against, may be NULL
> + * @pages: number of pages to account
> + * @inc: %true if @pages should be considered positive, %false if not
> + *
> + * Assumes a non-NULL @mm is valid (i.e. at least one reference on it).
> + *
> + * Return:
> + * * 0 on success, or if mm is NULL
> + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded.
> + */
> +int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc)
> +{
> + int ret;
> +
> + if (pages = 0 || !mm)
> + return 0;
> +
> + down_write(&mm->mmap_sem);
> + ret = __account_locked_vm(mm, pages, inc, current,
> + capable(CAP_IPC_LOCK));
> + up_write(&mm->mmap_sem);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(account_locked_vm);
> +
> unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
> unsigned long len, unsigned long prot,
> unsigned long flag, unsigned long pgoff)
>
> base-commit: cd6c84d8f0cdc911df435bb075ba22ce3c605b07
^ permalink raw reply [flat|nested] 42+ messages in thread
end of thread, other threads:[~2019-06-03 19:17 UTC | newest]
Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-03 20:16 [PATCH] mm: add account_locked_vm utility function Daniel Jordan
2019-05-03 20:16 ` Daniel Jordan
2019-05-03 20:16 ` Daniel Jordan
2019-05-03 23:28 ` Jason Gunthorpe
2019-05-03 23:28 ` Jason Gunthorpe
2019-05-03 23:28 ` Jason Gunthorpe
2019-05-07 3:09 ` Daniel Jordan
2019-05-07 3:09 ` Daniel Jordan
2019-05-07 3:09 ` Daniel Jordan
2019-05-20 6:19 ` Alexey Kardashevskiy
2019-05-20 6:19 ` Alexey Kardashevskiy
2019-05-20 6:19 ` Alexey Kardashevskiy
2019-05-20 15:30 ` Daniel Jordan
2019-05-20 15:30 ` Daniel Jordan
2019-05-20 15:30 ` Daniel Jordan
2019-05-24 6:43 ` Alexey Kardashevskiy
2019-05-24 6:43 ` Alexey Kardashevskiy
2019-05-24 6:43 ` Alexey Kardashevskiy
2019-05-24 17:50 ` [PATCH v2] " Daniel Jordan
2019-05-24 17:50 ` Daniel Jordan
2019-05-24 17:50 ` Daniel Jordan
2019-05-25 21:51 ` Andrew Morton
2019-05-25 21:51 ` Andrew Morton
2019-05-25 21:51 ` Andrew Morton
2019-05-28 15:04 ` Daniel Jordan
2019-05-28 15:04 ` Daniel Jordan
2019-05-28 15:04 ` Daniel Jordan
2019-05-29 18:56 ` Alex Williamson
2019-05-29 18:56 ` Alex Williamson
2019-05-29 18:56 ` Alex Williamson
2019-05-29 20:50 ` [PATCH v3] " Daniel Jordan
2019-05-29 20:50 ` Daniel Jordan
2019-05-29 20:50 ` Daniel Jordan
2019-06-03 19:15 ` Alex Williamson
2019-06-03 19:15 ` Alex Williamson
2019-06-03 19:15 ` Alex Williamson
2019-05-29 18:05 ` [PATCH v2] " Ira Weiny
2019-05-29 18:05 ` Ira Weiny
2019-05-29 18:05 ` Ira Weiny
2019-05-29 18:35 ` Daniel Jordan
2019-05-29 18:35 ` Daniel Jordan
2019-05-29 18:35 ` Daniel Jordan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.