* [PATCH 0/4] lib/iov_iter: Fixes and documentation for pmem user copies
@ 2018-07-08 20:45 Dan Williams
2018-07-08 20:46 ` [PATCH 1/4] lib/iov_iter: Document _copy_to_iter_mcsafe() Dan Williams
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Dan Williams @ 2018-07-08 20:45 UTC (permalink / raw)
To: mingo
Cc: Tony Luck, Peter Zijlstra, Linus Torvalds, Borislav Petkov,
Thomas Gleixner, Ross Zwisler, Andy Lutomirski, Al Viro,
Andrew Morton, linux-kernel, x86
* Add missing documentation for the 'flushcache' operation for writing to
persistent memory, and the 'mcsafe' operation for reading from
potentially poisoned persistent memory.
* Introduce copy_pipe_to_iter_mcsafe() to properly handle ITER_PIPE for
_copy_to_iter_mcsafe().
* Make sure copy_to_user_mcsafe() falls back to an exception handling
capable copy routine, not plain memcpy().
---
Dan Williams (4):
lib/iov_iter: Document _copy_to_iter_mcsafe()
lib/iov_iter: Document _copy_to_iter_flushcache()
lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe
x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling
arch/x86/Kconfig | 2 -
arch/x86/include/asm/uaccess_64.h | 7 +++
lib/iov_iter.c | 77 +++++++++++++++++++++++++++++++++++--
3 files changed, 80 insertions(+), 6 deletions(-)
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/4] lib/iov_iter: Document _copy_to_iter_mcsafe()
2018-07-08 20:45 [PATCH 0/4] lib/iov_iter: Fixes and documentation for pmem user copies Dan Williams
@ 2018-07-08 20:46 ` Dan Williams
2018-07-15 23:30 ` [tip:core/urgent] " tip-bot for Dan Williams
2018-07-08 20:46 ` [PATCH 2/4] lib/iov_iter: Document _copy_to_iter_flushcache() Dan Williams
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Dan Williams @ 2018-07-08 20:46 UTC (permalink / raw)
To: mingo
Cc: Andrew Morton, Andy Lutomirski, Borislav Petkov, Linus Torvalds,
Peter Zijlstra, Thomas Gleixner, Tony Luck, Al Viro,
linux-kernel, x86
Add some theory of operation documentation to _copy_to_iter_mcsafe().
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Reported-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
lib/iov_iter.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 7e43cd54c84c..94fa361be7bb 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -596,6 +596,32 @@ static unsigned long memcpy_mcsafe_to_page(struct page *page, size_t offset,
return ret;
}
+/**
+ * _copy_to_iter_mcsafe - copy to user with source-read error exception handling
+ * @addr: source kernel address
+ * @bytes: total transfer length
+ * @iter: destination iterator
+ *
+ * The pmem driver arranges for filesystem-dax to use this facility via
+ * dax_copy_to_iter() for protecting read/write to persistent memory.
+ * Unless / until an architecture can guarantee identical performance
+ * between _copy_to_iter_mcsafe() and _copy_to_iter() it would be a
+ * performance regression to switch more users to the mcsafe version.
+ *
+ * Otherwise, the main differences between this and typical _copy_to_iter().
+ *
+ * * Typical tail/residue handling after a fault retries the copy
+ * byte-by-byte until the fault happens again. Re-triggering machine
+ * checks is potentially fatal so the implementation uses source
+ * alignment and poison alignment assumptions to avoid re-triggering
+ * hardware exceptions.
+ *
+ * * ITER_KVEC, ITER_PIPE, and ITER_BVEC can return short copies.
+ * Compare to copy_to_iter() where only ITER_IOVEC attempts might return
+ * a short copy.
+ *
+ * See MCSAFE_TEST for self-test.
+ */
size_t _copy_to_iter_mcsafe(const void *addr, size_t bytes, struct iov_iter *i)
{
const char *from = addr;
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/4] lib/iov_iter: Document _copy_to_iter_flushcache()
2018-07-08 20:45 [PATCH 0/4] lib/iov_iter: Fixes and documentation for pmem user copies Dan Williams
2018-07-08 20:46 ` [PATCH 1/4] lib/iov_iter: Document _copy_to_iter_mcsafe() Dan Williams
@ 2018-07-08 20:46 ` Dan Williams
2018-07-15 23:31 ` [tip:core/urgent] " tip-bot for Dan Williams
2018-07-08 20:46 ` [PATCH 3/4] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe Dan Williams
2018-07-08 20:46 ` [PATCH 4/4] x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling Dan Williams
3 siblings, 1 reply; 9+ messages in thread
From: Dan Williams @ 2018-07-08 20:46 UTC (permalink / raw)
To: mingo
Cc: Andrew Morton, Andy Lutomirski, Borislav Petkov, Linus Torvalds,
Peter Zijlstra, Thomas Gleixner, Tony Luck, Al Viro,
linux-kernel, x86
Add some theory of operation documentation to _copy_to_iter_flushcache().
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Reported-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
lib/iov_iter.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 94fa361be7bb..09fb73ad9d54 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -727,6 +727,20 @@ size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i)
EXPORT_SYMBOL(_copy_from_iter_nocache);
#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE
+/**
+ * _copy_from_iter_flushcache - write destination through cpu cache
+ * @addr: destination kernel address
+ * @bytes: total transfer length
+ * @iter: source iterator
+ *
+ * The pmem driver arranges for filesystem-dax to use this facility via
+ * dax_copy_from_iter() for ensuring that writes to persistent memory
+ * are flushed through the CPU cache. It is differentiated from
+ * _copy_from_iter_nocache() in that guarantees all data is flushed for
+ * all iterator types. The _copy_from_iter_nocache() only attempts to
+ * bypass the cache for the ITER_IOVEC case, and on some archs may use
+ * instructions that strand dirty-data in the cache.
+ */
size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i)
{
char *to = addr;
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 3/4] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe
2018-07-08 20:45 [PATCH 0/4] lib/iov_iter: Fixes and documentation for pmem user copies Dan Williams
2018-07-08 20:46 ` [PATCH 1/4] lib/iov_iter: Document _copy_to_iter_mcsafe() Dan Williams
2018-07-08 20:46 ` [PATCH 2/4] lib/iov_iter: Document _copy_to_iter_flushcache() Dan Williams
@ 2018-07-08 20:46 ` Dan Williams
2018-07-15 23:31 ` [tip:core/urgent] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe() tip-bot for Dan Williams
2018-07-08 20:46 ` [PATCH 4/4] x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling Dan Williams
3 siblings, 1 reply; 9+ messages in thread
From: Dan Williams @ 2018-07-08 20:46 UTC (permalink / raw)
To: mingo
Cc: Al Viro, Andrew Morton, Andy Lutomirski, Borislav Petkov,
Linus Torvalds, Peter Zijlstra, Thomas Gleixner, Tony Luck,
Ross Zwisler, Ross Zwisler, linux-kernel, x86
By mistake the ITER_PIPE early-exit / warning from copy_from_iter() was
cargo-culted in _copy_to_iter_mcsafe() rather than a machine-check-safe
version of copy_to_iter_pipe().
Implement copy_pipe_to_iter_mcsafe() being careful to return the
indication of short copies due to a CPU exception.
Without this regression-fix all splice reads to dax-mode files fail.
Fixes: 8780356ef630 ("x86/asm/memcpy_mcsafe: Define copy_to_iter_mcsafe()")
Acked-by: Al Viro <viro@zeniv.linux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Reported-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Tested-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
lib/iov_iter.c | 37 +++++++++++++++++++++++++++++++++----
1 file changed, 33 insertions(+), 4 deletions(-)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 09fb73ad9d54..8be175df3075 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -596,6 +596,37 @@ static unsigned long memcpy_mcsafe_to_page(struct page *page, size_t offset,
return ret;
}
+static size_t copy_pipe_to_iter_mcsafe(const void *addr, size_t bytes,
+ struct iov_iter *i)
+{
+ struct pipe_inode_info *pipe = i->pipe;
+ size_t n, off, xfer = 0;
+ int idx;
+
+ if (!sanity(i))
+ return 0;
+
+ bytes = n = push_pipe(i, bytes, &idx, &off);
+ if (unlikely(!n))
+ return 0;
+ for ( ; n; idx = next_idx(idx, pipe), off = 0) {
+ size_t chunk = min_t(size_t, n, PAGE_SIZE - off);
+ unsigned long rem;
+
+ rem = memcpy_mcsafe_to_page(pipe->bufs[idx].page, off, addr,
+ chunk);
+ i->idx = idx;
+ i->iov_offset = off + chunk - rem;
+ xfer += chunk - rem;
+ if (rem)
+ break;
+ n -= chunk;
+ addr += chunk;
+ }
+ i->count -= xfer;
+ return xfer;
+}
+
/**
* _copy_to_iter_mcsafe - copy to user with source-read error exception handling
* @addr: source kernel address
@@ -627,10 +658,8 @@ size_t _copy_to_iter_mcsafe(const void *addr, size_t bytes, struct iov_iter *i)
const char *from = addr;
unsigned long rem, curr_addr, s_addr = (unsigned long) addr;
- if (unlikely(i->type & ITER_PIPE)) {
- WARN_ON(1);
- return 0;
- }
+ if (unlikely(i->type & ITER_PIPE))
+ return copy_pipe_to_iter_mcsafe(addr, bytes, i);
if (iter_is_iovec(i))
might_fault();
iterate_and_advance(i, bytes, v,
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 4/4] x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling
2018-07-08 20:45 [PATCH 0/4] lib/iov_iter: Fixes and documentation for pmem user copies Dan Williams
` (2 preceding siblings ...)
2018-07-08 20:46 ` [PATCH 3/4] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe Dan Williams
@ 2018-07-08 20:46 ` Dan Williams
2018-07-15 23:32 ` [tip:core/urgent] " tip-bot for Dan Williams
3 siblings, 1 reply; 9+ messages in thread
From: Dan Williams @ 2018-07-08 20:46 UTC (permalink / raw)
To: mingo
Cc: Al Viro, Andrew Morton, Andy Lutomirski, Borislav Petkov,
Linus Torvalds, Peter Zijlstra, Thomas Gleixner, Tony Luck,
Ross Zwisler, Ross Zwisler, linux-kernel, x86
All copy_to_user() implementations need to be prepared to handle faults
accessing userspace. The __memcpy_mcsafe() implementation handles both
mmu-faults on the user destination and machine-check-exceptions on the
source buffer. However, the memcpy_mcsafe() wrapper may silently
fallback to memcpy() depending on build options and cpu-capabilities.
Force copy_to_user_mcsafe() to always use __memcpy_mcsafe() when
available, and otherwise disable all of the copy_to_user_mcsafe()
infrastructure when __memcpy_mcsafe() is not available, i.e.
CONFIG_X86_MCE=n.
This fixes crashes of the form:
run fstests generic/323 at 2018-07-02 12:46:23
BUG: unable to handle kernel paging request at 00007f0d50001000
RIP: 0010:__memcpy+0x12/0x20
[..]
Call Trace:
copyout_mcsafe+0x3a/0x50
_copy_to_iter_mcsafe+0xa1/0x4a0
? dax_alive+0x30/0x50
dax_iomap_actor+0x1f9/0x280
? dax_iomap_rw+0x100/0x100
iomap_apply+0xba/0x130
? dax_iomap_rw+0x100/0x100
dax_iomap_rw+0x95/0x100
? dax_iomap_rw+0x100/0x100
xfs_file_dax_read+0x7b/0x1d0 [xfs]
xfs_file_read_iter+0xa7/0xc0 [xfs]
aio_read+0x11c/0x1a0
Fixes: 8780356ef630 ("x86/asm/memcpy_mcsafe: Define copy_to_iter_mcsafe()")
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Reported-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Tested-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
arch/x86/Kconfig | 2 +-
arch/x86/include/asm/uaccess_64.h | 7 ++++++-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f1dbb4ee19d7..887d3a7bb646 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -63,7 +63,7 @@ config X86
select ARCH_HAS_PTE_SPECIAL
select ARCH_HAS_REFCOUNT
select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64
- select ARCH_HAS_UACCESS_MCSAFE if X86_64
+ select ARCH_HAS_UACCESS_MCSAFE if X86_64 && X86_MCE
select ARCH_HAS_SET_MEMORY
select ARCH_HAS_SG_CHAIN
select ARCH_HAS_STRICT_KERNEL_RWX
diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
index 62acb613114b..a9d637bc301d 100644
--- a/arch/x86/include/asm/uaccess_64.h
+++ b/arch/x86/include/asm/uaccess_64.h
@@ -52,7 +52,12 @@ copy_to_user_mcsafe(void *to, const void *from, unsigned len)
unsigned long ret;
__uaccess_begin();
- ret = memcpy_mcsafe(to, from, len);
+ /*
+ * Note, __memcpy_mcsafe() is explicitly used since it can
+ * handle exceptions / faults. memcpy_mcsafe() may fall back to
+ * memcpy() which lacks this handling.
+ */
+ ret = __memcpy_mcsafe(to, from, len);
__uaccess_end();
return ret;
}
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [tip:core/urgent] lib/iov_iter: Document _copy_to_iter_mcsafe()
2018-07-08 20:46 ` [PATCH 1/4] lib/iov_iter: Document _copy_to_iter_mcsafe() Dan Williams
@ 2018-07-15 23:30 ` tip-bot for Dan Williams
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot for Dan Williams @ 2018-07-15 23:30 UTC (permalink / raw)
To: linux-tip-commits
Cc: mingo, tglx, torvalds, bp, hpa, linux-kernel, tony.luck,
dan.j.williams, viro, peterz, akpm, luto
Commit-ID: bf3eeb9b5f2a1a05b3a68c6d82112babd58d6a39
Gitweb: https://git.kernel.org/tip/bf3eeb9b5f2a1a05b3a68c6d82112babd58d6a39
Author: Dan Williams <dan.j.williams@intel.com>
AuthorDate: Sun, 8 Jul 2018 13:46:02 -0700
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 16 Jul 2018 00:05:05 +0200
lib/iov_iter: Document _copy_to_iter_mcsafe()
Add some theory of operation documentation to _copy_to_iter_mcsafe().
Reported-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Link: http://lkml.kernel.org/r/153108276256.37979.1689794213845539316.stgit@dwillia2-desk3.amr.corp.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
lib/iov_iter.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 7e43cd54c84c..94fa361be7bb 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -596,6 +596,32 @@ static unsigned long memcpy_mcsafe_to_page(struct page *page, size_t offset,
return ret;
}
+/**
+ * _copy_to_iter_mcsafe - copy to user with source-read error exception handling
+ * @addr: source kernel address
+ * @bytes: total transfer length
+ * @iter: destination iterator
+ *
+ * The pmem driver arranges for filesystem-dax to use this facility via
+ * dax_copy_to_iter() for protecting read/write to persistent memory.
+ * Unless / until an architecture can guarantee identical performance
+ * between _copy_to_iter_mcsafe() and _copy_to_iter() it would be a
+ * performance regression to switch more users to the mcsafe version.
+ *
+ * Otherwise, the main differences between this and typical _copy_to_iter().
+ *
+ * * Typical tail/residue handling after a fault retries the copy
+ * byte-by-byte until the fault happens again. Re-triggering machine
+ * checks is potentially fatal so the implementation uses source
+ * alignment and poison alignment assumptions to avoid re-triggering
+ * hardware exceptions.
+ *
+ * * ITER_KVEC, ITER_PIPE, and ITER_BVEC can return short copies.
+ * Compare to copy_to_iter() where only ITER_IOVEC attempts might return
+ * a short copy.
+ *
+ * See MCSAFE_TEST for self-test.
+ */
size_t _copy_to_iter_mcsafe(const void *addr, size_t bytes, struct iov_iter *i)
{
const char *from = addr;
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [tip:core/urgent] lib/iov_iter: Document _copy_to_iter_flushcache()
2018-07-08 20:46 ` [PATCH 2/4] lib/iov_iter: Document _copy_to_iter_flushcache() Dan Williams
@ 2018-07-15 23:31 ` tip-bot for Dan Williams
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot for Dan Williams @ 2018-07-15 23:31 UTC (permalink / raw)
To: linux-tip-commits
Cc: mingo, dan.j.williams, torvalds, bp, tony.luck, viro, akpm,
peterz, hpa, linux-kernel, luto, tglx
Commit-ID: abd08d7d245397bcbded8c6c29ff79a36b3875b0
Gitweb: https://git.kernel.org/tip/abd08d7d245397bcbded8c6c29ff79a36b3875b0
Author: Dan Williams <dan.j.williams@intel.com>
AuthorDate: Sun, 8 Jul 2018 13:46:07 -0700
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 16 Jul 2018 00:05:05 +0200
lib/iov_iter: Document _copy_to_iter_flushcache()
Add some theory of operation documentation to _copy_to_iter_flushcache().
Reported-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Link: http://lkml.kernel.org/r/153108276767.37979.9462477994086841699.stgit@dwillia2-desk3.amr.corp.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
lib/iov_iter.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 94fa361be7bb..09fb73ad9d54 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -727,6 +727,20 @@ size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i)
EXPORT_SYMBOL(_copy_from_iter_nocache);
#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE
+/**
+ * _copy_from_iter_flushcache - write destination through cpu cache
+ * @addr: destination kernel address
+ * @bytes: total transfer length
+ * @iter: source iterator
+ *
+ * The pmem driver arranges for filesystem-dax to use this facility via
+ * dax_copy_from_iter() for ensuring that writes to persistent memory
+ * are flushed through the CPU cache. It is differentiated from
+ * _copy_from_iter_nocache() in that guarantees all data is flushed for
+ * all iterator types. The _copy_from_iter_nocache() only attempts to
+ * bypass the cache for the ITER_IOVEC case, and on some archs may use
+ * instructions that strand dirty-data in the cache.
+ */
size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i)
{
char *to = addr;
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [tip:core/urgent] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe()
2018-07-08 20:46 ` [PATCH 3/4] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe Dan Williams
@ 2018-07-15 23:31 ` tip-bot for Dan Williams
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot for Dan Williams @ 2018-07-15 23:31 UTC (permalink / raw)
To: linux-tip-commits
Cc: mingo, luto, peterz, tony.luck, hpa, linux-kernel, ross.zwisler,
dan.j.williams, torvalds, bp, viro, akpm, tglx
Commit-ID: ca146f6f091e47b3fd18d6a7e76ec0297d202e0f
Gitweb: https://git.kernel.org/tip/ca146f6f091e47b3fd18d6a7e76ec0297d202e0f
Author: Dan Williams <dan.j.williams@intel.com>
AuthorDate: Sun, 8 Jul 2018 13:46:12 -0700
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 16 Jul 2018 00:05:05 +0200
lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe()
By mistake the ITER_PIPE early-exit / warning from copy_from_iter() was
cargo-culted in _copy_to_iter_mcsafe() rather than a machine-check-safe
version of copy_to_iter_pipe().
Implement copy_pipe_to_iter_mcsafe() being careful to return the
indication of short copies due to a CPU exception.
Without this regression-fix all splice reads to dax-mode files fail.
Reported-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Tested-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Al Viro <viro@zeniv.linux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Fixes: 8780356ef630 ("x86/asm/memcpy_mcsafe: Define copy_to_iter_mcsafe()")
Link: http://lkml.kernel.org/r/153108277278.37979.3327916996902264102.stgit@dwillia2-desk3.amr.corp.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
lib/iov_iter.c | 37 +++++++++++++++++++++++++++++++++----
1 file changed, 33 insertions(+), 4 deletions(-)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 09fb73ad9d54..8be175df3075 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -596,6 +596,37 @@ static unsigned long memcpy_mcsafe_to_page(struct page *page, size_t offset,
return ret;
}
+static size_t copy_pipe_to_iter_mcsafe(const void *addr, size_t bytes,
+ struct iov_iter *i)
+{
+ struct pipe_inode_info *pipe = i->pipe;
+ size_t n, off, xfer = 0;
+ int idx;
+
+ if (!sanity(i))
+ return 0;
+
+ bytes = n = push_pipe(i, bytes, &idx, &off);
+ if (unlikely(!n))
+ return 0;
+ for ( ; n; idx = next_idx(idx, pipe), off = 0) {
+ size_t chunk = min_t(size_t, n, PAGE_SIZE - off);
+ unsigned long rem;
+
+ rem = memcpy_mcsafe_to_page(pipe->bufs[idx].page, off, addr,
+ chunk);
+ i->idx = idx;
+ i->iov_offset = off + chunk - rem;
+ xfer += chunk - rem;
+ if (rem)
+ break;
+ n -= chunk;
+ addr += chunk;
+ }
+ i->count -= xfer;
+ return xfer;
+}
+
/**
* _copy_to_iter_mcsafe - copy to user with source-read error exception handling
* @addr: source kernel address
@@ -627,10 +658,8 @@ size_t _copy_to_iter_mcsafe(const void *addr, size_t bytes, struct iov_iter *i)
const char *from = addr;
unsigned long rem, curr_addr, s_addr = (unsigned long) addr;
- if (unlikely(i->type & ITER_PIPE)) {
- WARN_ON(1);
- return 0;
- }
+ if (unlikely(i->type & ITER_PIPE))
+ return copy_pipe_to_iter_mcsafe(addr, bytes, i);
if (iter_is_iovec(i))
might_fault();
iterate_and_advance(i, bytes, v,
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [tip:core/urgent] x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling
2018-07-08 20:46 ` [PATCH 4/4] x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling Dan Williams
@ 2018-07-15 23:32 ` tip-bot for Dan Williams
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot for Dan Williams @ 2018-07-15 23:32 UTC (permalink / raw)
To: linux-tip-commits
Cc: dan.j.williams, luto, akpm, viro, tglx, hpa, linux-kernel,
tony.luck, peterz, mingo, ross.zwisler, bp, torvalds
Commit-ID: 092b31aa2048cf7561a39697974adcd147fbb27b
Gitweb: https://git.kernel.org/tip/092b31aa2048cf7561a39697974adcd147fbb27b
Author: Dan Williams <dan.j.williams@intel.com>
AuthorDate: Sun, 8 Jul 2018 13:46:17 -0700
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 16 Jul 2018 00:05:05 +0200
x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling
All copy_to_user() implementations need to be prepared to handle faults
accessing userspace. The __memcpy_mcsafe() implementation handles both
mmu-faults on the user destination and machine-check-exceptions on the
source buffer. However, the memcpy_mcsafe() wrapper may silently
fallback to memcpy() depending on build options and cpu-capabilities.
Force copy_to_user_mcsafe() to always use __memcpy_mcsafe() when
available, and otherwise disable all of the copy_to_user_mcsafe()
infrastructure when __memcpy_mcsafe() is not available, i.e.
CONFIG_X86_MCE=n.
This fixes crashes of the form:
run fstests generic/323 at 2018-07-02 12:46:23
BUG: unable to handle kernel paging request at 00007f0d50001000
RIP: 0010:__memcpy+0x12/0x20
[..]
Call Trace:
copyout_mcsafe+0x3a/0x50
_copy_to_iter_mcsafe+0xa1/0x4a0
? dax_alive+0x30/0x50
dax_iomap_actor+0x1f9/0x280
? dax_iomap_rw+0x100/0x100
iomap_apply+0xba/0x130
? dax_iomap_rw+0x100/0x100
dax_iomap_rw+0x95/0x100
? dax_iomap_rw+0x100/0x100
xfs_file_dax_read+0x7b/0x1d0 [xfs]
xfs_file_read_iter+0xa7/0xc0 [xfs]
aio_read+0x11c/0x1a0
Reported-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Tested-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Fixes: 8780356ef630 ("x86/asm/memcpy_mcsafe: Define copy_to_iter_mcsafe()")
Link: http://lkml.kernel.org/r/153108277790.37979.1486841789275803399.stgit@dwillia2-desk3.amr.corp.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/Kconfig | 2 +-
arch/x86/include/asm/uaccess_64.h | 7 ++++++-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f1dbb4ee19d7..887d3a7bb646 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -63,7 +63,7 @@ config X86
select ARCH_HAS_PTE_SPECIAL
select ARCH_HAS_REFCOUNT
select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64
- select ARCH_HAS_UACCESS_MCSAFE if X86_64
+ select ARCH_HAS_UACCESS_MCSAFE if X86_64 && X86_MCE
select ARCH_HAS_SET_MEMORY
select ARCH_HAS_SG_CHAIN
select ARCH_HAS_STRICT_KERNEL_RWX
diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
index 62acb613114b..a9d637bc301d 100644
--- a/arch/x86/include/asm/uaccess_64.h
+++ b/arch/x86/include/asm/uaccess_64.h
@@ -52,7 +52,12 @@ copy_to_user_mcsafe(void *to, const void *from, unsigned len)
unsigned long ret;
__uaccess_begin();
- ret = memcpy_mcsafe(to, from, len);
+ /*
+ * Note, __memcpy_mcsafe() is explicitly used since it can
+ * handle exceptions / faults. memcpy_mcsafe() may fall back to
+ * memcpy() which lacks this handling.
+ */
+ ret = __memcpy_mcsafe(to, from, len);
__uaccess_end();
return ret;
}
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-07-15 23:33 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-08 20:45 [PATCH 0/4] lib/iov_iter: Fixes and documentation for pmem user copies Dan Williams
2018-07-08 20:46 ` [PATCH 1/4] lib/iov_iter: Document _copy_to_iter_mcsafe() Dan Williams
2018-07-15 23:30 ` [tip:core/urgent] " tip-bot for Dan Williams
2018-07-08 20:46 ` [PATCH 2/4] lib/iov_iter: Document _copy_to_iter_flushcache() Dan Williams
2018-07-15 23:31 ` [tip:core/urgent] " tip-bot for Dan Williams
2018-07-08 20:46 ` [PATCH 3/4] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe Dan Williams
2018-07-15 23:31 ` [tip:core/urgent] lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe() tip-bot for Dan Williams
2018-07-08 20:46 ` [PATCH 4/4] x86/asm/memcpy_mcsafe: Fix copy_to_user_mcsafe() exception handling Dan Williams
2018-07-15 23:32 ` [tip:core/urgent] " tip-bot for Dan Williams
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).