From: Tony Luck <tony.luck@intel.com> To: Ingo Molnar <mingo@kernel.org> Cc: Borislav Petkov <bp@alien8.de>, Andrew Morton <akpm@linux-foundation.org>, Andy Lutomirski <luto@kernel.org>, Dan Williams <dan.j.williams@intel.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@ml01.01.org, x86@kernel.org Subject: [PATCHV2 3/3] x86, ras: Add mcsafe_memcpy() function to recover from machine checks Date: Thu, 10 Dec 2015 16:21:50 -0800 [thread overview] Message-ID: <23b2515da9d06b198044ad83ca0a15ba38c24e6e.1449861203.git.tony.luck@intel.com> (raw) In-Reply-To: <cover.1449861203.git.tony.luck@intel.com> Using __copy_user_nocache() as inspiration create a memory copy routine for use by kernel code with annotations to allow for recovery from machine checks. Notes: 1) Unlike the original we make no attempt to copy all the bytes up to the faulting address. The original achieves that by re-executing the failing part as a byte-by-byte copy, which will take another page fault. We don't want to have a second machine check! 2) Likewise the return value for the original indicates exactly how many bytes were not copied. Instead we provide the physical address of the fault (thanks to help from do_machine_check() 3) Provide helpful macros to decode the return value. Signed-off-by: Tony Luck <tony.luck@intel.com> --- arch/x86/include/asm/uaccess_64.h | 5 +++ arch/x86/kernel/x8664_ksyms_64.c | 2 + arch/x86/lib/copy_user_64.S | 91 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+) diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h index f2f9b39b274a..779cb0e77ecc 100644 --- a/arch/x86/include/asm/uaccess_64.h +++ b/arch/x86/include/asm/uaccess_64.h @@ -216,6 +216,11 @@ __copy_to_user_inatomic(void __user *dst, const void *src, unsigned size) extern long __copy_user_nocache(void *dst, const void __user *src, unsigned size, int zerorest); +extern u64 mcsafe_memcpy(void *dst, const void __user *src, + unsigned size); +#define COPY_HAD_MCHECK(ret) ((ret) & BIT(63)) +#define COPY_MCHECK_PADDR(ret) ((ret) & ~BIT(63)) + static inline int __copy_from_user_nocache(void *dst, const void __user *src, unsigned size) { diff --git a/arch/x86/kernel/x8664_ksyms_64.c b/arch/x86/kernel/x8664_ksyms_64.c index a0695be19864..ec988c92c055 100644 --- a/arch/x86/kernel/x8664_ksyms_64.c +++ b/arch/x86/kernel/x8664_ksyms_64.c @@ -37,6 +37,8 @@ EXPORT_SYMBOL(__copy_user_nocache); EXPORT_SYMBOL(_copy_from_user); EXPORT_SYMBOL(_copy_to_user); +EXPORT_SYMBOL(mcsafe_memcpy); + EXPORT_SYMBOL(copy_page); EXPORT_SYMBOL(clear_page); diff --git a/arch/x86/lib/copy_user_64.S b/arch/x86/lib/copy_user_64.S index 982ce34f4a9b..ffce93cbc9a5 100644 --- a/arch/x86/lib/copy_user_64.S +++ b/arch/x86/lib/copy_user_64.S @@ -319,3 +319,94 @@ ENTRY(__copy_user_nocache) _ASM_EXTABLE(21b,50b) _ASM_EXTABLE(22b,50b) ENDPROC(__copy_user_nocache) + +/* + * mcsafe_memcpy - Uncached memory copy with machine check exception handling + * Note that we only catch machine checks when reading the source addresses. + * Writes to target are posted and don't generate machine checks. + * This will force destination/source out of cache for more performance. + */ +ENTRY(mcsafe_memcpy) + cmpl $8,%edx + jb 20f /* less then 8 bytes, go to byte copy loop */ + + /* check for bad alignment of destination */ + movl %edi,%ecx + andl $7,%ecx + jz 102f /* already aligned */ + subl $8,%ecx + negl %ecx + subl %ecx,%edx +0: movb (%rsi),%al + movb %al,(%rdi) + incq %rsi + incq %rdi + decl %ecx + jnz 100b +102: + movl %edx,%ecx + andl $63,%edx + shrl $6,%ecx + jz 17f +1: movq (%rsi),%r8 +2: movq 1*8(%rsi),%r9 +3: movq 2*8(%rsi),%r10 +4: movq 3*8(%rsi),%r11 + movnti %r8,(%rdi) + movnti %r9,1*8(%rdi) + movnti %r10,2*8(%rdi) + movnti %r11,3*8(%rdi) +9: movq 4*8(%rsi),%r8 +10: movq 5*8(%rsi),%r9 +11: movq 6*8(%rsi),%r10 +12: movq 7*8(%rsi),%r11 + movnti %r8,4*8(%rdi) + movnti %r9,5*8(%rdi) + movnti %r10,6*8(%rdi) + movnti %r11,7*8(%rdi) + leaq 64(%rsi),%rsi + leaq 64(%rdi),%rdi + decl %ecx + jnz 1b +17: movl %edx,%ecx + andl $7,%edx + shrl $3,%ecx + jz 20f +18: movq (%rsi),%r8 + movnti %r8,(%rdi) + leaq 8(%rsi),%rsi + leaq 8(%rdi),%rdi + decl %ecx + jnz 18b +20: andl %edx,%edx + jz 23f + movl %edx,%ecx +21: movb (%rsi),%al + movb %al,(%rdi) + incq %rsi + incq %rdi + decl %ecx + jnz 21b +23: xorl %eax,%eax + sfence + ret + + .section .fixup,"ax" +30: + sfence + /* do_machine_check() sets %eax return value */ + ret + .previous + + _ASM_MCEXTABLE(0b,30b) + _ASM_MCEXTABLE(1b,30b) + _ASM_MCEXTABLE(2b,30b) + _ASM_MCEXTABLE(3b,30b) + _ASM_MCEXTABLE(4b,30b) + _ASM_MCEXTABLE(9b,30b) + _ASM_MCEXTABLE(10b,30b) + _ASM_MCEXTABLE(11b,30b) + _ASM_MCEXTABLE(12b,30b) + _ASM_MCEXTABLE(18b,30b) + _ASM_MCEXTABLE(21b,30b) +ENDPROC(mcsafe_memcpy) -- 2.1.4
WARNING: multiple messages have this Message-ID (diff)
From: Tony Luck <tony.luck@intel.com> To: Ingo Molnar <mingo@kernel.org> Cc: Borislav Petkov <bp@alien8.de>, Andrew Morton <akpm@linux-foundation.org>, Andy Lutomirski <luto@kernel.org>, Dan Williams <dan.j.williams@intel.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@ml01.01.org, x86@kernel.org Subject: [PATCHV2 3/3] x86, ras: Add mcsafe_memcpy() function to recover from machine checks Date: Thu, 10 Dec 2015 16:21:50 -0800 [thread overview] Message-ID: <23b2515da9d06b198044ad83ca0a15ba38c24e6e.1449861203.git.tony.luck@intel.com> (raw) In-Reply-To: <cover.1449861203.git.tony.luck@intel.com> Using __copy_user_nocache() as inspiration create a memory copy routine for use by kernel code with annotations to allow for recovery from machine checks. Notes: 1) Unlike the original we make no attempt to copy all the bytes up to the faulting address. The original achieves that by re-executing the failing part as a byte-by-byte copy, which will take another page fault. We don't want to have a second machine check! 2) Likewise the return value for the original indicates exactly how many bytes were not copied. Instead we provide the physical address of the fault (thanks to help from do_machine_check() 3) Provide helpful macros to decode the return value. Signed-off-by: Tony Luck <tony.luck@intel.com> --- arch/x86/include/asm/uaccess_64.h | 5 +++ arch/x86/kernel/x8664_ksyms_64.c | 2 + arch/x86/lib/copy_user_64.S | 91 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+) diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h index f2f9b39b274a..779cb0e77ecc 100644 --- a/arch/x86/include/asm/uaccess_64.h +++ b/arch/x86/include/asm/uaccess_64.h @@ -216,6 +216,11 @@ __copy_to_user_inatomic(void __user *dst, const void *src, unsigned size) extern long __copy_user_nocache(void *dst, const void __user *src, unsigned size, int zerorest); +extern u64 mcsafe_memcpy(void *dst, const void __user *src, + unsigned size); +#define COPY_HAD_MCHECK(ret) ((ret) & BIT(63)) +#define COPY_MCHECK_PADDR(ret) ((ret) & ~BIT(63)) + static inline int __copy_from_user_nocache(void *dst, const void __user *src, unsigned size) { diff --git a/arch/x86/kernel/x8664_ksyms_64.c b/arch/x86/kernel/x8664_ksyms_64.c index a0695be19864..ec988c92c055 100644 --- a/arch/x86/kernel/x8664_ksyms_64.c +++ b/arch/x86/kernel/x8664_ksyms_64.c @@ -37,6 +37,8 @@ EXPORT_SYMBOL(__copy_user_nocache); EXPORT_SYMBOL(_copy_from_user); EXPORT_SYMBOL(_copy_to_user); +EXPORT_SYMBOL(mcsafe_memcpy); + EXPORT_SYMBOL(copy_page); EXPORT_SYMBOL(clear_page); diff --git a/arch/x86/lib/copy_user_64.S b/arch/x86/lib/copy_user_64.S index 982ce34f4a9b..ffce93cbc9a5 100644 --- a/arch/x86/lib/copy_user_64.S +++ b/arch/x86/lib/copy_user_64.S @@ -319,3 +319,94 @@ ENTRY(__copy_user_nocache) _ASM_EXTABLE(21b,50b) _ASM_EXTABLE(22b,50b) ENDPROC(__copy_user_nocache) + +/* + * mcsafe_memcpy - Uncached memory copy with machine check exception handling + * Note that we only catch machine checks when reading the source addresses. + * Writes to target are posted and don't generate machine checks. + * This will force destination/source out of cache for more performance. + */ +ENTRY(mcsafe_memcpy) + cmpl $8,%edx + jb 20f /* less then 8 bytes, go to byte copy loop */ + + /* check for bad alignment of destination */ + movl %edi,%ecx + andl $7,%ecx + jz 102f /* already aligned */ + subl $8,%ecx + negl %ecx + subl %ecx,%edx +0: movb (%rsi),%al + movb %al,(%rdi) + incq %rsi + incq %rdi + decl %ecx + jnz 100b +102: + movl %edx,%ecx + andl $63,%edx + shrl $6,%ecx + jz 17f +1: movq (%rsi),%r8 +2: movq 1*8(%rsi),%r9 +3: movq 2*8(%rsi),%r10 +4: movq 3*8(%rsi),%r11 + movnti %r8,(%rdi) + movnti %r9,1*8(%rdi) + movnti %r10,2*8(%rdi) + movnti %r11,3*8(%rdi) +9: movq 4*8(%rsi),%r8 +10: movq 5*8(%rsi),%r9 +11: movq 6*8(%rsi),%r10 +12: movq 7*8(%rsi),%r11 + movnti %r8,4*8(%rdi) + movnti %r9,5*8(%rdi) + movnti %r10,6*8(%rdi) + movnti %r11,7*8(%rdi) + leaq 64(%rsi),%rsi + leaq 64(%rdi),%rdi + decl %ecx + jnz 1b +17: movl %edx,%ecx + andl $7,%edx + shrl $3,%ecx + jz 20f +18: movq (%rsi),%r8 + movnti %r8,(%rdi) + leaq 8(%rsi),%rsi + leaq 8(%rdi),%rdi + decl %ecx + jnz 18b +20: andl %edx,%edx + jz 23f + movl %edx,%ecx +21: movb (%rsi),%al + movb %al,(%rdi) + incq %rsi + incq %rdi + decl %ecx + jnz 21b +23: xorl %eax,%eax + sfence + ret + + .section .fixup,"ax" +30: + sfence + /* do_machine_check() sets %eax return value */ + ret + .previous + + _ASM_MCEXTABLE(0b,30b) + _ASM_MCEXTABLE(1b,30b) + _ASM_MCEXTABLE(2b,30b) + _ASM_MCEXTABLE(3b,30b) + _ASM_MCEXTABLE(4b,30b) + _ASM_MCEXTABLE(9b,30b) + _ASM_MCEXTABLE(10b,30b) + _ASM_MCEXTABLE(11b,30b) + _ASM_MCEXTABLE(12b,30b) + _ASM_MCEXTABLE(18b,30b) + _ASM_MCEXTABLE(21b,30b) +ENDPROC(mcsafe_memcpy) -- 2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2015-12-11 19:32 UTC|newest] Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-12-11 19:13 [PATCHV2 0/3] Machine check recovery when kernel accesses poison Tony Luck 2015-12-11 19:13 ` Tony Luck 2015-12-10 21:58 ` [PATCHV2 1/3] x86, ras: Add new infrastructure for machine check fixup tables Tony Luck 2015-12-10 21:58 ` Tony Luck 2015-12-11 20:06 ` Andy Lutomirski 2015-12-11 20:06 ` Andy Lutomirski 2015-12-11 21:01 ` Luck, Tony 2015-12-11 21:01 ` Luck, Tony 2015-12-12 10:11 ` Borislav Petkov 2015-12-12 10:11 ` Borislav Petkov 2015-12-14 17:58 ` Ross Zwisler 2015-12-14 17:58 ` Ross Zwisler 2015-12-14 22:27 ` Borislav Petkov 2015-12-14 22:27 ` Borislav Petkov 2015-12-15 1:00 ` Luck, Tony 2015-12-15 1:00 ` Luck, Tony 2015-12-15 9:46 ` Borislav Petkov 2015-12-15 9:46 ` Borislav Petkov 2015-12-15 10:44 ` Borislav Petkov 2015-12-15 10:44 ` Borislav Petkov 2015-12-11 0:14 ` [PATCHV2 2/3] x86, ras: Extend machine check recovery code to annotated ring0 areas Tony Luck 2015-12-11 0:14 ` Tony Luck 2015-12-11 20:08 ` Andy Lutomirski 2015-12-11 20:08 ` Andy Lutomirski 2015-12-15 11:43 ` Borislav Petkov 2015-12-15 11:43 ` Borislav Petkov 2015-12-15 23:46 ` Luck, Tony 2015-12-15 23:46 ` Luck, Tony 2015-12-11 0:21 ` Tony Luck [this message] 2015-12-11 0:21 ` [PATCHV2 3/3] x86, ras: Add mcsafe_memcpy() function to recover from machine checks Tony Luck 2015-12-11 20:09 ` Andy Lutomirski 2015-12-11 20:09 ` Andy Lutomirski 2015-12-11 21:19 ` Luck, Tony 2015-12-11 21:19 ` Luck, Tony 2015-12-11 21:32 ` Konrad Rzeszutek Wilk 2015-12-11 21:50 ` Andy Lutomirski 2015-12-11 21:50 ` Andy Lutomirski 2015-12-11 22:17 ` Luck, Tony 2015-12-11 22:17 ` Luck, Tony 2015-12-11 22:20 ` Dan Williams 2015-12-11 22:20 ` Dan Williams 2015-12-11 22:26 ` Andy Lutomirski 2015-12-11 22:26 ` Andy Lutomirski 2015-12-11 22:35 ` Luck, Tony 2015-12-11 22:35 ` Luck, Tony 2015-12-11 22:38 ` Andy Lutomirski 2015-12-11 22:38 ` Andy Lutomirski 2015-12-11 22:45 ` Luck, Tony 2015-12-11 22:45 ` Luck, Tony 2015-12-11 22:55 ` Andy Lutomirski 2015-12-11 22:55 ` Andy Lutomirski 2015-12-14 8:36 ` Ingo Molnar 2015-12-14 8:36 ` Ingo Molnar 2015-12-14 19:46 ` Luck, Tony 2015-12-14 19:46 ` Luck, Tony 2015-12-14 20:11 ` Andy Lutomirski 2015-12-14 20:11 ` Andy Lutomirski 2015-12-15 13:11 ` Borislav Petkov 2015-12-15 13:11 ` Borislav Petkov 2015-12-15 17:45 ` Dan Williams 2015-12-15 17:45 ` Dan Williams 2015-12-15 17:53 ` Luck, Tony 2015-12-15 17:53 ` Luck, Tony 2015-12-15 18:21 ` Borislav Petkov 2015-12-15 18:21 ` Borislav Petkov 2015-12-15 18:27 ` Dan Williams 2015-12-15 18:27 ` Dan Williams 2015-12-15 18:35 ` Dan Williams 2015-12-15 18:35 ` Dan Williams 2015-12-15 18:39 ` Borislav Petkov 2015-12-15 18:39 ` Borislav Petkov 2015-12-15 19:19 ` Elliott, Robert (Persistent Memory) 2015-12-15 19:19 ` Elliott, Robert (Persistent Memory) 2015-12-15 19:28 ` Borislav Petkov 2015-12-15 19:28 ` Borislav Petkov 2015-12-15 20:25 ` Elliott, Robert (Persistent Memory) 2015-12-15 20:25 ` Elliott, Robert (Persistent Memory) 2015-12-21 17:33 ` Borislav Petkov 2015-12-21 17:33 ` Borislav Petkov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=23b2515da9d06b198044ad83ca0a15ba38c24e6e.1449861203.git.tony.luck@intel.com \ --to=tony.luck@intel.com \ --cc=akpm@linux-foundation.org \ --cc=bp@alien8.de \ --cc=dan.j.williams@intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-nvdimm@ml01.01.org \ --cc=luto@kernel.org \ --cc=mingo@kernel.org \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.