[PATCH 0/5] RFC: x86: Early exception table support

[PATCH 0/5] RFC: x86: Early exception table support

[H. Peter Anvin]

If we get an exception during early boot, walk the exception table to
see if we should intercept it.  The main use case for this is to allow
rdmsr_safe()/wrmsr_safe() during CPU initialization.

Since the exception table is currently sorted at runtime, and fairly
late in startup, this code walks the exception table linearly.  We
obviously don't need to worry about modules, however: none have been
loaded at this point.

In the future it would be better to have the table sorted at compile
time, or even better, turned into a perfect hash.  At that point this
code should be changed out from doing a linear search.

This patchset also makes the early exception handling a little more
similar between x86-64 and i386, but a lot of unification could (and
should) still be done: in particular, setup done in C as in x86-64,
and verbose error dump as in i386.

[PATCH 1/5] x86, nop: Make the ASM_NOP* macros work from assembly

[H. Peter Anvin]

From: "H. Peter Anvin" <hpa@zytor.com>

Make the ASM_NOP* macros work in actual assembly files.

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
---
 arch/x86/include/asm/nops.h |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/nops.h b/arch/x86/include/asm/nops.h
index 405b403..aff2b33 100644
--- a/arch/x86/include/asm/nops.h
+++ b/arch/x86/include/asm/nops.h
@@ -87,7 +87,11 @@
 #define P6_NOP8	0x0f,0x1f,0x84,0x00,0,0,0,0
 #define P6_NOP5_ATOMIC P6_NOP5
 
+#ifdef __ASSEMBLY__
+#define _ASM_MK_NOP(x) .byte x
+#else
 #define _ASM_MK_NOP(x) ".byte " __stringify(x) "\n"
+#endif
 
 #if defined(CONFIG_MK7)
 #define ASM_NOP1 _ASM_MK_NOP(K7_NOP1)
-- 
1.7.6.5

[PATCH 2/5] x86: Add symbolic constant for exceptions with error code

[H. Peter Anvin]

From: "H. Peter Anvin" <hpa@zytor.com>

Add a symbolic constant for the bitmask which states which exceptions
carry an error code.

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
---
 arch/x86/include/asm/segment.h |    2 ++
 arch/x86/kernel/head_64.S      |    2 +-
 2 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 1654662..58c1e6c 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -205,6 +205,8 @@
 
 #define IDT_ENTRIES 256
 #define NUM_EXCEPTION_VECTORS 32
+/* Bitmask of exception vectors which push an error code on the stack */
+#define EXCEPTION_ERRCODE_MASK  0x00027d00
 #define GDT_SIZE (GDT_ENTRIES * 8)
 #define GDT_ENTRY_TLS_ENTRIES 3
 #define TLS_SIZE (GDT_ENTRY_TLS_ENTRIES * 8)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 40f4eb3..adf52e8 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -295,7 +295,7 @@ ENTRY(early_idt_handler)
 	ja 0f
 	movl $1,%eax
 	salq %cl,%rax
-	testl $0x27d00,%eax
+	testl $EXCEPTION_ERRCODE_MASK,%eax
 	je 0f
 	popq %r8		# get error code
 0:	movq 0(%rsp),%rcx	# get ip
-- 
1.7.6.5

[PATCH 3/5] x86, paravirt: Replace GET_CR2_INTO_RCX with GET_CR2_INTO_RAX

[H. Peter Anvin]

From: "H. Peter Anvin" <hpa@zytor.com>

GET_CR2_INTO_RCX is asinine: it is only used in one place, the actual
paravirt call returns the value in %rax, not %rcx; and the one place
that wants it wants the result in %r9.  We actually generate as a
result of this call:

       call ...
       movq %rax, %rcx
       xorq %rax, %rax		/* this value isn't even used... */
       movq %rcx, %r9

At least make the macro do what the paravirt call does, which is put
the value into %rax.

Nevermind the fact that the macro clobbers all the volatile registers.

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Cc: Glauber de Oliveira Costa <gcosta@redhat.com>
---
 arch/x86/include/asm/paravirt.h |    6 ++----
 arch/x86/kernel/head_64.S       |    6 +++---
 2 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index aa0f913..6cbbabf 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -1023,10 +1023,8 @@ extern void default_banner(void);
 		  call PARA_INDIRECT(pv_cpu_ops+PV_CPU_swapgs)		\
 		 )
 
-#define GET_CR2_INTO_RCX				\
-	call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2);	\
-	movq %rax, %rcx;				\
-	xorq %rax, %rax;
+#define GET_CR2_INTO_RAX				\
+	call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2)
 
 #define PARAVIRT_ADJUST_EXCEPTION_FRAME					\
 	PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_adjust_exception_frame), \
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index adf52e8..d1e112c 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -23,8 +23,9 @@
 #ifdef CONFIG_PARAVIRT
 #include <asm/asm-offsets.h>
 #include <asm/paravirt.h>
+#define GET_CR2_INTO(reg) GET_CR2_INTO_RAX ; movq %rax, reg
 #else
-#define GET_CR2_INTO_RCX movq %cr2, %rcx
+#define GET_CR2_INTO(reg) movq %cr2, reg
 #endif
 
 /* we are not able to switch in one step to the final KERNEL ADDRESS SPACE
@@ -286,8 +287,7 @@ ENTRY(early_idt_handler)
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
 	incl early_recursion_flag(%rip)
-	GET_CR2_INTO_RCX
-	movq %rcx,%r9
+	GET_CR2_INTO(%r9)
 	xorl %r8d,%r8d		# zero for error code
 	movl %esi,%ecx		# get vector number
 	# Test %ecx against mask of vectors that push error code.
-- 
1.7.6.5

[PATCH 4/5] x86-64: Handle exception table entries during early boot

[H. Peter Anvin]

From: "H. Peter Anvin" <hpa@zytor.com>

If we get an exception during early boot, walk the exception table to
see if we should intercept it.  The main use case for this is to allow
rdmsr_safe()/wrmsr_safe() during CPU initialization.

Since the exception table is currently sorted at runtime, and fairly
late in startup, this code walks the exception table linearly.  We
obviously don't need to worry about modules, however: none have been
loaded at this point.

In the future it would be better to have the table sorted at compile
time, or even better, turned into a perfect hash.  At that point this
code should be changed out from doing a linear search.
---
 arch/x86/include/asm/segment.h |    2 +-
 arch/x86/kernel/head_64.S      |   72 +++++++++++++++++++++++++++++----------
 2 files changed, 54 insertions(+), 20 deletions(-)

diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 58c1e6c..c48a950 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -213,7 +213,7 @@
 
 #ifdef __KERNEL__
 #ifndef __ASSEMBLY__
-extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
+extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][2+2+5];
 
 /*
  * Load a segment. Fall back on loading the zero
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index d1e112c..8b89239 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -19,6 +19,7 @@
 #include <asm/cache.h>
 #include <asm/processor-flags.h>
 #include <asm/percpu.h>
+#include <asm/nops.h>
 
 #ifdef CONFIG_PARAVIRT
 #include <asm/asm-offsets.h>
@@ -26,6 +27,7 @@
 #define GET_CR2_INTO(reg) GET_CR2_INTO_RAX ; movq %rax, reg
 #else
 #define GET_CR2_INTO(reg) movq %cr2, reg
+#define INTERRUPT_RETURN iretq
 #endif
 
 /* we are not able to switch in one step to the final KERNEL ADDRESS SPACE
@@ -271,35 +273,56 @@ bad_address:
 	jmp bad_address
 
 	.section ".init.text","ax"
-#ifdef CONFIG_EARLY_PRINTK
 	.globl early_idt_handlers
 early_idt_handlers:
+	# 56(%rsp) %rflags
+	# 48(%rsp) %cs
+	# 40(%rsp) %rip
+	# 32(%rsp) error code
 	i = 0
 	.rept NUM_EXCEPTION_VECTORS
-	movl $i, %esi
+	.if (EXCEPTION_ERRCODE_MASK >> i) & 1
+	ASM_NOP2
+	.else
+	pushq $0		# Dummy error code, to make stack frame uniform
+	.endif
+	pushq $i		# 24(%rsp) Vector number
 	jmp early_idt_handler
 	i = i + 1
 	.endr
-#endif
 
 ENTRY(early_idt_handler)
-#ifdef CONFIG_EARLY_PRINTK
+	cld
+
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
 	incl early_recursion_flag(%rip)
-	GET_CR2_INTO(%r9)
-	xorl %r8d,%r8d		# zero for error code
-	movl %esi,%ecx		# get vector number
-	# Test %ecx against mask of vectors that push error code.
-	cmpl $31,%ecx
-	ja 0f
-	movl $1,%eax
-	salq %cl,%rax
-	testl $EXCEPTION_ERRCODE_MASK,%eax
-	je 0f
-	popq %r8		# get error code
-0:	movq 0(%rsp),%rcx	# get ip
-	movq 8(%rsp),%rdx	# get cs
+
+	pushq %rax		# 16(%rsp)
+	pushq %rcx		#  8(%rsp)
+	pushq %rdx		#  0(%rsp)
+
+	cmpl $__KERNEL_CS,48(%rsp)
+	jne 10f
+
+	movq 40(%rsp),%rax	# %rip
+	leaq __start___ex_table(%rip),%rcx
+	leaq __stop___ex_table(%rip),%rdx
+
+11:
+	cmpq (%rcx),%rax
+	je 20f			# Found an exception entry
+	addq $16,%rcx
+	cmpq %rdx,%rcx
+	jb 11b
+
+10:
+#ifdef CONFIG_EARLY_PRINTK
+	GET_CR2_INTO(%r9)	# can clobber any volatile register if pv
+	movl 32(%rsp),%r8d	# error code
+	movl 24(%rsp),%esi	# vector number
+	movl 48(%rsp),%edx	# %cs
+	movq 40(%rsp),%rcx	# %rip
 	xorl %eax,%eax
 	leaq early_idt_msg(%rip),%rdi
 	call early_printk
@@ -308,17 +331,28 @@ ENTRY(early_idt_handler)
 	call dump_stack
 #ifdef CONFIG_KALLSYMS	
 	leaq early_idt_ripmsg(%rip),%rdi
-	movq 0(%rsp),%rsi	# get rip again
+	movq 40(%rsp),%rsi	# %rip again
 	call __print_symbol
 #endif
 #endif /* EARLY_PRINTK */
 1:	hlt
 	jmp 1b
 
-#ifdef CONFIG_EARLY_PRINTK
+20:	# Exception table entry found
+	movq 8(%rcx),%rax	# exception table target
+	movq %rax,40(%rsp)	# change %rip on stack
+	popq %rdx
+	popq %rcx
+	popq %rax
+	addq $16,%rsp		# drop vector number and error code
+	decl early_recursion_flag(%rip)
+	INTERRUPT_RETURN
+
+	.balign 4
 early_recursion_flag:
 	.long 0
 
+#ifdef CONFIG_EARLY_PRINTK
 early_idt_msg:
 	.asciz "PANIC: early exception %02lx rip %lx:%lx error %lx cr2 %lx\n"
 early_idt_ripmsg:
-- 
1.7.6.5

[PATCH 5/5] x86-32: Handle exception table entries during early boot

[H. Peter Anvin]

From: "H. Peter Anvin" <hpa@zytor.com>

If we get an exception during early boot, walk the exception table to
see if we should intercept it.  The main use case for this is to allow
rdmsr_safe()/wrmsr_safe() during CPU initialization.

Since the exception table is currently sorted at runtime, and fairly
late in startup, this code walks the exception table linearly.  We
obviously don't need to worry about modules, however: none have been
loaded at this point.

In the future it would be better to have the table sorted at compile
time, or even better, turned into a perfect hash.  At that point this
code should be changed out from doing a linear search.

This patch changes the early IDT setup to look a lot more like x86-64:
we now install handlers for all 32 exception vectors.  The output of
the early exception handler has changed somewhat as it directly
reflects the stack frame of the exception handler, and the stack frame
has been somewhat restructured.

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
---
 arch/x86/kernel/head_32.S |  142 +++++++++++++++++++++++++++++----------------
 1 files changed, 91 insertions(+), 51 deletions(-)

diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S
index ce0be7c..3a3cd0d 100644
--- a/arch/x86/kernel/head_32.S
+++ b/arch/x86/kernel/head_32.S
@@ -21,6 +21,7 @@
 #include <asm/msr-index.h>
 #include <asm/cpufeature.h>
 #include <asm/percpu.h>
+#include <asm/nops.h>
 
 /* Physical address */
 #define pa(X) ((X) - __PAGE_OFFSET)
@@ -508,68 +509,94 @@ check_x87:
  *  Warning: %esi is live across this function.
  */
 setup_idt:
-	lea ignore_int,%edx
+	movl $idt_table,%edi
+	movl $early_idt_handlers,%eax
+	movl $NUM_EXCEPTION_VECTORS,%ecx
+1:
+	movl %eax,(%edi)
+	movl %eax,4(%edi)
+	/* interrupt gate, dpl=0, present */
+	movl $(0x8E000000 + __KERNEL_CS),2(%edi)
+	addl $9,%eax
+	addl $8,%edi
+	loop 1b
+
+	movl $256 - NUM_EXCEPTION_VECTORS,%ecx
+	movl $ignore_int,%edx
 	movl $(__KERNEL_CS << 16),%eax
 	movw %dx,%ax		/* selector = 0x0010 = cs */
 	movw $0x8E00,%dx	/* interrupt gate - dpl=0, present */
-
-	lea idt_table,%edi
-	mov $256,%ecx
-rp_sidt:
+2:
 	movl %eax,(%edi)
 	movl %edx,4(%edi)
 	addl $8,%edi
-	dec %ecx
-	jne rp_sidt
-
-.macro	set_early_handler handler,trapno
-	lea \handler,%edx
-	movl $(__KERNEL_CS << 16),%eax
-	movw %dx,%ax
-	movw $0x8E00,%dx	/* interrupt gate - dpl=0, present */
-	lea idt_table,%edi
-	movl %eax,8*\trapno(%edi)
-	movl %edx,8*\trapno+4(%edi)
-.endm
-
-	set_early_handler handler=early_divide_err,trapno=0
-	set_early_handler handler=early_illegal_opcode,trapno=6
-	set_early_handler handler=early_protection_fault,trapno=13
-	set_early_handler handler=early_page_fault,trapno=14
-
+	loop 2b
+	
 	ret
 
-early_divide_err:
-	xor %edx,%edx
-	pushl $0	/* fake errcode */
-	jmp early_fault
+	.globl early_idt_handlers
+early_idt_handlers:
+	# 36(%esp) %eflags
+	# 32(%esp) %cs
+	# 28(%esp) %eip
+	# 24(%rsp) error code
+	i = 0
+	.rept NUM_EXCEPTION_VECTORS
+	.if (EXCEPTION_ERRCODE_MASK >> i) & 1
+	ASM_NOP2
+	.else
+	pushl $0		# Dummy error code, to make stack frame uniform
+	.endif
+	pushl $i		# 20(%esp) Vector number
+	jmp early_idt_handler
+	i = i + 1
+	.endr
+
+	/* This is global to keep gas from relaxing the jumps */
+	.globl	early_idt_handler
+early_idt_handler:
+	cld
+	cmpl $2,%ss:early_recursion_flag
+	je hlt_loop
+	incl %ss:early_recursion_flag
 
-early_illegal_opcode:
-	movl $6,%edx
-	pushl $0	/* fake errcode */
-	jmp early_fault
+	push %eax		# 16(%esp)
+	push %ecx		# 12(%esp)
+	push %edx		#  8(%esp)
+	push %ds		#  4(%esp)
+	push %es		#  0(%esp)
+	movl $(__KERNEL_DS),%eax
+	movl %eax,%ds
+	movl %eax,%es
 
-early_protection_fault:
-	movl $13,%edx
-	jmp early_fault
+	cmpl $(__KERNEL_CS),32(%esp)
+	jne 10f
 
-early_page_fault:
-	movl $14,%edx
-	jmp early_fault
+	movl 28(%esp),%eax	# %eip
+	movl $__start___ex_table,%ecx
+	movl $__stop___ex_table,%edx
+11:
+	cmpl (%ecx),%eax
+	je ex_entry		/* found an exception entry */
+	addl $8,%ecx
+	cmpl %edx,%ecx
+	jb 11b
 
-early_fault:
-	cld
+10:
 #ifdef CONFIG_PRINTK
-	pusha
-	movl $(__KERNEL_DS),%eax
-	movl %eax,%ds
-	movl %eax,%es
-	cmpl $2,early_recursion_flag
-	je hlt_loop
-	incl early_recursion_flag
+	xorl %eax,%eax
+	movw %ax,2(%esp)	/* clean up the segment values on some cpus */
+	movw %ax,6(%esp)
+	movw %ax,34(%esp)
+	leal  40(%esp),%eax
+	pushl %eax		/* %esp before the exception */
+	pushl %ebx
+	pushl %ebp
+	pushl %esi
+	pushl %edi
 	movl %cr2,%eax
 	pushl %eax
-	pushl %edx		/* trapno */
+	pushl (20+6*4)(%esp)	/* trapno */
 	pushl $fault_msg
 	call printk
 #endif
@@ -578,6 +605,18 @@ hlt_loop:
 	hlt
 	jmp hlt_loop
 
+ex_entry:
+	movl 4(%ecx),%eax
+	movl %eax,28(%esp)	/* change %eip on the stack */
+	pop %es
+	pop %ds
+	pop %edx
+	pop %ecx
+	pop %eax
+	addl $8,%esp		/* drop vector number and error code */
+	decl %ss:early_recursion_flag
+	iret
+
 /* This is the default interrupt "handler" :-) */
 	ALIGN
 ignore_int:
@@ -681,11 +720,12 @@ int_msg:
 fault_msg:
 /* fault info: */
 	.ascii "BUG: Int %d: CR2 %p\n"
-/* pusha regs: */
-	.ascii "     EDI %p  ESI %p  EBP %p  ESP %p\n"
-	.ascii "     EBX %p  EDX %p  ECX %p  EAX %p\n"
+/* regs pushed in early_idt_handler: */
+	.ascii "     EDI %p  ESI %p  EBP %p  EBX %p\n"
+	.ascii "     ESP %p   ES %p   DS %p\n"
+	.ascii "     EDX %p  ECX %p  EAX %p\n"
 /* fault frame: */
-	.ascii "     err %p  EIP %p   CS %p  flg %p\n"
+	.ascii "     vec %p  err %p  EIP %p   CS %p  flg %p\n"
 	.ascii "Stack: %p %p %p %p %p %p %p %p\n"
 	.ascii "       %p %p %p %p %p %p %p %p\n"
 	.asciz "       %p %p %p %p %p %p %p %p\n"
-- 
1.7.6.5

Re: [PATCH 0/5] RFC: x86: Early exception table support

[Borislav Petkov]

Haha,

I was looking into that too and you beat me to it. Now isn't that great!
:-)

On Wed, Apr 18, 2012 at 05:16:45PM -0700, H. Peter Anvin wrote:
> If we get an exception during early boot, walk the exception table to
> see if we should intercept it.  The main use case for this is to allow
> rdmsr_safe()/wrmsr_safe() during CPU initialization.
> 
> Since the exception table is currently sorted at runtime, and fairly
> late in startup, this code walks the exception table linearly.  We
> obviously don't need to worry about modules, however: none have been
> loaded at this point.
> 
> In the future it would be better to have the table sorted at compile
> time, or even better, turned into a perfect hash.  At that point this
> code should be changed out from doing a linear search.

FWIW, I was thinking that maybe we could sort the main extable
(i.e., no modules) before we assign the early idt handlers in
x86_64_start_kernel() and this way use the binary search through it in
search_extable().

I'm sending the patches I had as a reply to this mail so that you can
get a better idea of what I mean. They're rough and the asm is f*cked
up so don't look at that - I was still figuring out the exception frame
layout to do it properly.

> This patchset also makes the early exception handling a little more
> similar between x86-64 and i386, but a lot of unification could (and
> should) still be done: in particular, setup done in C as in x86-64,
> and verbose error dump as in i386.

Yeah, I'll look through those and give them a test drive on my boxes
here.

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

[PATCH 1/3] x86, extable: Cleanup fixup_exception

[Borislav Petkov]

>From 74643bebf6b592f179974caa033f2f095d28bb74 Mon Sep 17 00:00:00 2001
From: Borislav Petkov <bp@alien8.de>
Date: Sat, 31 Mar 2012 16:29:03 +0200
Subject: [PATCH 1/3] x86, extable: Cleanup fixup_exception

And especially, fork out the CONFIG_PNPBIOS lump into its own function
and salvage a tab by inverting error code checking.

No functionality change.

Signed-off-by: Borislav Petkov <bp@alien8.de>
---
 arch/x86/mm/extable.c |   33 ++++++++++++++++++---------------
 1 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1fb85dbe390a..08aecb0d3132 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -2,13 +2,10 @@
 #include <linux/spinlock.h>
 #include <asm/uaccess.h>
 
-
-int fixup_exception(struct pt_regs *regs)
+static void pnp_bios_is_crap(unsigned long cs)
 {
-	const struct exception_table_entry *fixup;
-
 #ifdef CONFIG_PNPBIOS
-	if (unlikely(SEGMENT_IS_PNP_CODE(regs->cs))) {
+	if (unlikely(SEGMENT_IS_PNP_CODE(cs))) {
 		extern u32 pnp_bios_fault_eip, pnp_bios_fault_esp;
 		extern u32 pnp_bios_is_utter_crap;
 		pnp_bios_is_utter_crap = 1;
@@ -20,18 +17,24 @@ int fixup_exception(struct pt_regs *regs)
 		panic("do_trap: can't hit this");
 	}
 #endif
+}
+
+int fixup_exception(struct pt_regs *regs)
+{
+	const struct exception_table_entry *fixup;
+
+	pnp_bios_is_crap(regs->cs);
 
 	fixup = search_exception_tables(regs->ip);
-	if (fixup) {
-		/* If fixup is less than 16, it means uaccess error */
-		if (fixup->fixup < 16) {
-			current_thread_info()->uaccess_err = 1;
-			regs->ip += fixup->fixup;
-			return 1;
-		}
-		regs->ip = fixup->fixup;
+	if (!fixup)
+		return 0;
+
+	/* If fixup is less than 16, it means uaccess error */
+	if (fixup->fixup < 16) {
+		current_thread_info()->uaccess_err = 1;
+		regs->ip += fixup->fixup;
 		return 1;
 	}
-
-	return 0;
+	regs->ip = fixup->fixup;
+	return 1;
 }
-- 
1.7.5.3.401.gfb674

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

[PATCH 2/3] x86, extable: Carve out the main extable searching routine

[Borislav Petkov]

From: Borislav Petkov <bp@alien8.de>
Date: Mon, 26 Mar 2012 23:11:11 +0200
Subject: [PATCH 2/3] x86, extable: Carve out the main extable searching
 routine

For early exception handling and fixup we don't need the modules extable
so fork out the routine that searches the kernel's main extable.

Signed-off-by: Borislav Petkov <bp@alien8.de>
---
 arch/x86/include/asm/uaccess.h |    1 +
 kernel/extable.c               |    7 ++++++-
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index 8be5f54d9360..66ef1c1d4087 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -96,6 +96,7 @@ struct exception_table_entry {
 };
 
 extern int fixup_exception(struct pt_regs *regs);
+extern const struct exception_table_entry *search_main_extable(unsigned long);
 
 /*
  * These are the main single-value transfer routines.  They automatically
diff --git a/kernel/extable.c b/kernel/extable.c
index 5339705b8241..d95188c2f723 100644
--- a/kernel/extable.c
+++ b/kernel/extable.c
@@ -41,12 +41,17 @@ void __init sort_main_extable(void)
 	sort_extable(__start___ex_table, __stop___ex_table);
 }
 
+const struct exception_table_entry *search_main_extable(unsigned long addr)
+{
+	return search_extable(__start___ex_table, __stop___ex_table-1, addr);
+}
+
 /* Given an address, look for it in the exception tables. */
 const struct exception_table_entry *search_exception_tables(unsigned long addr)
 {
 	const struct exception_table_entry *e;
 
-	e = search_extable(__start___ex_table, __stop___ex_table-1, addr);
+	e = search_main_extable(addr);
 	if (!e)
 		e = search_module_extables(addr);
 	return e;
-- 
1.7.5.3.401.gfb674


-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

[PATCH 3/3] x86, extable: Handle early exceptions

[Borislav Petkov]

From: Borislav Petkov <bp@alien8.de>
Date: Sun, 1 Apr 2012 11:33:29 +0200
Subject: [PATCH 3/3] x86, extable: Handle early exceptions

Now that all is in place, wire in exception handling into the early IDT
handler.

Also, move the sorting of the main exception table earlier in the boot
process now that we handle exceptions in the early IDT handler too.

Signed-off-by: Borislav Petkov <bp@alien8.de>
---
 arch/x86/kernel/head64.c  |    6 ++++++
 arch/x86/kernel/head_64.S |    5 +++++
 arch/x86/mm/extable.c     |   12 ++++++++++++
 init/main.c               |    1 -
 4 files changed, 23 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 3a3b779f41d3..20860aa44747 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -13,6 +13,7 @@
 #include <linux/start_kernel.h>
 #include <linux/io.h>
 #include <linux/memblock.h>
+#include <linux/module.h>	/* sort_main_extable() */
 
 #include <asm/processor.h>
 #include <asm/proto.h>
@@ -79,6 +80,11 @@ void __init x86_64_start_kernel(char * real_mode_data)
 
 	max_pfn_mapped = KERNEL_IMAGE_SIZE >> PAGE_SHIFT;
 
+	/* We're going to enable the initial exception handlers and we need the
+	 * exception table sorted prior to that.
+	 */
+	sort_main_extable();
+
 	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++) {
 #ifdef CONFIG_EARLY_PRINTK
 		set_intr_gate(i, &early_idt_handlers[i]);
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 40f4eb3766d1..7e6661d97e9d 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -282,6 +282,11 @@ early_idt_handlers:
 #endif
 
 ENTRY(early_idt_handler)
+	call early_fixup_exception
+	cmpq $0, %rax
+	jz 2f
+	retq
+2:
 #ifdef CONFIG_EARLY_PRINTK
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 08aecb0d3132..f8ed1ce07f4c 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -19,6 +19,18 @@ static void pnp_bios_is_crap(unsigned long cs)
 #endif
 }
 
+asmlinkage int early_fixup_exception(struct pt_regs *regs)
+{
+	const struct exception_table_entry *fixup;
+
+	fixup = search_main_extable(regs->ip);
+	if (!fixup)
+		return 0;
+
+	regs->ip = fixup->fixup;
+	return 1;
+}
+
 int fixup_exception(struct pt_regs *regs)
 {
 	const struct exception_table_entry *fixup;
diff --git a/init/main.c b/init/main.c
index ff49a6dacfbb..8ef10a2e2892 100644
--- a/init/main.c
+++ b/init/main.c
@@ -517,7 +517,6 @@ asmlinkage void __init start_kernel(void)
 	setup_log_buf(0);
 	pidhash_init();
 	vfs_caches_init_early();
-	sort_main_extable();
 	trap_init();
 	mm_init();
 
-- 
1.7.5.3.401.gfb674

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH 1/5] x86, nop: Make the ASM_NOP* macros work from assembly

[Borislav Petkov]

On Wed, Apr 18, 2012 at 05:16:46PM -0700, H. Peter Anvin wrote:
> From: "H. Peter Anvin" <hpa@zytor.com>
> 
> Make the ASM_NOP* macros work in actual assembly files.
> 
> Signed-off-by: H. Peter Anvin <hpa@zytor.com>

Acked-by: Borislav Petkov <borislav.petkov@amd.com>

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH 2/5] x86: Add symbolic constant for exceptions with error code

[Borislav Petkov]

On Wed, Apr 18, 2012 at 05:16:47PM -0700, H. Peter Anvin wrote:
> From: "H. Peter Anvin" <hpa@zytor.com>
> 
> Add a symbolic constant for the bitmask which states which exceptions
> carry an error code.
> 
> Signed-off-by: H. Peter Anvin <hpa@zytor.com>

Acked-by: Borislav Petkov <borislav.petkov@amd.com>

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH 4/5] x86-64: Handle exception table entries during early boot

[Borislav Petkov]

On Wed, Apr 18, 2012 at 05:16:49PM -0700, H. Peter Anvin wrote:
> From: "H. Peter Anvin" <hpa@zytor.com>
> 
> If we get an exception during early boot, walk the exception table to
> see if we should intercept it.  The main use case for this is to allow
> rdmsr_safe()/wrmsr_safe() during CPU initialization.
> 
> Since the exception table is currently sorted at runtime, and fairly
> late in startup, this code walks the exception table linearly.  We
> obviously don't need to worry about modules, however: none have been
> loaded at this point.
> 
> In the future it would be better to have the table sorted at compile
> time, or even better, turned into a perfect hash.  At that point this
> code should be changed out from doing a linear search.
> ---
>  arch/x86/include/asm/segment.h |    2 +-
>  arch/x86/kernel/head_64.S      |   72 +++++++++++++++++++++++++++++----------
>  2 files changed, 54 insertions(+), 20 deletions(-)
> 
> diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
> index 58c1e6c..c48a950 100644
> --- a/arch/x86/include/asm/segment.h
> +++ b/arch/x86/include/asm/segment.h
> @@ -213,7 +213,7 @@
>  
>  #ifdef __KERNEL__
>  #ifndef __ASSEMBLY__
> -extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
> +extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][2+2+5];
>  
>  /*
>   * Load a segment. Fall back on loading the zero
> diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
> index d1e112c..8b89239 100644
> --- a/arch/x86/kernel/head_64.S
> +++ b/arch/x86/kernel/head_64.S
> @@ -19,6 +19,7 @@
>  #include <asm/cache.h>
>  #include <asm/processor-flags.h>
>  #include <asm/percpu.h>
> +#include <asm/nops.h>
>  
>  #ifdef CONFIG_PARAVIRT
>  #include <asm/asm-offsets.h>
> @@ -26,6 +27,7 @@
>  #define GET_CR2_INTO(reg) GET_CR2_INTO_RAX ; movq %rax, reg
>  #else
>  #define GET_CR2_INTO(reg) movq %cr2, reg
> +#define INTERRUPT_RETURN iretq
>  #endif
>  
>  /* we are not able to switch in one step to the final KERNEL ADDRESS SPACE
> @@ -271,35 +273,56 @@ bad_address:
>  	jmp bad_address
>  
>  	.section ".init.text","ax"
> -#ifdef CONFIG_EARLY_PRINTK
>  	.globl early_idt_handlers
>  early_idt_handlers:
> +	# 56(%rsp) %rflags
> +	# 48(%rsp) %cs
> +	# 40(%rsp) %rip
> +	# 32(%rsp) error code

Just a question about understanding this: I was strugging with the
exact interrupt-handler stack layout, so how do you know the exact
offsets of rflags, cs et all? I've been staring at our APM v2, p. 144
(http://support.amd.com/us/Processor_TechDocs/24593_APM_v2.pdf) which
shows the interrupt stack with error code but it has different offsets
than those above:

[Return SS]	40(%rsp)
[Return RSP]	32(%rsp)
[rFLAGS]	24(%rsp)
[CS]		16(%rsp)
[RIP]		 8(%rsp)
[ErrorCode]	 0(%rsp)

but I see below that we have room for %rax, %rcx and %rdx on the stack.

Are we doing something different in Linux, or what am I missing?

Thanks.

>  	i = 0
>  	.rept NUM_EXCEPTION_VECTORS
> -	movl $i, %esi
> +	.if (EXCEPTION_ERRCODE_MASK >> i) & 1
> +	ASM_NOP2
> +	.else
> +	pushq $0		# Dummy error code, to make stack frame uniform
> +	.endif
> +	pushq $i		# 24(%rsp) Vector number
>  	jmp early_idt_handler
>  	i = i + 1
>  	.endr
> -#endif
>  
>  ENTRY(early_idt_handler)
> -#ifdef CONFIG_EARLY_PRINTK
> +	cld
> +
>  	cmpl $2,early_recursion_flag(%rip)
>  	jz  1f
>  	incl early_recursion_flag(%rip)
> -	GET_CR2_INTO(%r9)
> -	xorl %r8d,%r8d		# zero for error code
> -	movl %esi,%ecx		# get vector number
> -	# Test %ecx against mask of vectors that push error code.
> -	cmpl $31,%ecx
> -	ja 0f
> -	movl $1,%eax
> -	salq %cl,%rax
> -	testl $EXCEPTION_ERRCODE_MASK,%eax
> -	je 0f
> -	popq %r8		# get error code
> -0:	movq 0(%rsp),%rcx	# get ip
> -	movq 8(%rsp),%rdx	# get cs
> +
> +	pushq %rax		# 16(%rsp)
> +	pushq %rcx		#  8(%rsp)
> +	pushq %rdx		#  0(%rsp)
> +
> +	cmpl $__KERNEL_CS,48(%rsp)
> +	jne 10f
> +
> +	movq 40(%rsp),%rax	# %rip
> +	leaq __start___ex_table(%rip),%rcx
> +	leaq __stop___ex_table(%rip),%rdx
> +
> +11:
> +	cmpq (%rcx),%rax
> +	je 20f			# Found an exception entry
> +	addq $16,%rcx
> +	cmpq %rdx,%rcx
> +	jb 11b
> +
> +10:
> +#ifdef CONFIG_EARLY_PRINTK
> +	GET_CR2_INTO(%r9)	# can clobber any volatile register if pv
> +	movl 32(%rsp),%r8d	# error code
> +	movl 24(%rsp),%esi	# vector number
> +	movl 48(%rsp),%edx	# %cs
> +	movq 40(%rsp),%rcx	# %rip
>  	xorl %eax,%eax
>  	leaq early_idt_msg(%rip),%rdi
>  	call early_printk
> @@ -308,17 +331,28 @@ ENTRY(early_idt_handler)
>  	call dump_stack
>  #ifdef CONFIG_KALLSYMS	
>  	leaq early_idt_ripmsg(%rip),%rdi
> -	movq 0(%rsp),%rsi	# get rip again
> +	movq 40(%rsp),%rsi	# %rip again
>  	call __print_symbol
>  #endif
>  #endif /* EARLY_PRINTK */
>  1:	hlt
>  	jmp 1b
>  
> -#ifdef CONFIG_EARLY_PRINTK
> +20:	# Exception table entry found
> +	movq 8(%rcx),%rax	# exception table target
> +	movq %rax,40(%rsp)	# change %rip on stack
> +	popq %rdx
> +	popq %rcx
> +	popq %rax
> +	addq $16,%rsp		# drop vector number and error code
> +	decl early_recursion_flag(%rip)
> +	INTERRUPT_RETURN
> +
> +	.balign 4
>  early_recursion_flag:
>  	.long 0
>  
> +#ifdef CONFIG_EARLY_PRINTK
>  early_idt_msg:
>  	.asciz "PANIC: early exception %02lx rip %lx:%lx error %lx cr2 %lx\n"
>  early_idt_ripmsg:
> -- 
> 1.7.6.5
> 
> 

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH 4/5] x86-64: Handle exception table entries during early boot

[H. Peter Anvin]

On 04/19/2012 06:02 AM, Borislav Petkov wrote:>> +	# 56(%rsp) %rflags
>> +	# 48(%rsp) %cs
>> +	# 40(%rsp) %rip
>> +	# 32(%rsp) error code
> 
> Just a question about understanding this: I was strugging with the
> exact interrupt-handler stack layout, so how do you know the exact
> offsets of rflags, cs et all? I've been staring at our APM v2, p. 144
> (http://support.amd.com/us/Processor_TechDocs/24593_APM_v2.pdf) which
> shows the interrupt stack with error code but it has different offsets
> than those above:
> 
> [Return SS]	40(%rsp)
> [Return RSP]	32(%rsp)
> [rFLAGS]	24(%rsp)
> [CS]		16(%rsp)
> [RIP]		 8(%rsp)
> [ErrorCode]	 0(%rsp)
> 
> but I see below that we have room for %rax, %rcx and %rdx on the stack.
> 
> Are we doing something different in Linux, or what am I missing?

The offsets in my code refer to offsets after we push the vector number,
%rax, %rcx and %rdx onto the stack manually; that is 4*8 = 32 bytes,
hence the offset.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 02:26 AM, Borislav Petkov wrote:
> 
> Also, move the sorting of the main exception table earlier in the boot
> process now that we handle exceptions in the early IDT handler too.
> 

I would much rather use David Daney's patchset to sort the exception
table at compile time:

	https://lkml.org/lkml/2011/11/18/427

... and I would be *even happier* with an O(1) hash (which pretty much
*have* to be constructed at compile time.)

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [PATCH 4/5] x86-64: Handle exception table entries during early boot

[Borislav Petkov]

On Thu, Apr 19, 2012 at 09:59:07AM -0700, H. Peter Anvin wrote:
> The offsets in my code refer to offsets after we push the vector number,
> %rax, %rcx and %rdx onto the stack manually; that is 4*8 = 32 bytes,
> hence the offset.

Ah, sure, you push those upon entry into early_idt_handler, ok.

Btw, just tested the patchset here with the following hunk below and it
looks good.

Also, you can add a revert of ce37defc0f667 as the last one in your
patchset since the comment becomes invalid after it goes in.

diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index 0a44b90..a3a6299 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -448,6 +448,8 @@ static void __cpuinit bsp_init_amd(struct cpuinfo_x86 *c)
 
 static void __cpuinit early_init_amd(struct cpuinfo_x86 *c)
 {
+       u32 dummy1, dummy2;
+
        early_init_amd_mc(c);
 
        /*
@@ -479,6 +481,7 @@ static void __cpuinit early_init_amd(struct cpuinfo_x86 *c)
                        set_cpu_cap(c, X86_FEATURE_EXTD_APICID);
        }
 #endif
+       rdmsr_safe(0xdeadbeef, &dummy1, &dummy2);
 }
 
 static void __cpuinit init_amd(struct cpuinfo_x86 *c)

Thanks.

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Linus Torvalds]

On Thu, Apr 19, 2012 at 10:02 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>
> ... and I would be *even happier* with an O(1) hash (which pretty much
> *have* to be constructed at compile time.)

Taking relocations into account might be interesting for hashing. I
guess you could hash the relative offsets, though.

But yeah, a hash might be the way to go, and once you generate the
tables at compile-time, why not go all the way? It doesn't need to be
some complex perfect hash, it should be fairly straightforward to just
size the hash right and use some simple linear probing model for
collissions or whatever.

                  Linus

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Borislav Petkov]

On Thu, Apr 19, 2012 at 10:27:09AM -0700, Linus Torvalds wrote:
> On Thu, Apr 19, 2012 at 10:02 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> >
> > ... and I would be *even happier* with an O(1) hash (which pretty much
> > *have* to be constructed at compile time.)
> 
> Taking relocations into account might be interesting for hashing. I
> guess you could hash the relative offsets, though.
> 
> But yeah, a hash might be the way to go, and once you generate the
> tables at compile-time, why not go all the way? It doesn't need to be
> some complex perfect hash, it should be fairly straightforward to just
> size the hash right and use some simple linear probing model for
> collissions or whatever.

Yeah, simplicity is the key here. I don't think we're getting that many
early boot exceptions due to rdmsr or whatever to warrant adding a bunch
of code.

OTOH, if we can share early and normal exception handling lookup code, then a
perfect hash would make sense as those exceptions would pile up.

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 10:02 AM, H. Peter Anvin wrote:
> On 04/19/2012 02:26 AM, Borislav Petkov wrote:
>>
>> Also, move the sorting of the main exception table earlier in the boot
>> process now that we handle exceptions in the early IDT handler too.
>>
> 
> I would much rather use David Daney's patchset to sort the exception
> table at compile time:
> 
> 	https://lkml.org/lkml/2011/11/18/427
> 
> ... and I would be *even happier* with an O(1) hash (which pretty much
> *have* to be constructed at compile time.)
> 

The sorting is easier infrastructure-wise, since it can be done
in-place.  The O(1) hash needs additional space for the hash table
itself (the actual table can still be stashed where the compiler
generates it.)

The hash table generator I have posted at:

http://www.zytor.com/~hpa/phash/

Generally needs ~3.2 bytes per hash table entry, rounded up to a power
of two.  The rounding up is for performance.  The easiest way to do this
is probably to let the linker create a zero-filled section of the proper
size (since the linker knows the final size of the __ex_table section,
and the linker script can do at least a modicum of arithmetic) and then
use a tool to patch in the hash table auxilliary data.

	-hpa


P.S. Another modification which was talked about in the past and there
even were patches for was to make the exception table entries relative
to the start of the kernel, so we don't need two relocations per entry
for x86-32 and twice the amount of data that we can actually use for
x86-64.  As I recall we tried those patches and there was some bug that
never got resolved.

This is obviously a precondition for doing the O(1) hash, since the O(1)
hash can't be relocated once generated.

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 10:38 AM, Borislav Petkov wrote:
> 
> Yeah, simplicity is the key here. I don't think we're getting that many
> early boot exceptions due to rdmsr or whatever to warrant adding a bunch
> of code.
> 
> OTOH, if we can share early and normal exception handling lookup code, then a
> perfect hash would make sense as those exceptions would pile up.
> 

Obviously we should use the hash after the main kernel is up, too.
Anything else would be silly.

I would argue that the O(1) hash makes things simpler as there is no
need to deal with collisions at all.  The one advantage with "plain"
hashes is that can be runtime modified, which would be of some interest
if we create one single hash table that includes modules.

*That* seems complex to me, though.

	-hpa

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Linus Torvalds]

On Thu, Apr 19, 2012 at 10:38 AM, Borislav Petkov <bp@amd64.org> wrote:
>
> OTOH, if we can share early and normal exception handling lookup code, then a
> perfect hash would make sense as those exceptions would pile up.

Oh, any of this only makes sense if we can share it with the runtime
exception lookup.

They are *seldom* hugely performance-critical, but there are some
unusual loads where you do get a fair number of exceptions.

               Linus

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Linus Torvalds]

On Thu, Apr 19, 2012 at 10:59 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>
> I would argue that the O(1) hash makes things simpler as there is no
> need to deal with collisions at all.

Most of the O(1) hashes I have seen more than made up for the trivial
complexity of a few linear lookups by making the hash function way
more complicated.

A linear probe with a step of one really is pretty simple. Sure, you
might want to make the initial hash "good enough" to not often hit the
probing code, but doing a few linear probes is cheap.

In contrast, the perfect linear hashes do crazy things like having
table lookups *JUST TO COMPUTE THE HASH*.

Which is f*cking stupid, really. They'll miss in the cache just at
hash compute time, never mind at hash lookup. The table-driven
versions look beautiful in microbenchmarks that have the tables in the
L1 cache, but for something like the exception handling, I can
guarantee that *nothing* is in L1, and probably not even L2.

So what you want is:
 - no table lookups for hashing
 - simple code (ie a normal "a multiply and a shift/mask or two") to
keep the I$ footprint down too
 - you *will* take a cache miss on the actual hash table lookup, that
cannot be avoided, but linear probing at least hopefully keeps it to
that single cache miss even if you have to do a probe or two.

Remember: this is very much a "cold-cache behavior matters" case. We
would never ever call this in a loop, at most we have loads that get a
fair amount of exceptions (but will go through the exception code, so
the L1 is probably blown even then).

                         Linus

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

Either way I suggest picking up David's presorting patchset since it is already done and use its infrastructure for any further improvements.

As far as a linear probe you get an average of n lookups with a packing density of 1-1/n so you are right; a linear probe with a density of say 1/2 is probably best.

Linus Torvalds <torvalds@linux-foundation.org> wrote:

>On Thu, Apr 19, 2012 at 10:59 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>
>> I would argue that the O(1) hash makes things simpler as there is no
>> need to deal with collisions at all.
>
>Most of the O(1) hashes I have seen more than made up for the trivial
>complexity of a few linear lookups by making the hash function way
>more complicated.
>
>A linear probe with a step of one really is pretty simple. Sure, you
>might want to make the initial hash "good enough" to not often hit the
>probing code, but doing a few linear probes is cheap.
>
>In contrast, the perfect linear hashes do crazy things like having
>table lookups *JUST TO COMPUTE THE HASH*.
>
>Which is f*cking stupid, really. They'll miss in the cache just at
>hash compute time, never mind at hash lookup. The table-driven
>versions look beautiful in microbenchmarks that have the tables in the
>L1 cache, but for something like the exception handling, I can
>guarantee that *nothing* is in L1, and probably not even L2.
>
>So what you want is:
> - no table lookups for hashing
> - simple code (ie a normal "a multiply and a shift/mask or two") to
>keep the I$ footprint down too
> - you *will* take a cache miss on the actual hash table lookup, that
>cannot be avoided, but linear probing at least hopefully keeps it to
>that single cache miss even if you have to do a probe or two.
>
>Remember: this is very much a "cold-cache behavior matters" case. We
>would never ever call this in a loop, at most we have loads that get a
>fair amount of exceptions (but will go through the exception code, so
>the L1 is probably blown even then).
>
>                         Linus

-- 
Sent from my mobile phone. Please excuse brevity and lack of formatting.

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[David Daney]

On 04/19/2012 11:55 AM, H. Peter Anvin wrote:
> Either way I suggest picking up David's presorting patchset since it is already done and use its infrastructure for any further improvements.
>

It does have the advantage of already being implemented.  There was a 
little feedback on the kbuild portions of the patch.

If you would like, I will send an updated version of the patch.

> As far as a linear probe you get an average of n lookups with a packing density of 1-1/n so you are right; a linear probe with a density of say 1/2 is probably best.
>

I usually see exception table sizes on the order of 2^10 entries, so I 
have to wonder how much you really gain from an O(1) implementation.

David Daney

> Linus Torvalds<torvalds@linux-foundation.org>  wrote:
>
>> On Thu, Apr 19, 2012 at 10:59 AM, H. Peter Anvin<hpa@zytor.com>  wrote:
>>>
>>> I would argue that the O(1) hash makes things simpler as there is no
>>> need to deal with collisions at all.
>>
>> Most of the O(1) hashes I have seen more than made up for the trivial
>> complexity of a few linear lookups by making the hash function way
>> more complicated.
>>
>> A linear probe with a step of one really is pretty simple. Sure, you
>> might want to make the initial hash "good enough" to not often hit the
>> probing code, but doing a few linear probes is cheap.
>>
>> In contrast, the perfect linear hashes do crazy things like having
>> table lookups *JUST TO COMPUTE THE HASH*.
>>
>> Which is f*cking stupid, really. They'll miss in the cache just at
>> hash compute time, never mind at hash lookup. The table-driven
>> versions look beautiful in microbenchmarks that have the tables in the
>> L1 cache, but for something like the exception handling, I can
>> guarantee that *nothing* is in L1, and probably not even L2.
>>
>> So what you want is:
>> - no table lookups for hashing
>> - simple code (ie a normal "a multiply and a shift/mask or two") to
>> keep the I$ footprint down too
>> - you *will* take a cache miss on the actual hash table lookup, that
>> cannot be avoided, but linear probing at least hopefully keeps it to
>> that single cache miss even if you have to do a probe or two.
>>
>> Remember: this is very much a "cold-cache behavior matters" case. We
>> would never ever call this in a loop, at most we have loads that get a
>> fair amount of exceptions (but will go through the exception code, so
>> the L1 is probably blown even then).
>>
>>                          Linus
>

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 01:17 PM, David Daney wrote:
> On 04/19/2012 11:55 AM, H. Peter Anvin wrote:
>> Either way I suggest picking up David's presorting patchset since it
>> is already done and use its infrastructure for any further improvements.
> 
> It does have the advantage of already being implemented.  There was a
> little feedback on the kbuild portions of the patch.
> 
> If you would like, I will send an updated version of the patch.

Please.  It gets us 90% of the way, and we need the infrastructure
anyway to do any further work.

>> As far as a linear probe you get an average of n lookups with a
>> packing density of 1-1/n so you are right; a linear probe with a
>> density of say 1/2 is probably best.
>>
> 
> I usually see exception table sizes on the order of 2^10 entries, so I
> have to wonder how much you really gain from an O(1) implementation.

Well, for either variant of hash table you end up with ~2 serial memory
references as opposed to ~11.  No idea if there are workloads where this
actually matters.

	-hpa

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 01:20 PM, H. Peter Anvin wrote:
> On 04/19/2012 01:17 PM, David Daney wrote:
>> On 04/19/2012 11:55 AM, H. Peter Anvin wrote:
>>> Either way I suggest picking up David's presorting patchset since it
>>> is already done and use its infrastructure for any further improvements.
>>
>> It does have the advantage of already being implemented.  There was a
>> little feedback on the kbuild portions of the patch.
>>
>> If you would like, I will send an updated version of the patch.
> 
> Please.  It gets us 90% of the way, and we need the infrastructure
> anyway to do any further work.
> 

Linus, it's not really an x86 patchset, but would you object if I queue
up David's updated patchset in -tip?  I'll try to get an Ack from Ralf
for the MIPS portion.

	-hpa

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[David Daney]

On 04/19/2012 01:26 PM, H. Peter Anvin wrote:
> On 04/19/2012 01:20 PM, H. Peter Anvin wrote:
>> On 04/19/2012 01:17 PM, David Daney wrote:
>>> On 04/19/2012 11:55 AM, H. Peter Anvin wrote:
>>>> Either way I suggest picking up David's presorting patchset since it
>>>> is already done and use its infrastructure for any further improvements.
>>>
>>> It does have the advantage of already being implemented.  There was a
>>> little feedback on the kbuild portions of the patch.
>>>
>>> If you would like, I will send an updated version of the patch.
>>
>> Please.  It gets us 90% of the way, and we need the infrastructure
>> anyway to do any further work.
>>
>
> Linus, it's not really an x86 patchset, but would you object if I queue
> up David's updated patchset in -tip?  I'll try to get an Ack from Ralf
> for the MIPS portion.
>

I am looking at the patch right now to see if I want to revise it in any 
way.  Very Soon Now I will either send a new version, or indicate that I 
think the existing version is good enough.

David Daney

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Linus Torvalds]

On Thu, Apr 19, 2012 at 1:26 PM, H. Peter Anvin <hpa@zytor.com> wrote:
>
> Linus, it's not really an x86 patchset, but would you object if I queue
> up David's updated patchset in -tip?  I'll try to get an Ack from Ralf
> for the MIPS portion.

No objections.

             Linus

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 01:17 PM, David Daney wrote:
> 
> I usually see exception table sizes on the order of 2^10 entries, so I
> have to wonder how much you really gain from an O(1) implementation.
> 

One thing that probably would give more of a boost is to use a rbtree or
similar structure to figure out *which* extable (if any) we should be
looking at; right now it looks like we linearly walk the modules, and
don't even look to see if we are inside that module before we do a
bsearch in that module's extable...

	-hpa

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Tony Luck]

On Thu, Apr 19, 2012 at 3:16 PM, H. Peter Anvin <hpa@zytor.com> wrote:
> One thing that probably would give more of a boost is to use a rbtree or
> similar structure to figure out *which* extable (if any) we should be
> looking at; right now it looks like we linearly walk the modules, and
> don't even look to see if we are inside that module before we do a
> bsearch in that module's extable...

How many entries are in the extable for a typical module?  Perhaps it might
make sense to bundle them all into one sorted combined table? Of course
you would have to have a way to squeeze them back out of the combined
table at module unload time.

This moves the cost to module load/unload time ... which is hopefully rare
compared to table lookup.

-Tony

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Linus Torvalds]

On Thu, Apr 19, 2012 at 3:47 PM, Tony Luck <tony.luck@gmail.com> wrote:
>
> How many entries are in the extable for a typical module?  Perhaps it might
> make sense to bundle them all into one sorted combined table? Of course
> you would have to have a way to squeeze them back out of the combined
> table at module unload time.
>
> This moves the cost to module load/unload time ... which is hopefully rare
> compared to table lookup.

Using a traditional chained hash-table might be very amenable to this
kind of situation. It's much easier to populate and doesn't have the
size issues. And realistically, we can probably size the hash table
for just the built-in kernel, because modules seldom have nearly as
many exception table entries, so adding them later to a fixed-size
table likely won't be too painful.

In fact, doing an "objdump" on the few modules I have, I didn't find a
*single* exception table entry. Maybe I did something wrong? Isn't it
the __ex_table in modules too?

(Admittedly, I avoid modules like the plague, so I don't tend to have
very many modules, and the ones I do have tend to be pretty limited.
So my module usage is absolutely not representative even if I think it
should be ;^).

                   Linus

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 03:58 PM, Linus Torvalds wrote:
> 
> In fact, doing an "objdump" on the few modules I have, I didn't find a
> *single* exception table entry. Maybe I did something wrong? Isn't it
> the __ex_table in modules too?
> 

It is, but I suspect a lot of modules don't end up with any exception
table entries, as the common forms of user space access are out of line.

I suspect that just finding the module we're in or not in before
searching its ex_table is probably good enough, but a global hash would
be better.  That being said, some distros are complaining about the
amount of time it takes to load modules on boot...

	-hpa

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Tony Luck]

On Thu, Apr 19, 2012 at 4:10 PM, H. Peter Anvin <hpa@zytor.com> wrote:
> be better.  That being said, some distros are complaining about the
> amount of time it takes to load modules on boot...

That could be fixed in the installer for the distribution by rebuilding a kernel
with the target set of modules for the system built in.

-Tony

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[H. Peter Anvin]

On 04/19/2012 04:26 PM, Tony Luck wrote:
> On Thu, Apr 19, 2012 at 4:10 PM, H. Peter Anvin <hpa@zytor.com> wrote:
>> be better.  That being said, some distros are complaining about the
>> amount of time it takes to load modules on boot...
> 
> That could be fixed in the installer for the distribution by rebuilding a kernel
> with the target set of modules for the system built in.
> 

... assuming you have a well-defined "the system".  It isn't always you
will boot on the same hardware every time.

	-hpa

[tip:x86/extable] x86, nop: Make the ASM_NOP* macros work from assembly

[tip-bot for H. Peter Anvin]

Commit-ID:  46326013e34eb5c178a91f06c1f2e99e79eed924
Gitweb:     http://git.kernel.org/tip/46326013e34eb5c178a91f06c1f2e99e79eed924
Author:     H. Peter Anvin <hpa@zytor.com>
AuthorDate: Wed, 18 Apr 2012 17:16:46 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 19 Apr 2012 15:07:42 -0700

x86, nop: Make the ASM_NOP* macros work from assembly

Make the ASM_NOP* macros work in actual assembly files.

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Link: http://lkml.kernel.org/r/1334794610-5546-2-git-send-email-hpa@zytor.com
---
 arch/x86/include/asm/nops.h |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/nops.h b/arch/x86/include/asm/nops.h
index 405b403..aff2b33 100644
--- a/arch/x86/include/asm/nops.h
+++ b/arch/x86/include/asm/nops.h
@@ -87,7 +87,11 @@
 #define P6_NOP8	0x0f,0x1f,0x84,0x00,0,0,0,0
 #define P6_NOP5_ATOMIC P6_NOP5
 
+#ifdef __ASSEMBLY__
+#define _ASM_MK_NOP(x) .byte x
+#else
 #define _ASM_MK_NOP(x) ".byte " __stringify(x) "\n"
+#endif
 
 #if defined(CONFIG_MK7)
 #define ASM_NOP1 _ASM_MK_NOP(K7_NOP1)

[tip:x86/extable] x86: Add symbolic constant for exceptions with error code

[tip-bot for H. Peter Anvin]

Commit-ID:  84f4fc524eed040660bd4ebc8cba259d8afe8461
Gitweb:     http://git.kernel.org/tip/84f4fc524eed040660bd4ebc8cba259d8afe8461
Author:     H. Peter Anvin <hpa@zytor.com>
AuthorDate: Wed, 18 Apr 2012 17:16:47 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 19 Apr 2012 15:07:49 -0700

x86: Add symbolic constant for exceptions with error code

Add a symbolic constant for the bitmask which states which exceptions
carry an error code.

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Link: http://lkml.kernel.org/r/1334794610-5546-3-git-send-email-hpa@zytor.com
---
 arch/x86/include/asm/segment.h |    2 ++
 arch/x86/kernel/head_64.S      |    2 +-
 2 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 1654662..58c1e6c 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -205,6 +205,8 @@
 
 #define IDT_ENTRIES 256
 #define NUM_EXCEPTION_VECTORS 32
+/* Bitmask of exception vectors which push an error code on the stack */
+#define EXCEPTION_ERRCODE_MASK  0x00027d00
 #define GDT_SIZE (GDT_ENTRIES * 8)
 #define GDT_ENTRY_TLS_ENTRIES 3
 #define TLS_SIZE (GDT_ENTRY_TLS_ENTRIES * 8)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 40f4eb3..adf52e8 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -295,7 +295,7 @@ ENTRY(early_idt_handler)
 	ja 0f
 	movl $1,%eax
 	salq %cl,%rax
-	testl $0x27d00,%eax
+	testl $EXCEPTION_ERRCODE_MASK,%eax
 	je 0f
 	popq %r8		# get error code
 0:	movq 0(%rsp),%rcx	# get ip

[tip:x86/extable] x86, paravirt: Replace GET_CR2_INTO_RCX with GET_CR2_INTO_RAX

[tip-bot for H. Peter Anvin]

Commit-ID:  ffc4bc9c6fa4eaf935d96d139bfa7443cac0b88e
Gitweb:     http://git.kernel.org/tip/ffc4bc9c6fa4eaf935d96d139bfa7443cac0b88e
Author:     H. Peter Anvin <hpa@zytor.com>
AuthorDate: Wed, 18 Apr 2012 17:16:48 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 19 Apr 2012 15:07:56 -0700

x86, paravirt: Replace GET_CR2_INTO_RCX with GET_CR2_INTO_RAX

GET_CR2_INTO_RCX is asinine: it is only used in one place, the actual
paravirt call returns the value in %rax, not %rcx; and the one place
that wants it wants the result in %r9.  We actually generate as a
result of this call:

       call ...
       movq %rax, %rcx
       xorq %rax, %rax		/* this value isn't even used... */
       movq %rcx, %r9

At least make the macro do what the paravirt call does, which is put
the value into %rax.

Nevermind the fact that the macro clobbers all the volatile registers.

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Link: http://lkml.kernel.org/r/1334794610-5546-4-git-send-email-hpa@zytor.com
Cc: Glauber de Oliveira Costa <glommer@parallels.com>
---
 arch/x86/include/asm/paravirt.h |    6 ++----
 arch/x86/kernel/head_64.S       |    6 +++---
 2 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index aa0f913..6cbbabf 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -1023,10 +1023,8 @@ extern void default_banner(void);
 		  call PARA_INDIRECT(pv_cpu_ops+PV_CPU_swapgs)		\
 		 )
 
-#define GET_CR2_INTO_RCX				\
-	call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2);	\
-	movq %rax, %rcx;				\
-	xorq %rax, %rax;
+#define GET_CR2_INTO_RAX				\
+	call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2)
 
 #define PARAVIRT_ADJUST_EXCEPTION_FRAME					\
 	PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_adjust_exception_frame), \
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index adf52e8..d1e112c 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -23,8 +23,9 @@
 #ifdef CONFIG_PARAVIRT
 #include <asm/asm-offsets.h>
 #include <asm/paravirt.h>
+#define GET_CR2_INTO(reg) GET_CR2_INTO_RAX ; movq %rax, reg
 #else
-#define GET_CR2_INTO_RCX movq %cr2, %rcx
+#define GET_CR2_INTO(reg) movq %cr2, reg
 #endif
 
 /* we are not able to switch in one step to the final KERNEL ADDRESS SPACE
@@ -286,8 +287,7 @@ ENTRY(early_idt_handler)
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
 	incl early_recursion_flag(%rip)
-	GET_CR2_INTO_RCX
-	movq %rcx,%r9
+	GET_CR2_INTO(%r9)
 	xorl %r8d,%r8d		# zero for error code
 	movl %esi,%ecx		# get vector number
 	# Test %ecx against mask of vectors that push error code.

[tip:x86/extable] x86, extable: Add early_fixup_exception()

[tip-bot for H. Peter Anvin]

Commit-ID:  6a1ea279c210e7dc05de86dc29c0d4f577f484fb
Gitweb:     http://git.kernel.org/tip/6a1ea279c210e7dc05de86dc29c0d4f577f484fb
Author:     H. Peter Anvin <hpa@linux.intel.com>
AuthorDate: Thu, 19 Apr 2012 15:24:20 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 19 Apr 2012 15:31:04 -0700

x86, extable: Add early_fixup_exception()

Add a restricted version of fixup_exception() to be used during early
boot only.  In particular, this doesn't support the try..catch variant
since we may not have a thread_info set up yet.

This relies on the exception table being sorted already at build time.

Link: http://lkml.kernel.org/r/1334794610-5546-1-git-send-email-hpa@zytor.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
---
 arch/x86/mm/extable.c |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1fb85db..5555675 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -35,3 +35,20 @@ int fixup_exception(struct pt_regs *regs)
 
 	return 0;
 }
+
+/* Restricted version used during very early boot */
+int __init early_fixup_exception(unsigned long *ip)
+{
+	const struct exception_table_entry *fixup;
+
+	fixup = search_exception_tables(*ip);
+	if (fixup) {
+		if (fixup->fixup < 16)
+			return 0; /* Not supported during early boot */
+
+		*ip = fixup->fixup;
+		return 1;
+	}
+
+	return 0;
+}

[tip:x86/extable] x86-64: Handle exception table entries during early boot

[tip-bot for H. Peter Anvin]

Commit-ID:  9900aa2f95844eb81428c1d3d202c01b7f3ac77a
Gitweb:     http://git.kernel.org/tip/9900aa2f95844eb81428c1d3d202c01b7f3ac77a
Author:     H. Peter Anvin <hpa@zytor.com>
AuthorDate: Wed, 18 Apr 2012 17:16:49 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 19 Apr 2012 15:42:45 -0700

x86-64: Handle exception table entries during early boot

If we get an exception during early boot, walk the exception table to
see if we should intercept it.  The main use case for this is to allow
rdmsr_safe()/wrmsr_safe() during CPU initialization.

Since the exception table is currently sorted at runtime, and fairly
late in startup, this code walks the exception table linearly.  We
obviously don't need to worry about modules, however: none have been
loaded at this point.

[ v2: Use early_fixup_exception() instead of linear search ]

Link: http://lkml.kernel.org/r/1334794610-5546-5-git-send-email-hpa@zytor.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
---
 arch/x86/include/asm/segment.h |    2 +-
 arch/x86/kernel/head_64.S      |   76 ++++++++++++++++++++++++++++++----------
 2 files changed, 58 insertions(+), 20 deletions(-)

diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 58c1e6c..c48a950 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -213,7 +213,7 @@
 
 #ifdef __KERNEL__
 #ifndef __ASSEMBLY__
-extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
+extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][2+2+5];
 
 /*
  * Load a segment. Fall back on loading the zero
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index d1e112c..7a40f24 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -19,6 +19,7 @@
 #include <asm/cache.h>
 #include <asm/processor-flags.h>
 #include <asm/percpu.h>
+#include <asm/nops.h>
 
 #ifdef CONFIG_PARAVIRT
 #include <asm/asm-offsets.h>
@@ -26,6 +27,7 @@
 #define GET_CR2_INTO(reg) GET_CR2_INTO_RAX ; movq %rax, reg
 #else
 #define GET_CR2_INTO(reg) movq %cr2, reg
+#define INTERRUPT_RETURN iretq
 #endif
 
 /* we are not able to switch in one step to the final KERNEL ADDRESS SPACE
@@ -271,35 +273,56 @@ bad_address:
 	jmp bad_address
 
 	.section ".init.text","ax"
-#ifdef CONFIG_EARLY_PRINTK
 	.globl early_idt_handlers
 early_idt_handlers:
+	# 104(%rsp) %rflags
+	#  96(%rsp) %cs
+	#  88(%rsp) %rip
+	#  80(%rsp) error code
 	i = 0
 	.rept NUM_EXCEPTION_VECTORS
-	movl $i, %esi
+	.if (EXCEPTION_ERRCODE_MASK >> i) & 1
+	ASM_NOP2
+	.else
+	pushq $0		# Dummy error code, to make stack frame uniform
+	.endif
+	pushq $i		# 72(%rsp) Vector number
 	jmp early_idt_handler
 	i = i + 1
 	.endr
-#endif
 
 ENTRY(early_idt_handler)
-#ifdef CONFIG_EARLY_PRINTK
+	cld
+
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
 	incl early_recursion_flag(%rip)
-	GET_CR2_INTO(%r9)
-	xorl %r8d,%r8d		# zero for error code
-	movl %esi,%ecx		# get vector number
-	# Test %ecx against mask of vectors that push error code.
-	cmpl $31,%ecx
-	ja 0f
-	movl $1,%eax
-	salq %cl,%rax
-	testl $EXCEPTION_ERRCODE_MASK,%eax
-	je 0f
-	popq %r8		# get error code
-0:	movq 0(%rsp),%rcx	# get ip
-	movq 8(%rsp),%rdx	# get cs
+
+	pushq %rax		# 64(%rsp)
+	pushq %rcx		# 56(%rsp)
+	pushq %rdx		# 48(%rsp)
+	pushq %rsi		# 40(%rsp)
+	pushq %rdi		# 32(%rsp)
+	pushq %r8		# 24(%rsp)
+	pushq %r9		# 16(%rsp)
+	pushq %r10		#  8(%rsp)
+	pushq %r11		#  0(%rsp)
+
+	cmpl $__KERNEL_CS,96(%rsp)
+	jne 10f
+
+	leaq 88(%rsp),%rdi	# Pointer to %rip
+	call early_fixup_exception
+	andl %eax,%eax
+	jnz 20f			# Found an exception entry
+
+10:
+#ifdef CONFIG_EARLY_PRINTK
+	GET_CR2_INTO(%r9)	# can clobber any volatile register if pv
+	movl 80(%rsp),%r8d	# error code
+	movl 72(%rsp),%esi	# vector number
+	movl 96(%rsp),%edx	# %cs
+	movq 88(%rsp),%rcx	# %rip
 	xorl %eax,%eax
 	leaq early_idt_msg(%rip),%rdi
 	call early_printk
@@ -308,17 +331,32 @@ ENTRY(early_idt_handler)
 	call dump_stack
 #ifdef CONFIG_KALLSYMS	
 	leaq early_idt_ripmsg(%rip),%rdi
-	movq 0(%rsp),%rsi	# get rip again
+	movq 40(%rsp),%rsi	# %rip again
 	call __print_symbol
 #endif
 #endif /* EARLY_PRINTK */
 1:	hlt
 	jmp 1b
 
-#ifdef CONFIG_EARLY_PRINTK
+20:	# Exception table entry found
+	popq %r11
+	popq %r10
+	popq %r9
+	popq %r8
+	popq %rdi
+	popq %rsi
+	popq %rdx
+	popq %rcx
+	popq %rax
+	addq $16,%rsp		# drop vector number and error code
+	decl early_recursion_flag(%rip)
+	INTERRUPT_RETURN
+
+	.balign 4
 early_recursion_flag:
 	.long 0
 
+#ifdef CONFIG_EARLY_PRINTK
 early_idt_msg:
 	.asciz "PANIC: early exception %02lx rip %lx:%lx error %lx cr2 %lx\n"
 early_idt_ripmsg:

[tip:x86/extable] x86-32: Handle exception table entries during early boot

[tip-bot for H. Peter Anvin]

Commit-ID:  4c5023a3fa2ec12b7ed313b276b157917575745b
Gitweb:     http://git.kernel.org/tip/4c5023a3fa2ec12b7ed313b276b157917575745b
Author:     H. Peter Anvin <hpa@zytor.com>
AuthorDate: Wed, 18 Apr 2012 17:16:50 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 19 Apr 2012 16:45:02 -0700

x86-32: Handle exception table entries during early boot

If we get an exception during early boot, walk the exception table to
see if we should intercept it.  The main use case for this is to allow
rdmsr_safe()/wrmsr_safe() during CPU initialization.

Since the exception table is currently sorted at runtime, and fairly
late in startup, this code walks the exception table linearly.  We
obviously don't need to worry about modules, however: none have been
loaded at this point.

This patch changes the early IDT setup to look a lot more like x86-64:
we now install handlers for all 32 exception vectors.  The output of
the early exception handler has changed somewhat as it directly
reflects the stack frame of the exception handler, and the stack frame
has been somewhat restructured.

Finally, centralize the code that can and should be run only once.

[ v2: Use early_fixup_exception() instead of linear search ]

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Link: http://lkml.kernel.org/r/1334794610-5546-6-git-send-email-hpa@zytor.com
---
 arch/x86/kernel/head_32.S |  223 ++++++++++++++++++++++++++-------------------
 1 files changed, 129 insertions(+), 94 deletions(-)

diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S
index ce0be7c..463c979 100644
--- a/arch/x86/kernel/head_32.S
+++ b/arch/x86/kernel/head_32.S
@@ -21,6 +21,7 @@
 #include <asm/msr-index.h>
 #include <asm/cpufeature.h>
 #include <asm/percpu.h>
+#include <asm/nops.h>
 
 /* Physical address */
 #define pa(X) ((X) - __PAGE_OFFSET)
@@ -363,28 +364,23 @@ default_entry:
 	pushl $0
 	popfl
 
-#ifdef CONFIG_SMP
-	cmpb $0, ready
-	jnz checkCPUtype
-#endif /* CONFIG_SMP */
-
 /*
  * start system 32-bit setup. We need to re-do some of the things done
  * in 16-bit mode for the "real" operations.
  */
-	call setup_idt
-
-checkCPUtype:
-
-	movl $-1,X86_CPUID		#  -1 for no CPUID initially
-
+	movl setup_once_ref,%eax
+	andl %eax,%eax
+	jz 1f				# Did we do this already?
+	call *%eax
+1:
+	
 /* check if it is 486 or 386. */
 /*
  * XXX - this does a lot of unnecessary setup.  Alignment checks don't
  * apply at our cpl of 0 and the stack ought to be aligned already, and
  * we don't need to preserve eflags.
  */
-
+	movl $-1,X86_CPUID	# -1 for no CPUID initially
 	movb $3,X86		# at least 386
 	pushfl			# push EFLAGS
 	popl %eax		# get EFLAGS
@@ -450,21 +446,6 @@ is386:	movl $2,%ecx		# set MP
 	movl $(__KERNEL_PERCPU), %eax
 	movl %eax,%fs			# set this cpu's percpu
 
-#ifdef CONFIG_CC_STACKPROTECTOR
-	/*
-	 * The linker can't handle this by relocation.  Manually set
-	 * base address in stack canary segment descriptor.
-	 */
-	cmpb $0,ready
-	jne 1f
-	movl $gdt_page,%eax
-	movl $stack_canary,%ecx
-	movw %cx, 8 * GDT_ENTRY_STACK_CANARY + 2(%eax)
-	shrl $16, %ecx
-	movb %cl, 8 * GDT_ENTRY_STACK_CANARY + 4(%eax)
-	movb %ch, 8 * GDT_ENTRY_STACK_CANARY + 7(%eax)
-1:
-#endif
 	movl $(__KERNEL_STACK_CANARY),%eax
 	movl %eax,%gs
 
@@ -473,7 +454,6 @@ is386:	movl $2,%ecx		# set MP
 
 	cld			# gcc2 wants the direction flag cleared at all times
 	pushl $0		# fake return address for unwinder
-	movb $1, ready
 	jmp *(initial_code)
 
 /*
@@ -495,81 +475,122 @@ check_x87:
 	.byte 0xDB,0xE4		/* fsetpm for 287, ignored by 387 */
 	ret
 
+	
+#include "verify_cpu.S"
+
 /*
- *  setup_idt
+ *  setup_once
  *
- *  sets up a idt with 256 entries pointing to
- *  ignore_int, interrupt gates. It doesn't actually load
- *  idt - that can be done only after paging has been enabled
- *  and the kernel moved to PAGE_OFFSET. Interrupts
- *  are enabled elsewhere, when we can be relatively
- *  sure everything is ok.
+ *  The setup work we only want to run on the BSP.
  *
  *  Warning: %esi is live across this function.
  */
-setup_idt:
-	lea ignore_int,%edx
-	movl $(__KERNEL_CS << 16),%eax
-	movw %dx,%ax		/* selector = 0x0010 = cs */
-	movw $0x8E00,%dx	/* interrupt gate - dpl=0, present */
+__INIT
+setup_once:
+	/*
+	 * Set up a idt with 256 entries pointing to ignore_int,
+	 * interrupt gates. It doesn't actually load idt - that needs
+	 * to be done on each CPU. Interrupts are enabled elsewhere,
+	 * when we can be relatively sure everything is ok.
+	 */
 
-	lea idt_table,%edi
-	mov $256,%ecx
-rp_sidt:
+	movl $idt_table,%edi
+	movl $early_idt_handlers,%eax
+	movl $NUM_EXCEPTION_VECTORS,%ecx
+1:
 	movl %eax,(%edi)
-	movl %edx,4(%edi)
+	movl %eax,4(%edi)
+	/* interrupt gate, dpl=0, present */
+	movl $(0x8E000000 + __KERNEL_CS),2(%edi)
+	addl $9,%eax
 	addl $8,%edi
-	dec %ecx
-	jne rp_sidt
+	loop 1b
 
-.macro	set_early_handler handler,trapno
-	lea \handler,%edx
+	movl $256 - NUM_EXCEPTION_VECTORS,%ecx
+	movl $ignore_int,%edx
 	movl $(__KERNEL_CS << 16),%eax
-	movw %dx,%ax
+	movw %dx,%ax		/* selector = 0x0010 = cs */
 	movw $0x8E00,%dx	/* interrupt gate - dpl=0, present */
-	lea idt_table,%edi
-	movl %eax,8*\trapno(%edi)
-	movl %edx,8*\trapno+4(%edi)
-.endm
+2:
+	movl %eax,(%edi)
+	movl %edx,4(%edi)
+	addl $8,%edi
+	loop 2b
 
-	set_early_handler handler=early_divide_err,trapno=0
-	set_early_handler handler=early_illegal_opcode,trapno=6
-	set_early_handler handler=early_protection_fault,trapno=13
-	set_early_handler handler=early_page_fault,trapno=14
+#ifdef CONFIG_CC_STACKPROTECTOR
+	/*
+	 * Configure the stack canary. The linker can't handle this by
+	 * relocation.  Manually set base address in stack canary
+	 * segment descriptor.
+	 */
+	movl $gdt_page,%eax
+	movl $stack_canary,%ecx
+	movw %cx, 8 * GDT_ENTRY_STACK_CANARY + 2(%eax)
+	shrl $16, %ecx
+	movb %cl, 8 * GDT_ENTRY_STACK_CANARY + 4(%eax)
+	movb %ch, 8 * GDT_ENTRY_STACK_CANARY + 7(%eax)
+#endif
 
+	andl $0,setup_once_ref	/* Once is enough, thanks */
 	ret
 
-early_divide_err:
-	xor %edx,%edx
-	pushl $0	/* fake errcode */
-	jmp early_fault
+ENTRY(early_idt_handlers)
+	# 36(%esp) %eflags
+	# 32(%esp) %cs
+	# 28(%esp) %eip
+	# 24(%rsp) error code
+	i = 0
+	.rept NUM_EXCEPTION_VECTORS
+	.if (EXCEPTION_ERRCODE_MASK >> i) & 1
+	ASM_NOP2
+	.else
+	pushl $0		# Dummy error code, to make stack frame uniform
+	.endif
+	pushl $i		# 20(%esp) Vector number
+	jmp early_idt_handler
+	i = i + 1
+	.endr
+ENDPROC(early_idt_handlers)
+	
+	/* This is global to keep gas from relaxing the jumps */
+ENTRY(early_idt_handler)
+	cld
+	cmpl $2,%ss:early_recursion_flag
+	je hlt_loop
+	incl %ss:early_recursion_flag
 
-early_illegal_opcode:
-	movl $6,%edx
-	pushl $0	/* fake errcode */
-	jmp early_fault
+	push %eax		# 16(%esp)
+	push %ecx		# 12(%esp)
+	push %edx		#  8(%esp)
+	push %ds		#  4(%esp)
+	push %es		#  0(%esp)
+	movl $(__KERNEL_DS),%eax
+	movl %eax,%ds
+	movl %eax,%es
 
-early_protection_fault:
-	movl $13,%edx
-	jmp early_fault
+	cmpl $(__KERNEL_CS),32(%esp)
+	jne 10f
 
-early_page_fault:
-	movl $14,%edx
-	jmp early_fault
+	leal 28(%esp),%eax	# Pointer to %eip
+	call early_fixup_exception
+	andl %eax,%eax
+	jnz ex_entry		/* found an exception entry */
 
-early_fault:
-	cld
+10:
 #ifdef CONFIG_PRINTK
-	pusha
-	movl $(__KERNEL_DS),%eax
-	movl %eax,%ds
-	movl %eax,%es
-	cmpl $2,early_recursion_flag
-	je hlt_loop
-	incl early_recursion_flag
+	xorl %eax,%eax
+	movw %ax,2(%esp)	/* clean up the segment values on some cpus */
+	movw %ax,6(%esp)
+	movw %ax,34(%esp)
+	leal  40(%esp),%eax
+	pushl %eax		/* %esp before the exception */
+	pushl %ebx
+	pushl %ebp
+	pushl %esi
+	pushl %edi
 	movl %cr2,%eax
 	pushl %eax
-	pushl %edx		/* trapno */
+	pushl (20+6*4)(%esp)	/* trapno */
 	pushl $fault_msg
 	call printk
 #endif
@@ -578,6 +599,17 @@ hlt_loop:
 	hlt
 	jmp hlt_loop
 
+ex_entry:
+	pop %es
+	pop %ds
+	pop %edx
+	pop %ecx
+	pop %eax
+	addl $8,%esp		/* drop vector number and error code */
+	decl %ss:early_recursion_flag
+	iret
+ENDPROC(early_idt_handler)
+
 /* This is the default interrupt "handler" :-) */
 	ALIGN
 ignore_int:
@@ -611,13 +643,18 @@ ignore_int:
 	popl %eax
 #endif
 	iret
+ENDPROC(ignore_int)
+__INITDATA
+	.align 4
+early_recursion_flag:
+	.long 0
 
-#include "verify_cpu.S"
-
-	__REFDATA
-.align 4
+__REFDATA
+	.align 4
 ENTRY(initial_code)
 	.long i386_start_kernel
+ENTRY(setup_once_ref)
+	.long setup_once
 
 /*
  * BSS section
@@ -670,22 +707,19 @@ ENTRY(initial_page_table)
 ENTRY(stack_start)
 	.long init_thread_union+THREAD_SIZE
 
-early_recursion_flag:
-	.long 0
-
-ready:	.byte 0
-
+__INITRODATA
 int_msg:
 	.asciz "Unknown interrupt or fault at: %p %p %p\n"
 
 fault_msg:
 /* fault info: */
 	.ascii "BUG: Int %d: CR2 %p\n"
-/* pusha regs: */
-	.ascii "     EDI %p  ESI %p  EBP %p  ESP %p\n"
-	.ascii "     EBX %p  EDX %p  ECX %p  EAX %p\n"
+/* regs pushed in early_idt_handler: */
+	.ascii "     EDI %p  ESI %p  EBP %p  EBX %p\n"
+	.ascii "     ESP %p   ES %p   DS %p\n"
+	.ascii "     EDX %p  ECX %p  EAX %p\n"
 /* fault frame: */
-	.ascii "     err %p  EIP %p   CS %p  flg %p\n"
+	.ascii "     vec %p  err %p  EIP %p   CS %p  flg %p\n"
 	.ascii "Stack: %p %p %p %p %p %p %p %p\n"
 	.ascii "       %p %p %p %p %p %p %p %p\n"
 	.asciz "       %p %p %p %p %p %p %p %p\n"
@@ -699,6 +733,7 @@ fault_msg:
  * segment size, and 32-bit linear address value:
  */
 
+	.data
 .globl boot_gdt_descr
 .globl idt_descr
 

[tip:x86/extable] x86, doc: Revert "x86: Document rdmsr_safe restrictions"

[tip-bot for H. Peter Anvin]

Commit-ID:  060feb650010c261fcfbae9de9348b46cedcd3cd
Gitweb:     http://git.kernel.org/tip/060feb650010c261fcfbae9de9348b46cedcd3cd
Author:     H. Peter Anvin <hpa@linux.intel.com>
AuthorDate: Thu, 19 Apr 2012 17:07:34 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 19 Apr 2012 17:07:34 -0700

x86, doc: Revert "x86: Document rdmsr_safe restrictions"

This reverts commit ce37defc0f6673f5ca2c92ed5cfcaf290ae7dd16
"x86: Document rdmsr_safe restrictions", as these restrictions no longer apply.

Reported-by: Borislav Petkov <borislav.petkov@amd.com>
Link: http://lkml.kernel.org/r/20120419171609.GH3221@aftab.osrc.amd.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
---
 arch/x86/include/asm/msr.h |    9 +--------
 1 files changed, 1 insertions(+), 8 deletions(-)

diff --git a/arch/x86/include/asm/msr.h b/arch/x86/include/asm/msr.h
index 95203d4..084ef95 100644
--- a/arch/x86/include/asm/msr.h
+++ b/arch/x86/include/asm/msr.h
@@ -169,14 +169,7 @@ static inline int wrmsr_safe(unsigned msr, unsigned low, unsigned high)
 	return native_write_msr_safe(msr, low, high);
 }
 
-/*
- * rdmsr with exception handling.
- *
- * Please note that the exception handling works only after we've
- * switched to the "smart" #GP handler in trap_init() which knows about
- * exception tables - using this macro earlier than that causes machine
- * hangs on boxes which do not implement the @msr in the first argument.
- */
+/* rdmsr with exception handling */
 #define rdmsr_safe(msr, p1, p2)					\
 ({								\
 	int __err;						\

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Andreas Schwab]

Tony Luck <tony.luck@gmail.com> writes:

> On Thu, Apr 19, 2012 at 4:10 PM, H. Peter Anvin <hpa@zytor.com> wrote:
>> be better.  That being said, some distros are complaining about the
>> amount of time it takes to load modules on boot...
>
> That could be fixed in the installer for the distribution by rebuilding a kernel
> with the target set of modules for the system built in.

Which would also make it pretty much unsupportable.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [PATCH 3/3] x86, extable: Handle early exceptions

[Borislav Petkov]

On Thu, Apr 19, 2012 at 11:11:00AM -0700, Linus Torvalds wrote:
> > OTOH, if we can share early and normal exception handling lookup code, then a
> > perfect hash would make sense as those exceptions would pile up.
> 
> Oh, any of this only makes sense if we can share it with the runtime
> exception lookup.
> 
> They are *seldom* hugely performance-critical, but there are some
> unusual loads where you do get a fair number of exceptions.

I was thinking, and this is probably purely hypothetical and tangential
to the topic but, what happens if we get an exception right in the
middle between using the early_idt_handler's and switching to the normal
trap handlers in trap_init()?

write_idt_entry() is a simple memcpy so what happens if we have a
half-written descriptor? Won't we need some locking there to atomize the
switch?

Or am I dreaming?

Thanks.

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551